How to Integrate AI-Based Image Recognition in FlutterFlow

In FlutterFlow, go to Custom Code → Pubspec Dependencies → Add Dependency: google_mlkit_image_labeling: ^0.11.0 and image_picker: ^1.0.4. Create a Custom Action named classifyImageOnDevice. The action: (1) opens the image picker with ImagePicker().pickImage(source: ImageSource.gallery or camera), (2) creates an InputImage from the file path, (3) runs ImageLabeler with ImageLabelerOptions(confidenceThreshold: 0.5), (4) processes the image with imageLabeler.processImage(inputImage) to get a List<ImageLabel>, (5) filters to labels with confidence > 0.7, (6) sorts by confidence descending, (7) converts to a list of strings formatted as 'Dog (87%)' and stores in App State variable recognitionResults (List of String). Call this Custom Action from a Button's onTap action. Display App State recognitionResults in a ListView with one Text widget per item.

1// Custom Action: classifyImageOnDevice
2import 'package:image_picker/image_picker.dart';
3import 'package:google_mlkit_image_labeling/google_mlkit_image_labeling.dart';
4
5Future classifyImageOnDevice() async {
6  final picker = ImagePicker();
7  final pickedFile = await picker.pickImage(
8    source: ImageSource.gallery,
9    maxWidth: 1024,
10    maxHeight: 1024,
11  );
12  if (pickedFile == null) return;
13
14  final inputImage = InputImage.fromFilePath(pickedFile.path);
15  final labeler = ImageLabeler(
16    options: ImageLabelerOptions(confidenceThreshold: 0.5)
17  );
18  
19  final labels = await labeler.processImage(inputImage);
20  labeler.close();
21  
22  // Filter to useful labels, format nicely
23  final results = labels
24    .where((label) => label.confidence > 0.7)
25    .toList()
26    ..sort((a, b) => b.confidence.compareTo(a.confidence));
27  
28  final formatted = results
29    .take(5) // top 5 labels
30    .map((l) => '${l.label} (${(l.confidence * 100).toStringAsFixed(0)}%)')
31    .toList();
32  
33  // Store in App State: recognitionResults (List<String>)
34  FFAppState().update(() {
35    FFAppState().recognitionResults = formatted;
36    FFAppState().classifiedImagePath = pickedFile.path;
37  });
38}

Deploy a Cloud Function named analyzeImage that accepts a base64-encoded image and a detection type (LABEL_DETECTION, OBJECT_LOCALIZATION, TEXT_DETECTION, or LOGO_DETECTION). The function calls the Vision API POST https://vision.googleapis.com/v1/images:annotate with your API key. For label detection, it returns labels with descriptions and scores. For object localization, it returns object names with bounding polygon coordinates. Enable the Cloud Vision API in Google Cloud Console → APIs and Services → Enable. Store the API key in Cloud Function environment variables as VISION_API_KEY. In FlutterFlow, add an API Call to your Cloud Functions API Group for analyzeImage. After the Custom Action captures the image and encodes it to base64, call the Cloud Function API, then store the returned labels array in App State.

1// Cloud Function: analyzeImage
2const functions = require('firebase-functions');
3const fetch = require('node-fetch');
4
5exports.analyzeImage = functions.https.onCall(async (data, context) => {
6  const { imageBase64, detectionType = 'LABEL_DETECTION' } = data;
7  const apiKey = process.env.VISION_API_KEY;
8  
9  const response = await fetch(
10    `https://vision.googleapis.com/v1/images:annotate?key=${apiKey}`,
11    {
12      method: 'POST',
13      headers: { 'Content-Type': 'application/json' },
14      body: JSON.stringify({
15        requests: [{
16          image: { content: imageBase64 },
17          features: [
18            { type: detectionType, maxResults: 10 },
19          ]
20        }]
21      })
22    }
23  );
24  
25  const result = await response.json();
26  const annotations = result.responses[0];
27  
28  if (detectionType === 'LABEL_DETECTION') {
29    return {
30      labels: (annotations.labelAnnotations || []).map(l => ({
31        name: l.description,
32        confidence: Math.round(l.score * 100)
33      }))
34    };
35  }
36  
37  if (detectionType === 'OBJECT_LOCALIZATION') {
38    return {
39      objects: (annotations.localizedObjectAnnotations || []).map(o => ({
40        name: o.name,
41        confidence: Math.round(o.score * 100),
42        bounds: o.boundingPoly.normalizedVertices
43      }))
44    };
45  }
46  
47  return { raw: annotations };
48});

Create an ImageRecognition page. Add an Image widget bound to App State classifiedImagePath (initially show a placeholder). Below the image, add a Row with two Buttons: 'From Camera' and 'From Gallery' — each calls the classifyImageOnDevice Custom Action with the appropriate source. Add a CircularProgressIndicator that shows while the action runs (bind to App State isProcessing boolean). Below that, add a Column with a Text header 'Detected:' and a ListView bound to App State recognitionResults (List of String). Each list item: a Container with a Text widget displaying the label string (already formatted as 'Dog (87%)') and a LinearProgressIndicator showing the confidence numerically — use a Custom Function to parse the percentage from the string. Add a 'Classify Another' button that clears App State recognitionResults and classifiedImagePath.

When using OBJECT_LOCALIZATION, Cloud Vision returns normalized vertex coordinates for each detected object's bounding polygon. To draw these overlays in FlutterFlow, create a Custom Widget named BoundingBoxOverlay. It accepts the image file path and a list of detected objects (name, confidence, normalized bounds). The widget uses a Stack: an Image widget at the bottom, and a CustomPaint widget on top that draws colored rectangles scaled to the image dimensions. The normalizedVertices coordinates (0.0 to 1.0) are multiplied by the image width and height to get pixel coordinates. Each rectangle has a label drawn above it with the object name and confidence. This Custom Widget replaces the plain Image widget on the recognition results page.