How to Integrate FlutterFlow with Machine Learning APIs for Advanced Features

For OCR (reading text from images), start with Google ML Kit which runs entirely on-device — free, offline-capable, and privacy-preserving. In FlutterFlow → Custom Code → Pubspec Dependencies, add google_mlkit_text_recognition (version ^0.13.0). For iOS: go to Settings → iOS Settings → Info.plist and verify the minimum iOS deployment target is 14.0 or higher (ML Kit requirement). Create a Custom Action named recognizeText that takes an imagePath String parameter. In the action: create a TextRecognizer using TextRecognizer(script: TextRecognitionScript.latin), create an InputImage from the file path using InputImage.fromFilePath(imagePath), call recognizer.processImage(inputImage), concatenate all recognized text blocks into a single string, call recognizer.close() to free resources, and return the text. Use this action after image_picker captures or selects an image.

1import 'package:google_mlkit_text_recognition/google_mlkit_text_recognition.dart';
2
3Future<String> recognizeText(String imagePath) async {
4  final recognizer = TextRecognizer(script: TextRecognitionScript.latin);
5  final inputImage = InputImage.fromFilePath(imagePath);
6  
7  try {
8    final recognized = await recognizer.processImage(inputImage);
9    final text = recognized.blocks
10        .map((b) => b.text)
11        .join('\n');
12    return text.isEmpty ? 'No text found in image' : text;
13  } finally {
14    recognizer.close();
15  }
16}

Google Cloud Vision identifies objects, labels, faces, logos, text, landmarks, and explicit content in images. Create an HTTPS Cloud Function named analyzeImage. It receives a base64-encoded image string and a list of feature types. The function calls the Vision API at https://vision.googleapis.com/v1/images:annotate using a service account key (JSON key stored in Firebase Secret Manager as GOOGLE_VISION_KEY). Request body: {requests: [{image: {content: base64Image}, features: [{type: 'LABEL_DETECTION', maxResults: 10}, {type: 'OBJECT_LOCALIZATION', maxResults: 5}]}]}. Parse the response and return: a labels array (description + score), a objects array (name + score), and any detected text from TEXT_DETECTION. Before calling the API, check Firestore cache: hash the image (MD5 of first 100 chars of base64) and look up in ml_cache collection. If cached result is less than 24 hours old, return the cached result without calling the API.

1const functions = require('firebase-functions');
2const { defineSecret } = require('firebase-functions/params');
3const admin = require('firebase-admin');
4const axios = require('axios');
5const crypto = require('crypto');
6
7if (!admin.apps.length) admin.initializeApp();
8const visionKey = defineSecret('GOOGLE_VISION_KEY');
9
10exports.analyzeImage = functions
11  .runWith({ secrets: ['GOOGLE_VISION_KEY'], memory: '512MB' })
12  .https.onCall(async (data) => {
13    const { base64Image, features = ['LABEL_DETECTION'] } = data;
14    if (!base64Image) throw new functions.https.HttpsError('invalid-argument', 'Image required');
15
16    // Check cache
17    const hash = crypto.createHash('md5').update(base64Image.slice(0, 200)).digest('hex');
18    const db = admin.firestore();
19    const cached = await db.collection('ml_cache').doc(hash).get();
20    if (cached.exists) {
21      const data = cached.data();
22      const age = (Date.now() - data.cachedAt.toMillis()) / 1000 / 3600;
23      if (age < 24) return data.result;
24    }
25
26    const request = {
27      requests: [{
28        image: { content: base64Image },
29        features: features.map(f => ({ type: f, maxResults: 10 }))
30      }]
31    };
32    const { data: response } = await axios.post(
33      `https://vision.googleapis.com/v1/images:annotate?key=${visionKey.value()}`,
34      request
35    );
36
37    const result = response.responses[0];
38    const output = {
39      labels: result.labelAnnotations?.map(l => ({ name: l.description, score: l.score })) || [],
40      objects: result.localizedObjectAnnotations?.map(o => ({ name: o.name, score: o.score })) || [],
41      text: result.textAnnotations?.[0]?.description || ''
42    };
43
44    await db.collection('ml_cache').doc(hash).set({
45      result: output,
46      cachedAt: admin.firestore.FieldValue.serverTimestamp()
47    });
48
49    return output;
50  });

Create a Cloud Function named chatWithAI that accepts a messages array and systemPrompt. Store the OpenAI API key in Firebase Secret Manager as OPENAI_API_KEY. The function calls POST https://api.openai.com/v1/chat/completions with model: gpt-4o-mini (cost-effective for most use cases), temperature: 0.7, max_tokens: 500, and the messages array including the system message. Return the content of the first choice. In FlutterFlow, create an AI Chat page with: a ListView of messages bound to a Page State messages list, a TextField for user input, and a Send button. The Send button's action flow: add the user message to the messages list (for immediate display), call the chatWithAI Cloud Function with the updated messages, append the AI response to the messages list. Use a CircularProgressIndicator while the call is in progress by tracking isLoading in Page State.

High-resolution photos from modern smartphones (12MP+) are 4-8MB JPEG files. Sending these to Cloud Vision or other ML APIs takes 3-5 seconds for upload alone and costs more (Vision API charges per image, and larger images sometimes count as multiple units). The solution: resize images on-device before sending. Add the image package to Pubspec Dependencies (version ^4.1.3). Create a Custom Function named resizeImageForML that takes an imagePath String. Read the file, decode the image with decodeImage, resize to max 1024px on the longest side using copyResize with a null value for the other dimension (preserves aspect ratio), re-encode as JPEG at quality 85, write to a temporary file, and return the new file path. Call this Custom Function before converting to base64 for the ML API call.

1import 'package:image/image.dart' as img;
2import 'package:path_provider/path_provider.dart';
3import 'dart:io';
4
5Future<String> resizeImageForML(String imagePath) async {
6  final bytes = await File(imagePath).readAsBytes();
7  final original = img.decodeImage(bytes);
8  if (original == null) return imagePath;
9
10  final maxDim = 1024;
11  final resized = (original.width > original.height)
12      ? img.copyResize(original, width: maxDim)
13      : img.copyResize(original, height: maxDim);
14
15  final tmpDir = await getTemporaryDirectory();
16  final outPath = '${tmpDir.path}/ml_resized_${DateTime.now().millisecondsSinceEpoch}.jpg';
17  await File(outPath).writeAsBytes(img.encodeJpg(resized, quality: 85));
18  return outPath;
19}

After calling the analyzeImage Cloud Function, store the returned labels and objects arrays in Page State variables (List<JSON>). In FlutterFlow, add a Column below the analyzed image. Add a Text widget showing 'Detected in this image:'. Add a Wrap widget (for horizontal chip layout) with children generated dynamically from the labels Page State list. Each label is a Chip widget showing the label name and score as percentage (multiply score by 100, format to 1 decimal). Color the chip based on score: if score > 0.9, use green; if score > 0.7 use blue; else use gray. For OpenAI text responses, display in a styled Container with markdown-like formatting using the flutter_markdown Custom Widget if you want formatted text output.