How to Integrate Machine Learning for Voice Recognition in FlutterFlow

Go to Custom Code → Pubspec Dependencies and add record: ^5.0.0. This package handles microphone permission, audio capture, and file output. Create a Custom Action named recordVoiceSample. Import the package and implement: request microphone permission via Permission.microphone.request(), create a recorder instance, start recording to a temp file with AudioEncoder.wav, record for 5 seconds, stop recording, then return the file path as a string. In your FlutterFlow page, add a Record Voice button that calls this action and stores the returned file path in a Page State variable named voiceFilePath.

In the Azure Portal, create a Speech service resource and copy the Subscription Key and Region. In FlutterFlow Secrets (Settings → Secrets), add AZURE_SPEECH_KEY. Create a Cloud Function named enrollSpeaker that accepts userId and audioBase64. Inside the function: POST to https://{region}.api.cognitive.microsoft.com/speaker/verification/v2.0/text-independent/profiles with header Ocp-Apim-Subscription-Key. This creates a speaker profile. Then POST the base64 audio to /profiles/{profileId}/enrollments. Save the profileId to Firestore users/{userId}.speakerProfileId. Create a second Cloud Function named verifySpeaker that takes userId and audioBase64, fetches the profileId from Firestore, then calls /profiles/{profileId}/verify with the audio. Returns a JSON with result (Accept/Reject) and score (0.0-1.0). In FlutterFlow API Manager, create API calls to both Cloud Functions.

1// Cloud Function: verifySpeaker
2const functions = require('firebase-functions');
3const admin = require('firebase-admin');
4const axios = require('axios');
5
6exports.verifySpeaker = functions.https.onRequest(async (req, res) => {
7  const { userId, audioBase64 } = req.body;
8  const azureKey = process.env.AZURE_SPEECH_KEY;
9  const region = process.env.AZURE_REGION;
10
11  // Get stored profile ID
12  const userDoc = await admin.firestore().doc(`users/${userId}`).get();
13  const profileId = userDoc.data().speakerProfileId;
14
15  // Convert base64 to binary buffer
16  const audioBuffer = Buffer.from(audioBase64, 'base64');
17
18  // Verify against enrolled voice
19  const response = await axios.post(
20    `https://${region}.api.cognitive.microsoft.com/speaker/verification/v2.0/text-independent/profiles/${profileId}/verify`,
21    audioBuffer,
22    {
23      headers: {
24        'Ocp-Apim-Subscription-Key': azureKey,
25        'Content-Type': 'audio/wav'
26      }
27    }
28  );
29
30  res.json({
31    result: response.data.result,
32    score: response.data.score
33  });
34});

Create a Hume AI account at hume.ai and get your API key. Add HUME_API_KEY to FlutterFlow Secrets. Create a Cloud Function named detectEmotion that accepts audioBase64. The function POSTs to https://api.hume.ai/v0/batch/jobs with header X-Hume-Api-Key and a multipart form containing the audio file. Hume processes asynchronously — poll GET /v0/batch/jobs/{jobId}/predictions until status is COMPLETED. The response includes emotion scores for prosody (speech rhythm and tone): an array of {name, score} objects sorted by score descending. Common emotions returned: Joy, Sadness, Anger, Fear, Surprise, Disgust, Neutral. In FlutterFlow, display these as a Container with a Column of emotion bars: Text (emotion name) + LinearProgressIndicator (score 0.0-1.0).

Add tflite_flutter: ^0.10.4 to Pubspec Dependencies. Obtain or train a TFLite model for your custom commands (.tflite file) and place it in your project's assets folder by adding it in FlutterFlow → Custom Code → Assets. Create a Custom Widget named VoiceCommandListener. In the widget, load the model: Interpreter.fromAsset('assets/voice_commands.tflite'). Record short audio clips (1 second), convert to mel spectrogram float array (40 features × 98 time steps), run inference: interpreter.run(inputBuffer, outputBuffer). outputBuffer contains probabilities for each command class. Pick the argmax class if confidence > 0.85. Map class indices to command strings: {0: 'start', 1: 'stop', 2: 'help', 3: 'background'}. Trigger FlutterFlow actions via callbacks on recognized commands. For production apps, RapidDev has implemented this exact on-device voice command pipeline across multiple FlutterFlow projects requiring specialized audio preprocessing.

Sign up at ElevenLabs.io and get your API key. Add ELEVENLABS_API_KEY to FlutterFlow Secrets. Create a Cloud Function named synthesizeSpeech that accepts text and voiceId. POST to https://api.elevenlabs.io/v1/text-to-speech/{voiceId} with header xi-api-key and body {text, model_id: 'eleven_multilingual_v2', voice_settings: {stability: 0.5, similarity_boost: 0.75}}. The response is binary MP3 audio. Upload the audio to Firebase Storage and return the download URL. In FlutterFlow, add an API Call to your synthesizeSpeech Cloud Function. Use a Custom Action to play the returned audio URL with the audioplayers package (add audioplayers: ^6.0.0 to Pubspec Dependencies).