How to Set Up a User-Generated Content Moderation System in FlutterFlow

On your existing content collection (posts, comments, reviews, etc.), add a moderationStatus field (String: pending/approved/rejected/flagged) with a default value of 'pending'. Add moderatedBy (String, moderator UID), moderatedAt (Timestamp), and rejectionReason (String). Update all public-facing queries to filter by moderationStatus == 'approved' — this ensures only moderated content appears in feeds and listings. In FlutterFlow, update your Backend Queries on ListViews and GridViews to include this filter. Content creators should see their own pending content with a 'Pending Review' badge.

Create a Cloud Function triggered by Firestore onCreate on your content collection. The function reads the content text, calls the Google Perspective API with the TOXICITY attribute, and checks the summary score. If the toxicity score exceeds your threshold (e.g., 0.85), auto-set moderationStatus to 'rejected' with rejectionReason 'Automated: content flagged as potentially toxic'. If the score is moderate (0.5-0.85), set to 'flagged' for human review. If below 0.5, set to 'approved'. Log the toxicity score on the document for moderator reference. This pre-screens most content automatically, reducing the moderator workload.

1const functions = require('firebase-functions');
2const admin = require('firebase-admin');
3const fetch = require('node-fetch');
4admin.initializeApp();
5
6const API_KEY = functions.config().perspective.key;
7
8exports.screenContent = functions.firestore
9  .document('posts/{postId}')
10  .onCreate(async (snap, context) => {
11    const post = snap.data();
12    const text = post.content || post.body || '';
13    if (!text) return;
14
15    const response = await fetch(
16      `https://commentanalyzer.googleapis.com/v1alpha1/comments:analyze?key=${API_KEY}`,
17      {
18        method: 'POST',
19        headers: { 'Content-Type': 'application/json' },
20        body: JSON.stringify({
21          comment: { text },
22          languages: ['en'],
23          requestedAttributes: { TOXICITY: {} },
24        }),
25      }
26    );
27    const result = await response.json();
28    const score = result.attributeScores
29      ?.TOXICITY?.summaryScore?.value || 0;
30
31    let status = 'approved';
32    let reason = null;
33    if (score >= 0.85) {
34      status = 'rejected';
35      reason = 'Automated: high toxicity score';
36    } else if (score >= 0.5) {
37      status = 'flagged';
38    }
39
40    await snap.ref.update({
41      moderationStatus: status,
42      toxicityScore: score,
43      ...(reason && { rejectionReason: reason }),
44    });
45  });

Create an admin-only page called ModerationQueue. Add a ChoiceChips filter at the top with options: Pending, Flagged, Rejected, All. Below it, add a ListView bound to a Backend Query on the content collection filtered by moderationStatus matching the selected chip. Each list item shows: the content text (truncated to 2 lines), author name, submission timestamp, toxicity score if available, and a Row of three Buttons — Approve (green), Reject (red), and Flag (orange). On Approve tap, update moderationStatus to 'approved', set moderatedBy to the current admin's UID, and moderatedAt to now. On Reject tap, show a dialog with a DropDown for rejection reason (Spam, Hate Speech, Inappropriate, Off Topic) and a TextField for notes, then update the document. Add a Badge in your admin nav showing the count of pending + flagged items.

On every piece of user content in your app, add a Report button (flag icon). On tap, show a BottomSheet with a DropDown for reason (Spam, Harassment, Misinformation, Inappropriate, Other) and an optional TextField for details. On submit, create a document in a `reports` subcollection under the content document with: reporterId (current user UID), reason, details, timestamp. Also increment a reportCount field on the content document using FieldValue.increment(1). Create a Cloud Function that triggers on report creation: if the content's reportCount exceeds a threshold (e.g., 3), automatically set moderationStatus to 'flagged' so it enters the moderator queue. Prevent duplicate reports by checking if the current user already has a report document.

When a user views their own rejected content, show the rejectionReason in a red Container and an Appeal button below it. On tap, open a dialog with a TextField where the user explains why the rejection should be reconsidered. On submit, update the content document: set appealText to the user's explanation and appealStatus to 'pending_appeal'. In the moderator queue, add a filter for items with appealStatus == 'pending_appeal'. Show the original content, the rejection reason, and the user's appeal text. Moderators can then Approve (override rejection) or Deny Appeal (confirm rejection, set appealStatus to 'denied'). Limit appeals to one per content item to prevent spam.