How to Build a Web Scraping API with Lovable

1Build a web scraping API. Create these Supabase tables:
2
3- scrape_jobs: id, user_id, url (text), selector_config (jsonb, e.g. { fieldName: 'css-selector' }), priority (int default 5, 1=highest 10=lowest), status (pending|running|done|failed|retrying), attempt_count (int default 0), max_attempts (int default 3), use_firecrawl (bool default false), error_message (text), created_at, started_at, completed_at
4
5- scrape_results: id, job_id (FK scrape_jobs UNIQUE), extracted_data (jsonb), raw_html_url (text, Supabase Storage path), page_title (text), scraped_at, response_status_code (int), response_time_ms (int)
6
7RLS:
8- scrape_jobs: user_id = auth.uid() for all operations
9- scrape_results: accessible via job_id FK to user's jobs (check via EXISTS subquery)
10
11Create indexes:
12- CREATE INDEX idx_scrape_jobs_queue ON scrape_jobs(status, priority, created_at) WHERE status = 'pending'
13- CREATE INDEX idx_scrape_jobs_user ON scrape_jobs(user_id, created_at DESC)
14
15Enable pg_cron extension: it should already be enabled on your Supabase project. If not, ask in Supabase support chat.

1// supabase/functions/scrape-worker/index.ts
2import { serve } from 'https://deno.land/std@0.168.0/http/server.ts'
3import { createClient } from 'https://esm.sh/@supabase/supabase-js@2'
4import { load } from 'https://esm.sh/cheerio@1.0.0-rc.12'
5
6const corsHeaders = { 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Headers': 'authorization, apikey, content-type' }
7
8serve(async (req: Request) => {
9  if (req.method === 'OPTIONS') return new Response('ok', { headers: corsHeaders })
10
11  const supabase = createClient(Deno.env.get('SUPABASE_URL') ?? '', Deno.env.get('SUPABASE_SERVICE_ROLE_KEY') ?? '')
12  const { jobId } = await req.json()
13
14  const { data: job } = await supabase.from('scrape_jobs').select('*').eq('id', jobId).single()
15  if (!job) return new Response(JSON.stringify({ error: 'Job not found' }), { status: 404, headers: corsHeaders })
16
17  await supabase.from('scrape_jobs').update({ status: 'running', started_at: new Date().toISOString(), attempt_count: job.attempt_count + 1 }).eq('id', jobId)
18
19  const start = Date.now()
20  try {
21    let html = ''
22    let statusCode = 200
23
24    if (job.use_firecrawl) {
25      html = await scrapeWithFirecrawl(job.url)
26    } else {
27      const res = await fetch(job.url, {
28        headers: { 'User-Agent': 'Mozilla/5.0 (compatible; scraper/1.0)' },
29        signal: AbortSignal.timeout(15000),
30      })
31      statusCode = res.status
32      if (res.status === 403 || res.status === 429) {
33        html = await scrapeWithFirecrawl(job.url)
34      } else {
35        html = await res.text()
36      }
37    }
38
39    const $ = load(html)
40    const extractedData: Record<string, string> = {}
41    for (const [field, selector] of Object.entries(job.selector_config as Record<string, string>)) {
42      extractedData[field] = $(selector).first().text().trim()
43    }
44
45    await supabase.from('scrape_results').upsert({
46      job_id: jobId,
47      extracted_data: extractedData,
48      page_title: $('title').text().trim(),
49      scraped_at: new Date().toISOString(),
50      response_status_code: statusCode,
51      response_time_ms: Date.now() - start,
52    }, { onConflict: 'job_id' })
53
54    await supabase.from('scrape_jobs').update({ status: 'done', completed_at: new Date().toISOString() }).eq('id', jobId)
55    return new Response(JSON.stringify({ success: true, fields: Object.keys(extractedData).length }), { headers: corsHeaders })
56  } catch (err) {
57    const msg = err instanceof Error ? err.message : 'Scrape failed'
58    const nextStatus = job.attempt_count + 1 >= job.max_attempts ? 'failed' : 'pending'
59    await supabase.from('scrape_jobs').update({ status: nextStatus, error_message: msg, completed_at: nextStatus === 'failed' ? new Date().toISOString() : null }).eq('id', jobId)
60    return new Response(JSON.stringify({ error: msg }), { status: 500, headers: corsHeaders })
61  }
62})
63
64async function scrapeWithFirecrawl(url: string): Promise<string> {
65  const res = await fetch('https://api.firecrawl.dev/v1/scrape', {
66    method: 'POST',
67    headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${Deno.env.get('FIRECRAWL_API_KEY')}` },
68    body: JSON.stringify({ url, formats: ['html'] }),
69  })
70  const data = await res.json()
71  if (!res.ok) throw new Error(data.error ?? 'Firecrawl failed')
72  return data.data?.html ?? data.data?.markdown ?? ''
73}

1Set up the automated queue processor:
2
31. Create an Edge Function at supabase/functions/process-queue/index.ts that:
4   - Queries scrape_jobs WHERE status = 'pending' ORDER BY priority ASC, created_at ASC LIMIT 5
5   - For each job: updates status to 'running' (to claim it), then calls the scrape-worker Edge Function via fetch
6   - Uses Promise.allSettled to run up to 5 jobs concurrently
7   - Returns a JSON summary: { dispatched: number, jobIds: string[] }
8
92. Register the pg_cron schedule in Supabase SQL editor:
10SELECT cron.schedule(
11  'process-scrape-queue',
12  '* * * * *',
13  $$
14  SELECT net.http_post(
15    url:='https://YOUR_PROJECT.supabase.co/functions/v1/process-queue',
16    headers:=json_build_object('Authorization', 'Bearer YOUR_SERVICE_ROLE_KEY')::jsonb
17  ) AS request_id;
18  $$
19);
20
213. Add a job intake Edge Function at supabase/functions/submit-scrape-job/index.ts that:
22   - Accepts POST with { url, selectorConfig, priority?, useFirecrawl? }
23   - Validates the URL is a valid HTTP/HTTPS URL
24   - Inserts into scrape_jobs and returns the new job ID
25   - Can be called without auth (add an API key check using the same pattern as the api-backend guide) or with Supabase Auth

1Build the scraping dashboard at src/pages/ScrapingDashboard.tsx:
2
31. Summary Cards at top: Jobs Today, Queue Depth (pending count), Success Rate %, Average Response Time
4
52. Job submission form (Card or Sheet):
6   - URL Input (required, validated as URL format)
7   - Selector Config builder: a key-value list where users add field name (e.g. 'price') + CSS selector (e.g. '.price-tag'). Show Add Row Button and remove buttons per row.
8   - Priority Slider (1-10)
9   - Use Firecrawl Checkbox with label 'Use Firecrawl for JavaScript-heavy sites (uses credits)'
10   - Submit Button: calls the submit-scrape-job Edge Function
11
123. Job queue DataTable with columns: created_at (relative), URL (truncated with Tooltip for full URL), Priority Badge, Status Badge (pending=gray, running=blue with spinner, done=green, failed=red, retrying=yellow), attempt_count, Actions menu (View Results, Retry, Delete)
13
144. Results Sheet (opens when clicking View Results):
15   - URL and scraped_at
16   - Extracted data as a key-value table (field name → extracted text)
17   - Response status code Badge and response time
18   - Error message if failed
19
205. Recharts BarChart below the table: jobs per hour over the last 24 hours, colored by status (done=green, failed=red stacked)