How to Integrate Retool with Screaming Frog SEO Spider

Run a crawl in Screaming Frog SEO Spider by opening the application, entering your target URL in the address bar, and clicking Start. Wait for the crawl to complete. Once finished, export the data by going to File → Export → All Export Tabs as CSV or using the Bulk Export option. The most useful exports for an SEO dashboard are: All Inlinks (every link on the site), Response Codes (pages grouped by HTTP status code), Redirects (all redirect chains), and Page Titles (all pages with their title tag data). Before importing into Retool, you need a database table to store the crawl data. If you're using Retool Database (the managed PostgreSQL), go to the Retool Database tab in your Retool instance and create a new table named crawl_data. If using your own PostgreSQL, create the table via your SQL client. The recommended schema for a unified crawl table is: id (serial primary key), crawl_date (date), address (text), content_type (text), status_code (integer), status (text), indexability (text), title_1 (text), meta_description_1 (text), h1_1 (text), word_count (integer), response_time (numeric), redirect_uri (text), canonical_link_element (text), crawl_depth (integer). This schema covers the most commonly needed fields from Screaming Frog's Internal tab export. Alternatively, use the simpler approach of importing the CSV directly. Screaming Frog's Internal tab export contains the most comprehensive page-level data and maps directly to this schema.

1-- SQL to create the crawl_data table in PostgreSQL or Retool Database
2CREATE TABLE IF NOT EXISTS crawl_data (
3  id SERIAL PRIMARY KEY,
4  crawl_date DATE NOT NULL DEFAULT CURRENT_DATE,
5  address TEXT NOT NULL,
6  content_type TEXT,
7  status_code INTEGER,
8  status TEXT,
9  indexability TEXT,
10  indexability_status TEXT,
11  title_1 TEXT,
12  title_1_length INTEGER,
13  meta_description_1 TEXT,
14  meta_description_1_length INTEGER,
15  h1_1 TEXT,
16  h1_2 TEXT,
17  word_count INTEGER,
18  response_time NUMERIC,
19  redirect_uri TEXT,
20  canonical_link_element TEXT,
21  crawl_depth INTEGER,
22  inlinks INTEGER,
23  unique_inlinks INTEGER,
24  created_at TIMESTAMP DEFAULT NOW()
25);
26
27-- Index for fast filtering by date and status
28CREATE INDEX idx_crawl_date ON crawl_data(crawl_date);
29CREATE INDEX idx_status_code ON crawl_data(status_code);
30CREATE INDEX idx_indexability ON crawl_data(indexability);

With your database table created, import the Screaming Frog CSV export. There are three approaches depending on your database setup and automation needs. Approach 1 — Direct PostgreSQL COPY (fastest for one-time imports): If you have direct access to your PostgreSQL server, use the COPY command to bulk-insert the CSV. Remove the header row from the CSV file first (Screaming Frog includes column headers), then run the COPY command pointing to your file. Approach 2 — Retool Database CSV import (easiest for Retool Database users): In the Retool Database interface, navigate to your crawl_data table, click Import Data, and upload the CSV file directly. Retool Database handles the column mapping automatically based on header names. Before importing, edit the CSV headers to match your table column names exactly. Approach 3 — Retool app file upload (most flexible for recurring imports): Build a simple Retool app with a File Button component that allows your SEO team to upload a new CSV after each crawl. Use a JavaScript query to parse the uploaded CSV and insert rows into the database via a parameterized INSERT query. This creates a self-service import tool that doesn't require database access. For all approaches, after importing, run a quick verification query in Retool to confirm the row count: SELECT COUNT(*), crawl_date FROM crawl_data GROUP BY crawl_date ORDER BY crawl_date DESC LIMIT 10. Confirm the expected number of rows for the latest crawl date.

1-- SQL for importing a specific crawl date's data
2-- Run this after copying CSV data into a temporary import table
3INSERT INTO crawl_data (
4  crawl_date, address, content_type, status_code, status,
5  indexability, title_1, title_1_length, meta_description_1,
6  meta_description_1_length, h1_1, word_count,
7  response_time, redirect_uri, canonical_link_element,
8  crawl_depth, inlinks, unique_inlinks
9)
10SELECT
11  '{{ currentCrawlDate }}'::date,
12  address, content_type,
13  NULLIF(status_code, '')::integer,
14  status, indexability, title_1,
15  NULLIF(title_1_length, '')::integer,
16  meta_description_1,
17  NULLIF(meta_description_1_length, '')::integer,
18  h1_1,
19  NULLIF(word_count, '')::integer,
20  NULLIF(response_time, '')::numeric,
21  redirect_uri, canonical_link_element,
22  NULLIF(crawl_depth, '')::integer,
23  NULLIF(inlinks, '')::integer,
24  NULLIF(unique_inlinks, '')::integer
25FROM crawl_import_staging;

With crawl data in your database, create the core SEO analysis dashboard in Retool. Connect to your database by adding it as a Resource in the Resources tab — select PostgreSQL (or Retool Database if applicable) and configure the connection credentials. Create a query named getCrawlSummary that produces an issue summary by status code and issue type: This query powers a summary stat panel at the top of your dashboard. Add Stat components showing: total pages crawled (all rows for the latest crawl date), broken links (status_code = 404), server errors (status_code >= 500), redirect pages (status_code IN (301,302,307)), non-indexable pages (indexability != 'Indexable'), and pages with missing title tags (title_1 IS NULL OR title_1 = ''). Create a second query named getIssueList that retrieves individual pages matching the currently selected issue type. Add a Select component named select_issue_type with options: 'All Issues', '4xx Errors', '5xx Errors', 'Redirects', 'Missing Title', 'Missing Meta Description', 'Duplicate Title', 'Non-Indexable'. The getIssueList query uses a CASE or WHERE clause based on the selected option. Bind getIssueList results to a Table component with columns for URL, status code, issue type, title, meta description, crawl depth, and inlink count. Add a Text Input filter for URL pattern matching. Sort by inlink count descending by default so high-authority pages with issues appear first — this helps prioritize which issues to fix based on internal link equity.

1-- SQL query: getCrawlSummary — issue counts for latest crawl date
2SELECT
3  crawl_date,
4  COUNT(*) AS total_pages,
5  COUNT(*) FILTER (WHERE status_code BETWEEN 200 AND 299) AS ok_pages,
6  COUNT(*) FILTER (WHERE status_code BETWEEN 300 AND 399) AS redirect_pages,
7  COUNT(*) FILTER (WHERE status_code = 404) AS not_found,
8  COUNT(*) FILTER (WHERE status_code >= 500) AS server_errors,
9  COUNT(*) FILTER (WHERE indexability = 'Indexable') AS indexable_pages,
10  COUNT(*) FILTER (WHERE indexability != 'Indexable' AND status_code = 200) AS blocked_pages,
11  COUNT(*) FILTER (WHERE (title_1 IS NULL OR title_1 = '') AND status_code = 200) AS missing_title,
12  COUNT(*) FILTER (WHERE (meta_description_1 IS NULL OR meta_description_1 = '') AND status_code = 200) AS missing_meta,
13  COUNT(*) FILTER (WHERE title_1_length > 60 AND status_code = 200) AS title_too_long,
14  COUNT(*) FILTER (WHERE word_count < 300 AND status_code = 200 AND indexability = 'Indexable') AS thin_content
15FROM crawl_data
16WHERE crawl_date = (
17  SELECT MAX(crawl_date) FROM crawl_data
18)
19GROUP BY crawl_date;

Build a crawl comparison view that shows how SEO health has changed between crawls. This is where storing historical crawl data in your database pays off. Create a query named getCrawlHistory that retrieves issue counts for all stored crawl dates: Bind this to a Chart component configured as a multi-series line chart with crawl_date on the x-axis. Add separate series for not_found count, redirect_pages count, missing_title count, and indexable_pages count. This visualizes whether technical SEO health is improving with each site update. Create a query named compareCrawls that compares two selected crawl dates side by side: Add two Date Select components (select_date_1 and select_date_2) to allow users to choose which two crawl dates to compare. The compareCrawls query joins the two dates and calculates deltas — positive deltas for broken links and missing titles are bad (shown in red), positive deltas for indexable pages are good (shown in green). For the URL-level comparison — specific pages that changed status between crawls — create a query named getStatusChanges that finds pages where status_code changed between the two selected dates. This is the most actionable report for developers: pages that became 404 after a deploy, or pages that were fixed between crawls. Add a download button that exports the current filtered issue list as a CSV using Retool's built-in table download feature, so SEO managers can share specific issue lists with developers or clients.

1-- SQL query: getStatusChanges — pages that changed status between two crawl dates
2SELECT
3  c1.address,
4  c1.status_code AS old_status,
5  c2.status_code AS new_status,
6  c1.title_1 AS title,
7  c1.inlinks AS inlink_count,
8  CASE
9    WHEN c1.status_code = 200 AND c2.status_code != 200 THEN 'BROKEN'
10    WHEN c1.status_code != 200 AND c2.status_code = 200 THEN 'FIXED'
11    WHEN c1.status_code != c2.status_code THEN 'CHANGED'
12    ELSE 'SAME'
13  END AS change_type
14FROM crawl_data c1
15JOIN crawl_data c2
16  ON c1.address = c2.address
17  AND c1.crawl_date = {{ select_date_1.value }}::date
18  AND c2.crawl_date = {{ select_date_2.value }}::date
19WHERE c1.status_code != c2.status_code
20  AND content_type LIKE '%text/html%'
21ORDER BY
22  CASE change_type WHEN 'BROKEN' THEN 1 WHEN 'FIXED' THEN 2 ELSE 3 END,
23  c1.inlinks DESC NULLS LAST;

For teams that run Screaming Frog crawls regularly (weekly or monthly), manual CSV export and import becomes tedious. The Screaming Frog CLI — available in licensed versions — enables headless crawls that output directly to files. Combined with a Retool Workflow triggered by a webhook, you can fully automate the import pipeline. On your server or local machine, the Screaming Frog CLI command for a basic crawl with CSV output is: screaming-frog-seo-spider-cli --crawl https://example.com --headless --output-folder /path/to/output --export-tabs 'Internal:All' After the crawl completes, a script on the same machine can read the generated CSV and POST the data to a Retool Workflow webhook endpoint. In Retool, create a new Workflow and add a Webhook trigger. Retool generates a unique webhook URL with an API key in the format: https://api.retool.com/workflows/[id]/startTrigger. In the Workflow, add a Resource Query block connected to your database that runs a parameterized INSERT query. The webhook payload contains the crawl data rows as a JSON array (converted from CSV by the local script). The Loop block iterates over the rows and inserts each one. For the local conversion script, write a simple Python or Node.js script that reads the Screaming Frog CSV output, converts it to JSON, and posts it to the Retool Workflow webhook URL. Set this script as a cron job to run automatically after each scheduled crawl. For complex multi-site crawl pipelines with dozens of domains, data normalization across different site structures, and integration with SEO platforms like SEMrush or Google Search Console, RapidDev's team can help architect a robust automated SEO monitoring system built on Retool.

1# Python script — convert Screaming Frog CSV output and POST to Retool Workflow
2import csv
3import json
4import requests
5from datetime import date
6
7CSV_PATH = '/path/to/screaming-frog-output/internal_all.csv'
8WEBHOOK_URL = 'https://api.retool.com/workflows/YOUR_WORKFLOW_ID/startTrigger'
9WEBHOOK_API_KEY = 'retool_wk_YOUR_API_KEY'
10CRAWL_DATE = date.today().isoformat()
11
12def convert_csv_to_json(filepath):
13    rows = []
14    with open(filepath, encoding='utf-8-sig') as f:
15        reader = csv.DictReader(f)
16        for row in reader:
17            rows.append({
18                'crawl_date': CRAWL_DATE,
19                'address': row.get('Address', ''),
20                'content_type': row.get('Content Type', ''),
21                'status_code': int(row['Status Code']) if row.get('Status Code') else None,
22                'status': row.get('Status', ''),
23                'indexability': row.get('Indexability', ''),
24                'title_1': row.get('Title 1', ''),
25                'title_1_length': int(row['Title 1 Length']) if row.get('Title 1 Length') else None,
26                'meta_description_1': row.get('Meta Description 1', ''),
27                'h1_1': row.get('H1-1', ''),
28                'word_count': int(row['Word Count']) if row.get('Word Count') else None,
29                'crawl_depth': int(row['Crawl Depth']) if row.get('Crawl Depth') else None,
30                'inlinks': int(row['Inlinks']) if row.get('Inlinks') else None
31            })
32    return rows
33
34rows = convert_csv_to_json(CSV_PATH)
35
36response = requests.post(
37    WEBHOOK_URL,
38    headers={
39        'Content-Type': 'application/json',
40        'X-Workflow-Api-Key': WEBHOOK_API_KEY
41    },
42    json={'rows': rows, 'crawl_date': CRAWL_DATE}
43)
44print(f'Imported {len(rows)} rows, status: {response.status_code}')