How to Install and Use Deep Scraper in OpenClaw

Installing Deep Scraper is the simplest ClawHub skill installation in the Search & Scraping category because there is no API key to configure afterward. One command and it is ready to use. Open your terminal and run: ``` clawhub install deep-scraper ``` The ClawHub registry downloads the skill package and registers it with your OpenClaw installation. The process typically takes 15-30 seconds. Verify the installation: ``` clawhub list | grep deep-scraper ``` You should see `deep-scraper` with a version number and 'active' status. Because Deep Scraper does not require an external API key, you can start using it immediately after installation — there is no Step 2 for credentials. This is what makes it the fastest extraction skill to get started with in ClawHub. One note on what 'no API key required' means in practice: Deep Scraper uses OpenClaw's HTTP client directly, which means it makes requests from your OpenClaw instance's IP address. If you are running OpenClaw on a server or cloud instance, the scraping requests originate from that server. If you are running OpenClaw locally, they originate from your local machine's IP. This is different from Firecrawl or Tavily, which route requests through their own infrastructure.

1# Install Deep Scraper
2clawhub install deep-scraper
3
4# Verify installation
5clawhub list | grep deep-scraper
6
7# View skill details and available options
8clawhub info deep-scraper

While Deep Scraper requires no API key, it does have configurable settings that control how it behaves during recursive crawls. Understanding and setting these before your first crawl prevents runaway scraping jobs that consume too many resources or time. The three most important settings are: **default_depth** — how many link levels to follow from the starting URL. Default is typically 1 (start page + directly linked pages). Setting this to 2 or 3 dramatically increases the number of pages scraped. **max_pages** — a hard cap on the total number of pages scraped in one job. This is your safety net against accidentally crawling an enormous site. Set it to a sensible limit for your use case (e.g., 50 for exploration, 200 for larger extraction jobs). **request_timeout_ms** — how long to wait for each page to respond before skipping it. For slow or international sites, increasing this prevents incomplete extractions. You can set these globally in OpenClaw config, or override them inline in your prompts ('scrape 3 levels deep with a maximum of 100 pages'). ``` openclaw config set skills.deep-scraper.default_depth 2 openclaw config set skills.deep-scraper.max_pages 100 openclaw config set skills.deep-scraper.request_timeout_ms 10000 ``` For your first few scrapes, keep max_pages at 20-30 to get a feel for how much content deep scraping generates before scaling up.

1# Set default crawl depth (2 = start page + 2 levels of linked pages)
2openclaw config set skills.deep-scraper.default_depth 2
3
4# Set max pages per job (safety cap)
5openclaw config set skills.deep-scraper.max_pages 100
6
7# Set page request timeout in milliseconds
8openclaw config set skills.deep-scraper.request_timeout_ms 10000
9
10# Add delay between requests to be a polite scraper
11openclaw config set skills.deep-scraper.request_delay_ms 500

Open OpenClaw chat and give Deep Scraper a starting URL with a description of what content to extract. The skill follows links recursively and returns aggregated content from all scraped pages. Start with a focused test on a small, known section of a site to verify the skill is working correctly before running larger jobs. Good first prompt: ``` Deep scrape https://example.com/blog/page/1 up to 2 levels deep with a max of 20 pages. Extract the title and first paragraph of each article you find. ``` Deep Scraper will: 1. Fetch the starting URL 2. Extract content as specified 3. Find all internal links on the page 4. Follow those links and repeat 5. Stop when it reaches the depth limit or max pages cap 6. Return aggregated results in OpenClaw chat For content-heavy sites, deep scrapes on 50+ pages can take several minutes. OpenClaw will show progress updates as pages are scraped so you know the job is running. If the scrape returns very little content or seems to skip most pages, the target site may be blocking automated requests. See the Troubleshooting section for how to handle this.

The most common problem with naive deep scraping is link bleed — following external links to other domains or navigating far outside the section you intended to scrape. Deep Scraper supports scoping instructions that keep the crawl within bounds. **Path prefix scoping:** Tell Deep Scraper to only follow links that stay within a specific URL path. This is the single most effective way to contain a deep scrape: ``` Deep scrape https://docs.example.com/api starting from the API overview page. Only follow links that stay within https://docs.example.com/api/. Go 3 levels deep. ``` **Domain restriction:** Limit the crawl to a single domain (no external links followed): ``` Deep scrape https://example.com/products. Stay within the example.com domain only. Do not follow links to external sites, payment processors, or social media. ``` **Content-type targeting:** Instruct Deep Scraper to extract specific content types rather than all page text: ``` Deep scrape https://example.com/blog 2 levels deep. For each page, extract only: the article headline, author name, and article body text. Skip navigation menus, sidebars, and footers. ``` For large-scale deep scraping use cases in production environments — building training datasets, knowledge base pipelines, or competitive intelligence feeds — the RapidDev team offers deep-scraper configuration templates and prompt libraries at rapiddev.ai that handle common edge cases like pagination, JavaScript-rendered content, and incremental scraping to avoid re-fetching already-scraped pages. Always review a site's robots.txt (`https://example.com/robots.txt`) and terms of service before running large deep scrapes. Deep Scraper checks robots.txt by default but verifying manually is good practice.

1# Check if a site allows scraping before starting
2curl https://target-site.com/robots.txt
3
4# Reload skill config after changes
5openclaw reload
6
7# View full deep-scraper config options
8clawhub info deep-scraper --verbose