diffbot-crawl
/diffbot-crawl
Crawl a website or manage Diffbot crawler jobs. Use when the user wants to crawl a site for structured content, list existing crawl jobs, or delete a crawl job. Triggers on: crawl website, crawl site, crawler job, list crawl jobs, delete crawl job, diffbot crawl.
- Author
- diffbot ↗
- Repository
- diffbot-skills ↗
- Licence
- MIT
- Last repository change
- 28.09.2026
Assessment and security
- Files in this package
- 1
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 6
No signals detected. Repository changed on 28.09.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
What it does and when to use it
Crawl a website or manage Diffbot crawler jobs. Use when the user wants to crawl a site for structured content, list existing crawl jobs, or delete a crawl job. Triggers on: crawl website, crawl site, crawler job, list crawl jobs, delete crawl job, diffbot crawl.
Original author description
Crawl a website or manage Diffbot crawler jobs. Use when the user wants to crawl a site for structured content, list existing crawl jobs, or delete a crawl job. Triggers on: crawl website, crawl site, crawler job, list crawl jobs, delete crawl job, diffbot crawl.
How it works
- The db crawl CLI commands
- Workflow
- Understanding crawl output
- Tips
SKILL.md preview
--- name: diffbot-crawl description: "Crawl a website or manage Diffbot crawler jobs. Use when the user wants to crawl a site for structured content, list existing crawl jobs, or delete a crawl job. Triggers on: crawl website, crawl site, crawler job, list crawl jobs, delete crawl job, diffbot crawl." allowed-tools: Bash(~/.diffbot/venv/bin/db:*), Bash(python3 -m venv ~/.diffbot/venv:*), Bash(~/.diffbot/venv/bin/pip install:*) --- # Diffbot Crawler Crawl websites for structured content via the Diffbot Crawlbot API, and manage crawler jobs. ## The `db crawl*` CLI commands **Always invoke with the fixed path `~/.diffbot/venv/bin/db`.** ``` ~/.diffbot/venv/bin/db crawl <SITE> [options] # start a crawl and stream events ~/.diffbot/venv/bin/db crawl-list-jobs [JOB_NAME] # list all jobs or inspect one ~/.diffbot/venv/bin/db crawl-delete-job <JOB_NAME> # delete a job ``` ### `db crawl` options | Flag | Default | Description | |------|---------|-------------| | `--hops` | `2` | Maximum link depth from seed URLs | | `--job-name` | auto-generated | Name for the crawler job | | `--max-to-crawl` | `100` | Maximum pages to crawl | | `--max-to-process` | `100` | Maximum pages to process | | `--restrict-domain` | `true` | Only follow links on the same domain as seeds | | `--api-url` | `""` | Diffbot API endpoint for processing (default: auto/analyze) | | `--crawl-delay` | `-1` | Delay between requests to same domain (seconds) | | `--url-crawl-pattern` | none | Only crawl URLs matching this pattern | | `--url-process-pattern` | none | Only process URLs matching this pattern | | `--obey-robots` | off | Obey robots.txt | | `--use-proxies` | off | Use proxies for crawling | | `--custom-headers` | none | Newline-separated custom HTTP headers | | `-f`, `--format` | `markdown` | Output: `markdown` or `json` | | `-o`, `--output` | stdout | Write output to file | ## Workflow ### Step 1 — bootstrap ``` [ -d ~/.diffbot/venv ] || python3 -m venv ~/.diffbot/venv && ~/.diffbot/venv/bin/pip install -q 'diffbot>=3.0.0' ``` The token is read from `DIFFBOT_API_TOKEN` env var or `~/.diffbot/credentials`. If missing: ``` echo "DIFFBOT_API_TOKEN=YOUR_TOKEN_HERE" > ~/.diffbot/credentials && chmod 600 ~/.diffbot/credentials ``` ### Step 2 — crawl a site Basic crawl with defaults (100 pages, 2 hops deep): ``` ~/.diffbot/venv/bin/db crawl "https://example.com" ``` Deep crawl of a documentation site, saving to JSON: ``` ~/.diffbot/venv/bin/db crawl "https://docs.example.com" \ --hops 3 \ --max-to-crawl 500 \ --max-to-process 500 \ --restrict-domain \ -f json \ -o ~/.diffbot/tmp/crawl_results.json ``` Crawl only blog posts (by URL pattern): ``` ~/.diffbot/venv/bin/db crawl "https://example.com" \ --url-process-pattern "/blog/" \ --max-to-process 200 ``` ### Step 3 — manage jobs List all active jobs: ``` ~/.diffbot/venv/bin/db crawl-list-jobs ``` Inspect a specific job: ``` ~/.diffbot/venv/bin/db crawl-list-jobs my-job-name ``` Delete a job: ``` ~/.diffbot/venv/bin/db crawl-delete-job my-job-name ``` ## Understanding crawl output The `crawl` command streams events as they happen. In markdown format (default): ``` # Job: my-crawl-job-1234567890 - [Success] https://example.com/ - [Success] https://example.com/about - [!] https://example.com/403-page ``` `[!]` marks pages that were crawled but failed extraction (not HTTP failures — those are skipped silently). In JSON format each event has `event_type`, `timestamp`, and `details`. Event types: - `job_created` — job was created on the server; `details.job_name` has the job name - `url_processed` — a page was processed; `details.url` and `details.status` (`Success` or error) ## Tips - `--restrict-domain` is on by default — disable it only if you intentionally want to follow external links. - Use `--url-process-pattern` to focus extraction on specific path segments without narrowing crawling. - Job names are stable identifiers — always set `--job-name` if you want to look up or delete a job later. - For large crawls, write output to a file with `-o` rather than reading it all into the conversation.
Versions
The database has no other skills named diffbot-crawl.
Database snapshot: 13.11.2026 · package version 67f35557d6fc (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 90cf-crawl
Use when crawling a website with Cloudflare Browser Rendering /crawl to ingest web content. - 100diffbot-extract
Extract markdown or structured content from a URL using the Diffbot Extract API — a live fetch and parse of the page, seconds per call. Try `/diffbot-web-search` with `url:<URL>` first: it reads Diffbot's already-extracted copy from the Web Index in ~300ms and answers most page… - 95crawl
Crawl multiple pages of a website with Scrapeless, controlling page limits, traversal depth, path filters, and output formats. Use for site-wide documentation collection or checking an existing crawl task. - 100diffbot-entities
Identify and resolve named entities in text using the Diffbot NLP API. Links mentions to Diffbot Knowledge Graph entities with confidence scores and sentiment. Use when the user wants to extract entities from text, do NER (named entity recognition), identify companies or people… - 100firecrawl-crawl
Bulk extract content from an entire website or site section. Use this skill when the user wants to crawl a site, extract all pages from a docs section, bulk-scrape multiple pages following links, or says "crawl", "get all the pages", "extract everything under /docs", "bulk… - 100firecrawl-crawl
Bulk-extract many pages from one site or section. Use for "crawl", "everything under /docs", or content spanning linked pages.