diffbot-extract
/diffbot-extract
Extract markdown or structured content from a URL using the Diffbot Extract API — a live fetch and parse of the page, seconds per call. Try `/diffbot-web-search` with `url:<URL>` first: it reads Diffbot's already-extracted copy from the Web Index in ~300ms and answers most page questions outright. Use this skill when that lookup misses (URL not indexed) or when you need the complete document rather than the relevant chunks. Triggers on: extract URL, fetch page, parse webpage, get content from URL, extract article, extract structured data, full page text, whole article.
- Author
- diffbot ↗
- Repository
- diffbot-skills ↗
- Licence
- MIT
- Last repository change
- 28.09.2026
Assessment and security
- Files in this package
- 1
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 6
No signals detected. Repository changed on 28.09.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
What it does and when to use it
Extract markdown or structured content from a URL using the Diffbot Extract API — a live fetch and parse of the page, seconds per call. Try `/diffbot-web-search` with `url:<URL>` first: it reads Diffbot's already-extracted copy from the Web Index in ~300ms and answers most page questions outright. Use this skill when that lookup misses (URL not indexed) or when you need the complete document rather than the relevant chunks. Triggers on: extract URL, fetch page, parse webpage, get content from URL, extract article, extract structured data, full page text, whole article.
Original author description
Extract markdown or structured content from a URL using the Diffbot Extract API — a live fetch and parse of the page, seconds per call. Try `/diffbot-web-search` with `url:<URL>` first: it reads Diffbot's already-extracted copy from the Web Index in ~300ms and answers most page questions outright. Use this skill when that lookup misses (URL not indexed) or when you need the complete document rather than the relevant chunks. Triggers on: extract URL, fetch page, parse webpage, get content from URL, extract article, extract structured data, full page text, whole article.
How it works
- First: is a live fetch actually needed?
- The db extract CLI
- Workflow
- Common fields in the JSON response
- Tips
SKILL.md preview
--- name: diffbot-extract description: "Extract markdown or structured content from a URL using the Diffbot Extract API — a live fetch and parse of the page, seconds per call. Try `/diffbot-web-search` with `url:<URL>` first: it reads Diffbot's already-extracted copy from the Web Index in ~300ms and answers most page questions outright. Use this skill when that lookup misses (URL not indexed) or when you need the complete document rather than the relevant chunks. Triggers on: extract URL, fetch page, parse webpage, get content from URL, extract article, extract structured data, full page text, whole article." allowed-tools: Bash(~/.diffbot/venv/bin/db:*), Bash(python3 -m venv ~/.diffbot/venv:*), Bash(~/.diffbot/venv/bin/pip install:*) --- # Diffbot Extract Extract structured content from any URL via the Diffbot Extract API. The CLI returns a clean markdown rendering of the page by default. ## First: is a live fetch actually needed? Extract is a **live retrieval** — it fetches and parses the page on demand. Measured on this CLI: 1.3 s–9 s for most URLs, 24 s cold for one docs page (0.8 s on a repeat), with an occasional aborted run. That is the cost of going to the origin. Diffbot has usually already parsed the page, and that copy is one index lookup away: ``` ~/.diffbot/venv/bin/db web-search "url:<URL> <what you want to know>" -f json ``` ~300 ms, one record back, and the chunks matching your terms — see `/diffbot-web-search`, which also covers when a cached copy is *not* safe to trust. Run it first for any public URL, the same way you would before reaching for `WebFetch` or `curl`. Come back here when: - `search_results` came back **empty** — the URL isn't indexed (too new, paywalled, private, behind a login), so live retrieval is the only option; - you need the **whole document**, not the ~4–6 KB of chunks the index lookup returns — every row of a table, a full changelog, an entire spec, license, or transcript (that docs page: 4.3 KB of chunks from the index, 25.5 KB of text from Extract); - the page's **current state** is the question — a status page, a feed, a price, a version number — where a snapshot of unknown age is worse than no answer; - you need Extract-only fields the search index doesn't carry: `author`, `images`, `links`, `tags`, `sentiment`, or a typed `product`/`discussion` parse. ## The `db extract` CLI All work in this skill is driven by the `db` CLI from the [`diffbot-python`](https://github.com/diffbot/diffbot-python) library. **Always invoke with the fixed path `~/.diffbot/venv/bin/db`.** ``` ~/.diffbot/venv/bin/db extract <URL> [-f markdown|json] [-a <api>] [-o <outfile>] ``` | Flag | Default | Description | |------|---------|-------------| | `-f`, `--format` | `markdown` | Output format: `markdown` (cleaned text) or `json` (full raw API response) | | `-a`, `--api` | `analyze` | Diffbot API to use: `analyze` (auto-detect), `article`, `product`, `image`, `video`, `discussion` | | `-o`, `--output` | stdout | Write output to file instead of stdout | ## Workflow ### Step 1 — bootstrap ``` [ -d ~/.diffbot/venv ] || python3 -m venv ~/.diffbot/venv && ~/.diffbot/venv/bin/pip install -q 'diffbot>=3.0.0' ``` The token is read from `DIFFBOT_API_TOKEN` env var or `~/.diffbot/credentials`. If missing: ``` echo "DIFFBOT_API_TOKEN=YOUR_TOKEN_HERE" > ~/.diffbot/credentials && chmod 600 ~/.diffbot/credentials ``` ### Step 2 — extract content For a quick read of a page: ``` ~/.diffbot/venv/bin/db extract "https://example.com/article" ``` For the full raw JSON (useful to inspect all fields Diffbot returns): ``` ~/.diffbot/venv/bin/db extract "https://example.com/article" -f json -o ~/.diffbot/tmp/extracted.json ``` Force a specific extractor when auto-detect isn't right: ``` ~/.diffbot/venv/bin/db extract "https://example.com/product-page" -a product ~/.diffbot/venv/bin/db extract "https://example.com/article" -a article ``` ### Step 3 — display - For markdown format: the CLI renders `Title`, `URL`, and `Content` directly. Relay it to the user. - For JSON format: large payloads should be written to a file with `-o`. Read the file and summarize the key fields. ## Common fields in the JSON response The `objects[0]` structure from the `analyze`/`article` API includes: | Field | Description | |-------|-------------| | `title` | Page/article title | | `text` | Plain text of the main content | | `content` | Markdown-formatted content | | `pageUrl` | Canonical URL | | `date` | Publication date (ISO 8601) | | `author` | Author name | | `tags` | Entity tags (from Diffbot KG) | | `images` | Extracted images | | `links` | Outbound links | | `type` | Detected page type (`article`, `product`, etc.) | ## Tips - Re-check the index before extracting a batch of URLs: a `url:` lookup per URL costs ~300 ms and usually removes the need to extract most of them. - The CLI normalizes URLs automatically — you can omit `https://`. - For saving article content to disk for later processing, use `-o` to avoid bloating the conversation context. - If extraction fails with a 4xx/5xx error code, the page may be behind auth, Cloudflare, or a JS SPA. Try `-a article` explicitly as a fallback.
Versions
The database has no other skills named diffbot-extract.
Database snapshot: 13.11.2026 · package version d060eae851f5 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100diffbot-web-search
Use for any web search, documentation search, or research request not falling under the DQL categories (news, organizations, people, places, deals). Returns ranked results with relevance scores, publication dates, and relevant chunks per result in one call. ALSO USE before… - 100diffbot-entities
Identify and resolve named entities in text using the Diffbot NLP API. Links mentions to Diffbot Knowledge Graph entities with confidence scores and sentiment. Use when the user wants to extract entities from text, do NER (named entity recognition), identify companies or people… - 959router-web-fetch
Fetch URL → markdown / text / HTML via 9Router /v1/web/fetch using Ollama Cloud / Firecrawl / Jina Reader / Tavily Extract / Exa Contents. Use when the user wants to scrape a webpage, extract URL content, read article, or convert a URL to markdown. - 100content-parser
Extract and parse content from URLs. Triggers on: user provides a URL to extract content from, another skill needs to parse source material, "parse this URL", "extract content", "解析链接", "提取内容". - 90web-scraper
Intelligent web scraper that fetches any URL and returns clean Markdown content. Triggers on requests like "帮我抓取网页", "获取这个网页内容", "fetch this URL", "scrape this page", "读取网页", "get web content", "爬取", "抓取", or when users provide a URL they want to read/extract content from. - 100fetch
Default, free way to read a webpage or any URL's actual content: fetch the full page as clean markdown, HTML, or structured JSON, including JavaScript-rendered pages, for up to 10 URLs in parallel. Use to read or summarize a webpage or article, extract data from a website, web…