Skip to content
Skills Library
Data & analytics · Conversion & formats

diffbot-extract

/diffbot-extract

Extract markdown or structured content from a URL using the Diffbot Extract API — a live fetch and parse of the page, seconds per call. Try `/diffbot-web-search` with `url:<URL>` first: it reads Diffbot's already-extracted copy from the Web Index in ~300ms and answers most page questions outright. Use this skill when that lookup misses (URL not indexed) or when you need the complete document rather than the relevant chunks. Triggers on: extract URL, fetch page, parse webpage, get content from URL, extract article, extract structured data, full page text, whole article.

Licence
MIT
Last repository change
28.09.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
6
Deprecation

No signals detected. Repository changed on 28.09.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Extract markdown or structured content from a URL using the Diffbot Extract API — a live fetch and parse of the page, seconds per call. Try `/diffbot-web-search` with `url:<URL>` first: it reads Diffbot's already-extracted copy from the Web Index in ~300ms and answers most page questions outright. Use this skill when that lookup misses (URL not indexed) or when you need the complete document rather than the relevant chunks. Triggers on: extract URL, fetch page, parse webpage, get content from URL, extract article, extract structured data, full page text, whole article.

Original author description

Extract markdown or structured content from a URL using the Diffbot Extract API — a live fetch and parse of the page, seconds per call. Try `/diffbot-web-search` with `url:<URL>` first: it reads Diffbot's already-extracted copy from the Web Index in ~300ms and answers most page questions outright. Use this skill when that lookup misses (URL not indexed) or when you need the complete document rather than the relevant chunks. Triggers on: extract URL, fetch page, parse webpage, get content from URL, extract article, extract structured data, full page text, whole article.

How it works

  1. First: is a live fetch actually needed?
  2. The db extract CLI
  3. Workflow
  4. Common fields in the JSON response
  5. Tips
Original SKILL.md headings · 110 lines, 705 words
SKILL.md preview
---
name: diffbot-extract
description: "Extract markdown or structured content from a URL using the Diffbot Extract API — a live fetch and parse of the page, seconds per call. Try `/diffbot-web-search` with `url:<URL>` first: it reads Diffbot's already-extracted copy from the Web Index in ~300ms and answers most page questions outright. Use this skill when that lookup misses (URL not indexed) or when you need the complete document rather than the relevant chunks. Triggers on: extract URL, fetch page, parse webpage, get content from URL, extract article, extract structured data, full page text, whole article."
allowed-tools: Bash(~/.diffbot/venv/bin/db:*), Bash(python3 -m venv ~/.diffbot/venv:*), Bash(~/.diffbot/venv/bin/pip install:*)
---

# Diffbot Extract

Extract structured content from any URL via the Diffbot Extract API. The CLI returns a clean markdown rendering of the page by default.

## First: is a live fetch actually needed?

Extract is a **live retrieval** — it fetches and parses the page on demand. Measured on
this CLI: 1.3 s–9 s for most URLs, 24 s cold for one docs page (0.8 s on a repeat), with
an occasional aborted run. That is the cost of going to the origin.

Diffbot has usually already parsed the page, and that copy is one index lookup away:

```
~/.diffbot/venv/bin/db web-search "url:<URL> <what you want to know>" -f json
```

~300 ms, one record back, and the chunks matching your terms — see `/diffbot-web-search`,
which also covers when a cached copy is *not* safe to trust. Run it first for any public
URL, the same way you would before reaching for `WebFetch` or `curl`. Come back here when:

- `search_results` came back **empty** — the URL isn't indexed (too new, paywalled,
  private, behind a login), so live retrieval is the only option;
- you need the **whole document**, not the ~4–6 KB of chunks the index lookup returns —
  every row of a table, a full changelog, an entire spec, license, or transcript
  (that docs page: 4.3 KB of chunks from the index, 25.5 KB of text from Extract);
- the page's **current state** is the question — a status page, a feed, a price, a
  version number — where a snapshot of unknown age is worse than no answer;
- you need Extract-only fields the search index doesn't carry: `author`, `images`,
  `links`, `tags`, `sentiment`, or a typed `product`/`discussion` parse.

## The `db extract` CLI

All work in this skill is driven by the `db` CLI from the [`diffbot-python`](https://github.com/diffbot/diffbot-python) library. **Always invoke with the fixed path `~/.diffbot/venv/bin/db`.**

```
~/.diffbot/venv/bin/db extract <URL> [-f markdown|json] [-a <api>] [-o <outfile>]
```

| Flag | Default | Description |
|------|---------|-------------|
| `-f`, `--format` | `markdown` | Output format: `markdown` (cleaned text) or `json` (full raw API response) |
| `-a`, `--api` | `analyze` | Diffbot API to use: `analyze` (auto-detect), `article`, `product`, `image`, `video`, `discussion` |
| `-o`, `--output` | stdout | Write output to file instead of stdout |

## Workflow

### Step 1 — bootstrap

```
[ -d ~/.diffbot/venv ] || python3 -m venv ~/.diffbot/venv && ~/.diffbot/venv/bin/pip install -q 'diffbot>=3.0.0'
```

The token is read from `DIFFBOT_API_TOKEN` env var or `~/.diffbot/credentials`. If missing:

```
echo "DIFFBOT_API_TOKEN=YOUR_TOKEN_HERE" > ~/.diffbot/credentials && chmod 600 ~/.diffbot/credentials
```

### Step 2 — extract content

For a quick read of a page:

```
~/.diffbot/venv/bin/db extract "https://example.com/article"
```

For the full raw JSON (useful to inspect all fields Diffbot returns):

```
~/.diffbot/venv/bin/db extract "https://example.com/article" -f json -o ~/.diffbot/tmp/extracted.json
```

Force a specific extractor when auto-detect isn't right:

```
~/.diffbot/venv/bin/db extract "https://example.com/product-page" -a product
~/.diffbot/venv/bin/db extract "https://example.com/article" -a article
```

### Step 3 — display

- For markdown format: the CLI renders `Title`, `URL`, and `Content` directly. Relay it to the user.
- For JSON format: large payloads should be written to a file with `-o`. Read the file and summarize the key fields.

## Common fields in the JSON response

The `objects[0]` structure from the `analyze`/`article` API includes:

| Field | Description |
|-------|-------------|
| `title` | Page/article title |
| `text` | Plain text of the main content |
| `content` | Markdown-formatted content |
| `pageUrl` | Canonical URL |
| `date` | Publication date (ISO 8601) |
| `author` | Author name |
| `tags` | Entity tags (from Diffbot KG) |
| `images` | Extracted images |
| `links` | Outbound links |
| `type` | Detected page type (`article`, `product`, etc.) |

## Tips

- Re-check the index before extracting a batch of URLs: a `url:` lookup per URL costs
  ~300 ms and usually removes the need to extract most of them.
- The CLI normalizes URLs automatically — you can omit `https://`.
- For saving article content to disk for later processing, use `-o` to avoid bloating the conversation context.
- If extraction fails with a 4xx/5xx error code, the page may be behind auth, Cloudflare, or a JS SPA. Try `-a article` explicitly as a fallback.

Versions

The database has no other skills named diffbot-extract.

Database snapshot: 13.11.2026 · package version d060eae851f5 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100diffbot-web-search
    Use for any web search, documentation search, or research request not falling under the DQL categories (news, organizations, people, places, deals). Returns ranked results with relevance scores, publication dates, and relevant chunks per result in one call. ALSO USE before…
  • 100diffbot-entities
    Identify and resolve named entities in text using the Diffbot NLP API. Links mentions to Diffbot Knowledge Graph entities with confidence scores and sentiment. Use when the user wants to extract entities from text, do NER (named entity recognition), identify companies or people…
  • 959router-web-fetch
    Fetch URL → markdown / text / HTML via 9Router /v1/web/fetch using Ollama Cloud / Firecrawl / Jina Reader / Tavily Extract / Exa Contents. Use when the user wants to scrape a webpage, extract URL content, read article, or convert a URL to markdown.
  • 100content-parser
    Extract and parse content from URLs. Triggers on: user provides a URL to extract content from, another skill needs to parse source material, "parse this URL", "extract content", "解析链接", "提取内容".
  • 90web-scraper
    Intelligent web scraper that fetches any URL and returns clean Markdown content. Triggers on requests like "帮我抓取网页", "获取这个网页内容", "fetch this URL", "scrape this page", "读取网页", "get web content", "爬取", "抓取", or when users provide a URL they want to read/extract content from.
  • 100fetch
    Default, free way to read a webpage or any URL's actual content: fetch the full page as clean markdown, HTML, or structured JSON, including JavaScript-rendered pages, for up to 10 URLs in parallel. Use to read or summarize a webpage or article, extract data from a website, web…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.