docs-crawler
/docs-crawler
Crawl an entire documentation or design-system website into one LLM-ready Markdown corpus. Discovers every page from the site's sitemap.xml (with a same-origin link-following fallback when there is no sitemap), extracts each page's main content as clean Markdown, downloads each page's images (and inline base64 images) into a local folder referenced by relative paths, and renders JavaScript-heavy pages with a headless browser. Use this whenever someone wants the WHOLE of a multi-page docs site, design-system site, API reference, component library, or knowledge base turned into Markdown — for ex
- Author
- CaesiumY ↗
- Repository
- ko-design-md ↗
- Licence
- MIT
- Last repository change
- 25.10.2026
Assessment and security
- Files in this package
- 17
- Scripts
- 14
- HTML comments
- 0
- URLs in SKILL.md
- 1
No signals detected. Repository changed on 25.10.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
- 14 scripts in this package. Review them before installing, as Claude may execute them.
Scripts: scripts/crawl.ts, scripts/crawl/args.test.ts, scripts/crawl/args.ts, scripts/crawl/corpus.test.ts, scripts/crawl/corpus.ts, scripts/crawl/discover.test.ts, scripts/crawl/discover.ts, scripts/crawl/extract.test.ts
What it does and when to use it
Crawl an entire documentation or design-system website into one LLM-ready Markdown corpus. Discovers every page from the site's sitemap.xml (with a same-origin link-following fallback when there is no sitemap), extracts each page's main content as clean Markdown, downloads each page's images (and inline base64 images) into a local folder referenced by relative paths, and renders JavaScript-heavy pages with a headless browser. Use this whenever someone wants the WHOLE of a multi-page docs site, design-system site, API reference, component library, or knowledge base turned into Markdown — for ex
Triggers from the description
Original author description
Crawl an entire documentation or design-system website into one LLM-ready Markdown corpus. Discovers every page from the site's sitemap.xml (with a same-origin link-following fallback when there is no sitemap), extracts each page's main content as clean Markdown, downloads each page's images (and inline base64 images) into a local folder referenced by relative paths, and renders JavaScript-heavy pages with a headless browser. Use this whenever someone wants the WHOLE of a multi-page docs site, design-system site, API reference, component library, or knowledge base turned into Markdown — for ex
How it works
- When to use
- Inputs
- Running the crawl
- Output
- After crawling
- Output location and the design-md skill
- Notes and edge cases
SKILL.md preview
---
name: docs-crawler
metadata:
internal: true
description: >-
Crawl an entire documentation or design-system website into one LLM-ready
Markdown corpus. Discovers every page from the site's sitemap.xml (with a
same-origin link-following fallback when there is no sitemap), extracts each
page's main content as clean Markdown, downloads each page's images (and
inline base64 images) into a local folder referenced by relative paths, and
renders JavaScript-heavy pages with a headless browser.
Use this whenever someone wants the WHOLE of a multi-page docs site,
design-system site, API reference, component library, or knowledge base
turned into Markdown — for example "crawl this docs site", "turn this design
system site into markdown", "make an LLM or RAG corpus from these docs",
"mirror the entire documentation", "archive the whole knowledge base", or
"give an AI the full X documentation". It applies even when the user never
says the word "crawl" — any request to capture, ingest, archive, or convert
a whole site's documentation pages into text fits. The design-md skill also
calls it during research to gather a brand's published design-system docs.
Do not use it for reading or summarizing a single page, for scraping
structured data such as prices into a spreadsheet, or for generating a
sitemap.
---
# docs-crawler
Crawl a documentation website into a single Markdown corpus that an LLM can
consume as context.
## When to use
The user wants to capture a multi-page documentation, design-system, or
reference site as Markdown — to feed an LLM, archive it, or research a design
system. This skill is also invoked by the `design-md` skill's research phase.
## Inputs
- **Site URL** (required) — the documentation site, e.g.
`https://socarframe.socar.kr/`. Any page on the site works; page discovery
starts from the site's `sitemap.xml`.
- **Output directory** (optional) — where to write the corpus. Defaults to
`./{host}-docs/` in the current working directory.
If the user has not provided a URL, ask for one — do not guess.
## Running the crawl
From the repository root:
```
pnpm crawl:docs <site-url> [--out <dir>] [--external-images]
```
Equivalent direct form:
```
pnpm exec tsx .claude/skills/docs-crawler/scripts/crawl.ts <site-url> [--out <dir>]
```
The engine discovers pages via `sitemap.xml` (falling back to same-origin
link-following), fetches each page, extracts the main content, and converts it
to Markdown. It prints per-page progress and a final summary line.
By default it also **localizes images**: every external image and inline base64
`data:` image is downloaded into `crawl/images/` and the Markdown is rewritten
to relative paths, so the corpus is self-contained for renderers that can't
fetch external URLs (Claude Design, offline previews). Pass `--external-images`
to skip downloading and keep the original external URLs instead.
## Output
These artifacts are written under the output directory:
- **`crawl-corpus.md`** — every successful page merged into one document, with
a table of contents and a `Source:` URL per page. This is the primary
deliverable.
- **`crawl/pages/{NNN}-{slug}.md`** — one file per page, each with
`source_url` / `title` / `method` frontmatter.
- **`crawl/images/`** — downloaded image files (skipped with
`--external-images`), referenced from the corpus and page files by relative
paths.
- **`crawl/manifest.json`** — a per-URL audit log (status, fetch method,
extracted character count, error reason).
## After crawling
Report back to the user:
- How many pages succeeded versus failed (from the summary line or
`crawl/manifest.json`).
- The path to `crawl-corpus.md`.
- If any pages failed, list them from the manifest with the reason (e.g.
`disallowed by robots.txt`, `no extractable content`).
Do not paste the whole corpus into the conversation — point the user at the
file. The corpus can be large.
## Output location and the design-md skill
The `--out` argument controls where everything is written. When the `design-md`
skill calls this crawler during its research phase, it passes `--out` pointing
at its cache directory (`.claude/cache/design-md/{slug}/`), so the resulting
`crawl-corpus.md` (and the localized `crawl/images/`) land where
`research-collector` can read and cite them. The whole cache directory is
gitignored, so downloaded images stay out of version control. Nothing special
is needed for that case — just run the crawl with the given `--out`.
## Notes and edge cases
- **No sitemap** — discovery automatically falls back to following same-origin
links from the start URL.
- **JavaScript-rendered sites** — if a page returns an empty shell, the engine
re-fetches it through a headless browser. Chromium is installed automatically
on first need (~150 MB, one-time). Static sites never launch a browser.
- **Images** are downloaded into `crawl/images/` by default and referenced by
relative paths, so the corpus is self-contained — this covers external
`<img>` URLs and inline base64 `data:` images alike. A non-base64
(percent-encoded) `data:` URI, or any image whose bytes fail to download or
decode, instead becomes an `inline-image-omitted` placeholder (alt and title
text are kept) so raw base64 never bloats the corpus. With `--external-images`
the crawler downloads nothing: external URLs stay as-is and inline `data:`
images collapse to the placeholder.
- **Few or no images is normal, especially for design-system sites** — modern
docs and design-system sites (Docusaurus and similar) render icons and
component visuals as inline `<svg>`, CSS, or live DOM rather than `<img>`
files; any real `<img>` (e.g. a navbar logo) is usually site chrome the
content extractor correctly drops. An image-light or image-free corpus is
expected in that case — it is not a crawl failure. Capture visuals separately
(screenshots) if the research needs them.
- **robots.txt** is respected; disallowed paths are skipped.
- **Page cap** — at most 200 pages per crawl.
- **Zero pages crawled** — the engine exits non-zero and prints the reason
(bad URL, blocked, or an empty site). Surface that to the user rather than
reporting success.
Files in this package (17)
- SKILL.md 6 KB
- trigger-eval-queries.json 3 KB
- evals/evals.json 4 KB
- scripts/crawl.ts 11 KB
- scripts/crawl/args.test.ts 2 KB
- scripts/crawl/args.ts 3 KB
- scripts/crawl/corpus.test.ts 3 KB
- scripts/crawl/corpus.ts 3 KB
- scripts/crawl/discover.test.ts 3 KB
- scripts/crawl/discover.ts 7 KB
- scripts/crawl/extract.test.ts 7 KB
- scripts/crawl/extract.ts 5 KB
- scripts/crawl/fetch-page.test.ts 1 KB
- scripts/crawl/fetch-page.ts 5 KB
- scripts/crawl/images.test.ts 8 KB
- scripts/crawl/images.ts 10 KB
- scripts/crawl/types.ts 2 KB
Versions
The database has no other skills named docs-crawler.
Database snapshot: 13.11.2026 · package version 26341b13eb3d (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100scrapling
Fetches web pages and extracts structured data from them with Python, from a single HTTP request to a full crawl, including pages rendered by JavaScript and selectors that keep working after a site redesign. Use when a user asks to scrape a website, build a crawler or spider,… - 100algolia-crawler
Use this skill whenever a user wants to crawl one or more web pages or a whole site and turn them into an Algolia index using the Algolia CLI — especially for RAG, AI search, semantic search, or Agent Studio retrieval. Triggers: "index my website/docs with Algolia", "set up the… - 100webforai
Convert web pages or local HTML files to clean, LLM-ready Markdown with the webforai CLI. Use when a task needs page content as Markdown - reading documentation or articles, scraping, summarising URLs, crawling a docs site into files or llms.txt, building text corpora -… - 100read-web-page
Read a public web page as clean markdown with SnapRender, rendered in a real browser so JavaScript-built pages come through. Use before writing code against online docs, an API reference or a changelog, or when the user asks to read, summarize or quote a page. - 95docs-page-generator
When the user wants to create, optimize, or structure a documentation site. Also use when the user mentions "docs," "documentation site," "docs subdomain," "docs.yourdomain.com," "help center," "knowledge base," "Getting Started," "API Reference," "user guides," or "tutorials."… - 100read-web-page
Read a public web page as clean markdown with SnapRender, rendered in a real browser so pages built with JavaScript come through. Use before writing code against online docs, an API reference or a changelog, or when the user asks to read, summarize or quote a page.