crawlberg-headless-fallback
/crawlberg-headless-fallback
Use when a static fetch returns nothing useful and the page needs a real browser. Covers `--browser-mode auto|always|never`, external CDP via `--browser-endpoint`, symptoms of JS-only pages and WAF blocks, and the performance cost.
- Author
- xberg-io ↗
- Repository
- crawlberg ↗
- Licence
- MIT
- Last repository change
- 27.10.2026
Assessment and security
- Files in this package
- 2
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 4
No signals detected. Repository changed on 27.10.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
What it does and when to use it
Use when a static fetch returns nothing useful and the page needs a real browser. Covers `--browser-mode auto|always|never`, external CDP via `--browser-endpoint`, symptoms of JS-only pages and WAF blocks, and the performance cost.
Original author description
Use when a static fetch returns nothing useful and the page needs a real browser. Covers `--browser-mode auto|always|never`, external CDP via `--browser-endpoint`, symptoms of JS-only pages and WAF blocks, and the performance cost.
How it works
- Modes
- Symptoms that point to headless
- External CDP endpoint
- Performance cost
- Wait strategies
- Persistent profiles
SKILL.md preview
---
type: Playbook
title: Headless Fallback
description: Use when a static fetch returns nothing useful and the page needs a real browser. Covers `--browser-mode auto|always|never`, external CDP via `--browser-endpoint`, symptoms of JS-only pages and WAF blocks, and the performance cost.
x-ai-rulez:
kind: skill
id: crawlberg-headless-fallback
metadata:
name: crawlberg-headless-fallback
---
# Headless fallback
Some pages are unscrapable without a real browser — SPA shells, infinite
scroll, Cloudflare interstitials, JS-rendered article bodies. Crawlberg
ships with an optional headless-Chrome backend driven by chromiumoxide.
## Modes
```text
--browser-mode auto # default — try static first, fall back to browser on JS/WAF
--browser-mode always # skip static, go straight to browser
--browser-mode never # static only, fail closed
```
### `auto` (default)
The engine fetches statically, then inspects the response. It launches
headless Chrome and re-fetches when it sees:
- WAF responses from one of 8 detected vendor fingerprints (Cloudflare,
Akamai, AWS WAF, Imperva, DataDome, PerimeterX, F5, plus a generic
catch-all).
- SPA shells: `<noscript>` warnings, near-empty `<body>` with heavy JS.
- Heuristic JS-render-required signals.
This is the right default. The browser only spins up when needed.
### `always`
Skip the static probe entirely. Use when:
- The user already told you the page needs JS.
- You are scraping a site you know is React/Vue/Svelte SPA.
- You need `<script>`-emitted state that never lands in static HTML.
```bash
crawlberg scrape https://spa.example.com --browser-mode always --format markdown
```
### `never`
Static only — the browser path is disabled. Use when:
- You are in a hot loop where a stray Chrome launch would blow the budget.
- You are running in a sandbox without a Chrome binary.
- The user explicitly wants only static fetches.
In `never` mode, JS-only pages return empty/stub content. Inspect
`markdown.content` and `markdown.warnings` before treating the result as
final.
## Symptoms that point to headless
In `--browser-mode never` or when you suspect the auto detector missed a
signal:
- `markdown.content` is short, nav-only, or just a loading message.
- `status_code` is 200 but `metadata.headings` is empty on a page that
clearly has headings.
- `markdown.warnings` mentions JS-render-required or WAF detection.
- 403/406/503 with WAF response headers (`server: cloudflare`,
`cf-mitigated`, `x-amz-cf-id`, `set-cookie: __cf_bm=…`).
Re-run with `--browser-mode always`. If that succeeds, leave it set for
that host.
## External CDP endpoint
Point at an already-running Chrome (Browserless, Steel, your own) instead
of launching locally:
```bash
crawlberg scrape https://example.com \
--browser-mode always \
--browser-endpoint ws://browser.internal:9222/devtools/browser/<id> \
--format markdown
```
The endpoint must be a WebSocket URL — `ws://` or `wss://`. The CLI
rejects anything else with a clear error.
Use external CDP when:
- You are running in containers or CI without a local Chrome.
- You want a shared, warm browser pool across many crawl jobs.
- You need browser-side residential proxies or stealth configuration the
local Chrome cannot provide.
## Performance cost
Headless Chrome is expensive relative to a static fetch:
- Cold start: 1-3 seconds the first time it launches.
- Per-page overhead: 500 ms-2 s for `NetworkIdle` wait, plus the page's own
JS load time.
- Memory: each tab takes 100-300 MB; long crawls should bound
`--concurrent`.
Mitigations:
- Stay in `--browser-mode auto` — the engine only pays the cost when it
needs to.
- Use `--browser-endpoint` to share one warm browser across jobs.
- Drop `--concurrent` when you know the crawl will route through Chrome.
## Wait strategies
Pass via `--config` JSON when you need control:
```bash
crawlberg scrape https://example.com --browser-mode always \
--config '{"browser":{"wait":"selector","wait_selector":".article-body"}}'
```
Supported strategies (the `browser.wait` field is a string enum; pair
`"selector"` with a sibling `wait_selector`):
- `network_idle` (default) — wait until the network goes quiet.
- `selector` — wait until the CSS selector in `wait_selector` resolves.
- `fixed` — wait a fixed duration.
`extra_wait` adds milliseconds on top of the wait strategy if the page
keeps loading content after the primary signal.
## Persistent profiles
```bash
crawlberg scrape https://app.example.com --browser-mode always \
--config '{"browser_profile":"prod","save_browser_profile":true}'
```
Profile names are path-traversal-validated. Use them to keep cookies,
localStorage, and login state across runs without re-authenticating.
Files in this package (2)
- SKILL.md 5 KB
- index.md 1 KB
Versions
| Repository | Score | Licence |
|---|---|---|
| This page · xberg-io/crawlberg | 84 | MIT |
| xberg-io/crawlberg | 100 | MIT |
| xberg-io/crawlberg | 100 | MIT |
Database snapshot: 13.11.2026 · package version 47f85106a068 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 84crawlberg-automating-the-browser
Use when extracting a page needs scripted interaction first — click, type, press a key, scroll, wait, screenshot, or run JS before capturing the DOM. Covers `crawlberg interact URL --actions` with the real action schema, result shape, limits, and external-CDP options. - 100crawlberg
Crawl, scrape, and convert websites to Markdown using the local crawlberg CLI and its MCP server. Use when the user wants to fetch a page, follow links across a domain, enumerate URLs, or drive a real browser. Covers installation, the subcommands (scrape, crawl, map, interact,… - 100lightpanda
Operate lightpanda-io/browser, the headless browser for AI and automation: offline preflight, release selection, approved fetch and CDP/MCP integration. Use when users name Lightpanda or need its source-pinned setup and runtime boundaries; route generic browser automation to… - 86blocked-page-recovery
Use when a fetch fails: 403/429, paywall, WAF, bot wall. - 100fetch
Use when the user wants one known URL returned as content: "get me this page", read this article, this page as markdown, save the raw HTML, screenshot this page, or page text for a RAG index or model context. Also when a plain fetch of a URL came back blocked or challenged… - 100fetchmcp
Fetch a public web page live and get its main content as clean Markdown, plain text or HTML, through the FetchMCP API. Use when you have a specific URL and need what the page says now, for example docs, articles, product or listing pages, including pages that need a real browser to render.