Skip to content
Skills Library
Data & analytics · Automation

crawlberg-headless-fallback

/crawlberg-headless-fallback

Use when a static fetch returns nothing useful and the page needs a real browser. Covers `--browser-mode auto|always|never`, external CDP via `--browser-endpoint`, symptoms of JS-only pages and WAF blocks, and the performance cost.

Repository
crawlberg ↗
Licence
MIT
Last repository change
27.10.2026

Assessment and security

84/ 100Class: good
Format9 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
2
Scripts
0
HTML comments
0
URLs in SKILL.md
4
Deprecation

No signals detected. Repository changed on 27.10.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Use when a static fetch returns nothing useful and the page needs a real browser. Covers `--browser-mode auto|always|never`, external CDP via `--browser-endpoint`, symptoms of JS-only pages and WAF blocks, and the performance cost.

Original author description

Use when a static fetch returns nothing useful and the page needs a real browser. Covers `--browser-mode auto|always|never`, external CDP via `--browser-endpoint`, symptoms of JS-only pages and WAF blocks, and the performance cost.

How it works

  1. Modes
  2. Symptoms that point to headless
  3. External CDP endpoint
  4. Performance cost
  5. Wait strategies
  6. Persistent profiles
Original SKILL.md headings · 135 lines, 619 words
SKILL.md preview
---
type: Playbook
title: Headless Fallback
description: Use when a static fetch returns nothing useful and the page needs a real browser. Covers `--browser-mode auto|always|never`, external CDP via `--browser-endpoint`, symptoms of JS-only pages and WAF blocks, and the performance cost.
x-ai-rulez:
  kind: skill
  id: crawlberg-headless-fallback
  metadata:
    name: crawlberg-headless-fallback
---

# Headless fallback

Some pages are unscrapable without a real browser — SPA shells, infinite
scroll, Cloudflare interstitials, JS-rendered article bodies. Crawlberg
ships with an optional headless-Chrome backend driven by chromiumoxide.

## Modes

```text
--browser-mode auto    # default — try static first, fall back to browser on JS/WAF
--browser-mode always  # skip static, go straight to browser
--browser-mode never   # static only, fail closed
```

### `auto` (default)

The engine fetches statically, then inspects the response. It launches
headless Chrome and re-fetches when it sees:

- WAF responses from one of 8 detected vendor fingerprints (Cloudflare,
  Akamai, AWS WAF, Imperva, DataDome, PerimeterX, F5, plus a generic
  catch-all).
- SPA shells: `<noscript>` warnings, near-empty `<body>` with heavy JS.
- Heuristic JS-render-required signals.

This is the right default. The browser only spins up when needed.

### `always`

Skip the static probe entirely. Use when:

- The user already told you the page needs JS.
- You are scraping a site you know is React/Vue/Svelte SPA.
- You need `<script>`-emitted state that never lands in static HTML.

```bash
crawlberg scrape https://spa.example.com --browser-mode always --format markdown
```

### `never`

Static only — the browser path is disabled. Use when:

- You are in a hot loop where a stray Chrome launch would blow the budget.
- You are running in a sandbox without a Chrome binary.
- The user explicitly wants only static fetches.

In `never` mode, JS-only pages return empty/stub content. Inspect
`markdown.content` and `markdown.warnings` before treating the result as
final.

## Symptoms that point to headless

In `--browser-mode never` or when you suspect the auto detector missed a
signal:

- `markdown.content` is short, nav-only, or just a loading message.
- `status_code` is 200 but `metadata.headings` is empty on a page that
  clearly has headings.
- `markdown.warnings` mentions JS-render-required or WAF detection.
- 403/406/503 with WAF response headers (`server: cloudflare`,
  `cf-mitigated`, `x-amz-cf-id`, `set-cookie: __cf_bm=…`).

Re-run with `--browser-mode always`. If that succeeds, leave it set for
that host.

## External CDP endpoint

Point at an already-running Chrome (Browserless, Steel, your own) instead
of launching locally:

```bash
crawlberg scrape https://example.com \
  --browser-mode always \
  --browser-endpoint ws://browser.internal:9222/devtools/browser/<id> \
  --format markdown
```

The endpoint must be a WebSocket URL — `ws://` or `wss://`. The CLI
rejects anything else with a clear error.

Use external CDP when:

- You are running in containers or CI without a local Chrome.
- You want a shared, warm browser pool across many crawl jobs.
- You need browser-side residential proxies or stealth configuration the
  local Chrome cannot provide.

## Performance cost

Headless Chrome is expensive relative to a static fetch:

- Cold start: 1-3 seconds the first time it launches.
- Per-page overhead: 500 ms-2 s for `NetworkIdle` wait, plus the page's own
  JS load time.
- Memory: each tab takes 100-300 MB; long crawls should bound
  `--concurrent`.

Mitigations:

- Stay in `--browser-mode auto` — the engine only pays the cost when it
  needs to.
- Use `--browser-endpoint` to share one warm browser across jobs.
- Drop `--concurrent` when you know the crawl will route through Chrome.

## Wait strategies

Pass via `--config` JSON when you need control:

```bash
crawlberg scrape https://example.com --browser-mode always \
  --config '{"browser":{"wait":"selector","wait_selector":".article-body"}}'
```

Supported strategies (the `browser.wait` field is a string enum; pair
`"selector"` with a sibling `wait_selector`):

- `network_idle` (default) — wait until the network goes quiet.
- `selector` — wait until the CSS selector in `wait_selector` resolves.
- `fixed` — wait a fixed duration.

`extra_wait` adds milliseconds on top of the wait strategy if the page
keeps loading content after the primary signal.

## Persistent profiles

```bash
crawlberg scrape https://app.example.com --browser-mode always \
  --config '{"browser_profile":"prod","save_browser_profile":true}'
```

Profile names are path-traversal-validated. Use them to keep cookies,
localStorage, and login state across runs without re-authenticating.
Files in this package (2)
  • SKILL.md 5 KB
  • index.md 1 KB

Versions

RepositoryScoreLicence
This page · xberg-io/crawlberg84MIT
xberg-io/crawlberg100MIT
xberg-io/crawlberg100MIT

Database snapshot: 13.11.2026 · package version 47f85106a068 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 84crawlberg-automating-the-browser
    Use when extracting a page needs scripted interaction first — click, type, press a key, scroll, wait, screenshot, or run JS before capturing the DOM. Covers `crawlberg interact URL --actions` with the real action schema, result shape, limits, and external-CDP options.
  • 100crawlberg
    Crawl, scrape, and convert websites to Markdown using the local crawlberg CLI and its MCP server. Use when the user wants to fetch a page, follow links across a domain, enumerate URLs, or drive a real browser. Covers installation, the subcommands (scrape, crawl, map, interact,…
  • 100lightpanda
    Operate lightpanda-io/browser, the headless browser for AI and automation: offline preflight, release selection, approved fetch and CDP/MCP integration. Use when users name Lightpanda or need its source-pinned setup and runtime boundaries; route generic browser automation to…
  • 86blocked-page-recovery
    Use when a fetch fails: 403/429, paywall, WAF, bot wall.
  • 100fetch
    Use when the user wants one known URL returned as content: "get me this page", read this article, this page as markdown, save the raw HTML, screenshot this page, or page text for a RAG index or model context. Also when a plain fetch of a URL came back blocked or challenged…
  • 100fetchmcp
    Fetch a public web page live and get its main content as clean Markdown, plain text or HTML, through the FetchMCP API. Use when you have a specific URL and need what the page says now, for example docs, articles, product or listing pages, including pages that need a real browser to render.

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.