scrape
/scrape
Pobiera treści z sieci jako czysty Markdown, HTML lub JSON przez Bright Data CLI: pojedyncze strony, listy URL-i i paginowane listingi, z przekazaniem do data-feeds lub search.
- Autor
- brightdata ↗
- Repozytorium
- skills ↗
- Licencja
- MIT
- Ostatnia zmiana repozytorium
- 15.10.2026
Ocena i bezpieczeństwo
- Pliki w paczce
- 4
- Skrypty
- 0
- Komentarze HTML
- 0
- Adresy URL w SKILL.md
- 5
Brak sygnałów. Repozytorium zmienione 15.10.2026, bez odwołań do wycofanych modeli i API.
MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.
Co robi i kiedy
Pobiera treści z sieci jako czysty Markdown, HTML lub JSON przez Bright Data CLI: pojedyncze strony, listy URL-i i paginowane listingi, z przekazaniem do data-feeds lub search.
Oryginalny opis (EN)
Scrape web content as clean markdown/HTML/JSON via the Bright Data CLI (`bdata scrape`). Use when the user wants to fetch a page, extract content from a list of URLs, or crawl paginated listings. Hands off to `data-feeds` for supported platforms (Amazon, LinkedIn, TikTok, Instagram, YouTube, Reddit, etc.) and to `search` when URLs must be discovered first. Requires the Bright Data CLI; proactively guides install + login if missing.
Jak działa
- Setup gate (run first)
- Pick your path
- Action
- Verification gate (run before claiming success)
- Red flags
- References
Podgląd SKILL.md
---
name: scrape
description: Scrape web content as clean markdown/HTML/JSON via the Bright Data CLI (`bdata scrape`). Use when the user wants to fetch a page, extract content from a list of URLs, or crawl paginated listings. Hands off to `data-feeds` for supported platforms (Amazon, LinkedIn, TikTok, Instagram, YouTube, Reddit, etc.) and to `search` when URLs must be discovered first. Requires the Bright Data CLI; proactively guides install + login if missing.
---
# Bright Data — Scrape
Get clean content (markdown, HTML, JSON, screenshot) from one or more URLs via the Bright Data CLI. This skill owns the "fetch raw or lightly-structured content" job. For platform-specific structured data (Amazon, LinkedIn, TikTok, etc.), **stop and use `data-feeds` instead** — you'll get clean JSON without selector logic.
## Setup gate (run first)
Before any scrape, verify the CLI is installed and authenticated:
```bash
if ! command -v bdata >/dev/null 2>&1; then
echo "bdata CLI not installed — see bright-data-best-practices/references/cli-setup.md"
elif ! bdata zones >/dev/null 2>&1; then
echo "bdata not authenticated — run: bdata login (or: bdata login --device for SSH)"
fi
```
If either check fails, halt and route the user to `skills/bright-data-best-practices/references/cli-setup.md`. Do not attempt the legacy `curl` fallback silently — ask the user first.
## Pick your path
| Situation | Action |
|---|---|
| Single URL | `bdata scrape <url> -f markdown` |
| Small list (≤ ~20 URLs) | shell loop, 1 at a time (see `references/patterns.md`) |
| Larger list (dozens+) | `xargs -P 4` with parallelism cap (see `references/patterns.md`) |
| Paginated listing | scrape page 1 → extract next-page URL → append → repeat (see `references/examples.md`) |
| JS-heavy / login-gated / interaction-required | escalate to `bdata browser` (see `brightdata-cli` skill) |
| Amazon, LinkedIn, TikTok, Instagram, YouTube, Reddit, … | **stop — hand off to `data-feeds`** |
| No URL yet, just a topic | **hand off to `search`** |
## Action
Core commands:
```bash
# Clean markdown (default)
bdata scrape "https://example.com/article" -f markdown -o article.md
# Raw HTML (when you need the DOM)
bdata scrape "https://example.com" -f html -o page.html
# Structured JSON (when the Unlocker returns parsed fields)
bdata scrape "https://example.com" -f json --pretty -o page.json
# Visual snapshot (saves PNG)
bdata scrape "https://example.com" -f screenshot -o page.png
# Geo-targeted (override the exit country)
bdata scrape "https://example.com" --country de -f markdown
```
Full flag reference: [`references/flags.md`](references/flags.md).
## Verification gate (run before claiming success)
1. **Non-empty output:** `test -s "$out_path"` — or, for stdout, at least 200 bytes of content.
2. **Not a block page** — grep the output for any of these signatures (case-insensitive):
- `Access Denied`
- `Just a moment`
- `Attention Required`
- `Checking your browser`
- `captcha`
- `cf-browser-verification`
- `cloudflare` *(with < 2KB total body)*
3. **Expected markers present** for the task: e.g., a product page should contain a price pattern (`\$\d`); an article should contain at least one `<h1>` or `# ` heading.
4. **On failure, escalation ladder:**
- Retry with a different `--country` (e.g., `--country de` if the origin site is US)
- Escalate to `bdata browser` for full JS rendering (hand off to `brightdata-cli` skill)
Do not report success until all checks above pass.
## Red flags
- Claiming success without inspecting the output.
- Silencing errors with `2>/dev/null` — you'll miss auth failures and rate-limit errors.
- Running `bdata scrape` on Amazon/LinkedIn/TikTok/Instagram/YouTube/Reddit URLs — these are supported by `data-feeds` and return structured data directly. Scraping loses the structure.
- Scraping the same URL repeatedly in the same task — cache the first result.
- Looping `bdata scrape` sequentially for large lists instead of using `xargs -P 4` (or similar) with a parallelism cap.
- Using `curl` against `api.brightdata.com` directly — legacy path; only when the CLI isn't available.
## References
- [`references/flags.md`](references/flags.md) — every flag with when-to-use notes.
- [`references/patterns.md`](references/patterns.md) — shell-loop batching, `xargs` parallelism, pagination recipe, retry/backoff, block-page recovery chain, legacy `curl` fallback.
- [`references/examples.md`](references/examples.md) — (1) single page → markdown, (2) batch a list of URLs with parallelism cap, (3) paginated listing, (4) block-page recovery.
Pliki w paczce (4)
- SKILL.md 5 KB
- references/examples.md 2 KB
- references/flags.md 2 KB
- references/patterns.md 4 KB
Wersje
| Repozytorium | Wynik | Licencja |
|---|---|---|
| Ta strona · brightdata/skills | 100 | MIT |
| 5uck1ess/devkit | 100 | MIT |
| davekilleen/Dex | 100 | MIT |
| mikegianfelice/GitHired | 95 | MIT |
| Mingrui-Ai/ai-job-search-cn | 95 | MIT |
| NekitKibersp/my-skills | 95 | MIT |
| madhug17/test-ai-job | 95 | MIT |
| Aditya7615/ai-job-search | 95 | MIT |
| OutlineDriven/outline-driven-development | 90 | Apache-2.0 |
Migawka bazy z 13.11.2026 · wersja paczki acaf5afae206 (skrót SHA-256 całej paczki)
Pobrania w ostatnich 30 dniach: 0 · ranking
Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.
Podobne skille
- 95data-feeds
Wyciąga ustrukturyzowane dane z ponad 40 platform, jak Amazon, LinkedIn czy YouTube, przez Bright Data CLI w formacie JSON. - 100data-feeds
Pobiera ustrukturyzowane dane z gotowych scraperów Bright Data (e-commerce, social, oferty pracy, mapy, nieruchomości) przez Bright Data CLI i zwraca czysty JSON. - 100search
Przeszukuje sieć przez CLI Bright Data: wyniki SERP z Google, Bing i Yandex oraz semantyczne wyszukiwanie, przekazując wybrane adresy URL do scrapowania. - 95bright-data-mcp
Zbiera dane z sieci przez Bright Data MCP: pobiera strony, wyszukuje w internecie i wyciąga dane z Amazon, LinkedIn i innych serwisów. - 100bright-data-mcp
Obsługuje wszystkie operacje na danych z sieci przez Bright Data MCP: strony, wyszukiwanie, dane z Amazon, LinkedIn i innych serwisów oraz automatyzację przeglądarki. - 95BrightData
Pobiera treść stron w czterech etapach, od zwykłego curl po przeglądarkę i zarządzane pobieranie, z zamianą wyniku na markdown.