diffbot-crawl
/diffbot-crawl
Przeszukuje całe witryny przez Diffbot i zarządza zadaniami crawlera: tworzy, wyświetla i usuwa joby pobierające ustrukturyzowaną treść.
- Autor
- diffbot ↗
- Repozytorium
- diffbot-skills ↗
- Licencja
- MIT
- Ostatnia zmiana repozytorium
- 28.09.2026
Ocena i bezpieczeństwo
- Pliki w paczce
- 1
- Skrypty
- 0
- Komentarze HTML
- 0
- Adresy URL w SKILL.md
- 6
Brak sygnałów. Repozytorium zmienione 28.09.2026, bez odwołań do wycofanych modeli i API.
MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.
Co robi i kiedy
Przeszukuje całe witryny przez Diffbot i zarządza zadaniami crawlera: tworzy, wyświetla i usuwa joby pobierające ustrukturyzowaną treść.
Oryginalny opis (EN)
Crawl a website or manage Diffbot crawler jobs. Use when the user wants to crawl a site for structured content, list existing crawl jobs, or delete a crawl job. Triggers on: crawl website, crawl site, crawler job, list crawl jobs, delete crawl job, diffbot crawl.
Jak działa
- The db crawl CLI commands
- Workflow
- Understanding crawl output
- Tips
Podgląd SKILL.md
--- name: diffbot-crawl description: "Crawl a website or manage Diffbot crawler jobs. Use when the user wants to crawl a site for structured content, list existing crawl jobs, or delete a crawl job. Triggers on: crawl website, crawl site, crawler job, list crawl jobs, delete crawl job, diffbot crawl." allowed-tools: Bash(~/.diffbot/venv/bin/db:*), Bash(python3 -m venv ~/.diffbot/venv:*), Bash(~/.diffbot/venv/bin/pip install:*) --- # Diffbot Crawler Crawl websites for structured content via the Diffbot Crawlbot API, and manage crawler jobs. ## The `db crawl*` CLI commands **Always invoke with the fixed path `~/.diffbot/venv/bin/db`.** ``` ~/.diffbot/venv/bin/db crawl <SITE> [options] # start a crawl and stream events ~/.diffbot/venv/bin/db crawl-list-jobs [JOB_NAME] # list all jobs or inspect one ~/.diffbot/venv/bin/db crawl-delete-job <JOB_NAME> # delete a job ``` ### `db crawl` options | Flag | Default | Description | |------|---------|-------------| | `--hops` | `2` | Maximum link depth from seed URLs | | `--job-name` | auto-generated | Name for the crawler job | | `--max-to-crawl` | `100` | Maximum pages to crawl | | `--max-to-process` | `100` | Maximum pages to process | | `--restrict-domain` | `true` | Only follow links on the same domain as seeds | | `--api-url` | `""` | Diffbot API endpoint for processing (default: auto/analyze) | | `--crawl-delay` | `-1` | Delay between requests to same domain (seconds) | | `--url-crawl-pattern` | none | Only crawl URLs matching this pattern | | `--url-process-pattern` | none | Only process URLs matching this pattern | | `--obey-robots` | off | Obey robots.txt | | `--use-proxies` | off | Use proxies for crawling | | `--custom-headers` | none | Newline-separated custom HTTP headers | | `-f`, `--format` | `markdown` | Output: `markdown` or `json` | | `-o`, `--output` | stdout | Write output to file | ## Workflow ### Step 1 — bootstrap ``` [ -d ~/.diffbot/venv ] || python3 -m venv ~/.diffbot/venv && ~/.diffbot/venv/bin/pip install -q 'diffbot>=3.0.0' ``` The token is read from `DIFFBOT_API_TOKEN` env var or `~/.diffbot/credentials`. If missing: ``` echo "DIFFBOT_API_TOKEN=YOUR_TOKEN_HERE" > ~/.diffbot/credentials && chmod 600 ~/.diffbot/credentials ``` ### Step 2 — crawl a site Basic crawl with defaults (100 pages, 2 hops deep): ``` ~/.diffbot/venv/bin/db crawl "https://example.com" ``` Deep crawl of a documentation site, saving to JSON: ``` ~/.diffbot/venv/bin/db crawl "https://docs.example.com" \ --hops 3 \ --max-to-crawl 500 \ --max-to-process 500 \ --restrict-domain \ -f json \ -o ~/.diffbot/tmp/crawl_results.json ``` Crawl only blog posts (by URL pattern): ``` ~/.diffbot/venv/bin/db crawl "https://example.com" \ --url-process-pattern "/blog/" \ --max-to-process 200 ``` ### Step 3 — manage jobs List all active jobs: ``` ~/.diffbot/venv/bin/db crawl-list-jobs ``` Inspect a specific job: ``` ~/.diffbot/venv/bin/db crawl-list-jobs my-job-name ``` Delete a job: ``` ~/.diffbot/venv/bin/db crawl-delete-job my-job-name ``` ## Understanding crawl output The `crawl` command streams events as they happen. In markdown format (default): ``` # Job: my-crawl-job-1234567890 - [Success] https://example.com/ - [Success] https://example.com/about - [!] https://example.com/403-page ``` `[!]` marks pages that were crawled but failed extraction (not HTTP failures — those are skipped silently). In JSON format each event has `event_type`, `timestamp`, and `details`. Event types: - `job_created` — job was created on the server; `details.job_name` has the job name - `url_processed` — a page was processed; `details.url` and `details.status` (`Success` or error) ## Tips - `--restrict-domain` is on by default — disable it only if you intentionally want to follow external links. - Use `--url-process-pattern` to focus extraction on specific path segments without narrowing crawling. - Job names are stable identifiers — always set `--job-name` if you want to look up or delete a job later. - For large crawls, write output to a file with `-o` rather than reading it all into the conversation.
Wersje
W bazie nie ma innych skilli o nazwie diffbot-crawl.
Migawka bazy z 13.11.2026 · wersja paczki 67f35557d6fc (skrót SHA-256 całej paczki)
Pobrania w ostatnich 30 dniach: 0 · ranking
Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.
Podobne skille
- 90cf-crawl
Pobiera zawartość witryny przez funkcję crawl w Cloudflare Browser Rendering, by zebrać strony do dalszego przetwarzania. - 100diffbot-extract
Pobiera z podanego adresu URL treść w markdown lub w formie ustrukturyzowanej przez Diffbot Extract API, gdy strony nie ma w indeksie albo potrzebny jest cały dokument. - 95crawl
Przeszukuje wiele stron jednej witryny z limitem stron, głębokością i filtrami ścieżek, na przykład przy zbieraniu dokumentacji. - 100diffbot-entities
Rozpoznaje nazwane encje w tekście i łączy je z Diffbot Knowledge Graph, zwracając poziom pewności oraz sentyment. - 100firecrawl-crawl
Pobiera hurtowo treść z całej witryny lub jej sekcji przez Firecrawl, z limitem głębokości i filtrem ścieżek. - 100firecrawl-crawl
Pobiera wiele stron z jednej witryny lub sekcji, np. całej dokumentacji, do dalszego przetwarzania.