algolia-crawler
/algolia-crawler
Indeksuje strony lub całe witryny w Algolia przez CLI i Crawler, na przykład pod RAG, wyszukiwanie semantyczne i Agent Studio, z pisaniem i debugowaniem konfiguracji crawlera.
Ocena i bezpieczeństwo
- Pliki w paczce
- 9
- Skrypty
- 0
- Komentarze HTML
- 0
- Adresy URL w SKILL.md
- 1
Brak sygnałów. Repozytorium zmienione 26.10.2026, bez odwołań do wycofanych modeli i API.
MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.
Co robi i kiedy
Indeksuje strony lub całe witryny w Algolia przez CLI i Crawler, na przykład pod RAG, wyszukiwanie semantyczne i Agent Studio, z pisaniem i debugowaniem konfiguracji crawlera.
Wyzwalacze z opisu
Oryginalny opis (EN)
Use this skill whenever a user wants to crawl one or more web pages or a whole site and turn them into an Algolia index using the Algolia CLI — especially for RAG, AI search, semantic search, or Agent Studio retrieval. Triggers: "index my website/docs with Algolia", "set up the Algolia Crawler", "crawl this page for RAG", "scrape my site into Algolia", "build a knowledge base for my AI agent from these URLs", writing or debugging a crawler recordExtractor, or handling JavaScript-rendered pages that won't index. It guides ingestion end-to-end with `algolia crawler` commands: inspect the page, w
Jak działa
- When to use this skill vs. the others
- The RAG record mental model
- The workflow
- Two things that will bite you
- Driving it with the CLI
- References
Podgląd SKILL.md
---
name: algolia-crawler
description: >-
Use this skill whenever a user wants to crawl one or more web pages or a
whole site and turn them into an Algolia index using the Algolia CLI —
especially for RAG, AI search, semantic search, or Agent Studio retrieval.
Triggers: "index my website/docs with Algolia", "set up the Algolia Crawler",
"crawl this page for RAG", "scrape my site into Algolia", "build a knowledge
base for my AI agent from these URLs", writing or debugging a crawler
recordExtractor, or handling JavaScript-rendered pages that won't index. It
guides ingestion end-to-end with `algolia crawler` commands: inspect the
page, write a RAG-optimized recordExtractor, validate with `algolia crawler
test` BEFORE indexing, apply index settings explicitly, then reindex. Do NOT
use for building the chatbot/agent layer itself (use algobot-cli), raw
record/synonym/settings ops on an existing index (use algolia-cli), frontend
search UI (use instantsearch), or read-only search/analytics (use
algolia-mcp).
license: MIT
metadata:
author: algolia
version: "1.0"
---
# Algolia Crawler
Turn web pages into a **RAG-optimized Algolia index** with the [Algolia Crawler](https://www.algolia.com/doc/tools/crawler/getting-started/overview), driven entirely by the **Algolia CLI** (`algolia crawler …`). The Crawler visits URLs, extracts content with a JavaScript `recordExtractor`, and writes Algolia records on a schedule — no scraping code to maintain.
The single most important idea: **crawling for RAG is not the same as crawling for site search.** A RAG record is a chunk of text that will be dropped into an LLM's context window *in isolation*. That changes how you shape every record (see [The RAG record mental model](#the-rag-record-mental-model)).
## When to use this skill vs. the others
| Need | Skill |
|------|-------|
| Crawl/ingest web pages into an index (this workflow) | **algolia-crawler** |
| Build the chatbot / agent / RAG app on top of the index | **algobot-cli** |
| Raw record/settings/synonym/rule ops on an existing index | **algolia-cli** |
| Read-only search, analytics, recommendations | **algolia-mcp** |
| Frontend search UI (autocomplete, results, facets) | **instantsearch** |
**Rule of thumb:** if the user wants to *get web content into Algolia*, you're in the right skill. Once the index exists and they want to *ask questions over it*, hand off to **algobot-cli**.
## The RAG record mental model
Each record is retrieved and shown to an LLM alone, without the surrounding page. So optimize for that:
1. **Small and single-topic.** One idea, one fact, one Q&A per record — not a whole page. Granular records retrieve precisely and waste less of the context window.
2. **Self-contained.** Every record carries the context needed to stand alone: the entity it's about, the page title, the section heading. The LLM won't see the rest of the page.
3. **Natural-language `content`, even for structured data.** This is the highest-leverage move. Don't store only `{model: "X", score: 0.87}` — also synthesize a sentence: *"Model X scores 0.87 on relevance…"*. Both keyword and vector retrieval, and the LLM itself, want prose.
4. **Structured attributes for filtering.** Alongside `content`, keep clean fields (`category`, `type`, `date`, numeric values) so retrieval can be scoped with `filters` and facets.
If you internalize only one thing from this skill, make it this list. Everything else is mechanics.
## The workflow
Follow these steps in order. Full commands, config, and code live in the references — read them as you reach each step.
1. **Set up the CLI and credentials.** Install the CLI and export your Crawler credentials (a Crawler *user ID* + *API key*, distinct from your app keys), plus an Algolia **write** API key for the target index and your **App ID**. See [cli.md](references/cli.md#setup-and-credentials).
2. **Inspect the page first — is it JavaScript-rendered?** Many modern pages load their real content via XHR after load, so a naive crawl indexes empty shells. Detect this and enable rendering. See [javascript-rendering.md](references/javascript-rendering.md).
3. **Write a `recordExtractor` that emits RAG records.** For docs and prose, the recommended default is Algolia's Markdown helpers (`helpers.markdown` + `helpers.splitTextIntoRecords`) — they preserve headings/lists/code and are Algolia's own AskAI/RAG pattern. Hand-roll an extractor when you need per-entity structured records (tables, catalogs). Either way, follow the mental model above. See [record-extractor.md](references/record-extractor.md).
4. **Validate with `algolia crawler test` BEFORE indexing.** It runs your config against a live URL and returns the records it *would* create, without writing anything. Use it as your feedback loop — and as a DOM inspector to fix selectors against the real rendered markup. See [workflow.md](references/workflow.md#validate-before-you-index).
5. **Apply index settings explicitly.** Do not rely on `initialIndexSettings` to configure the index — in practice it frequently does not apply. Set `searchableAttributes`, `attributesForFaceting`, etc. yourself with `algolia settings import` after the first crawl. See [rag-index-settings.md](references/rag-index-settings.md).
6. **Reindex, then verify** by searching the index (`algolia search`) for a few realistic RAG questions before trusting it.
7. **Schedule** a recurring crawl so the index stays fresh as the source pages change.
**Crawling a whole site or section (one URL → all its sub-pages)?** The crawler discovers sub-pages via `startUrls`/`sitemaps`/`discoveryPatterns` — but the config must be shaped from *all* the page types it will hit, not just the start page. A config fitted to the landing page produces junk on the article/reference/blog templates it never saw. Discover the URL set, group it into templates, sample a representative page per template, and validate the extractor against **each**. See [site-crawls.md](references/site-crawls.md).
The complete, copy-adaptable end-to-end run (create → test → settings → reindex → verify) is in [workflow.md](references/workflow.md).
## Two things that will bite you
These are the non-obvious failures. Both references cover them, but they're worth stating up front:
- **JavaScript-rendered data.** If the page shows `Loading…` placeholders in its initial HTML, or the numbers/table appear only after load, you must enable `renderJavaScript`. A crawl without it indexes nothing useful. ([javascript-rendering.md](references/javascript-rendering.md))
- **Values hidden in attributes.** Rendered pages often keep the real value in an attribute (e.g. `data-tip="Score: 79.8%"`) while the visible cell shows only a bar or a delta. Extract from the attribute, not the visible text. ([record-extractor.md](references/record-extractor.md#reading-values-from-attributes))
## Driving it with the CLI
Everything runs through `algolia crawler …` (and `algolia settings` / `algolia search` for the index side). Full cheatsheet and the gotchas below are in [cli.md](references/cli.md).
```bash
algolia crawler create <name> -F config.json # create (prints nothing on success)
algolia crawler test <id> --url <url> [-F cfg] # extract records WITHOUT indexing
algolia crawler reindex <id> # start a crawl that writes records
algolia crawler get <id> # inspect crawler + config/status
algolia crawler stats <id> # crawl status summary
```
Two CLI realities to plan around (details in [cli.md](references/cli.md#gotchas)):
- **Set `renderJavaScript: true` (boolean).** The CLI only accepts the boolean form; the object form (`{ enabled, patterns, waitTime }`) makes `get`/`list`/`create -F` fail to parse. Boolean `true` uses the default render wait, which is enough for most pages — confirm with `crawler test` (empty values mean the page needs more render time).
- **`create` prints nothing and there's no config-update or delete command.** After `create`, recover the id from `algolia crawler list` (`get` needs a UUID — it doesn't accept a name). If `list` errors because another crawler in the account uses a non-boolean `renderJavaScript`, there's no pure-CLI recovery — look the id up via the Crawler REST list endpoint (`GET /1/crawlers?name=<name>`). To change a config, re-create; to delete, use the dashboard or REST `DELETE`.
## References
- [workflow.md](references/workflow.md) — the full end-to-end CLI run: config, test-before-index loop, settings, reindex, verify, schedule.
- [site-crawls.md](references/site-crawls.md) — crawling a whole site/section: discover sub-pages, group page templates, sample and validate each, cover them with a branching extractor or multiple actions.
- [record-extractor.md](references/record-extractor.md) — RAG record patterns, helpers, reading values from attributes, chunking long prose.
- [rag-index-settings.md](references/rag-index-settings.md) — index settings tuned for retrieval, NeuralSearch/vector, query-time filtering for RAG.
- [javascript-rendering.md](references/javascript-rendering.md) — detecting and handling JavaScript-rendered pages.
- [cli.md](references/cli.md) — setup, credentials, the `algolia crawler` command cheatsheet, and the CLI gotchas in detail.
Pliki w paczce (9)
- SKILL.md 9 KB
- evals/EVAL_RESULTS.md 5 KB
- evals/evals.json 5 KB
- references/cli.md 5 KB
- references/javascript-rendering.md 2 KB
- references/rag-index-settings.md 3 KB
- references/record-extractor.md 9 KB
- references/site-crawls.md 5 KB
- references/workflow.md 6 KB
Wersje
W bazie nie ma innych skilli o nazwie algolia-crawler.
Migawka bazy z 13.11.2026 · wersja paczki 47aa3589d1f2 (skrót SHA-256 całej paczki)
Pobrania w ostatnich 30 dniach: 0 · ranking
Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.
Podobne skille
- 100algolia-common-errors
Diagnozuje błędy żądań, poświadczeń, indeksów i zapytań Algolia na podstawie konkretnych dowodów, na przykład odpowiedzi 4xx i 5xx albo pustych wyników. - 100algolia-hello-world
Tworzy minimalny przykład indeksowania i wyszukiwania w Algolia z klientem JavaScript v5 na jednorazowym indeksie, aby sprawdzić poświadczenia i połączenie. - 100algolia-quickstart
Uruchamia Algolia od zera: zakłada konto, loguje Algolia CLI i tworzy aplikację, by uzyskać App ID i klucz API, gdy ktoś nie ma jeszcze konta ani poświadczeń. - 100algolia-audit
Audytuje istniejące wdrożenie Algolia i kieruje do właściwych skilli, gdy trzeba sprawdzić konfigurację, spadek jakości wyszukiwania albo gotowość przed startem. - 90algolia-search
Pomaga dodać wyszukiwanie Algolia do aplikacji: indeksowanie, React InstantSearch, API wyszukiwania i strojenie trafności wyników. - 90algolia-search
Dostarcza wzorce wdrożenia wyszukiwania Algolia: strategie indeksowania, React InstantSearch i strojenie trafności.