Przejdź do treści
Biblioteka skilli
Programowanie · Automatyzacja

algolia-crawler

/algolia-crawler

Indeksuje strony lub całe witryny w Algolia przez CLI i Crawler, na przykład pod RAG, wyszukiwanie semantyczne i Agent Studio, z pisaniem i debugowaniem konfiguracji crawlera.

Repozytorium
skills ↗
Licencja
MIT
Ostatnia zmiana repozytorium
26.10.2026

Ocena i bezpieczeństwo

100/ 100Klasa: dobry
Format25 / 25
Frontmatter z nazwą i opisem, poprawna nazwa, opis do 1024 znaków, nazwa zgodna z folderem.
Odkrywalność25 / 25
Opis mówi, kiedy użyć skilla, ma sensowną długość i jest w 3. osobie.
Treść35 / 35
Co najmniej 150 słów i do 500 linii, nagłówki, listy kroków, przykłady albo kod.
Higiena15 / 15
Bez zaślepek typu TODO, bez ścieżek Windows, z licencją.
Ryzykobrak sygnałów w skanie
Pliki w paczce
9
Skrypty
0
Komentarze HTML
0
Adresy URL w SKILL.md
1
Przestarzałość

Brak sygnałów. Repozytorium zmienione 26.10.2026, bez odwołań do wycofanych modeli i API.

Licencja

MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.

Co robi i kiedy

Indeksuje strony lub całe witryny w Algolia przez CLI i Crawler, na przykład pod RAG, wyszukiwanie semantyczne i Agent Studio, z pisaniem i debugowaniem konfiguracji crawlera.

Wyzwalacze z opisu

index my website/docs with Algoliaset up the Algolia Crawlercrawl this page for RAGscrape my site into Algoliabuild a knowledge base for my AI agent from these URLs
Oryginalny opis (EN)

Use this skill whenever a user wants to crawl one or more web pages or a whole site and turn them into an Algolia index using the Algolia CLI — especially for RAG, AI search, semantic search, or Agent Studio retrieval. Triggers: "index my website/docs with Algolia", "set up the Algolia Crawler", "crawl this page for RAG", "scrape my site into Algolia", "build a knowledge base for my AI agent from these URLs", writing or debugging a crawler recordExtractor, or handling JavaScript-rendered pages that won't index. It guides ingestion end-to-end with `algolia crawler` commands: inspect the page, w

Jak działa

  1. When to use this skill vs. the others
  2. The RAG record mental model
  3. The workflow
  4. Two things that will bite you
  5. Driving it with the CLI
  6. References
Sekcje z nagłówków SKILL.md, w oryginale · 79 linii, 1 139 słów
Podgląd SKILL.md
---
name: algolia-crawler
description: >-
  Use this skill whenever a user wants to crawl one or more web pages or a
  whole site and turn them into an Algolia index using the Algolia CLI —
  especially for RAG, AI search, semantic search, or Agent Studio retrieval.
  Triggers: "index my website/docs with Algolia", "set up the Algolia Crawler",
  "crawl this page for RAG", "scrape my site into Algolia", "build a knowledge
  base for my AI agent from these URLs", writing or debugging a crawler
  recordExtractor, or handling JavaScript-rendered pages that won't index. It
  guides ingestion end-to-end with `algolia crawler` commands: inspect the
  page, write a RAG-optimized recordExtractor, validate with `algolia crawler
  test` BEFORE indexing, apply index settings explicitly, then reindex. Do NOT
  use for building the chatbot/agent layer itself (use algobot-cli), raw
  record/synonym/settings ops on an existing index (use algolia-cli), frontend
  search UI (use instantsearch), or read-only search/analytics (use
  algolia-mcp).
license: MIT
metadata:
  author: algolia
  version: "1.0"
---

# Algolia Crawler

Turn web pages into a **RAG-optimized Algolia index** with the [Algolia Crawler](https://www.algolia.com/doc/tools/crawler/getting-started/overview), driven entirely by the **Algolia CLI** (`algolia crawler …`). The Crawler visits URLs, extracts content with a JavaScript `recordExtractor`, and writes Algolia records on a schedule — no scraping code to maintain.

The single most important idea: **crawling for RAG is not the same as crawling for site search.** A RAG record is a chunk of text that will be dropped into an LLM's context window *in isolation*. That changes how you shape every record (see [The RAG record mental model](#the-rag-record-mental-model)).

## When to use this skill vs. the others

| Need | Skill |
|------|-------|
| Crawl/ingest web pages into an index (this workflow) | **algolia-crawler** |
| Build the chatbot / agent / RAG app on top of the index | **algobot-cli** |
| Raw record/settings/synonym/rule ops on an existing index | **algolia-cli** |
| Read-only search, analytics, recommendations | **algolia-mcp** |
| Frontend search UI (autocomplete, results, facets) | **instantsearch** |

**Rule of thumb:** if the user wants to *get web content into Algolia*, you're in the right skill. Once the index exists and they want to *ask questions over it*, hand off to **algobot-cli**.

## The RAG record mental model

Each record is retrieved and shown to an LLM alone, without the surrounding page. So optimize for that:

1. **Small and single-topic.** One idea, one fact, one Q&A per record — not a whole page. Granular records retrieve precisely and waste less of the context window.
2. **Self-contained.** Every record carries the context needed to stand alone: the entity it's about, the page title, the section heading. The LLM won't see the rest of the page.
3. **Natural-language `content`, even for structured data.** This is the highest-leverage move. Don't store only `{model: "X", score: 0.87}` — also synthesize a sentence: *"Model X scores 0.87 on relevance…"*. Both keyword and vector retrieval, and the LLM itself, want prose.
4. **Structured attributes for filtering.** Alongside `content`, keep clean fields (`category`, `type`, `date`, numeric values) so retrieval can be scoped with `filters` and facets.

If you internalize only one thing from this skill, make it this list. Everything else is mechanics.

## The workflow

Follow these steps in order. Full commands, config, and code live in the references — read them as you reach each step.

1. **Set up the CLI and credentials.** Install the CLI and export your Crawler credentials (a Crawler *user ID* + *API key*, distinct from your app keys), plus an Algolia **write** API key for the target index and your **App ID**. See [cli.md](references/cli.md#setup-and-credentials).
2. **Inspect the page first — is it JavaScript-rendered?** Many modern pages load their real content via XHR after load, so a naive crawl indexes empty shells. Detect this and enable rendering. See [javascript-rendering.md](references/javascript-rendering.md).
3. **Write a `recordExtractor` that emits RAG records.** For docs and prose, the recommended default is Algolia's Markdown helpers (`helpers.markdown` + `helpers.splitTextIntoRecords`) — they preserve headings/lists/code and are Algolia's own AskAI/RAG pattern. Hand-roll an extractor when you need per-entity structured records (tables, catalogs). Either way, follow the mental model above. See [record-extractor.md](references/record-extractor.md).
4. **Validate with `algolia crawler test` BEFORE indexing.** It runs your config against a live URL and returns the records it *would* create, without writing anything. Use it as your feedback loop — and as a DOM inspector to fix selectors against the real rendered markup. See [workflow.md](references/workflow.md#validate-before-you-index).
5. **Apply index settings explicitly.** Do not rely on `initialIndexSettings` to configure the index — in practice it frequently does not apply. Set `searchableAttributes`, `attributesForFaceting`, etc. yourself with `algolia settings import` after the first crawl. See [rag-index-settings.md](references/rag-index-settings.md).
6. **Reindex, then verify** by searching the index (`algolia search`) for a few realistic RAG questions before trusting it.
7. **Schedule** a recurring crawl so the index stays fresh as the source pages change.

**Crawling a whole site or section (one URL → all its sub-pages)?** The crawler discovers sub-pages via `startUrls`/`sitemaps`/`discoveryPatterns` — but the config must be shaped from *all* the page types it will hit, not just the start page. A config fitted to the landing page produces junk on the article/reference/blog templates it never saw. Discover the URL set, group it into templates, sample a representative page per template, and validate the extractor against **each**. See [site-crawls.md](references/site-crawls.md).

The complete, copy-adaptable end-to-end run (create → test → settings → reindex → verify) is in [workflow.md](references/workflow.md).

## Two things that will bite you

These are the non-obvious failures. Both references cover them, but they're worth stating up front:

- **JavaScript-rendered data.** If the page shows `Loading…` placeholders in its initial HTML, or the numbers/table appear only after load, you must enable `renderJavaScript`. A crawl without it indexes nothing useful. ([javascript-rendering.md](references/javascript-rendering.md))
- **Values hidden in attributes.** Rendered pages often keep the real value in an attribute (e.g. `data-tip="Score: 79.8%"`) while the visible cell shows only a bar or a delta. Extract from the attribute, not the visible text. ([record-extractor.md](references/record-extractor.md#reading-values-from-attributes))

## Driving it with the CLI

Everything runs through `algolia crawler …` (and `algolia settings` / `algolia search` for the index side). Full cheatsheet and the gotchas below are in [cli.md](references/cli.md).

```bash
algolia crawler create <name> -F config.json     # create (prints nothing on success)
algolia crawler test   <id> --url <url> [-F cfg] # extract records WITHOUT indexing
algolia crawler reindex <id>                      # start a crawl that writes records
algolia crawler get    <id>                        # inspect crawler + config/status
algolia crawler stats  <id>                        # crawl status summary
```

Two CLI realities to plan around (details in [cli.md](references/cli.md#gotchas)):

- **Set `renderJavaScript: true` (boolean).** The CLI only accepts the boolean form; the object form (`{ enabled, patterns, waitTime }`) makes `get`/`list`/`create -F` fail to parse. Boolean `true` uses the default render wait, which is enough for most pages — confirm with `crawler test` (empty values mean the page needs more render time).
- **`create` prints nothing and there's no config-update or delete command.** After `create`, recover the id from `algolia crawler list` (`get` needs a UUID — it doesn't accept a name). If `list` errors because another crawler in the account uses a non-boolean `renderJavaScript`, there's no pure-CLI recovery — look the id up via the Crawler REST list endpoint (`GET /1/crawlers?name=<name>`). To change a config, re-create; to delete, use the dashboard or REST `DELETE`.

## References

- [workflow.md](references/workflow.md) — the full end-to-end CLI run: config, test-before-index loop, settings, reindex, verify, schedule.
- [site-crawls.md](references/site-crawls.md) — crawling a whole site/section: discover sub-pages, group page templates, sample and validate each, cover them with a branching extractor or multiple actions.
- [record-extractor.md](references/record-extractor.md) — RAG record patterns, helpers, reading values from attributes, chunking long prose.
- [rag-index-settings.md](references/rag-index-settings.md) — index settings tuned for retrieval, NeuralSearch/vector, query-time filtering for RAG.
- [javascript-rendering.md](references/javascript-rendering.md) — detecting and handling JavaScript-rendered pages.
- [cli.md](references/cli.md) — setup, credentials, the `algolia crawler` command cheatsheet, and the CLI gotchas in detail.
Pliki w paczce (9)
  • SKILL.md 9 KB
  • evals/EVAL_RESULTS.md 5 KB
  • evals/evals.json 5 KB
  • references/cli.md 5 KB
  • references/javascript-rendering.md 2 KB
  • references/rag-index-settings.md 3 KB
  • references/record-extractor.md 9 KB
  • references/site-crawls.md 5 KB
  • references/workflow.md 6 KB

Wersje

W bazie nie ma innych skilli o nazwie algolia-crawler.

Migawka bazy z 13.11.2026 · wersja paczki 47aa3589d1f2 (skrót SHA-256 całej paczki)

Pobrania w ostatnich 30 dniach: 0 · ranking

Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.

Podobne skille

  • 100algolia-common-errors
    Diagnozuje błędy żądań, poświadczeń, indeksów i zapytań Algolia na podstawie konkretnych dowodów, na przykład odpowiedzi 4xx i 5xx albo pustych wyników.
  • 100algolia-hello-world
    Tworzy minimalny przykład indeksowania i wyszukiwania w Algolia z klientem JavaScript v5 na jednorazowym indeksie, aby sprawdzić poświadczenia i połączenie.
  • 100algolia-quickstart
    Uruchamia Algolia od zera: zakłada konto, loguje Algolia CLI i tworzy aplikację, by uzyskać App ID i klucz API, gdy ktoś nie ma jeszcze konta ani poświadczeń.
  • 100algolia-audit
    Audytuje istniejące wdrożenie Algolia i kieruje do właściwych skilli, gdy trzeba sprawdzić konfigurację, spadek jakości wyszukiwania albo gotowość przed startem.
  • 90algolia-search
    Pomaga dodać wyszukiwanie Algolia do aplikacji: indeksowanie, React InstantSearch, API wyszukiwania i strojenie trafności wyników.
  • 90algolia-search
    Dostarcza wzorce wdrożenia wyszukiwania Algolia: strategie indeksowania, React InstantSearch i strojenie trafności.

Uwagi i zgłoszenia

Wiadomość trafi do Bartosza Sokolińskiego. Adres e-mail jest potrzebny tylko do odpowiedzi. Szczegóły w polityce prywatności.