Skip to content
Skills Library
AI & agents · Building & coding

rag

/rag

Build a RAG (retrieval-augmented generation) pipeline using railtracks. Use when the user wants to ingest documents into a vector store and retrieve relevant passages to answer questions.

Repository
railtracks ↗
Licence
MIT
Last repository change
12.11.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
2
Deprecation

No signals detected. Repository changed on 12.11.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Build a RAG (retrieval-augmented generation) pipeline using railtracks. Use when the user wants to ingest documents into a vector store and retrieve relevant passages to answer questions.

Arguments

/rag [describe the data source and what you want to retrieve]
Original author description

Build a RAG (retrieval-augmented generation) pipeline using railtracks. Use when the user wants to ingest documents into a vector store and retrieve relevant passages to answer questions.

How it works

  1. How railtracks retrieval works
  2. Component reference
  3. Steps
  4. Patterns to follow
  5. Rules
Original SKILL.md headings · 314 lines, 1,385 words
SKILL.md preview
---
name: rag
description: Build a RAG (retrieval-augmented generation) pipeline using railtracks. Use when the user wants to ingest documents into a vector store and retrieve relevant passages to answer questions.
argument-hint: '[describe the data source and what you want to retrieve]'
---

# Build a Railtracks RAG Pipeline

The user wants to build a retrieval-augmented generation (RAG) pipeline using the railtracks framework: $ARGUMENTS

## How railtracks retrieval works

The pipeline has two paths:

**Write path (ingestion):** `Loader → Chunker → Embedder → VectorStore`
**Read path (retrieval):** `query → Embedder → VectorStore.search → RetrievalResult`

`RetrievalRuntime` orchestrates both paths. It takes a chunker, embedder, and store at construction time, then exposes `ingest_all()` / `ingest()` for writes and `retrieve()` for reads.

### Core data flow
| Stage | Input | Output |
|-------|-------|--------|
| Loader | source (file, URL, dataset) | `Document` |
| Chunker | `Document` | `Chunk` |
| Embedder | `Chunk.content` | `EmbeddedChunk` |
| Store | `EmbeddedChunk` | stored `StoreEntry` |
| Retrieve | query string | `RetrievalResult` (ranked `RetrievedChunk` list) |

---

## Component reference

### Loaders
```python
from railtracks.retrieval.loaders import (
    TextLoader,  # .txt / .md files
    CSVLoader,  # rows → documents; content_columns, id_column, ignore_columns
    JSONLoader,  # .json / .jsonl files
    PyPDFLoader,  # PDF file or folder; breakdown_strategy="page" (default), "paragraph", or "document"
    HuggingFaceDatasetLoader,  # HF Hub datasets; pass dataset_name, split, content_columns
    LangChainLoaderAdapter,  # wrap any LangChain document loader
)
```

### Chunkers
```python
from railtracks.retrieval.chunking import (
    RecursiveCharacterChunker,  # general text; chunk_size (chars), overlap
    MarkdownHeaderChunker,  # heading-aware splits for .md content
    SentenceChunker,  # sentence windows; chunk_size (# sentences), overlap
    SemanticChunker,  # embedding-driven breakpoints; variable chunk size
    FixedTokenChunker,  # hard token budget; chunk_size (tokens), overlap
)
```

### Embedders
```python
from railtracks.retrieval.embedding import (
    OpenAIEmbedding,  # default model: "text-embedding-3-small"
    AzureEmbedding,  # Azure OpenAI routing
    OllamaEmbedding,  # local dev; model, api_base
    LiteLLMEmbedding,  # any LiteLLM-supported provider
)
```

### Store backends
```python
from railtracks.retrieval.stores import (
    VectorStore,  # store implementation; wraps a backend
    InMemoryVectorBackend,  # tests and small corpora; no persistence by default
    ChromaBackend,  # local / persistent / HTTP modes
    ChromaCloudBackend,  # managed Chroma Cloud
    PgvectorBackend,  # Postgres + pgvector; production
)
```

### Installation extras
```
pip install "railtracks[retrieval-core]"   # minimum (no connectors)
pip install "railtracks[retrieval]"        # all connectors
pip install "railtracks[chroma]"           # Chroma only
pip install "railtracks[huggingface]"      # HuggingFace datasets
pip install "railtracks[ocr]"             # PyPDF + OCR (requires Tesseract)
```

---

## Steps

1. **Read the existing code** — check what files already exist. Understand what data source, chunking strategy, and query pattern the user needs before writing anything.
2. **Choose a loader** — match the source type (file, PDF, CSV, HuggingFace dataset, etc.). Ask the user to clarify if not obvious from `$ARGUMENTS`.
3. **Choose a chunker** — default to `RecursiveCharacterChunker` for plain text, `MarkdownHeaderChunker` for structured docs, `SentenceChunker` for narrative text. Use `SemanticChunker` only when chunk boundaries matter semantically.
4. **Choose an embedder** — if the project already has a populated collection, use the embedder and model it was built with (mixing models raises `EmbeddingModelMismatchError`). Otherwise match the provider the project already uses, or one whose API key is configured (embedders and their settings: https://docs.railtracks.org/retrieval/components/embeddings/methods/); otherwise ask the user. Use `OllamaEmbedding` for local/offline dev.
5. **Choose a backend** — `InMemoryVectorBackend` for demos/tests, `ChromaBackend` for local persistence, `PgvectorBackend` for production.
6. **Construct `RetrievalRuntime`** — pass chunker, embedder, store. Initialize async backends with `await backend.initialize()` or the `ChromaBackend.create()` factory.
7. **Ingest documents** — call `runtime.ingest_all(loader=...)`. Print stats. Handle `documents_failed` gracefully.
8. **Add a retrieve function** — call `runtime.retrieve(query, top_k=...)`. Return or format `result.chunks`.
9. **Wire into an agent (if needed)** — expose `retrieve` as a `@rt.function_node` tool, or call it in a pre-invoke hook.
10. **Add a `if __name__ == "__main__":` block** — include both an ingest path and a query example so the user can run it immediately.

---

## Patterns to follow

### Minimal pipeline (in-memory, for demos/tests)
```python
import asyncio
from railtracks.retrieval import RetrievalRuntime
from railtracks.retrieval.loaders import TextLoader
from railtracks.retrieval.chunking import RecursiveCharacterChunker
from railtracks.retrieval.embedding import OpenAIEmbedding
from railtracks.retrieval.stores import VectorStore, InMemoryVectorBackend


async def main():
    runtime = RetrievalRuntime(
        chunker=RecursiveCharacterChunker(chunk_size=1000, overlap=200),
        embedder=OpenAIEmbedding(),
        store=VectorStore(InMemoryVectorBackend()),
        batch_size=100,
    )
    stats = await runtime.ingest_all(loader=TextLoader("data/my_file.txt"))
    print(f"chunks embedded: {stats.chunks_embedded}")

    result = await runtime.retrieve("your question here", top_k=5)
    for rc in result.chunks:
        print(f"[rank={rc.rank} score={rc.score:.3f}] {rc.chunk.content[:200]}")


if __name__ == "__main__":
    asyncio.run(main())
```

### Persistent pipeline (Chroma, recommended for local dev)
```python
import asyncio
from railtracks.retrieval import RetrievalRuntime
from railtracks.retrieval.loaders import TextLoader
from railtracks.retrieval.chunking import RecursiveCharacterChunker
from railtracks.retrieval.embedding import OpenAIEmbedding
from railtracks.retrieval.stores import VectorStore, ChromaBackend


async def create_runtime() -> RetrievalRuntime:
    backend = await ChromaBackend.create(
        collection_name="my-collection",
        path="./my_db",
    )
    return RetrievalRuntime(
        chunker=RecursiveCharacterChunker(chunk_size=1000, overlap=200),
        embedder=OpenAIEmbedding(model="text-embedding-3-small"),
        store=VectorStore(backend),
        batch_size=100,
    )


async def main():
    runtime = await create_runtime()
    stats = await runtime.ingest_all(loader=TextLoader("data/my_file.txt"))
    print(
        f"loaded={stats.documents_loaded} chunks={stats.chunks_embedded} skipped={stats.documents_skipped}"
    )

    result = await runtime.retrieve("your question here", top_k=5)
    for rc in result.chunks:
        print(f"[rank={rc.rank} score={rc.score:.3f}]\n{rc.chunk.content}\n")


if __name__ == "__main__":
    asyncio.run(main())
```

### Streaming ingestion events (for progress / error handling)
```python
from railtracks.retrieval import RetrievalRuntime
from railtracks.retrieval.runtime import BatchIngested, DocumentFailed, DocumentSkipped


async def ingest_with_progress(runtime: RetrievalRuntime, loader) -> None:
    async for event in runtime.ingest(loader=loader):
        if isinstance(event, BatchIngested):
            print(f"batch {event.batch_index}: {len(event.embedded_chunks)} chunks written")
        elif isinstance(event, DocumentFailed):
            # stage is "load", "chunk", "embed", or "store"; document_id is None for "load"
            print(f"FAILED at {event.stage}: {event.source} — {event.errors!r}")
        elif isinstance(event, DocumentSkipped):
            print(f"skipped (unchanged): {event.document_id}")
```

### PDF ingestion
```python
from railtracks.retrieval.loaders import PyPDFLoader
from railtracks.retrieval.chunking import RecursiveCharacterChunker

# breakdown_strategy="page" (default): one Document per page
# breakdown_strategy="paragraph": one Document per non-empty paragraph on each page
# breakdown_strategy="document": one Document for the whole PDF
# Pass a folder instead of a file to load every .pdf in it, recursively
loader = PyPDFLoader("data/report.pdf", breakdown_strategy="page")
chunker = RecursiveCharacterChunker(chunk_size=800, overlap=100)
```

### CSV / structured data ingestion
```python
from railtracks.retrieval.loaders import CSVLoader

# content_columns: the columns whose text gets embedded (joined in order)
# id_column: a column that uniquely identifies each row, for stable upserts
# every other column is stored as metadata, unless listed in ignore_columns
loader = CSVLoader(
    file_path="data/products.csv",
    content_columns=["description"],
    id_column="product_id",
)
```

### HuggingFace dataset ingestion
```python
from railtracks.retrieval.loaders import HuggingFaceDatasetLoader

loader = HuggingFaceDatasetLoader(
    dataset_name="squad",
    split="train",
    content_columns=["context"],
)
```

### Multi-tenancy with StoreScope
```python
from railtracks.retrieval.stores import StoreScope

user_scope = StoreScope(labels={"user_id": "u123", "org_id": "acme"})

# Write: ingest only into this scope
stats = await runtime.ingest_all(loader=loader, scope=user_scope)

# Read: retrieve only from this scope (no scope = unscoped entries only)
result = await runtime.retrieve("question", top_k=5, scope=user_scope)

# Admin search across every tenant must be explicit
from railtracks.retrieval import ALL_SCOPES
everything = await runtime.retrieve("question", top_k=5, scope=ALL_SCOPES)

# Delete all documents for exactly this scope (labels must match exactly:
# clearing {"org_id": "acme"} does not touch {"user_id": "u123", "org_id": "acme"})
await runtime.store.clear(user_scope)
```

### RAG as an agent tool
```python
import railtracks as rt
from railtracks.retrieval import RetrievalRuntime

runtime: RetrievalRuntime = ...  # constructed at startup


@rt.function_node
async def search_knowledge_base(query: str) -> str:
    """Search the knowledge base for relevant context.
    Args:
        query: The natural language question to look up.
    Returns:
        Relevant excerpts from the knowledge base, ranked by similarity.
    """
    result = await runtime.retrieve(query, top_k=5)
    if not result.chunks:
        return "No relevant documents found."
    return "\n\n---\n\n".join(
        f"[score={rc.score:.3f}]\n{rc.chunk.content}" for rc in result.chunks
    )


RagAgent = rt.agent_node(
    "RAG Agent",
    tool_nodes=[search_knowledge_base],
    llm=rt.llm.AnthropicLLM("claude-sonnet-5-5"),
    system_message="You are a helpful assistant. Always search the knowledge base before answering.",
)
flow = rt.Flow(name="RAG Flow", entry_point=RagAgent)
```

### Sentence chunking (narrative / long-form text)
```python
from railtracks.retrieval.chunking import SentenceChunker

# chunk_size: number of sentences per chunk
# overlap: number of sentences shared between consecutive chunks
chunker = SentenceChunker(chunk_size=6, overlap=1)
```

### Local/offline embeddings with Ollama
```python
from railtracks.retrieval.embedding import OllamaEmbedding

embedder = OllamaEmbedding(model="nomic-embed-text", api_base="http://localhost:11434")
```

### Delete a document from the store
```python
# Re-ingest will upsert (content-hash skips unchanged docs)
# To explicitly remove (document_id is the Document.id UUID):
from uuid import UUID

await runtime.delete_document(document_id=UUID("1f0c9b6e-0000-4000-8000-000000000000"))
# Pass the scope it was ingested with to delete only that tenant's copy
await runtime.delete_document(document_id=UUID("1f0c9b6e-0000-4000-8000-000000000000"), scope=user_scope)
```

---

## Rules
- Create Chroma backends with `await ChromaBackend.create(...)`, which handles initialization.
- Use the same embedding model for ingest and retrieve on a collection; railtracks raises `EmbeddingModelMismatchError` on a mismatch.
- Ingest with `ingest_all()` and a loader that streams (`astream()`), so documents are processed as they load.
- Check `documents_failed` in the ingestion stats and surface any failures to the user. One unreadable file or failing document is reported as a `DocumentFailed` (see its `stage` and `source`) and the rest still ingest; pass `on_error="raise"` to `RetrievalRuntime` only when a single bad document should stop the run.
- In production, store vectors in `ChromaBackend` with a `path`, or in `PgvectorBackend`; `InMemoryVectorBackend` keeps them only until the process restarts.
- Build `RetrievalRuntime` once at startup and reuse it across requests.
- Pass `retrieve()` the user's natural-language question as the query.

Versions

RepositoryScoreLicence
This page · railtownai/railtracks100MIT
ericrisco/rsc-harness100MIT
railtownai/railtracks100MIT
automateyournetwork/netclaw100Apache-2.0
nimadorostkar/Claude-Skills-collection100MIT
giuseppe-trisciuoglio/developer-kit100MIT
Alejo2075/claude-code-aws90MIT
clawic/skills90MIT
dkbnull/hello-skill86MIT

Database snapshot: 13.11.2026 · package version 0bf703ae3a4a (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100rag-pipeline
    Build a RAG (retrieval-augmented generation) pipeline or a custom search engine on top of Bright Data's Discover API — using intent-ranked web results + parsed page content as the retrieval/ingestion layer for an LLM or vector store. Use when the user wants to "build a RAG…
  • 100rag-tool
    Store and search text or markdown as semantic chunks in a reusable local vector store. Use when a user wants to ingest documents or retrieve relevant passages from previously stored material.
  • 100retrieval-augmented-generation-rag-pipeline
    Use this skill when constructing end-to-end RAG architectures combining embeddings, vector indexing, retrieval, and contextual generation.
  • 100agent-builder
    Build an agent using the railtracks Python framework. Use when the user wants to create an AI agent, tool-calling workflow, or multi-agent system with railtracks.
  • 100rag-engineer
    Provides Retrieval-Augmented Generation patterns covering embedding models, vector databases, chunking strategies, and retrieval optimization. Use when building RAG systems or when the user mentions RAG, vector search, embeddings, or retrieval-augmented generation.
  • 100rag-architecture-designer
    Design a Retrieval-Augmented Generation (RAG) pipeline architecture: ingestion, indexing, retrieval, generation, and evaluation components. Use when working on RAG architecture, retrieval pipelines, vector stores.

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.