rag
/rag
Build a RAG (retrieval-augmented generation) pipeline using railtracks. Use when the user wants to ingest documents into a vector store and retrieve relevant passages to answer questions.
- Author
- railtownai ↗
- Repository
- railtracks ↗
- Licence
- MIT
- Last repository change
- 12.11.2026
Assessment and security
- Files in this package
- 1
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 2
No signals detected. Repository changed on 12.11.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
What it does and when to use it
Build a RAG (retrieval-augmented generation) pipeline using railtracks. Use when the user wants to ingest documents into a vector store and retrieve relevant passages to answer questions.
Arguments
/rag [describe the data source and what you want to retrieve]
Original author description
Build a RAG (retrieval-augmented generation) pipeline using railtracks. Use when the user wants to ingest documents into a vector store and retrieve relevant passages to answer questions.
How it works
- How railtracks retrieval works
- Component reference
- Steps
- Patterns to follow
- Rules
SKILL.md preview
---
name: rag
description: Build a RAG (retrieval-augmented generation) pipeline using railtracks. Use when the user wants to ingest documents into a vector store and retrieve relevant passages to answer questions.
argument-hint: '[describe the data source and what you want to retrieve]'
---
# Build a Railtracks RAG Pipeline
The user wants to build a retrieval-augmented generation (RAG) pipeline using the railtracks framework: $ARGUMENTS
## How railtracks retrieval works
The pipeline has two paths:
**Write path (ingestion):** `Loader → Chunker → Embedder → VectorStore`
**Read path (retrieval):** `query → Embedder → VectorStore.search → RetrievalResult`
`RetrievalRuntime` orchestrates both paths. It takes a chunker, embedder, and store at construction time, then exposes `ingest_all()` / `ingest()` for writes and `retrieve()` for reads.
### Core data flow
| Stage | Input | Output |
|-------|-------|--------|
| Loader | source (file, URL, dataset) | `Document` |
| Chunker | `Document` | `Chunk` |
| Embedder | `Chunk.content` | `EmbeddedChunk` |
| Store | `EmbeddedChunk` | stored `StoreEntry` |
| Retrieve | query string | `RetrievalResult` (ranked `RetrievedChunk` list) |
---
## Component reference
### Loaders
```python
from railtracks.retrieval.loaders import (
TextLoader, # .txt / .md files
CSVLoader, # rows → documents; content_columns, id_column, ignore_columns
JSONLoader, # .json / .jsonl files
PyPDFLoader, # PDF file or folder; breakdown_strategy="page" (default), "paragraph", or "document"
HuggingFaceDatasetLoader, # HF Hub datasets; pass dataset_name, split, content_columns
LangChainLoaderAdapter, # wrap any LangChain document loader
)
```
### Chunkers
```python
from railtracks.retrieval.chunking import (
RecursiveCharacterChunker, # general text; chunk_size (chars), overlap
MarkdownHeaderChunker, # heading-aware splits for .md content
SentenceChunker, # sentence windows; chunk_size (# sentences), overlap
SemanticChunker, # embedding-driven breakpoints; variable chunk size
FixedTokenChunker, # hard token budget; chunk_size (tokens), overlap
)
```
### Embedders
```python
from railtracks.retrieval.embedding import (
OpenAIEmbedding, # default model: "text-embedding-3-small"
AzureEmbedding, # Azure OpenAI routing
OllamaEmbedding, # local dev; model, api_base
LiteLLMEmbedding, # any LiteLLM-supported provider
)
```
### Store backends
```python
from railtracks.retrieval.stores import (
VectorStore, # store implementation; wraps a backend
InMemoryVectorBackend, # tests and small corpora; no persistence by default
ChromaBackend, # local / persistent / HTTP modes
ChromaCloudBackend, # managed Chroma Cloud
PgvectorBackend, # Postgres + pgvector; production
)
```
### Installation extras
```
pip install "railtracks[retrieval-core]" # minimum (no connectors)
pip install "railtracks[retrieval]" # all connectors
pip install "railtracks[chroma]" # Chroma only
pip install "railtracks[huggingface]" # HuggingFace datasets
pip install "railtracks[ocr]" # PyPDF + OCR (requires Tesseract)
```
---
## Steps
1. **Read the existing code** — check what files already exist. Understand what data source, chunking strategy, and query pattern the user needs before writing anything.
2. **Choose a loader** — match the source type (file, PDF, CSV, HuggingFace dataset, etc.). Ask the user to clarify if not obvious from `$ARGUMENTS`.
3. **Choose a chunker** — default to `RecursiveCharacterChunker` for plain text, `MarkdownHeaderChunker` for structured docs, `SentenceChunker` for narrative text. Use `SemanticChunker` only when chunk boundaries matter semantically.
4. **Choose an embedder** — if the project already has a populated collection, use the embedder and model it was built with (mixing models raises `EmbeddingModelMismatchError`). Otherwise match the provider the project already uses, or one whose API key is configured (embedders and their settings: https://docs.railtracks.org/retrieval/components/embeddings/methods/); otherwise ask the user. Use `OllamaEmbedding` for local/offline dev.
5. **Choose a backend** — `InMemoryVectorBackend` for demos/tests, `ChromaBackend` for local persistence, `PgvectorBackend` for production.
6. **Construct `RetrievalRuntime`** — pass chunker, embedder, store. Initialize async backends with `await backend.initialize()` or the `ChromaBackend.create()` factory.
7. **Ingest documents** — call `runtime.ingest_all(loader=...)`. Print stats. Handle `documents_failed` gracefully.
8. **Add a retrieve function** — call `runtime.retrieve(query, top_k=...)`. Return or format `result.chunks`.
9. **Wire into an agent (if needed)** — expose `retrieve` as a `@rt.function_node` tool, or call it in a pre-invoke hook.
10. **Add a `if __name__ == "__main__":` block** — include both an ingest path and a query example so the user can run it immediately.
---
## Patterns to follow
### Minimal pipeline (in-memory, for demos/tests)
```python
import asyncio
from railtracks.retrieval import RetrievalRuntime
from railtracks.retrieval.loaders import TextLoader
from railtracks.retrieval.chunking import RecursiveCharacterChunker
from railtracks.retrieval.embedding import OpenAIEmbedding
from railtracks.retrieval.stores import VectorStore, InMemoryVectorBackend
async def main():
runtime = RetrievalRuntime(
chunker=RecursiveCharacterChunker(chunk_size=1000, overlap=200),
embedder=OpenAIEmbedding(),
store=VectorStore(InMemoryVectorBackend()),
batch_size=100,
)
stats = await runtime.ingest_all(loader=TextLoader("data/my_file.txt"))
print(f"chunks embedded: {stats.chunks_embedded}")
result = await runtime.retrieve("your question here", top_k=5)
for rc in result.chunks:
print(f"[rank={rc.rank} score={rc.score:.3f}] {rc.chunk.content[:200]}")
if __name__ == "__main__":
asyncio.run(main())
```
### Persistent pipeline (Chroma, recommended for local dev)
```python
import asyncio
from railtracks.retrieval import RetrievalRuntime
from railtracks.retrieval.loaders import TextLoader
from railtracks.retrieval.chunking import RecursiveCharacterChunker
from railtracks.retrieval.embedding import OpenAIEmbedding
from railtracks.retrieval.stores import VectorStore, ChromaBackend
async def create_runtime() -> RetrievalRuntime:
backend = await ChromaBackend.create(
collection_name="my-collection",
path="./my_db",
)
return RetrievalRuntime(
chunker=RecursiveCharacterChunker(chunk_size=1000, overlap=200),
embedder=OpenAIEmbedding(model="text-embedding-3-small"),
store=VectorStore(backend),
batch_size=100,
)
async def main():
runtime = await create_runtime()
stats = await runtime.ingest_all(loader=TextLoader("data/my_file.txt"))
print(
f"loaded={stats.documents_loaded} chunks={stats.chunks_embedded} skipped={stats.documents_skipped}"
)
result = await runtime.retrieve("your question here", top_k=5)
for rc in result.chunks:
print(f"[rank={rc.rank} score={rc.score:.3f}]\n{rc.chunk.content}\n")
if __name__ == "__main__":
asyncio.run(main())
```
### Streaming ingestion events (for progress / error handling)
```python
from railtracks.retrieval import RetrievalRuntime
from railtracks.retrieval.runtime import BatchIngested, DocumentFailed, DocumentSkipped
async def ingest_with_progress(runtime: RetrievalRuntime, loader) -> None:
async for event in runtime.ingest(loader=loader):
if isinstance(event, BatchIngested):
print(f"batch {event.batch_index}: {len(event.embedded_chunks)} chunks written")
elif isinstance(event, DocumentFailed):
# stage is "load", "chunk", "embed", or "store"; document_id is None for "load"
print(f"FAILED at {event.stage}: {event.source} — {event.errors!r}")
elif isinstance(event, DocumentSkipped):
print(f"skipped (unchanged): {event.document_id}")
```
### PDF ingestion
```python
from railtracks.retrieval.loaders import PyPDFLoader
from railtracks.retrieval.chunking import RecursiveCharacterChunker
# breakdown_strategy="page" (default): one Document per page
# breakdown_strategy="paragraph": one Document per non-empty paragraph on each page
# breakdown_strategy="document": one Document for the whole PDF
# Pass a folder instead of a file to load every .pdf in it, recursively
loader = PyPDFLoader("data/report.pdf", breakdown_strategy="page")
chunker = RecursiveCharacterChunker(chunk_size=800, overlap=100)
```
### CSV / structured data ingestion
```python
from railtracks.retrieval.loaders import CSVLoader
# content_columns: the columns whose text gets embedded (joined in order)
# id_column: a column that uniquely identifies each row, for stable upserts
# every other column is stored as metadata, unless listed in ignore_columns
loader = CSVLoader(
file_path="data/products.csv",
content_columns=["description"],
id_column="product_id",
)
```
### HuggingFace dataset ingestion
```python
from railtracks.retrieval.loaders import HuggingFaceDatasetLoader
loader = HuggingFaceDatasetLoader(
dataset_name="squad",
split="train",
content_columns=["context"],
)
```
### Multi-tenancy with StoreScope
```python
from railtracks.retrieval.stores import StoreScope
user_scope = StoreScope(labels={"user_id": "u123", "org_id": "acme"})
# Write: ingest only into this scope
stats = await runtime.ingest_all(loader=loader, scope=user_scope)
# Read: retrieve only from this scope (no scope = unscoped entries only)
result = await runtime.retrieve("question", top_k=5, scope=user_scope)
# Admin search across every tenant must be explicit
from railtracks.retrieval import ALL_SCOPES
everything = await runtime.retrieve("question", top_k=5, scope=ALL_SCOPES)
# Delete all documents for exactly this scope (labels must match exactly:
# clearing {"org_id": "acme"} does not touch {"user_id": "u123", "org_id": "acme"})
await runtime.store.clear(user_scope)
```
### RAG as an agent tool
```python
import railtracks as rt
from railtracks.retrieval import RetrievalRuntime
runtime: RetrievalRuntime = ... # constructed at startup
@rt.function_node
async def search_knowledge_base(query: str) -> str:
"""Search the knowledge base for relevant context.
Args:
query: The natural language question to look up.
Returns:
Relevant excerpts from the knowledge base, ranked by similarity.
"""
result = await runtime.retrieve(query, top_k=5)
if not result.chunks:
return "No relevant documents found."
return "\n\n---\n\n".join(
f"[score={rc.score:.3f}]\n{rc.chunk.content}" for rc in result.chunks
)
RagAgent = rt.agent_node(
"RAG Agent",
tool_nodes=[search_knowledge_base],
llm=rt.llm.AnthropicLLM("claude-sonnet-5-5"),
system_message="You are a helpful assistant. Always search the knowledge base before answering.",
)
flow = rt.Flow(name="RAG Flow", entry_point=RagAgent)
```
### Sentence chunking (narrative / long-form text)
```python
from railtracks.retrieval.chunking import SentenceChunker
# chunk_size: number of sentences per chunk
# overlap: number of sentences shared between consecutive chunks
chunker = SentenceChunker(chunk_size=6, overlap=1)
```
### Local/offline embeddings with Ollama
```python
from railtracks.retrieval.embedding import OllamaEmbedding
embedder = OllamaEmbedding(model="nomic-embed-text", api_base="http://localhost:11434")
```
### Delete a document from the store
```python
# Re-ingest will upsert (content-hash skips unchanged docs)
# To explicitly remove (document_id is the Document.id UUID):
from uuid import UUID
await runtime.delete_document(document_id=UUID("1f0c9b6e-0000-4000-8000-000000000000"))
# Pass the scope it was ingested with to delete only that tenant's copy
await runtime.delete_document(document_id=UUID("1f0c9b6e-0000-4000-8000-000000000000"), scope=user_scope)
```
---
## Rules
- Create Chroma backends with `await ChromaBackend.create(...)`, which handles initialization.
- Use the same embedding model for ingest and retrieve on a collection; railtracks raises `EmbeddingModelMismatchError` on a mismatch.
- Ingest with `ingest_all()` and a loader that streams (`astream()`), so documents are processed as they load.
- Check `documents_failed` in the ingestion stats and surface any failures to the user. One unreadable file or failing document is reported as a `DocumentFailed` (see its `stage` and `source`) and the rest still ingest; pass `on_error="raise"` to `RetrievalRuntime` only when a single bad document should stop the run.
- In production, store vectors in `ChromaBackend` with a `path`, or in `PgvectorBackend`; `InMemoryVectorBackend` keeps them only until the process restarts.
- Build `RetrievalRuntime` once at startup and reuse it across requests.
- Pass `retrieve()` the user's natural-language question as the query.
Versions
| Repository | Score | Licence |
|---|---|---|
| This page · railtownai/railtracks | 100 | MIT |
| ericrisco/rsc-harness | 100 | MIT |
| railtownai/railtracks | 100 | MIT |
| automateyournetwork/netclaw | 100 | Apache-2.0 |
| nimadorostkar/Claude-Skills-collection | 100 | MIT |
| giuseppe-trisciuoglio/developer-kit | 100 | MIT |
| Alejo2075/claude-code-aws | 90 | MIT |
| clawic/skills | 90 | MIT |
| dkbnull/hello-skill | 86 | MIT |
Database snapshot: 13.11.2026 · package version 0bf703ae3a4a (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100rag-pipeline
Build a RAG (retrieval-augmented generation) pipeline or a custom search engine on top of Bright Data's Discover API — using intent-ranked web results + parsed page content as the retrieval/ingestion layer for an LLM or vector store. Use when the user wants to "build a RAG… - 100rag-tool
Store and search text or markdown as semantic chunks in a reusable local vector store. Use when a user wants to ingest documents or retrieve relevant passages from previously stored material. - 100retrieval-augmented-generation-rag-pipeline
Use this skill when constructing end-to-end RAG architectures combining embeddings, vector indexing, retrieval, and contextual generation. - 100agent-builder
Build an agent using the railtracks Python framework. Use when the user wants to create an AI agent, tool-calling workflow, or multi-agent system with railtracks. - 100rag-engineer
Provides Retrieval-Augmented Generation patterns covering embedding models, vector databases, chunking strategies, and retrieval optimization. Use when building RAG systems or when the user mentions RAG, vector search, embeddings, or retrieval-augmented generation. - 100rag-architecture-designer
Design a Retrieval-Augmented Generation (RAG) pipeline architecture: ingestion, indexing, retrieval, generation, and evaluation components. Use when working on RAG architecture, retrieval pipelines, vector stores.