A laboratory for evaluating RAG strategies
Compare 3 chunking strategies × 4 retrieval modes across 7 evaluation metrics. Inspect retrieved chunks, measure latency, cost, and identify failure patterns with a built-in taxonomy.
Everything you need to evaluate a RAG pipeline
RAGBench runs a controlled experiment across all config combinations, collecting metrics that matter for production RAG: relevance, faithfulness, citation accuracy, and cost.
Recursive character splitting, semantic chunking via embedding similarity, and parent-child hierarchical chunking. Each strategy produces a different chunk topology for retrieval.
Full-text search (BM25/tsvector), dense vector retrieval (Voyage embeddings + pgvector), hybrid RRF fusion, and hybrid with Voyage rerank-2.5-lite re-scoring.
Context Precision, Context Recall, Answer Faithfulness, Answer Relevance, Citation Accuracy, Abstention Rate, and End-to-end Latency. All tracked per config.
Every query result is classified with failure tags: retrieval miss, hallucination, over-abstention, citation gap, and more. Enables targeted debugging of RAG pipelines.
How RAGBench works
Each query runs through a reproducible pipeline: documents are ingested, chunked with a chosen strategy, embedded, retrieved with a chosen mode, then generated and evaluated.
Source documents are loaded from the data directory and parsed into a normalized format ready for chunking.
Each document is split using one of three strategies: recursive (fixed-size with overlap), semantic (embedding-based boundaries), or parent-child (hierarchical).
Chunks are embedded with Voyage AI's voyage-3-lite model and stored in PostgreSQL with pgvector for similarity search.
At query time, the selected retrieval mode (FTS, Dense, Hybrid, or Hybrid+Rerank) fetches the top-k most relevant chunks.
Claude claude-sonnet-4-6 receives the retrieved chunks as context and generates an answer with inline citations referencing specific chunks.
The answer is scored across 7 metrics using an LLM judge and exact-match rules. Results are stored and compared across all 12 configs.
Run the chat interface to interactively test any chunker + retrieval combination, or view aggregated metrics in the eval dashboard.