If I were building a production RAG system in 2026, I wouldn't start by choosing a framework. I'd start by deciding how much abstraction I actually need.
My general preference is:
| Scenario | What I'd choose |
|---|
| Small prototype (1-2 weeks) | LlamaIndex |
| Production application with many integrations | LangChain (mostly LangGraph) |
| Large-scale production where retrieval is the core product | Thin custom orchestration + Haystack components + your own retrieval pipeline |
| Research-heavy retrieval | Haystack |
My preferred production architecture
Documents
│
Parsing pipeline
│
Chunking + metadata extraction
│
Dense embeddings (BGE, Jina, etc.)
│
Hybrid Index
┌──────────┴──────────┐
Vector DB BM25
└──────────┬──────────┘
│
Candidate retrieval
│
Cross-encoder reranker
│
Context compression/filtering
│
LLM generation
│
Citation verification step
Notice that the orchestration framework isn't the interesting part.
Retrieval quality usually dominates performance.
Framework comparison
LlamaIndex
Pros:
- extremely fast to build
- excellent ingestion pipeline
- lots of retrieval techniques built in
- strong support for structured data
- good query engines
Weaknesses:
- abstraction can become difficult once customization grows
- some APIs evolve quickly
I still think it's the fastest route from "documents" to "working RAG."
LangChain / LangGraph
I no longer think of LangChain as just chains.
The interesting piece is LangGraph.
It gives:
- deterministic workflows
- retries
- checkpoints
- state management
- human approval
- branching
- agents
For enterprise workflows that's valuable.
I would use LangGraph for orchestration, not retrieval.
Haystack
Haystack shines when retrieval itself is the product.
Examples:
- advanced retrievers
- multiple rankers
- pipelines
- evaluation
- experimentation
Its retrieval abstractions are excellent.
If I were building:
- legal search
- medical search
- internal enterprise search
- scientific search
I'd probably lean Haystack.
Retrieval quality at scale
This is where most RAG systems succeed or fail.
A common beginner pipeline looks like:
embed query
↓
top 5 cosine similarity
↓
LLM
That leaves a lot of performance on the table.
A stronger pipeline is:
Query
↓
Query rewriting
↓
Hybrid retrieval
↓
Dense retrieval
+
Sparse retrieval
↓
Merge
↓
Rerank
↓
Context compression
↓
LLM
1. Hybrid search
Always.
Dense vectors alone miss exact matches.
Sparse search alone misses semantics.
I almost always combine:
using Reciprocal Rank Fusion (RRF) or weighted fusion.
This consistently outperforms either approach alone.
2. Query rewriting
Many user questions are poor retrieval queries.
Instead of
"How do I fix this?"
rewrite to
"Troubleshooting authentication timeout after OAuth callback"
The LLM generates a retrieval-friendly query.
Quality usually improves significantly.
3. Multi-query retrieval
Instead of one embedding:
Generate 3–5 alternative formulations.
Example:
"What are GPU memory optimizations?"
↓
"VRAM optimization"
↓
"Reduce CUDA memory"
↓
"Memory-efficient inference"
↓
retrieve all
↓
merge
This improves recall.
4. Parent-child retrieval
Instead of embedding huge documents:
Store
Retrieve
Return
You get precise retrieval without losing context.
This works very well.
5. Cross-encoder reranking
This is probably the highest-ROI improvement after hybrid search.
Pipeline:
retrieve 100
↓
rerank
↓
keep top 8
Instead of vector similarity,
the reranker actually reads
query + document
and scores relevance.
This often improves answer quality dramatically.
6. Metadata filtering
Instead of searching everything:
department = HR
year > 2024
product = API
language = English
Search becomes:
metadata filter
↓
vector search
Much higher precision.
7. Context compression
Retrieved chunks often contain irrelevant material.
Use a small model to extract only the relevant spans.
Instead of:
8 × 800 tokens
you send:
8 × 120 tokens
Benefits:
- lower cost
- better focus
- fewer hallucinations
Evaluation
Most teams evaluate the LLM.
The better approach is to evaluate retrieval separately.
Useful metrics include:
- Recall@k
- MRR (Mean Reciprocal Rank)
- NDCG
- Hit Rate
- Context precision
- Context recall
- Groundedness
- Faithfulness
If retrieval isn't finding the right documents, a better LLM won't fix the problem.
Embeddings
I wouldn't default to OpenAI embeddings anymore.
Strong choices depend on your constraints:
- OpenAI: high quality, managed API, easy integration.
- BGE (BAAI): excellent open-weight embedding models.
- Jina AI: strong multilingual and long-context embedding models.
- Nomic: solid open embeddings with permissive licensing.
The best choice depends on your latency, privacy, multilingual needs, and hosting model. It's worth benchmarking on your own corpus rather than assuming one model is universally best.
Vector databases
Any mature vector store that supports hybrid search, metadata filtering, and efficient indexing can work well. Common production choices include:
- PostgreSQL + pgvector for teams already invested in Postgres.
- Qdrant for a focused, open-source vector database with strong filtering.
- Weaviate when you want a feature-rich vector platform.
- Pinecone as a managed service with minimal operational overhead.
Operational factors (cost, scaling, backup strategy, regional availability, and team expertise) often matter more than small benchmark differences.
If I were starting today
For a greenfield production system, my stack would look something like:
- Orchestration: LangGraph (or lightweight custom orchestration if workflows are simple)
- Parsing & ingestion: LlamaIndex
- Embeddings: benchmark BGE, Jina, Nomic, and OpenAI on the target corpus
- Vector store: Qdrant or PostgreSQL + pgvector
- Retrieval: hybrid (BM25 + dense) with metadata filtering
- Reranking: a cross-encoder reranker
- Evaluation: a dedicated retrieval benchmark (Recall@k, NDCG, hit rate) plus end-to-end answer evaluation
- Observability: tracing and offline evaluation of retrieval quality before optimizing prompts
The biggest lesson from production RAG systems is that gains usually come from improving retrieval rather than prompt engineering. A solid retrieval pipeline with hybrid search, reranking, and continuous evaluation will often outperform a much larger model paired with a simplistic "top-k vector search" approach.
