The short version is that the default recommendation has shifted over the past couple of years:
- If you're building a product with a handful of LLM workflows, build your own lightweight orchestration.
- If you're building a sophisticated agent platform with long-running workflows, LangGraph is often worth it.
- If your primary challenge is enterprise search and RAG over lots of data sources, LlamaIndex is still one of the strongest choices.
The biggest lesson many teams learned is that orchestration frameworks introduce an abstraction layer that eventually becomes part of your infrastructure. That has real operational cost.
Here's how I'd think about it in 2026.
| Factor | Lightweight | LangGraph | LlamaIndex |
|---|
| Initial complexity | Low | Medium-High | Medium |
| Production operations | Low | Medium | Medium |
| Vendor lock-in | Low | Medium | Medium |
| Debugging | Easy | Moderate | Moderate |
| Agent workflows | Manual | Excellent | Good |
| RAG | Manual | Good | Excellent |
| Multi-step state | Manual | Excellent | Good |
| Long-term maintenance | Low | Medium | Medium |
Option 1: Lightweight orchestration (my default recommendation)
This usually means:
- direct SDK calls (OpenAI, Anthropic, etc.)
- your own Python/TypeScript functions
- async queues
- database for state
- tracing via OpenTelemetry or an observability platform
- explicit workflow code
Example:
answer = await classify(question)
if answer.intent == "support":
docs = await retrieve_docs(...)
response = await generate(...)
elif answer.intent == "sales":
...
Not:
Agent
↓
Router Chain
↓
Tool Node
↓
Conditional Graph
↓
Prompt Node
Advantages:
- almost zero magic
- every engineer understands it
- easier testing
- easier profiling
- easier retries
- easier migration between model providers
Most production SaaS products today still resemble normal backend services that happen to call LLMs.
Option 2: LangGraph
LangGraph has become the mature successor to the "classic LangChain" style.
It's genuinely useful when you have:
- long-running agents
- resumable workflows
- human approval
- persistent state
- many tools
- cycles
- retries
- interruptions
For example:
Research Agent
↓
Search
↓
Analyze
↓
Need More Data?
↙ ↘
yes no
↓ ↓
Search Report
That's where a graph starts making sense.
It also provides:
- checkpointing
- durable execution
- state transitions
- visualization
- replay
- branch debugging
Those are difficult to recreate well.
The downside is conceptual overhead.
A new engineer has to understand:
- graph state
- nodes
- edges
- reducers
- messages
- checkpointing
- execution semantics
instead of just reading Python.
Option 3: LlamaIndex
LlamaIndex has become much more than "an indexing library."
It's strongest when your product is fundamentally about knowledge.
Examples:
- internal company search
- enterprise assistants
- document QA
- RAG
- multi-index routing
- SQL + documents
- vector + graph retrieval
Its retrieval ecosystem is deeper than what most teams want to build themselves.
If retrieval quality is your differentiator, LlamaIndex saves months.
If retrieval is only 15% of your product, it may be unnecessary.
The hidden production costs
This is where teams often underestimate the tradeoffs.
1. Learning curve
Lightweight:
- every backend engineer contributes immediately
LangGraph:
- people need to learn framework concepts
LlamaIndex:
- people need to learn its retrieval abstractions
The cost compounds as the team grows.
2. Debugging
Suppose a customer says:
"The AI gave a bad answer."
With lightweight orchestration:
API request
↓
retrieve()
↓
prompt()
↓
LLM()
↓
response
Easy.
With graph systems:
API
↓
Graph
↓
State
↓
Router
↓
Node
↓
Checkpoint
↓
Reducer
↓
Prompt
↓
LLM
There are simply more moving parts.
3. Version upgrades
Frameworks evolve.
Every major release may require:
- migration
- adapting deprecated APIs
- changing graph definitions
- updating plugins
A small orchestration layer you own usually changes only when you choose.
4. Framework leakage
Over time you start writing code like:
GraphState(...)
Node(...)
Runnable(...)
Message(...)
instead of plain business logic.
Eventually your application becomes coupled to the framework's execution model.
Where frameworks save enormous time
They really shine for:
- memory
- checkpointing
- human-in-the-loop
- tool calling
- graph visualization
- retries
- durable execution
- multi-agent systems
Rebuilding those well isn't trivial.
What many successful teams look like
A pattern that's become increasingly common:
API
↓
Business Logic
↓
Your Orchestrator
↓
LLM SDK
↓
Models
Where "Your Orchestrator" is only a few hundred lines of code.
It handles:
- prompt versioning
- retries
- logging
- evaluation hooks
- provider abstraction
- structured outputs
- metrics
- caching
That's often enough for products that have dozens of LLM endpoints.
When I would choose each
Choose lightweight orchestration if:
- You have 1–20 LLM workflows.
- Most workflows are linear or have simple branching.
- Your team is experienced with backend engineering and values explicit code.
- You expect to switch model providers or evolve your architecture over time.
Choose LangGraph if:
- You're building agentic systems with long-running, stateful, or interruptible workflows.
- You need durable execution, human approvals, replay, or complex tool orchestration.
- You're comfortable investing in a framework to gain those capabilities.
Choose LlamaIndex if:
- Retrieval quality is central to your product.
- You need connectors to many data sources, sophisticated indexing, or advanced retrieval strategies.
- Document and knowledge management are a major part of the system.
What I'd recommend for most existing SaaS products
Since you're adding LLM features to an existing product (rather than building an AI platform from scratch), I'd start with a lightweight orchestration layer:
- Keep workflow logic as ordinary application code.
- Use the model provider SDKs directly with structured outputs.
- Add robust observability, prompt versioning, evaluations, retries, and caching from day one.
- Introduce LangGraph only if you later discover you need durable, graph-based agent workflows.
- Adopt LlamaIndex selectively if retrieval becomes a core capability rather than a supporting feature.
That approach minimizes operational complexity while preserving a straightforward path to adopt specialized frameworks later, if your product's requirements genuinely outgrow a simpler architecture.
