Yes. This is one of the biggest pain points with current LLM workflows. The trick is to stop treating the model's context window like permanent memory. Instead, make it retrieve only what's relevant.
Here are the approaches that scale well, roughly from simplest to most sophisticated.
1. Keep a "project brain" (my favorite)
Instead of feeding your whole project every time, maintain a structured knowledge base:
project/
├── README.md # High-level overview
├── architecture.md # System design
├── decisions.md # ADRs and important choices
├── api.md # APIs/interfaces
├── tasks.md # Current work
├── glossary.md
└── docs/
Then only load:
- the current file
- relevant documentation
- the current task
This often reduces context by 90% or more.
2. Hierarchical summaries
Have AI summarize information at multiple levels.
Example:
Entire repo
↓
Subsystem summaries
↓
Directory summaries
↓
File summaries
↓
Function summaries
When working on a function, you might load:
- project summary
- subsystem summary
- file summary
- target function
instead of 500 files.
3. Retrieval (RAG)
Instead of dumping everything into context:
- index every document
- split into chunks
- embed them
- retrieve the top 10–30 relevant chunks
The prompt becomes:
Question
Relevant docs:
- chunk 14
- chunk 283
- chunk 91
Answer.
This is how many production AI assistants work.
4. Use Git as memory
Your repository already contains valuable context.
Useful references include:
- commit history
- pull requests
- design documents
- issue tracker
- ADRs (Architecture Decision Records)
Instead of asking:
Why do we use Redis?
Have the assistant search:
- ADRs
- commits mentioning Redis
- issues discussing caching
You retrieve the rationale only when needed.
5. Working sets
Humans don't think about an entire codebase simultaneously.
Create a working set for each task:
Task:
Implement OAuth refresh.
Relevant files:
✓ auth.py
✓ middleware.py
✓ user.py
✓ config.py
Ignore:
✗ billing
✗ analytics
✗ notifications
Many IDE AI tools build this automatically.
6. Long-term memory
Store reusable facts separately:
Facts:
- We use PostgreSQL.
- JWT expires in 15 minutes.
- Frontend is React.
- API version is v3.
Then prepend only these stable facts instead of rediscovering them every session.
7. Build a knowledge graph
Instead of storing raw text, store relationships.
UserService
↓
depends on
↓
AuthService
↓
uses
↓
Redis
When editing UserService, retrieve connected nodes rather than unrelated documentation.
8. Agentic retrieval
Modern coding agents don't send the entire repository to the model.
Instead they:
- search filenames
- search symbols
- search references
- open matching files
- read only relevant sections
- answer
This keeps token usage surprisingly low even on very large repositories.
9. Compress aggressively
Large documents often contain redundancy.
Instead of:
50 pages
store
2-page summary
Important APIs
Breaking assumptions
Open questions
Keep the full version on disk for retrieval if needed.
10. Treat context as a cache
Think of the context window like CPU cache:
- permanent knowledge → stored externally
- current task → loaded into context
- irrelevant information → evicted
This mindset scales much better than trying to fit everything into a huge prompt.
A practical stack
For a large software project, a common architecture looks like this:
Repository
│
▼
Indexer
│
▼
Embeddings / Search Index
│
▼
Retriever
│
▼
LLM Context
│
▼
Response
Alongside that, keep concise human-maintained documents for architecture, decisions, and conventions, plus automatically generated summaries for files and modules. The retriever assembles only the pieces relevant to the current task.
This approach generally scales far better than relying on a million-token context window. Even if a model can technically accept enormous prompts, sending your entire project on every request is slower, more expensive, and often less effective than targeted retrieval combined with well-maintained project documentation.
