For AI infrastructure, agent platform, and inference engineering interviews, there's a fairly consistent set of "deep dive" questions that come up beyond standard system design. These are intended to test whether you understand what actually happens during LLM inference and agent execution.
1. KV Cache
This is probably the most common low-level LLM systems question.
Typical questions:
- What is the KV cache?
- Why does it reduce latency?
- Why does it increase memory usage?
- Why can't two unrelated conversations share a KV cache?
- When should the cache be invalidated?
- How does it interact with long contexts?
- What happens if the system prompt changes?
- How do you cache prefixes?
- What is prefix caching?
Good answer:
During autoregressive generation, every transformer layer computes key and value tensors for all processed tokens. Without a KV cache, generating each new token would require recomputing attention over the entire prefix. The cache stores those key/value tensors so subsequent tokens only need to compute attention against the cached representations plus the new token.
Tradeoff:
- Lower latency
- Higher GPU memory
- One cache per active sequence
A common follow-up:
Which becomes the bottleneck first—compute or memory?
For long contexts, it's often GPU memory bandwidth and cache size rather than raw FLOPs.
2. Continuous batching
Very common.
Question:
How do inference servers achieve high throughput?
Expected topics:
- dynamic batching
- continuous batching
- scheduling
- token-level scheduling
Explain:
Instead of waiting for complete requests, modern servers batch requests at every decoding step.
Req A: token 34
Req B: token 10
Req C: token 102
↓
GPU executes one forward pass
This dramatically improves GPU utilization.
3. Prefill vs Decode
Interviewers love this.
Question:
What's the difference?
Prefill
- process prompt
- highly parallel
- compute-bound
Decode
- generate one token at a time
- sequential
- memory-bound
Large prompts dominate prefill.
Long outputs dominate decode.
4. Tool Calling
Questions:
How does tool calling work?
Expected explanation:
LLM
↓
Produces structured function call
↓
Runtime validates JSON
↓
Execute tool
↓
Return tool output
↓
LLM continues reasoning
Important point:
The model never executes tools.
The orchestration layer does.
Follow-ups:
How do you prevent hallucinated tools?
Answer:
- schema validation
- whitelist
- runtime validation
- authorization
- retries
5. Structured Outputs
Question:
How do you force valid JSON?
Talk about:
- JSON schema
- constrained decoding
- grammar decoding
- retries
- validation
6. Context Window
Common questions:
Why can't we have infinite context?
Discuss:
- quadratic attention
- latency
- KV cache growth
- GPU memory
Also discuss alternatives:
- RAG
- summarization
- memory
- chunking
7. RAG
Interviewers often ask:
Why not just use a larger context window?
Good answer:
Large contexts:
RAG retrieves only the relevant information, reducing cost and improving focus.
8. Hallucinations
Questions:
How do you reduce hallucinations?
Expected ideas:
- retrieval
- grounding
- tool use
- citations
- verification
- self-consistency
- evaluation
9. Agent Loop
Classic:
Reason
↓
Call Tool
↓
Observe
↓
Reason
↓
Call Tool
↓
Finish
Follow-up:
How do you stop infinite loops?
Answer:
- max iterations
- timeout
- cost budget
- confidence threshold
- human approval
10. Memory
Question:
How does ChatGPT "remember"?
Differentiate:
Conversation history
↓
Summaries
↓
Vector retrieval
↓
Persistent user profile
11. Model Routing
Question:
How would you reduce costs?
Typical architecture:
Small model
↓
If confidence high
↓
Return
Else
↓
Large model
Also:
- classify request
- route accordingly
- cache common responses
12. Prompt Injection
Almost guaranteed.
Question:
How do you stop prompt injection?
Answer:
Treat retrieved content as untrusted input.
Separate:
System instructions
User input
Retrieved documents
Never let retrieved text override system policy. Restrict tool permissions, validate tool inputs, and require explicit authorization for sensitive actions.
13. Evaluation Harness
Question:
How do you know your agent improved?
Discuss:
Offline
- benchmark datasets
- regression tests
Online
- A/B tests
- human ratings
- production metrics
Metrics:
- success rate
- latency
- token cost
- tool accuracy
- user satisfaction
14. Multi-Agent Systems
Question:
When would you use multiple agents?
Good answer:
Only when tasks naturally decompose into specialized roles or require independent reasoning. Examples include separate retrieval, planning, coding, and review agents.
Avoid unnecessary multi-agent designs because they add:
- latency
- token cost
- coordination complexity
- debugging challenges
15. Streaming
Question:
Why stream responses?
Benefits:
- lower perceived latency
- earlier user feedback
- cancellation support
- better UX
Implementation:
LLM
↓
Token stream
↓
WebSocket or Server-Sent Events (SSE)
↓
Client renders incrementally
16. Speculative Decoding
This is becoming more common.
Idea:
A smaller draft model proposes several future tokens.
The larger model verifies them in a single pass. If they match, multiple tokens are accepted at once; if not, the larger model corrects the sequence.
Benefit:
- Faster generation with similar output quality.
- Better utilization of the larger model when the draft model is accurate.
A concise "cheat sheet" of interview buzzwords
Interviewers often expect you to naturally bring up concepts like:
- Prefill vs. decode
- KV cache and prefix caching
- Continuous batching
- Paged attention / efficient KV cache management
- Speculative decoding
- Tensor, pipeline, and data parallelism
- Quantization (e.g., INT8, FP8, 4-bit)
- Mixture-of-Experts (MoE) routing
- Tool calling and structured outputs
- Retrieval-Augmented Generation (RAG)
- Prompt injection defenses
- Context engineering
- Offline vs. online evaluation
- Tracing and observability
- Model routing and cascading
- Token budgeting and cost optimization
- Rate limiting and backpressure
- Streaming inference
- Checkpointing and retry logic for long-running agents
One pattern I've seen in stronger interviews is that they move fluidly between ML concepts (attention, decoding, embeddings), distributed systems (scheduling, caching, queues), and product concerns (latency, reliability, cost, and evaluation). Demonstrating those connections is often more valuable than giving isolated definitions.
