For production deployment, the order is usually driven by the tradeoff you're optimizing (latency, memory, energy, or accuracy), but a common workflow is:
- Train a strong teacher model.
- Distill into a smaller student (if you're reducing architecture size).
- Apply pruning (if appropriate).
- Fine-tune to recover accuracy.
- Quantize.
- Benchmark on the target hardware.
- Iterate.
The reason quantization often comes last is that it's tightly coupled to the deployment hardware, while distillation and pruning change the model itself.
Here's how each technique fits.
| Technique | Best for | Typical accuracy impact | Typical speedup |
|---|
| Distillation | Shrinking model architecture | Low if done well | High |
| Pruning | Removing redundant weights/channels | Moderate unless structured | Moderate |
| Quantization | Lower precision arithmetic | Usually very low (INT8), higher for INT4 | Often highest practical gain |
1. Distillation first
If you're willing to retrain, this is usually the highest-leverage optimization.
Instead of trying to compress a huge model, train a smaller architecture to imitate the larger one.
Advantages:
- Reduces parameter count
- Reduces FLOPs
- Usually easier to deploy
- Often preserves most accuracy
For example:
- Teacher: 1B parameters
- Student: 300M parameters
- Accuracy loss: perhaps only 1–3%
Now every later optimization starts from a much smaller model.
2. Pruning next (when useful)
Pruning removes unnecessary weights.
There are two major categories.
Unstructured pruning
- Removes individual weights
- High sparsity possible
- Little real-world speedup unless hardware supports sparse kernels
Structured pruning
- Removes channels
- Removes attention heads
- Removes filters
- Removes entire layers
Structured pruning is usually much more production-friendly because standard inference libraries can exploit it.
Typical workflow:
- Prune 20–50%
- Fine-tune
- Measure
- Repeat
3. Quantization last
Quantization changes numerical precision.
Common options:
In production, INT8 is often the "safe default."
Reasons:
- Broad hardware support
- Small accuracy loss
- Significant reductions in memory bandwidth
- Faster inference on CPUs, NPUs, and many accelerators
For more aggressive deployments:
- INT4 for LLMs
- Mixed precision
- Weight-only quantization
These require more careful evaluation.
Post-training vs. quantization-aware training
Start with:
Post-training quantization (PTQ)
because it's inexpensive and often works surprisingly well.
If accuracy drops too much:
Quantization-aware training (QAT)
During training, the model simulates low-precision arithmetic, usually recovering much of the lost accuracy.
Fine-tuning between stages
A common production pipeline looks like:
Train teacher
↓
Distill student
↓
Prune
↓
Fine-tune
↓
Quantize
↓
Evaluate
Each compression step can introduce some accuracy loss, and a short fine-tuning stage often recovers much of it.
Always benchmark on the deployment hardware
A common mistake is assuming fewer parameters always means faster inference.
Not necessarily.
For example:
- 50% unstructured sparsity may yield almost no speedup on standard GPUs.
- INT8 often provides significant gains on CPUs with vector extensions.
- FP16 may outperform INT8 on some GPUs because of optimized tensor cores.
- Memory bandwidth can become the bottleneck before compute.
The deployment target (CPU, GPU, NPU, edge accelerator, mobile SoC) largely determines which optimization is most effective.
Practical recommendations by scenario
- Cloud GPU inference: Start with FP16 or BF16, then explore INT8 or weight-only quantization if supported. Prune only if your serving stack benefits from structured sparsity.
- CPU serving: INT8 quantization typically delivers the biggest improvement. Structured pruning can help if your inference engine exploits it.
- Mobile or edge devices: Distillation is often the highest-impact step because it reduces both compute and memory. Follow with INT8 quantization, and consider structured pruning if the runtime supports it.
- Large language models: Weight-only quantization (e.g., INT8 or INT4) is commonly the first optimization because it greatly reduces memory usage with relatively small quality degradation. Distillation for generative LLMs is possible but generally more involved than for classifiers.
In practice, distillation → structured pruning (if beneficial) → fine-tuning → quantization is a strong default pipeline. That said, many successful production systems skip pruning entirely because the engineering complexity outweighs the gains, relying instead on a well-distilled model plus hardware-friendly quantization, which often provides the best balance of accuracy, latency, and operational simplicity.
