The right choice depends more on your deployment target than on the model itself. After deploying models across phones, embedded Linux, and industrial hardware, a few patterns consistently emerge.
Here's a practical comparison:
| Runtime | Best for | Strengths | Weaknesses |
|---|
| TensorFlow Lite | Android, microcontrollers, mobile | Excellent mobile tooling, quantization, Edge TPU support | Less pleasant if your training isn't in TensorFlow |
| ONNX Runtime | Cross-platform (Windows, Linux, embedded, x86, ARM) | Supports models from PyTorch, TensorFlow, scikit-learn, etc.; many hardware execution providers | Operator compatibility occasionally requires adjustments |
| PyTorch ExecuTorch | Mobile and embedded PyTorch deployments | Native PyTorch workflow | Ecosystem still maturing compared to TFLite/ONNX |
| NVIDIA TensorRT | NVIDIA Jetson and GPUs | Outstanding inference performance | NVIDIA-only |
| OpenVINO | Intel CPUs, GPUs, NPUs | Excellent optimization for Intel hardware | Primarily Intel-focused |
| TVM | Research and highly optimized deployments | Can produce very efficient binaries | Steeper learning curve |
My general recommendation
If you're starting a new project today:
- Embedded Linux (Raspberry Pi, ARM SBCs): ONNX Runtime
- Android: TensorFlow Lite or ONNX Runtime Mobile
- iOS: Core ML (often converted from ONNX or PyTorch)
- Jetson: TensorRT
- Microcontrollers (256 KB–2 MB RAM): TensorFlow Lite for Microcontrollers
- Mixed hardware fleet: ONNX Runtime
ONNX has become something of a common interchange format, making it easier to move models between ecosystems.
Lessons learned
1. Quantization usually matters more than architecture
Many teams spend weeks trying different model architectures when an optimized version of the existing model delivers a larger improvement.
Typical gains:
- INT8: 2–4× faster
- 4× smaller model
- Much lower memory bandwidth
For many vision and tabular tasks, accuracy drops only slightly with proper calibration.
2. Measure end-to-end latency
Model inference is often only part of the pipeline.
Example:
- Camera capture: 10 ms
- Resize/preprocess: 15 ms
- Model inference: 18 ms
- Post-processing: 25 ms
Total latency: 68 ms
Optimizing inference by 20% only saves about 4 ms overall.
3. Memory is usually the real constraint
On edge devices, RAM is often more limiting than compute.
Watch for:
- Multiple copies of tensors
- Image format conversions
- Dynamic allocations
- Large intermediate activations
Peak memory can exceed the model size several times over.
4. Operator support is never guaranteed
Just because a model exports doesn't mean it runs efficiently.
Common issues include:
- Unsupported operators
- Dynamic tensor shapes
- Custom layers
- Non-Maximum Suppression implementations
- Certain transformer attention variants
Before committing to an architecture, verify that your target runtime supports the required operators.
5. Batch size is usually 1
Unlike server inference, edge applications typically process one input at a time.
Optimize for:
- Single-image latency
- Cold-start performance
- Memory footprint
High throughput is often less important.
6. Hardware acceleration isn't automatic
Many devices advertise NPUs or AI accelerators, but they may support only subsets of operations.
It's common to see part of the graph execute on the accelerator while unsupported operators fall back to the CPU, reducing the expected speedup.
Always inspect execution logs to confirm where each operator runs.
7. Profiling early saves time
A simple benchmark on the target hardware can reveal:
- CPU utilization
- Memory usage
- Thermal throttling
- Accelerator utilization
- Power consumption
Desktop benchmarks rarely predict edge-device performance accurately.
8. Preprocessing consistency matters
A surprising number of deployment issues come from mismatched preprocessing:
- RGB vs. BGR
- NHWC vs. NCHW layouts
- Mean/std normalization
- Image resizing method
- Data types (
uint8 vs. float32)
Keeping preprocessing identical between training and deployment avoids subtle accuracy regressions.
If you're using PyTorch
A common production workflow is:
PyTorch
↓
Export to ONNX
↓
Optimize
↓
Deploy using:
• ONNX Runtime
• TensorRT
• OpenVINO
• Core ML
This keeps training flexible while allowing deployment to multiple hardware targets.
If I had to choose one runtime today
For the broadest compatibility across CPUs, ARM devices, embedded Linux systems, cloud, and edge accelerators, I'd choose ONNX Runtime. It has a mature ecosystem, supports execution providers for many hardware backends, and works well with models originating from multiple frameworks.
If your deployment is primarily Android or microcontrollers, TensorFlow Lite remains an excellent choice due to its mature tooling and optimized mobile ecosystem.
The biggest practical takeaway is that deployment success depends less on the inference runtime alone and more on matching the runtime to the target hardware, validating operator support early, quantizing when appropriate, and profiling on the actual device rather than relying on desktop performance.
