On-Device Edge AI
On-Device Edge AI executes machine learning models (computer vision, object detection, audio processing, and Small Language Models like Gemma 2B or Phi-3) directly on the user’s mobile hardware using LiteRT (TensorFlow Lite), MediaPipe, CoreML, and llama.cpp FFI bindings.
1. On-Device Edge AI vs. Cloud AI APIs
| Feature | Cloud AI APIs (e.g. Gemini API, OpenAI) | On-Device Edge AI (LiteRT, Gemma 2B) |
|---|---|---|
| Privacy | Data transmitted over network to cloud | 100% Privacy (Data never leaves device) |
| Latency | Network latency (200ms - 2,000ms) | Ultra-Low Latency (5ms - 50ms inferencing) |
| Offline Capability | Requires active internet connection | 100% Offline (Zero network dependency) |
| Hardware Costs | Pay-per-token API cloud costs | Zero cloud API costs (Leverages user’s NPU) |
| Resource Limits | Unrestricted cloud GPU clusters | Constrained by mobile RAM, battery, & NPU |
2. Under The Hood: Model Quantization & NPU Acceleration
Running modern Small Language Models (SLMs) on mobile devices requires overcoming severe memory constraints.
1. Model Quantization (INT4 / INT8)
Standard ML models use 32-bit floating point weights (FP32). A 2-billion parameter model in FP32 requires 8GB of RAM—exceeding standard mobile memory allocations.
Quantization compresses model weights into 4-bit (INT4) or 8-bit (INT8) integers:
- RAM Reduction: Reduces Gemma 2B RAM footprint from 8GB to ~1.3GB, fitting within mobile device memory limits.
- Accuracy Trade-off: Minimal loss in output precision (1-3% accuracy reduction) in exchange for 4x-6x memory reduction.
2. Hardware NPU Acceleration
Modern mobile SoCs (Apple A17/M-series Neural Engine, Qualcomm Snapdragon NPU, Google Tensor TPU) contain hardware accelerators optimized for matrix multiplication.
- Delegates: LiteRT and MediaPipe route model graph execution to hardware delegates (
GpuDelegate,NnApiDelegate,CoreMlDelegate).
┌─────────────────────┐ Loads Quantized Model (.tflite / .bin) ┌─────────────────────┐
│ Flutter App (Dart) │ ────────────────────────────────────────────────► │ LiteRT / MediaPipe │
└─────────────────────┘ └──────────┬──────────┘
│
Routes to NPU Delegate
│
▼
┌─────────────────────┐
│ Apple Neural Engine │
│ Snapdragon NPU │
└─────────────────────┘3. Integrating LiteRT / MediaPipe in Flutter
To prevent model inferencing from blocking the Flutter UI thread, offload model execution to a background Isolate or native C++ thread via dart:ffi:
import 'dart:isolate';
import 'package:tflite_flutter/tflite_flutter.dart';
class LocalClassifier {
late Interpreter _interpreter;
Future<void> loadModel() async {
// Load quantized model with GPU delegate acceleration
final options = InterpreterOptions()..addDelegate(GpuDelegate());
_interpreter = await Interpreter.fromAsset('models/mobilenet_v3_int8.tflite', options: options);
}
Future<List<double>> runInferenceInIsolate(List<double> inputBytes) async {
// Offload model execution to background Isolate to maintain 120 FPS UI
return await Isolate.run(() {
var output = List.filled(1000, 0.0).reshape([1, 1000]);
_interpreter.run(inputBytes, output);
return output[0] as List<double>;
});
}
}4. Trade-offs & Production Considerations
- Thermal Throttling: Running continuous LLM generation on mobile hardware causes device heating and OS thermal throttling, reducing CPU/NPU clock speeds by up to 50%. Limit local generation bursts.
- Binary & Asset Footprint: Bundling a 1.5GB quantized Gemma 2B model directly inside the app APK/IPA bloats download sizes. Use background asset downloaders (
path_provider+Dio) to fetch model weights on demand after app installation.