Edge AI
Edge AI is the deployment of artificial intelligence — primarily neural network inference — directly on hardware devices (“at the edge”) rather than in centralized cloud data centers. Instead of streaming sensor data to the cloud for processing, edge AI runs the trained model locally on the device’s CPU, GPU, NPU (Neural Processing Unit), or FPGA accelerator. The result is single-digit-millisecond latency, raw sensor data that can stay on the device, and the ability to operate without an internet connection. Demand for edge AI hardware is driven by smartphones, automotive ADAS, industrial vision, and consumer IoT.
Key Facts
| Aspect | Detail |
|---|---|
| Definition | AI inference executed on local hardware, not transmitted to cloud |
| Typical latency | 1–50 ms (vs. 100–500 ms for cloud round-trip) |
| Power envelope | 1 mW (TinyML on Cortex-M) to 60 W (Jetson AGX Orin) |
| Common silicon | NPU (Google Edge TPU, Hailo-8/10H), FPGA (AMD Versal AI Edge), GPU (Jetson) |
| Software stacks | LiteRT for Microcontrollers (formerly TensorFlow Lite Micro), ONNX Runtime, AMD Vitis AI, NVIDIA TensorRT |
| Typical model sizes | 100 KB (TinyML) to 100 MB (compressed image models) |
| Privacy | Raw sensor data can stay on the device — major advantage for cameras, microphones |
| Compliance | EU AI Act applies; edge processing simplifies GDPR data minimization |
Why Edge AI?
Edge AI is replacing cloud-only AI architectures in many product categories for five core reasons:
- Latency — Cloud round-trip introduces 100–500 ms; safety-critical decisions (autonomous driving, factory robotics) need sub-10 ms responses
- Privacy — Cameras, microphones, biometric sensors: raw data can stay on the device, which supports data minimization under GDPR and sectoral privacy rules but does not by itself establish compliance
- Bandwidth — Streaming HD video, multi-sensor industrial telemetry, or LiDAR clouds to the cloud is economically and technically impractical at scale
- Reliability — Edge AI keeps working when network is degraded, intermittent, or absent (mining, agriculture, remote infrastructure, vehicles in tunnels)
- Energy — Surprisingly, doing inference locally often consumes less total energy than sending raw data to cloud, processing it, and returning the result
Edge AI Hardware Categories
The right edge AI silicon depends on your workload, power budget, and latency requirement:
| Tier | Silicon class | Power | Throughput | Typical use |
|---|---|---|---|---|
| TinyML | Cortex-M4/M7, ESP32-S3 | 1–50 mW | < 1 GOPS | Keyword spotting, vibration anomaly, simple gesture |
| Low-power MPU | Cortex-A53/A72 + NPU (e.g. NXP i.MX 8M Plus) | 1–5 W | 1–10 TOPS | Person detection, simple object classification |
| Edge NPU/SoC | Hailo-8/10H, Coral Edge TPU, Ambarella | 2–10 W | 4–50 TOPS | Real-time multi-camera object detection, semantic segmentation |
| High-end edge GPU | NVIDIA Jetson Orin Nano / AGX Orin | 7–60 W | 34–275 TOPS | LiDAR fusion, autonomous robotics, multi-modal LLM at edge |
| FPGA AI accelerator | AMD Versal AI Edge, Lattice CrossLink-NX | 3–30 W | 1–100 TOPS | Custom signal-processing+AI pipelines, deterministic latency, defense radar |
Implementation Approaches
There is no single “correct” edge AI architecture — the best fit depends on the workload:
- TinyML on microcontrollers — Quantized 8-bit neural networks running on Cortex-M microcontrollers (STM32, nRF) and on ESP32 (Xtensa or RISC-V cores). Frameworks: LiteRT for Microcontrollers (formerly TensorFlow Lite Micro), Edge Impulse. Typical models: 50 KB – 1 MB. Ideal for battery-powered IoT sensors.
- NPU accelerators on SoC — Dedicated AI hardware blocks (Apple Neural Engine, Qualcomm Hexagon, Google Edge TPU) running quantized models at 1–50 TOPS. Used in smartphones and edge AI products.
- FPGA-based AI — Deep Learning Processor Unit (DPU) overlays on AMD/Xilinx FPGAs run 8-bit quantized convolutional networks; network-specific dataflow accelerators (e.g. FINN) allow custom topologies and mixed bit widths. Inovasense FPGA design services can implement this approach for industrial systems.
- Edge GPU — NVIDIA Jetson family for high-throughput multi-stream inference; widely used in robotics and autonomous systems.
- Custom ASIC AI accelerator — Highest performance and lowest power for fixed workloads at volumes >100K units; examples include the Google Edge TPU and the Tesla FSD chip.
EU AI Act and Edge AI
Under the EU AI Act (Regulation 2024/1689), AI systems are classified by risk level — and the requirements apply regardless of whether inference happens in cloud or at the edge. Edge processing can make some obligations easier to meet, although local execution alone does not establish compliance:
- Data minimization (GDPR) — Personal data can be processed locally and need not be transmitted or retained
- Transparency — Easier to document model behavior on a known hardware platform
- Auditability — Deterministic inference on locked-down hardware vs. drifting cloud models
- High-risk AI systems (Annex III) — Biometric identification, critical infrastructure, employment decisions — face strict requirements where edge deployment with on-device logging is often the cleanest compliance path
For products in the high-risk category, Inovasense provides EU compliance services covering both AI Act and CRA conformity assessment.
Common Use Cases
- Industrial vision — Defect detection on production lines, robotic guidance, predictive maintenance from vibration signatures
- Smart cameras — Person/vehicle counting, intrusion detection, license plate recognition (all without sending video to cloud)
- Predictive maintenance — Acoustic and vibration analysis on motors, pumps, bearings — detecting faults before failure
- Voice interfaces — Always-on keyword spotting (Alexa, Siri wake words) running on TinyML at about 1 mW to a few mW
- Healthcare wearables — ECG arrhythmia detection, fall detection, gait analysis — running locally to protect patient data
- Automotive ADAS — Object detection, lane keeping, driver monitoring; safety-critical inference at sub-10 ms
- Defense surveillance — Persistent ISR (intelligence, surveillance, reconnaissance) with offline target classification
Related Terms
- FPGA — Reconfigurable silicon for custom AI accelerators (DPU overlays)
- SoC — Modern SoCs integrate dedicated NPU blocks for edge AI
- TinyML — Subset of edge AI running on microcontroller-class hardware (1 mW – 100 mW)
- EU AI Act — Regulation governing AI systems sold in the EU
- CRA — Cyber Resilience Act, which also applies to AI-enabled products
Inovasense Edge AI Capabilities
Inovasense designs production-grade Edge AI and sensing solutions across the full hardware spectrum — TinyML on Cortex-M (battery-powered industrial sensors), NPU integration on Linux-class SoCs (smart cameras, gateways), and FPGA-accelerated custom DPU pipelines (for example radar signal processing, imaging, high-bandwidth multi-sensor fusion). Our work covers model quantization, hardware-aware training, custom RTL accelerators, and full system integration. For products entering the EU market, we manage EU AI Act conformity assessment alongside CE marking and CRA compliance.
Official References
- Regulation (EU) 2024/1689 (EU AI Act) — EUR-Lex, requirements for AI systems including edge-deployed AI
- Regulation (EU) 2024/2847 (CRA) — Cyber Resilience Act, applies to AI-enabled connected products
- GDPR (Regulation 2016/679) — Data protection regulation; edge AI supports data minimization but does not by itself establish compliance