Autopilots need parachutes: Lessons learned from LLM-automated embedded ML pipelines

We set out to build an Autopilot that automatically executes key stages of the embedded machine learning lifecycle, using large language models (LLMs). Rather than demonstrating seamless automation, our study exposes the trade-offs between feasibility and dependability in LLM-driven embedded ML pipelines. We present

an empirical, system-level evaluation based on a fully implemented middleware framework that orchestrates the complete lifecycle, integrates validation and recovery mechanisms, and interacts with multiple LLMs. Using this framework, we execute hundreds of end-to-end runs across a wide spectrum of deployment targets, from

highly constrained microcontroller-class devices to less constrained edge boards. Our results show that the primary challenge is not code generation itself, but managing silent failures, repeated retries, and unstable convergence across heterogeneous targets. Small variations in prompt structure and system constraints lead to drastically different outcomes, ranging from immediate success to costly iterative failures. While success rates improve as device constraints are relaxed, reliable automation consistently requires explicit structure, validation, and fallback mechanisms.We contribute (i) a systematic feasibility and dependability analysis of LLM-automated embedded

ML pipelines, (ii) empirical insights into convergence behavior, cost, and semantic alignment across devices and models, and (iii) a fully released framework with execution traces and artifacts to support reproducibility. Lesson learned: automation is possible, but only when the system assumes failure from the start. Autopilots need

parachutes.