Everyone has a demo where the LLM agent works. Few people have an agent that keeps working when the MQTT broker times out mid-tool-call, when the local model's real context window is half what the registry claims, and when nobody is watching a dashboard in a server room because the "server" is a box in a factory cabinet.
NeoMind is an open-source, Rust-powered edge AI platform for IoT automation — single binary, embedded MQTT broker, rule engine, and an AI agent runtime that drives real devices. We just shipped v0.9.20, and the release's spine is agent execution reliability. This post is the engineering story: the failure modes we found in production-shaped testing, and the fixes that now sit in the loop.
Everything below reflects the repository at commit 8a75c57ee09b.
Failure mode 1: the dedup brake that punished retries
The agent loop runs up to 30 rounds of tool-calling. To stop infinite loops, we keep a dedup set of execution signatures — if the model proposes a call it already made, we end the loop.
The bug: signatures were inserted before execution. So when a tool call failed transiently — an MQTT timeout, an extension hiccup — and the model correctly retried, the retry looked like a "duplicate". The loop ended via AllDuplicate, with the original error still in hand and the task unfinished. The anti-loop mechanism was punishing the model for doing the right thing.
The fix, in the 0.9.20 release notes:
The cross-round dedup set now records only successful executions. Failed calls can retry — and a failed signature that keeps failing (budget: 3 consecutive failures) is blacklisted so the loop brakes instead of burning all 30 rounds.
We also deleted our StuckDetector entirely. It implemented five OpenHands-style stuck patterns — but they were mathematically unreachable behind the dedup logic. The docs described a brake that never fired; the dedup IS the brake. Sometimes reliability work is subtraction.
Failure mode 2: "context overflow" that was really a config mismatch
On local backends, context overflow is marked permanent per is_permanent(). But we found hosts where the running model's window was smaller than the registry default — e.g. a backend started with -c 16384 while the model entry claimed 128K. Result: every round overflowed, and small-model execution "inevitably failed".
The fix is a self-heal: on overflow, one hard-compaction retry with a halved effective window. That single retry converts "can never run" into "completes". Alongside it, custom backends now carry an explicit max_context on create/update, so a 16K backend never receives 128K-budgeted prompts, and the catalog caps imported models at 128K because a GGUF header claiming 1M would OOM the KV allocation.
Failure mode 3: the agent didn't know it was out of runway
Within the last 3 rounds, the loop now injects a [System] note telling the model how much runway is left. It sounds trivial; the effect is not. Without it, the model would start a multi-step chain at round 28 that the cap cuts off mid-flight — a half-finished task with no summary. With the countdown, it wraps up.
Related shutdown fixes: cancellation now exits the whole round loop (previously it only broke the tool-concurrency batch), and Phase-2 summarization is skipped during shutdown — no LLM calls while the process is going down.
Failure mode 4: the metrics lied
Our agent journal recorded a success_rate for every execution. It was pinned at 1.0, always. Reason: tool results returning Ok { success: false } were counted as successes — the transport succeeded, so the outcome flag was ignored.
Now the journal reflects real outcomes. This matters beyond dashboards: the journal is the learning signal the agent system trains and steers on. Honest metrics are a prerequisite for any self-improvement story.
Similarly, invoke used to await execution inline with a 60s timeout — which dropped the future. Long runs died mid-flight with no execution record and no journal entry: a ghost execution the agent could never learn from. The execution now runs in its own task; past the wait window the caller gets still_executing with poll pointers while the run completes and writes its record.
Failure mode 5: one bad schedule killed the whole scheduler
Found by our own design review: the frontend's "on-demand" option encodes manual-only as interval_seconds: 0. That value made the agent due on the very next tick, which hit a division by zero in update_next_execution — a panic inside the unguarded scheduler loop that stopped every agent on the platform. Process alive, no restart, one panic line in the logs.
Three-layer fix: 0 is now a first-class "manual-only" value (schedules to a never-due time), reschedule math guards against 0 for legacy rows, and the tick's reservation phase runs in its own spawned task so any future panic there is logged and skipped instead of unwinding the loop. We also promoted one-shot agents to a real type — ScheduleType::Manual with a Completed ready-state between runs — so "on-demand" no longer needs a numeric hack.
Per-model sampling: stop sharing one global temperature
The scheduled-agent loop hardcoded temperature 0.7 for everything. But 2026's small-model zoo does not share one optimal sampling point. 0.9.20 wires /api/settings/agent into both chat and scheduled paths, and each built-in model now carries its own best-known point, applied at llama-server startup and on the request side:
- Qwen 3.5 (non-thinking): 0.7 / 0.8 / 20 — official values; the 4B tier (~3.4 GB quant, ~6 GB RAM, Apache 2.0) is currently one of the strongest laptop-class agent models
- Gemma 4: 1.0 / 0.95 / 64 — Google's model card recommends temperature 1.0 for all modes; we were running 0.6
- Ling-3.0-tiny: 1.0 / 0.95 / 20 — official
- LFM: 0.6 / 0.85 / 20 — our measured best, which beat the official card values in a 154-case A/B
The catalog schema carries temperature/top_p/top_k; absent means legacy default. Registry tests lock all four values so a future "let's just tweak this" doesn't silently regress them.
The payoff: which small model actually drives agents?
We validate agent capability on a 30-case agent suite (tool selection, multi-step device control, recovery from failed calls). The 0.9.20 headline:
Ling-3.0-tiny Q4_K_M: 77% — ties Qwen 3.5 4B, while generating ~45% faster (~110–116 tok/s on M4-class hardware).
Ling-3.0-tiny ships through the open model catalog (4.8 GB Q4_K_M, 128K ctx, min 6 GB RAM), and our bundled runtime already carries the architecture support merged upstream, so the download runs out of the box. Tying the quality of a proven 4B agent model at roughly 1.5x the speed is exactly the trade an edge deployment wants.
What we'd tell anyone building agent loops
- Dedup on outcomes, not intentions. Recording a signature before knowing if it succeeded turns transient errors into permanent dead-ends.
- Make failure modes diagnosable. "Sorry, the model could not produce a response" now carries the actual reason instead of a bare "Please retry". Your future self debugging at 2am is the customer.
- Budget the loop and tell the model about it. A countdown in the last rounds changes behavior more than any prompt tweak.
- Measure what actually happened. If your success rate is 1.0, your metric is broken, not your agent.
- Guard the scheduler from its own inputs. One malformed row should degrade one agent, not the platform.
If you're building or evaluating agent systems that must run where there is no cloud fallback, come look at the code or file an issue: github.com/camthink-ai/NeoMind. The project wiki covers the platform itself.
Facts in this article reflect the NeoMind repository at commit 8a75c57ee09b (release v0.9.20, 2026-08-26) and its official release notes; third-party model specs (Qwen 3.5 family, Gemma 4 sampling guidance) are from their respective 2026 publications.