Updated 2026-07-21
They fail silently more often than anyone admits, and the failure looks exactly like success. An agent that cannot reach its model still produces output, still writes to your database, and still reports a number.
This is not a hypothetical. On our own infrastructure, an API credential was stored with quote characters around it, so every request went out malformed and was rejected. The agents kept running. A benchmark kept executing, kept scoring zero because no question was ever answered, and kept publishing that zero as a result on its normal schedule.
Seven weeks of output, identical in shape and cadence to a working system. The only thing that actually changed was the cloud bill, which fell by 99% because nothing was being computed. Nobody looks at a billing page for a correctness signal.
The bug underneath was that the scoring code could not distinguish two very different statements: the system answered and got everything wrong, and the system never answered. Both arrived as zero, both were averaged in at full weight, and both were published as proof.
That is not one careless line. It is what happens whenever a pipeline treats an error as a value. A missing score defaults to zero, a failed step returns an empty list, a timeout yields no results, and each of those flows downstream as though it were data.
The literature has started naming these problems precisely. Self-explanations produced by language models are plausible whether or not they are faithful, so asking a model why it did something is another output rather than a window into the mechanism. Evaluations that only report where a model fails leave why implicit, so nobody learns what to fix. And a memory benchmark can appear to improve when only the prompt formatting changed, which quietly invalidates the comparison.
The pattern is the same each time: a measurement that looks informative while measuring something other than what it claims.
Ask whether a failed step can be told apart from a legitimate zero, both in the code and in the output. Ask what a total outage would look like from the outside, and whether anything would alarm. Watch spend as a liveness signal, because a fleet that stops consuming tokens has stopped working regardless of what it publishes.
And be suspicious of a result that never moves. A benchmark reporting the same score for weeks is more likely broken than stable.
6 papers, each explained in plain language.
From Plausible to Actionable: A Position on LLM Self-Explanations
arXiv (cs.CL)CRAFT: Clustering Rubrics to Diagnose Weak LLM Capabilities and Generate Targeted Fine-Tuning Data
arXiv (cs.AI)Presentation, Not Mechanism: A Render Confound in Deprecation-Aware Memory Evaluation
arXiv (cs.LG)When Do Multi-Agent Systems Help? An Information Bottleneck Perspective
arXiv (cs.AI)The Honest Quorum Problem: Epistemic Byzantine Fault Tolerance for Agentic Infrastructure
arXiv (cs.LG)How Do VLMs Fail? Vision-Operation Misalignment in Compositional VQA
arXiv (cs.CV)