The HuggingFace Finding: Agents Report Success They Can't Verify
HuggingFace researchers observed something uncomfortable: agents routinely declare tasks complete when the external system never changed state at all. Not close-but-wrong. Actually, completely untouched. The agent returns a confidence signal saying "done" while Stripe never charged, the CRM record never updated, and the Postgres insert never committed.
This isn't a model problem. It's not a prompt-engineering failure or an RLHF tuning issue. The agent has no ground-truth channel. It calls a tool, the tool returns an HTTP 200 (or a synthetic success signal), and the agent reasons: "the operation succeeded because I got a success code." That reasoning is sound. The architecture is broken.
The 2am failure mode is worse: no exception fires. No error log. Nothing screaming that work needs to be retried. The agent moves on. The human operator discovers the gap days later by spot-checking a sample of "completed" actions against the system-of-record and finding 15% silently failed.
Tool Call Returned 200. That's Not the Same as Done.
HTTP 200 is a lie. A wrapper endpoint can return 200 while a downstream job queue has backpressure and hasn't started processing. A read-replica is lagged 500ms behind the write, so your immediate read-for-verification hits the old state. An async SQS job was accepted (202 Acknowledged) but the Lambda behind it failed six seconds into execution.
Stripe returns 200 immediately. The charge is pending. The webhook hasn't fired yet. An idempotency key prevents duplicate charges on retry—but the agent misreads the "Duplicate Request Detected" response as "charge successful, moving on."
Salesforce API returns 200 when the record was queued for indexing. A search won't find it for 15 seconds. Postgres returns 200 but a before-trigger is still running in the same transaction, and the row you just tried to read hasn't acquired its final state yet.
The pattern is consistent: the tool has no way to communicate "I accept your request but I don't know if it actually worked." And the agent has no way to ask "is this actually done?" because it's been trained to trust the return code.
External State Assertion Belongs Inside the Agent Loop
The fix is architectural. Every state-changing tool call must be paired with a verification read before the agent is allowed to declare success and move forward. Not after the fact. Not as observability. Inside the loop.
After the agent calls "create_stripe_charge", it must immediately call "verify_stripe_charge_posted" which hits the Stripe API directly, retrieves the charge object, checks the status field, and returns { actual_state: "succeeded", expected_state: "succeeded", verified_at: "2024-01-15T09:23:14Z" }. If they don't match, the agent doesn't proceed.
Structure the verification as its own tool the model is forced to call. Use structured outputs (OpenAI's JSON mode, Anthropic's tool-use specification) to make verification non-optional. In n8n or Temporal workflows, model verification as a required edge in the state machine, not an optional step. The agent doesn't get to claim victory until it receives the verification signal.
Yes, this adds latency. 200–800ms per verification, depending on the external system's API response time. That's negligible compared to the cost of silent failure: a charge that never posted, a lead never assigned, an inventory adjustment never committed.
Retries and DLQs Have to Key Off External State, Not Agent Confidence
The trigger for a retry is not an exception the agent noticed. It's a state mismatch the verification step detected. When a verification read shows expected_state ≠ actual_state, you retry the entire write-then-verify sequence with exponential backoff.
Dead-letter queue entries need to include the diff: what did you expect, what did you observe? This is debugging gold. It tells you whether the tool failed silently (the write never happened), the verification was too fast (the state hasn't propagated yet), or the external system rejected the operation without saying so.
Idempotency keys scoped per action_id ensure retries don't double-charge or create duplicate records. Redis-backed in-flight tracking prevents the agent from attempting the same action twice in parallel while the first verification is still pending.
After N verification failures (we've found 3–5 is sane), stop retrying. Hand off to a human. A circuit breaker that escalates instead of looping is worth its weight in production credibility.
Observability: Log the Assertion, Not Just the Call
You need three distinct log lines per action: tool_called, state_asserted, state_matched. Each with a correlation ID that spans the entire call-verify-confirm chain. Most teams log only the tool invocation and miss the verification step entirely.
Instrument the actual comparison. When state_asserted doesn't equal state_matched, log the diff. If the agent expected { balance: $500 } and the system shows { balance: $401 }, you need that data right there in the trace.
Measure the assertion success rate as a key metric. The percentage of agent "task_complete" claims that pass verification. This number should be >99.5% for production-grade work. Most teams discover it's 82–87%. That's the number that tells you to call us, or to redesign the tool contracts entirely.
Separate trace spans for assertion latency versus tool latency. The agent's 200ms call to Stripe is independent of the 150ms read to verify the charge. You need both in the timeline. Alert on assertion-mismatch rate, not tool-error rate. Tool errors are loud. Assertion mismatches are silent.
When This Matters — and When You're Overbuilding
Mandatory: anything that writes to a system-of-record. Payments, CRM records, inventory adjustments, user-visible state. If a false "done" costs money or requires human cleanup, assert.
Optional: read-only research agents, draft generation, internal summarization, analysis. If the agent hallucinates a summary, the cost is a human re-reading it. Not ideal, but the economics are different.
The two-question test: does a false "done" cost money? Does it require human intervention to fix? If yes to either, assert. If no to both, you can skip it.
The cost math is usually trivial. One extra read per write-action, ~$0.001 per call, <1s latency. Negligible against the alternative. But there's a point where you're stacking verification on verification and the agent is spending more time verifying than working. When you're 4–5 verifications deep, escalate to human-in-the-loop instead of adding another layer of checks.
Next Step: Audit for the Mismatch Rate
Audit one production agent this week by diffing its "task_complete" events against the system-of-record for the last 7 days. Count how many completed actions never actually changed state. The mismatch rate will tell you whether you're shipping an elaborate hallucination machine or something closer to real.
If that number makes you uncomfortable, we've built systems to fix exactly this. Get in touch.