General LLM evals don't predict agent success on your codebase
HumanEval and MMLU measure single-turn reasoning over isolated problems. Your agent doesn't work that way. It chains 50 tool calls across a 50k-line codebase, reading and rewriting files it half-remembers, recovering from compile failures, deciding whether to retry or escalate. SWE-Bench Verified gets closer—it includes real Python repos and multi-step reasoning—but it's still bounded to small PRs and a single language family.
ScarfBench targets enterprise Java migrations: Spring 4 to Spring 6, Java 8 to 17, Jakarta EE namespace shifts. The kind of work that lives in production banking systems, insurance platforms, telco backends. A model that scores 90% on HumanEval can still hallucinate imports, leave dead code across 200 files, and restart from scratch after hitting a compilation error. The gap between eval pass@1 and production reliability is where agents die.
Why does this matter? Because your production agent—whether it's orchestrating internal APIs, migrating a codebase, or triaging support tickets—lives in the same world. It has to maintain state across 20+ tool calls without re-hallucinating facts it already discovered. That's not what coding benchmarks measure.
What ScarfBench actually measures: tool reliability, not intelligence
ScarfBench isolates three failure axes that map directly to production risk. The first is tool-use reliability: does the agent call read_file, apply_patch, run_tests in the right order without malformed arguments? A model can be smart but careless—passing a line number when a byte offset is required, or forgetting to include the full context in a patch. These small errors cascade across a 45-minute migration run.
The second is context window thrashing. A single Java file can exceed 8k tokens. When the agent has to chunk it, retrieve multiple pieces, and rebuild context repeatedly, the latency math breaks: 40 tool calls × 800ms retrieval per call = 32 seconds before the first token of reasoning arrives. By the time the agent thinks again, it's forgotten what it discovered two files ago. Symbol-level indexing—using tree-sitter, LSP, or Sourcegraph SCIP to pass only the relevant callers and callees—is what separates shipping from demos.
The third is partial-refactor recovery. The agent starts migrating a package, hits a compile error mid-way, and has to decide: retry the current file, checkpoint and move on, or escalate to a human. ScarfBench includes runs where migrations fail intentionally midway. Can the agent resume without re-hallucinating already-migrated code? Can it detect that javax.persistence is now jakarta.persistence and not reapply the same fix twice? Dead code left behind—unused methods that weren't cleaned up—is the tell that state was lost mid-run.
Context thrashing is the failure mode nobody benchmarks
Marketing talks about 200k-token context windows. In practice, an agent re-reads the same file 12 times per task. The window size doesn't matter if retrieval is naive and expensive.
We've shipped enough retrieval-scoped prompts to know the pattern: pass only the symbols the agent needs to make the next decision. If the agent is rewriting a Spring @Configuration class, include the public methods and their callsites—not the entire package. At scale, orchestration is solved. The real bottleneck is token routing—deciding what to fetch, when to cache it, and when to let an older version of context go stale.
Anthropic's prompt caching cuts repeated system-context cost by roughly 90% on re-reads. Use it. Log the cache hit rate alongside token spend; if it's below 60%, your retrieval strategy is broken. Latency budgets matter more than model speed here: if you burn 800ms retrieving symbols, a 1ms faster model doesn't save you.
Recovery from partial state is what separates agents from demos
Every tool call needs an idempotency key. Retries without idempotency are how you end up with a file patched three times. Checkpoint after each file—via a git commit, a Postgres row, or a message in a durable queue. Never trust in-memory state across a 45-minute run. A customer deploys your agent on Friday afternoon and leaves. By Monday, someone notices the run crashed at 4am Sunday, and the agent had already migrated 80% of the files. You need to know exactly where it stopped.
The ground truth is the compile-and-test loop. mvn verify exit code tells you if a migration worked. The agent's self-assessment—"I believe this file is now Jakarta EE compatible"—is worth less than the build log. When to build an agent vs. a workflow comes down to this: can your task be verified automatically and fast enough to retry? Migrations can. Support tickets can't. Design accordingly.
Use a dead-letter queue for files the agent couldn't migrate cleanly. Route them to a human review lane. Retry tool errors with exponential backoff, but cap at 3 attempts. Infinite loops burn tokens faster than they fix bugs. If the agent has retried the same file three times, it's time to give up and escalate.
Why migration failures predict production agent failures
The same failure modes that break ScarfBench runs break customer-facing agents. A support agent that loses context across a 20-turn conversation is the same bug as one that forgets which files it migrated. An orchestration agent calling 15 internal APIs fails on the same tool-schema drift ScarfBench surfaces. If your eval doesn't include partial-failure recovery, you're shipping demos.
The teams winning on agents run a domain-specific ScarfBench-style harness in CI. Not vibes-based prompt tweaks. 20 real tasks from your codebase, scored on completion + compile + test pass. Run it on every prompt change and model swap. Track tool-call success rate, context re-reads per file, recovery rate after failure, wall-clock time to green build. If one metric regresses, you know before deploying.
Cost signal matters too: a flaky agent that retries 4x on average is a 4x infra bill. That's not a UX problem, that's a budget problem. Your CFO will notice.
How we'd instrument your agent before it hits production
Build a domain harness with 20 real tasks from your codebase. For a Java migration, that's 20 packages of varying size and complexity. For an orchestration agent, it's 20 workflows that exercise all the APIs it will call. Score on completion, outcome (compile pass, test pass, no dead code), and wall-clock time to result.
Log every tool call with its arguments, latency, and outcome to a structured store—Postgres or ClickHouse work fine. Wire up four observability metrics: tool-call success rate (did the tool return 200?), context re-reads per file (how many times did the agent fetch the same symbol?), recovery rate after failure (did the agent resume cleanly?), and wall-clock to green build (total elapsed time, not token time).
Run the harness on every prompt change and model swap. Not just at launch. A new system prompt might cut latency by 30% and break recovery entirely. You won't know if you're not measuring.
Before you upgrade to the next frontier model, run your own ScarfBench—20 real tasks from your repo, scored on compile-and-test, not vibes. If you want a hand building that harness, that's exactly the kind of work we ship. Here's what agent observability actually looks like in production. Talk to us at /contact.