What Vercel's Is Agentic Views Actually Reveal
Vercel shipped Is Agentic last month with a deceptively simple feature: audit views segmented by site type. The Commerce view flags payment standards—Apple Pay conformance, Stripe schema compliance, structured product data. The App view flags authentication patterns and SDK conventions. The Docs view flags llms.txt structure and content indexing. Each view runs the same set of tests but routes them by domain before scoring.
The accidental insight: agents and workflows that touch multiple surfaces need separate observability contexts, not a single aggregated score. Vercel doesn't call this an eval problem. It calls it an audit problem. But for teams shipping agents that read documentation, call REST APIs, and execute payments—it is exactly the eval problem we've gotten wrong.
The Monolithic Eval Score Is Lying to You
Every eval harness we've built starts the same way: run a test set, count passes and failures, report a percentage. A 78% pass rate looks acceptable. Then you get paged at 2am because a Stripe webhook retry storm is maxing out your rate limits. You pull the logs. The agent passed 78% of eval traces. It failed 0% of payment flows. The aggregate was lying.
Here's the technical reason: ML systems generalize from data, but eval sets rarely represent the distribution of surfaces your agent actually encounters. A single test set with 20 docs questions, 15 API calls, and 5 payment flows will weight each surface equally in the final score. If your agent gets docs right 95% of the time but payments right 40%, the aggregate hides it behind a passable number. The surface with the lowest tolerance for failure disappears into the noise.
We've found that re-slicing historical eval data by interaction surface reveals gaps no single pass rate ever flagged. An agent that "passes 82% of tests" often splits into something like docs at 91%, APIs at 78%, and transactions at 41%. One of those is a production emergency. An aggregated score will not tell you which.
Interaction Surfaces Have Incompatible Failure Modes
The reason site-type routing matters is structural, not cosmetic. Each surface has a completely different failure taxonomy and tolerance profile.
Documentation and RAG surfaces care about hallucination, citation grounding, and recall@k. A wrong answer is bad, but retrievable via retry. Latency budget is generous—20 seconds is fine. Idempotency is not a concern.
API and tool-call surfaces care about schema conformance, 4xx/5xx handling, idempotency keys, and dead-letter queues. A retry is expensive but safe if you set Idempotency-Key headers correctly. Latency budget is tight—200ms tail latency kills the user experience.
Transactional and commerce surfaces care about PCI boundaries, exactly-once semantics, refund reversibility, and PSD2/SCA flows. A retry is dangerous. A hallucination about a refund amount is a compliance violation. Latency budget is somewhere in between, but the cost of failure is highest. The observability granularity each surface needs is completely different—docs need citation chains, APIs need correlation IDs, transactions need trace IDs that span to your payment processor's logs.
One eval pipeline cannot score all three without either watering down the checks or obscuring which one is broken. You need domain-specific signal routing.
What Domain-Specific Signal Routing Looks Like in Practice
The simplest version starts at the tool-call layer. Tag every trace with an interaction_surface field—either in Vercel AI SDK middleware or OpenAI Responses API metadata. Anthropic and OpenAI both expose tool_use metadata cheaply. A documentation retrieval is one surface. A REST API call is another. A Stripe charge is a third. You emit one tag per tool invocation.
Route traces to surface-specific evaluators. For docs, spin up an LLM-as-judge that grounds on retrieved sources. For APIs, use contract tests—did the agent emit a valid schema? For commerce, replay against Stripe test mode and check charge idempotency. Each evaluator has different golden sets, different scoring rubrics, and different thresholds.
Store per-surface pass rates in Postgres with a rollup view. Never let the aggregate be the deploy gate. Wire per-surface thresholds into CI: block deploys when transactional eval drops below 95% even if the aggregate is 82%. Redis-backed caching of golden traces keeps per-surface replays under a minute.
The teams we've shipped this with treat per-surface eval results the same way they treat production SLOs—if any surface drops below threshold, the deploy is blocked, period.
The Eval Infrastructure Change This Forces
This is not a configuration change. It is a topology change. Move from a single eval CLI that runs one big test suite to a fan-out architecture: one dispatcher that routes incoming traces to N surface-scoped runners.
Golden sets now version independently per surface. Your commerce golden set changes when Stripe API changes, not when your docs get rewritten. Your docs golden set evolves with your content strategy, independent of your payment flows. This decoupling saves months of coordination work.
Per-surface SLOs live on the same dashboard as production metrics—Grafana, Datadog, whatever you use. Eval regressions and production incidents share vocabulary. When your eval shows a commerce drop, it lives next to your prod Stripe error rate. The correlation is immediate.
Cost-wise, surface-scoped evals run more frequently but on smaller sets. We've found this is usually cheaper than nightly monolithic runs. Org-wise, the payments team owns the commerce eval suite, the DX team owns docs. No more central eval team bottleneck deciding priority.
Where to Start This Week
You do not need to rebuild everything at once. We've shipped this in three steps, always starting with the highest-risk surface first.
Step one: Instrument existing traces with a surface tag. Takes an afternoon with AI SDK middleware. If you're using OpenAI Responses API, set a custom field in metadata. Log the tag on every tool invocation.
Step two: Pull your last 30 days of eval results and re-slice by surface. Do not overthink the bucketing—docs, API, transactions, anything else. Calculate per-surface pass rates. You will find at least one surface hiding a 20+ point deficit. Write it down.
Step three: Pick the highest-risk surface (usually transactional) and stand up a dedicated evaluator with its own deploy gate. Get that one working before you build the others. The routing topology is the hard part, not the individual evaluators. Prove the pattern works on one surface, then replicate.
If you want a second pair of eyes on the routing layer or the golden set strategy, let's talk.
Start with your next eval run: split by interaction surface and compare pass rates side-by-side. If the deltas are bigger than 15 points, your aggregated score has been lying to you. That gap is where production failures hide.