Most companies still ship AI agents the way they shipped scripts. Someone tries eight or ten prompts, decides the answers look reasonable, and it goes live in front of customers. That worked when a failed test produced a stack trace. It does not work for an agent that talks to your customers, reads your systems, and acts on your behalf. July 2026 made the new standard explicit: OpenAI announced Presence on July 22 and describes it as bundling pre-deployment simulation, graders, and runtime guardrails into the deployment product itself, and Google says its Threat Intelligence agentic capabilities reached general availability the same month. Every vendor description in this article is that vendor's positioning, not verified proof, and you should treat it that way in a buying conversation.
Demo testing fails because it measures whether the output reads well rather than whether the agent reached the right outcome. An agent can produce a fluent, confident answer that is wrong on the facts, right on the facts but outside your refund policy, right on policy but calling the wrong tool, or right on everything except that it should have handed the conversation to a person three turns ago. All four look identical in a demo. They look like a good answer.
The right mental model is evaluating an employee, not QA of a feature. You do not grade a new support hire on the grammar of their emails. You ask whether they resolved the problem, followed policy on credits and exceptions, used the right systems, and escalated what needed a supervisor. Those four axes are what modern evaluation tooling checks. Manual prompts also sample only the easy middle of your request distribution, and agents regress on their own when a model version updates or a knowledge source is refreshed.
Testing an agent means scoring whether it reached the correct outcome, followed your policy, used its tools appropriately, and escalated when it should have, across a bank of scenarios you can rerun every time anything changes.
A scenario bank is a maintained set of realistic test conversations with correct outcomes defined in advance, and you build one by mining your own closed tickets, escalations, complaints, and adversarial attempts rather than inventing cases. The raw material is already in your ticketing system, your call recordings, and your chat transcripts. Your edge cases are specific to your business, and no vendor ships them.
A scenario is not just a prompt. It needs the opening message and likely follow-ups, the account state, the correct outcome in plain language, the policy rules that apply, which tools should and should not be called, and whether escalation is the right ending. Write the outcome first, or you will grade the answer against itself.
Two hundred well-chosen scenarios beats two thousand scraped ones: roughly half common requests, a third edge cases, the rest higher-risk situations involving money, personal data, or legal exposure. Build it alongside the knowledge base the agent draws from, because most wrong answers trace back to missing or stale source content rather than to the model.
Graders check four distinct things on every scenario: whether the agent reached the correct outcome, followed your policy, used its tools appropriately, and escalated when it should have. OpenAI describes the graders in Presence as evaluating that set. Score each axis separately: an agent that resolves 90% of scenarios correctly but violates policy on 4% of them is not a 90% agent, and averaging the two hides the problem that generates the complaint. Escalation is the axis teams under-test most, because a smooth wrong answer scores better on the other three than an appropriate handoff does.
Graders are usually a mix: deterministic checks for anything you can assert programmatically (was a refund issued, was the correct API called, did the transcript contain an unapproved commitment), and model-based judges for the softer criteria. Prefer deterministic checks: cheaper, faster, no drift. Validate model judges against human review on a sample, because an unvalidated judge is just another untested agent.
Yes: simulation should run automatically on every change and block the release when results fall below thresholds you set in advance, the same way a failing unit test blocks a merge. A suite that runs when someone remembers to run it produces no assurance. OpenAI says Presence simulations test the agent against common requests, edge cases, and higher-risk scenarios, which is the right shape, but the gate is a process decision rather than a product feature: your deployment pipeline has to enforce it. Set thresholds per axis, with near-zero tolerance on policy and escalation for higher-risk scenarios and a hard stop on any unauthorized tool call, and publish them so the argument happens before the release, not after the incident.
Tests tell you what the agent did on scenarios you thought of; guardrails intervene on the ones you did not. OpenAI describes Presence guardrails as intervening mid-conversation when a live interaction moves outside defined boundaries: a layer that can stop, redirect, or escalate before the agent acts. The consequence is that guardrails belong to whoever owns the systems the agent acts through, not to the team writing prompts, because that is where an action can actually be blocked.
The rest of the July 2026 launches landed in that layer. Alterion announced Draco on July 16 and positions it as a runtime control platform giving security, risk, and compliance leaders enforced governance over agents across clouds, vendors, and endpoints. Codenotary says AgentMon 3 adapts runtime security policy to observed agent behavior. Both descriptions are vendor positioning rather than measured results, so ask for a proof of concept against your own scenarios before you believe either. Make guardrail configuration part of your security program, paired with real observability into what your agents do in production.
Vendor tooling covers the machinery, meaning simulation infrastructure, grader frameworks, and runtime enforcement, while your policies, scenarios, test data, escalation paths, and retained evidence stay yours. The demo will not draw that line, so draw it before the contract.
Portability matters more than it sounds. OpenAI says Presence is in limited rollout to selected enterprise customers, and every platform here is moving fast. A bank that lives only inside one vendor's console hands that vendor your switching costs. Keep the definitions in version control with the rest of your agent configuration.
Three parties own different pieces, a named business owner, risk or compliance, and IT, and the first 90 days run in three stages: define and collect, grade and gate, then guardrails and feedback. The business owner defines correct outcomes and approves policy rules, because only they can say what a right answer is. Risk sets thresholds and defines higher risk. IT builds the harness and maintains runtime controls. The failure mode is IT owning all three: pass criteria written by engineering drift toward what is easy to measure, and nobody outside the team believes the results. Walk an agent governance checklist before the first release, as you would for any serious process automation program.
Track outcome, policy, tool use, and escalation as separate rates plus how fast production failures become new scenarios, and expect programs to fail where a single averaged score replaces reading the individual failures. The vanity metric is containment rate, the share of conversations handled without a human. OpenAI reports it runs its own English-language phone support line on Presence and that 75% of inbound calls resolve without a human. Useful as a reference point, and exactly the number that gets quoted out of context: containment without a policy adherence figure says nothing about whether those resolutions were correct.
The most common failure is a suite that is measured but never read: teams watch the aggregate score and stop investigating individual failures once it looks acceptable. The score is a summary, the failures are the information. Close behind: banks that never grow after launch; grader criteria written by whoever wrote the prompt, producing tests the agent passes by construction; and treating vendor-suggested improvements as approved changes. If you are earlier in this arc, the harder problem is usually getting from a working demo to something you can deploy, and testing is the bridge.
Have a documented agent test program in place before you buy agent tooling, because the platforms now assume you arrive with the scenarios, policies, and thresholds they run against. The July 2026 launches did not invent agent testing, they made it non-optional. The question in your next vendor conversation is not whether you need a test program but whether yours is good enough to use what you are buying. Writing the policies, mining your own history for the scenarios that will break the agent, and deciding who signs off is work only you can do. If you want help building that program, our consulting team works on site across Dallas–Fort Worth and delivers remotely for companies nationwide.
Infonaligy builds, tests, and governs production AI agents for mid-market companies, on site across Dallas–Fort Worth and delivered remotely nationwide.
Infonaligy builds and tests production AI agents: scenario banks drawn from your own tickets, graders tied to your policy, and release gates wired into your pipeline. Let's talk about the agent you are about to put in front of customers.
Infonaligy