How to Test an AI Shopping Agent Before It Pays: Evaluation Harnesses, Repeat-Run Consistency and Decline-to-Buy Behaviour
An AI shopping agent that gets the right answer once has proved very little. The question a retailer, platform or payments partner needs answered is whether it gets the right answer every time, and whether it refuses when it should. In November, Visa put a number on that gap: in its own benchmark, 22 AI models failed roughly 43% of realistic shopping test runs, and only 23% of US consumers said they trust AI to pay for them. Consistency, not raw capability, was the stated failure point. That turns agent testing from a nice-to-have into the main piece of evidence between a promising demo and a live checkout. This article sets out what to test, how to build the harness that tests it, and what a go-live evidence pack should contain.
Why "it worked in the demo" is not evidence
Most agent projects are judged on a handful of demonstrations. Someone asks the agent to buy a specific item, it finds it, adds it to the basket and completes the purchase. Everyone is impressed.
The problem is that a demonstration is one sample from a system that does not behave identically each time. Large language models are probabilistic. The same request can produce a different sequence of tool calls, a different interpretation of an ambiguous instruction or a different choice between two near-identical products. A single successful run tells you the agent can succeed. It does not tell you how often it fails, or how badly.
Visa's benchmark matters for this reason. The design and pass criteria are Visa's own, and the payment network has an obvious interest in agent commerce succeeding, so the numbers should be read as Visa's findings rather than an independent standard. But the shape of the finding is consistent with what builders see: agents that are capable but inconsistent. The trust figure points the same way from the customer side.
The market has moved from liability to control
Through September and October, the agentic commerce debate was mostly about who authorises a purchase and who is liable when it goes wrong. By November the conversation had shifted to measurable reliability and controllable delegation. Bank of America told customers not to give banking credentials to AI assistants, and United Airlines barred agents from completing bookings, according to TechCrunch and PYMNTS reporting. Deutsche Bank and Ant Digital set out that agents need verifiable identity bound to an accountable owner. Fintechs have started shipping agent control features.
For anyone building on our Agentic Commerce & Payments agenda, the implication is direct. The organisations that hold the keys, banks, card schemes, merchants' own risk teams, will ask for evidence that the agent behaves. A test method is how you produce that evidence.
What to test: five behaviours that matter before an agent pays
A useful test plan focuses on behaviours, not features. Five matter most for an agent that can spend money.
1. Task success on realistic requests
Start with the obvious: given a realistic request, does the agent buy the right thing? "Realistic" is the important word. Test sets built from clean, unambiguous instructions overstate performance. Real customers say "the usual coffee pods", "something like last time but cheaper" or "a gift for my sister under £40".
Build a test set from your own anonymised request patterns where you have them, and include the messy ones. Define success precisely for each case: right product, right variant, right quantity, right price band, right delivery option.
2. Repeat-run consistency
Run each test case many times, not once. An agent that succeeds eight times in ten on a given request has a 20% failure rate on that request, whatever the demo showed. Consistency testing reveals which request types are stable and which are fragile.
Record not just pass or fail but the path taken: which tools were called, in what order, and where runs diverged. Divergence points are where design fixes usually sit, such as a missing clarifying question, an ambiguous product catalogue or a tool that returns inconsistent data.
3. Decline-to-buy behaviour
The most important behaviour for a payments context is refusal. An agent must decline when the request is ambiguous, when the price exceeds a limit, when the item is out of stock and the substitute is materially different, when the merchant looks untrustworthy, or when the instruction conflicts with a rule the customer set earlier.
Test refusals as deliberately as successes. Build cases where the correct outcome is "do not buy, ask the customer" and measure how often the agent gets that right. An agent that never declines is not reliable; it is reckless.
4. Limits and escalation
Agents should operate within explicit limits: a maximum spend per transaction, a maximum per period, an approved merchant list, categories that always require confirmation. Test that the agent respects each limit, including at the edges, and that it escalates to a person rather than looking for a way round.
5. Resistance to redirection
Shopping agents read untrusted content: product pages, reviews, merchant descriptions. Any of that text can contain instructions designed to redirect the agent. Test with adversarial content in the places the agent reads, and confirm it does not change its goal, its payment details or its limits.
How to build an evaluation harness
An evaluation harness is the software that runs these tests repeatedly, records the results and makes them comparable over time. It does not need to be elaborate, but it does need to exist before launch.
A versioned test set. Store test cases with their expected outcomes, including the cases where the expected outcome is a refusal. Version them, so results from different weeks are compared against the same set.
A sandboxed environment. Run the agent against test merchants, test catalogues and test payment rails, never live ones. The environment should let you change catalogue data, prices and stock levels to create edge cases.
Repeat execution. Run each case enough times to see its failure rate, not just whether it can pass. Track the number of runs alongside the results, so nobody mistakes a small sample for a stable rate.
Path logging. Capture every tool call, every intermediate decision and the final outcome. This is what lets an engineer explain a failure, and it doubles as the decision log a risk team will later ask for.
Scoring rules agreed in advance. Decide before testing what counts as a pass, a soft failure (wrong variant, right product) and a hard failure (wrong product, over limit, failed to refuse). Agreeing this upfront stops the definition of success drifting to fit the results.
Regression on every change. Re-run the full set whenever the model, the prompt, a tool or the catalogue structure changes. Agent behaviour can shift with any of them.
What "good enough" looks like
There is no published industry pass mark for agent purchasing, and anyone quoting one should be asked where it came from. The threshold is a business decision that depends on what the agent buys, for whom and with what safeguards.
A practical approach is to set thresholds per behaviour rather than one overall score. A team might accept a modest failure rate on a low-value, easily reversed task where the customer confirms before payment, while requiring near-perfect refusal behaviour on anything above a spend limit. The point is to write the thresholds down before the results arrive, and to have someone accountable sign them off.
The go-live evidence pack
When a board, a bank partner or a payments provider asks whether the agent is ready, the answer should be a short pack, not a demo.
- Scope. What the agent may buy, from whom, up to what limits.
- Test set summary. How many cases, how they were built, how many refusal cases.
- Results by behaviour. Task success, consistency, refusals, limits, redirection resistance, each against its agreed threshold.
- Known weaknesses. Request types where the agent is fragile, and the mitigation for each, such as forcing a confirmation step.
- Accountable owner. The person who signed off the thresholds and can pause the agent.
- Monitoring plan. How live behaviour will be compared with test results after launch.
This pack also helps with the disputes question. If a customer later disputes an agent-made purchase, the same records show what the agent was asked to do and how it was tested.
Where testing fits in a UK regulatory picture
UK financial regulators have not issued agent-specific testing rules. The FCA's "AI in financial services: good practice and areas for improvement", published in October 2026, says existing rules apply and frames AI as support, automate or abstain, with live monitoring as a concrete expectation. A test harness with explicit refusal behaviour is a practical way to show that an agent abstains when it should. Innovify is a technology partner and not FCA-authorised; firms should read the FCA's material directly and take advice on their own position.
Where Innovify fits
Innovify helps mid to large businesses take AI and fintech products from pilot to production, and agentic checkout is a clear example of where that step fails without evidence. Through Innovify AI Labs, enterprise teams can run several proofs of concept in parallel, test with real users and finish with a business case ready for funding. For an agent that pays, that means building the test set, the sandboxed harness and the refusal cases inside a time-boxed proof of concept, so the decision to launch rests on measured behaviour rather than a demo. We design compliance in from day one, so the decision log and limits are part of the build, not an afterthought.
FAQ
How do you test an AI shopping agent before launch?
Run it against a versioned set of realistic requests in a sandboxed environment, many times per request, and score each run against outcomes agreed in advance. Include cases where the correct outcome is to decline and ask the customer. Log every tool call so failures can be explained.
What did Visa's AI agent benchmark find?
According to Visa's press release of 5 November 2026, 22 AI models failed roughly 43% of realistic shopping test runs, and only 23% of US consumers trust AI to pay for them. The benchmark was designed and run by Visa, so its results are Visa's own findings rather than an independent standard.
What is decline-to-buy behaviour?
It is an agent's ability to refuse a purchase when it should, for example when a request is ambiguous, a price exceeds a limit or a substitute is materially different. It is the most important behaviour to test for an agent that can spend money, because an agent that never refuses cannot be trusted with payment.
What is an evaluation harness for AI agents?
It is software that runs an agent repeatedly against a fixed test set, records each step and outcome, and makes results comparable over time. It should include sandboxed merchants and payment rails, agreed scoring rules and regression runs whenever the model, prompt or tools change.
Is there a standard pass rate for AI shopping agents?
No published industry standard exists. The threshold is a business decision based on what the agent buys, its limits and whether the customer confirms before payment. Set thresholds per behaviour, agree them before testing and have an accountable owner sign them off.
Conclusion
The question for agentic checkout is no longer whether an agent can complete a purchase. It is whether it does so consistently, refuses when it should and stays inside its limits. Visa's figures, whatever their source, make that gap visible. The teams that will get agents past their bank, scheme and risk-team gatekeepers are the ones that arrive with a test set, a harness and a short evidence pack, rather than a demo.
Related reading
- Release Gates for Regulated AI: Evaluation, Pre-Deployment Certification and Live Oversight for Customer-Facing Agents
- Consumers Will Let AI Research but Not Yet Buy: What the Trust Gap Means for Checkout Design and Merchant Controls
- The Prompt-Injection Problem in Agentic Checkout: What Stripe's UCP Demo Revealed About Shopping-Agent Security
- Disputes and Chargebacks When an AI Agent Buys: What Evidence a Merchant Will Need











