Visa has launched an Agentic Commerce Benchmark and tested 22 AI models from 12 labs on 15 shopping tasks. The models passed 57.2% of runs, which means 42.8% ended in failure. The best model passed 82.2% of attempts, the worst 28.9%. The result matters for business because payment delegation depends on consistent requirement matching, not single successful answers.
How Visa tested shopping agents
The benchmark covered purchases of a TV, a laptop and a personal care appliance, according to the company statement released on Thursday. The customer did not disclose all conditions in advance, so each model had to clarify needs through dialogue. If no suitable product was available, the correct decision was to refuse the purchase. A run counted as passed only when the model found every set requirement and made the right buy or no-buy call.
Each model received three attempts per scenario, and justification of the choice was part of the test. The design was meant to exclude lucky guesses by demanding enough information to support the decision. One scenario involved a TV request with a mentioned brand and a third-party supplier membership, while budget, format support, HDR10+ and gaming features had to be uncovered. In 54% of failed runs the agent still completed a purchase, either selecting an unsuitable product or buying before all needs were identified.
Visa describes the project as the first benchmark of its kind for agentic commerce. The company plans to add scenarios and product categories and extend testing to checkout, payment and ordering. The benchmark has been shared with leading model providers for feedback. Anyone can try the test on the Payments Frontier site, which keeps the methodology open for inspection.
What failed purchases mean for commerce
For companies deploying shopping assistants, the gap between single attempts and repeatable behavior is the main operational factor. Only 34.2% of scenario series passed on all three tries, so the same task can succeed and then fail without changes. That pattern affects support load, returns and dispute handling in stores where agents select goods. Larger retailers will need repeated-run checks before connecting an agent to assortment and ordering.
Question quality and run cost correlated with scores in this round. The five models that uncovered the most hidden buying factors passed 70.2% of the time, against 44.4% for the five weakest at discovery. The five most expensive models per run passed 67.1%, compared with 50.2% for the five cheapest. Small firms should therefore test discovery dialogue rather than demo accuracy, while large firms should compare price per run against stability gains.
The signal to watch is the next Visa benchmark expansion into checkout, payment and ordering. A second marker is repeat-pass rates and named model scores, since Visa did not name models and rankings came via Fortune reporting. Google and Meta agents reportedly shared the top score around 82%, a Claude agent took third with 76% and an OpenAI agent was sixth with 64%. If repeatability rises without higher failure purchases, delegation of buying tasks will become a practical option.
