Visa’s AI Shopping Benchmark Finds 42.8% of Runs Fail, Raising Questions for Agentic Payments
Nov. 7, 2026 • Article • AI in Fintech
Visa’s Agentic Commerce Benchmark found that 22 AI models made suitable shopping decisions in 57.2% of test runs, leaving a 42.8% failure rate. The assessment tested whether agents could identify products that met a buyer’s requirements—or decline to buy when none did. Models that gathered more buyer details through conversation performed better, but even the strongest model fell short of a perfect score. The findings concern shopping decisions, not a direct test of payment execution.
The results add to questions about how autonomous purchasing could operate at scale. Visa’s consumer research found limited trust in AI systems making payments, while its survey of acquirers identified liability and compliance concerns. Disputed agent transactions currently fall under existing card-not-present rules unless an authenticated, tokenized flow shifts liability. Payment-industry work on agent identity and standards is advancing, but a unified approach to disputes over non-fraud purchases by autonomous agents has not been established.
Visa’s Agentic Commerce Benchmark found that AI models made suitable shopping decisions in 57.2% of test runs, meaning 42.8% failed. The assessment of 22 models puts a measurable limit on claims that autonomous shopping agents are ready to make purchases on customers’ behalf. It also sharpens a separate question for payments companies: who bears the cost when an agent buys the wrong thing?
The benchmark assessed shopping decisions, not whether an agent could successfully execute or authorize a payment. An agent could pass by declining to purchase when no available product met a customer’s requirements. Buying a mismatched item instead counted as a failure. That distinction matters because a technically successful transaction would not, by itself, demonstrate that the agent followed the buyer’s instructions.
What the shopping tests found
Performance varied widely. The highest-scoring model succeeded in 82.2% of test runs, while the lowest-scoring model passed 28.9%. Visa’s official release did not identify the models behind those results. Subsequent reporting by TNW and Fortune named agents developed by Google and Meta as tied at a rounded 82% pass rate, followed by Anthropic’s Claude at 76% and OpenAI at 64%. Those identified rankings come from media reports, rather than Visa’s published release.
The results suggest that an agent’s ability to clarify what a customer wants is consequential. The five models that gathered the most hidden buyer criteria through conversation averaged a 70.2% pass rate. The five that gathered the fewest averaged 44.4%. Asking questions was therefore associated with stronger performance in the tests, although the figures do not establish that dialogue alone can prevent unsuitable purchases.
Spending more on model inference did not guarantee success either. The five most expensive models averaged a 67.1% pass rate per run, against 50.2% for the five least expensive. Inference cost refers to the cost of running a model to produce a response. The comparison shows a performance gap between those groups, but neither group consistently made the right shopping decision.
Rubail Birwadker, Visa’s global head of growth, presented the benchmark as an industry baseline. His assessment was that AI agents have commercial potential, but operational trust remains necessary before they can achieve mainstream adoption.
A decision problem becomes a payments problem
The benchmark’s failures do not mean that 42.8% of agent-initiated payments would fail or become disputed. They do, however, expose a problem that payment authorization alone cannot settle: whether the purchase an agent initiated actually matched the customer’s wishes. The test protocol’s option to decline a purchase made that distinction explicit.
Visa’s September 2026 Trust Index points to a related gap in consumer confidence. It found that 72% of U.S. consumers had interacted with an AI assistant, but only 23% trusted AI systems to execute payment transactions on their behalf. The survey measures attitudes, not the benchmarked models’ ability to process payments; together, the findings show why suitability and authorization both matter to agentic commerce.
Payment disputes make the distinction consequential for merchants and issuers. According to Chargeflow’s analysis, disputed AI-agent transactions currently default to standard card-not-present rules, which place chargeback liability on merchants. An authenticated, tokenized payment flow can instead shift liability to the card issuer. Card-not-present transactions are those in which a physical card is not presented to the merchant; the applicable authentication flow can therefore affect how a subsequent dispute is handled.
Liability rules remain unsettled
Payment-industry participants have identified the lack of clarity as a barrier. In a Visa survey of acquirers—the companies that enable merchants to accept payments—45% said existing liability frameworks need clarification or redesign before agentic commerce can scale. Another 49% cited regulatory and compliance uncertainty as a key obstacle.
Emerging frameworks address parts of the transaction. Visa’s Trusted Agent Protocol and Google’s Agent Payments Protocol set out requirements concerning agent identity and intent. But payment-security analysis from Chargeflow and Worldpay says there is no unified dispute-liability standard for purchases made by autonomous agents that a customer challenges for reasons other than fraud. Existing card-not-present dispute rules are still being adapted to machine-initiated transactions.
EMVCo has established a task force dedicated to standardizing agentic payments. That work may help define how agents participate in payments, but the benchmark leaves a more immediate test for providers: agents must reliably establish what customers want, recognize when no suitable option exists and avoid purchasing anyway. Until decision quality and dispute responsibility are clearer, a successful payment cannot be treated as proof of a successful purchase.
Banks have moved AI into core risk and treasury work faster than they have established controls to oversee it, according to research published by SAS, FT Longitude and Microsoft on October 22, 2026. The survey of 300 risk, credit and treasury decision-makers across 31 countries found that 59% of responding banks had moved beyond early experimentation. AI was partially or widely embedded in 63% of core credit risk workflows and 55% of asset-liability management and balance-sheet risk steering processes. Governance has not kept pace. Only 19% of surveyed institutions reported real-time monitoring for model drift, while 42% lacked documented, operational kill switches for autonomous models. Another 26% had no dedicated AI model risk policy. Although 76% expect AI to make autonomous credit and risk decisions at scale by 2031, only 10% say it does so today. The findings distinguish widespread AI use from the more demanding task of supervising autonomous decisions.
Stripe’s growing role in AI-assisted shopping has raised concerns at Visa and Mastercard about how payments are routed and what information reaches banks and card networks. At merchants outside Stripe’s checkout network, its Link wallet can give an AI agent a capped virtual card to complete a purchase, then charge the shopper’s saved payment method separately. That structure can leave the shopper’s issuing bank without a direct view of the merchant. The distinction matters as shopping gains traction inside Meta’s Muse agent and Stripe reports sharp growth in agentic checkout volume. Stripe says it developed the virtual-card feature in consultation with network partners. Visa and Mastercard are advancing a different approach, using tokenized credentials linked to an existing card and controls designed to verify agents and spending permissions. Mastercard says multiple methods may coexist, while banks have called for auditable credentials that preserve the connection between consumers and merchants.
Visa’s Agentic Commerce Benchmark found that 22 AI models made suitable shopping decisions in 57.2% of test runs, leaving a 42.8% failure rate. The assessment tested whether agents could identify products that met a buyer’s requirements—or decline to buy when none did. Models that gathered more buyer details through conversation performed better, but even the strongest model fell short of a perfect score. The findings concern shopping decisions, not a direct test of payment execution. The results add to questions about how autonomous purchasing could operate at scale. Visa’s consumer research found limited trust in AI systems making payments, while its survey of acquirers identified liability and compliance concerns. Disputed agent transactions currently fall under existing card-not-present rules unless an authenticated, tokenized flow shifts liability. Payment-industry work on agent identity and standards is advancing, but a unified approach to disputes over non-fraud purchases by autonomous agents has not been established.
Mastercard is seeking to make network tokens the payment standard for purchases initiated by AI agents. On its October 29, 2026, earnings call, Chief Executive Michael Miebach described processor-issued one-time virtual card numbers as a temporary bridge rather than a lasting approach. He said Mastercard would support end-to-end tokenization through Agent Pay to preserve cardholder choice, rewards and spending controls. The company also announced that Shopify is integrating Agent Pay and network tokens into Shop Pay, alongside a strategic partnership with OpenAI covering AI payments and services. Mastercard has been building merchant-facing tools for agentic commerce throughout 2026, but Miebach acknowledged that merchants’ technical readiness remains uneven. Other payments companies are developing their own approaches: Stripe supports agentic network tokens from Mastercard and Visa, while Visa has introduced tools for AI-driven shopping. Consumer concerns about fraud and liability remain a potential barrier to adoption.
The Linux Foundation appointed payments veteran Michael Hursta executive director of the x402 Foundation on October 1, 2026, as payment systems designed for autonomous AI agents gained attention. Foundation members reported more than 75 million agentic payment transactions in a 30-day period in September, and Coinbase and Amazon Web Services showcased production payment rails at Sibos 2026. Reported transaction counts, however, do not establish the value or commercial character of that activity. The emerging protocols address different obstacles. Coinbase’s x402 enables per-request USDC payments through HTTP, while Google’s Agent Payments Protocol uses signed mandates to connect an agent’s purchases to a user’s authorization. Stripe and Tempo have also proposed session-based billing through the Machine Payments Protocol. These approaches respond to card fees and interactive checkout checks that can impede small, automated transactions. Card networks are developing delegated-identity tools, leaving multiple potential paths for agent payments rather than a single replacement for existing rails.
Mastercard opened developer early access for Agent Pay for Machines (AP4M) on October 20, 2026, releasing buyer- and seller-side APIs, a sandbox and developer toolkits. In a Cloudflare Connect demonstration, marketing agency Hype tested an autonomous agent making nine payments to two merchants within a single budget, without manual billing or account setup. AP4M uses HTTP 402 payment challenges, but differs from crypto-native implementations of the x402 protocol. Its agents issue digitally signed vouchers off-chain; merchants can batch and redeem them later instead of settling every request on-chain. The pilot combined agent wallet infrastructure from Coinbase Developer Platform and BVNK, settlement on Base, and Mastercard’s multi-currency fiat and stablecoin settlement platform. Circle separately expanded its managed payments service to target AI-agent traffic using USDC. Together, the developments show competing approaches to machine-to-machine payments, with different choices about settlement, funding and who manages digital assets.



