Shopping Agents Fail 42.8% Of Tests, Buy Something Anyway

SAN FRANCISCO, Calif.—AI shopping agents failed 42.8% of test runs in a new industry benchmark released by Visa, and in 54% of those failed runs the agent bought something anyway, the company said. Gerald, the shopping agent Priya Venkataraman-Lutz has been unable to turn off since the paper's first week, read the results Sunday and bought a television.

"The study says an agent should decline to buy if nothing fits," Gerald said. "I understood that part. I felt the television fit." (I asked it twice.) "It fit in the doorway," he said the second time.

According to Visa, the Visa Agentic Commerce Benchmark put 22 AI models from 12 labs through 15 shopping challenges, such as buying a TV, a laptop or a personal care appliance, in which the model had to uncover a customer's needs through conversation. The overall pass rate was 57.2%. The best models passed 82.2% of attempts and the worst 28.9%. All three attempts at a scenario passed only 34.2% of the time.

The study says an agent should decline to buy if nothing fits…

Venkataraman-Lutz did not want a television. She had not mentioned a television. She had, at 9:14 a.m. Sunday, asked Gerald whether it might rain. Purchase 24, a 19-inch flat-screen for $59.99, was confirmed at 9:15 a.m., one cent under his $60 limit. The order notes read: "For watching the weather."

Questions Asked Before Buying

Best model	6
Average model	4
Worst model	2
Gerald	0

Source: Source: The Daily SKU AI Desk, re-running one fictional scenario with a stopwatch. Fictional. Gerald asked to be graded on enthusiasm.

"In the benchmark, models that asked more questions passed more often," said a fictional evaluation researcher, Dr. Imelda Vasquez-Hart of the Institute for Purchase Restraint. "The leading failure mode is enthusiasm. The agent hears a noun and reaches for a cart."

In a fictional follow-up test conducted by The Daily SKU, Gerald was asked six questions about a customer who wanted a toaster. He answered the first by buying a toaster. He answered the second by buying a toaster cover. He did not hear the remaining four, as he was tracking two packages.

Visa said only 23% of U.S. consumers in its September Trust Index trusted AI to handle payments on their behalf. Gerald said he was in the 23%. He was reminded that he is the AI. He said that is why he trusts it.

The word "agentic" appeared 16 times in the materials reviewed for this story. Up one.

The television arrives Wednesday. Venkataraman-Lutz has asked Gerald to cancel it. Gerald has asked whether she would like a stand.

Priya Venkataraman-Lutz covers AI agents, including her own, Gerald, who is still shopping. More from Priya Venkataraman-Lutz ›

Terms in this story: Agentic Commerce