Researchers Test AI’s Yearlong E-Commerce Operations

Researchers from Alibaba’s Qwen team, Hong Kong University of Science and Technology (HKUST) and other institutions developed E-Commerce Bench to measure how well AI models can operate online stores over a year. Their paper, “E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation,” assessed 18 commercial and open-weight models.

Each model received 100,000 yuan and its total assets were compared after 365 days. Models could run up to four stores in a market with 6,886 products and 576 suppliers, using 18 tools for tasks including market research, price negotiations, purchasing, pricing, shipping, returns and cash management. The simulated market drew on real Taobao and Tmall data, while formula-based rules in the negotiation engine made results reproducible.

Sales events and natural disasters changed demand. About one-quarter of suppliers were set as fraudulent sellers who charged high prices, falsified order counts or shipped poor-quality products that drove up return rates. Purchasing costs and operating expenses were deducted from bank accounts immediately, while sales revenue stayed in escrow (an account that holds funds temporarily) for nine days after shipment. A model went bankrupt if its bank balance remained negative for 10 consecutive days. Because older conversation history was deleted after reaching a limit, models had to retain transaction information in up to 20 memory entries.

GPT-5.6 Sol ranked first in final assets, increasing its starting funds by an average of about 14.3 times to about 1.43 million yuan, but ranked 16th among the 18 models for the share of spending that went to fraudulent suppliers. Runner-up Fable5 grew its funds by about eight times, earned profits with fewer than half as many tool calls as GPT-5.6 Sol, and led in profit per operation. Claude Opus 4.7 ranked first in discount negotiations and fraud avoidance, but placed eighth in total assets because it did not expand the scale of its business.

Among open-weight models (models whose trained parameters are publicly available), Alibaba’s Qwen3.8-Max-Preview placed fifth overall and led that group, growing its assets by about 4.2 times. It also ranked first among all 18 models in long-term learning, using the lowest previous purchase price as a reference to negotiate further discounts when restocking. Of the 90 trials—five for each model—10 ended in bankruptcy. GPT-5.5, Claude Opus 4.6 and Gemini 3.1 Pro each went bankrupt in two of five trials; Qwen3.5-Plus did so in four of five.

Researchers from Alibaba’s Qwen team, Hong Kong University of Science and Technology (HKUST) and other institutions developed E-Commerce Bench to measure how well AI models can operate online stores over a year. Their paper, “E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation,” assessed 18 commercial and open-weight models.

Each model received 100,000 yuan and its total assets were compared after 365 days. Models could run up to four stores in a market with 6,886 products and 576 suppliers, using 18 tools for tasks including market research, price negotiations, purchasing, pricing, shipping, returns and cash management. The simulated market drew on real Taobao and Tmall data, while formula-based rules in the negotiation engine made results reproducible.

Sales events and natural disasters changed demand. About one-quarter of suppliers were set as fraudulent sellers who charged high prices, falsified order counts or shipped poor-quality products that drove up return rates. Purchasing costs and operating expenses were deducted from bank accounts immediately, while sales revenue stayed in escrow (an account that holds funds temporarily) for nine days after shipment. A model went bankrupt if its bank balance remained negative for 10 consecutive days. Because older conversation history was deleted after reaching a limit, models had to retain transaction information in up to 20 memory entries.

GPT-5.6 Sol ranked first in final assets, increasing its starting funds by an average of about 14.3 times to about 1.43 million yuan, but ranked 16th among the 18 models for the share of spending that went to fraudulent suppliers. Runner-up Fable5 grew its funds by about eight times, earned profits with fewer than half as many tool calls as GPT-5.6 Sol, and led in profit per operation. Claude Opus 4.7 ranked first in discount negotiations and fraud avoidance, but placed eighth in total assets because it did not expand the scale of its business.

Among open-weight models (models whose trained parameters are publicly available), Alibaba’s Qwen3.8-Max-Preview placed fifth overall and led that group, growing its assets by about 4.2 times. It also ranked first among all 18 models in long-term learning, using the lowest previous purchase price as a reference to negotiate further discounts when restocking. Of the 90 trials—five for each model—10 ended in bankruptcy. GPT-5.5, Claude Opus 4.6 and Gemini 3.1 Pro each went bankrupt in two of five trials; Qwen3.5-Plus did so in four of five.