Commerce Agent Bench

Commerce Agent Bench, renamed from RealReplicaBench, tests whether an AI agent can finish real business work instead of just talking about it. Its 107 tasks (publishing products, booking freight, editing storefronts) run against local replicas of commerce and SaaS systems, each graded by its own verifier. Code is Apache 2.0, tasks CC BY 4.0, both fine for commercial use.

Running it takes Python 3.11+, Docker on linux/amd64, a pinned OpenClaw image, an API key for the model under test and a second key for the LLM judge. The real bill is tokens: the published runs averaged between 0.8 million and 11 million tokens per task depending on model and harness.

This is an instrument for teams building agents or choosing models, not something you deploy. Accio will run your model on request, with no published pricing. OSWorld, WebArena and tau-bench are the peers.

The catch: Alibaba's Accio team builds the benchmark, and its own Accio harness competes on the leaderboard. The published scores came through Accio-managed endpoints and the raw task-level results aren't public yet, so you can't reproduce the board run for run. Mocks also measure skill against Accio's model of these services, not the real ones.