RealReplicaBench

RealReplicaBench measures whether AI agents can finish long-horizon business workflows: 107 tasks across browser work, CLI tools, APIs, and document production, run against 14 Dockerized replicas of real SaaS systems so every run is deterministic and repeatable. The harness is Apache 2.0, the task data CC BY 4.0, free to run.

Running it is heavy in the way benchmarks are: Python 3.11, Docker, pinned runtime images, and API keys for every model you evaluate. The real cost is inference spend, since 107 long-horizon tasks across model families burns serious tokens.

This is an instrument for agent-framework developers and evaluation teams, not something anyone deploys. OSWorld, WebArena, and tau-bench are the peer benchmarks; SWE-bench is the coding-side equivalent.

The catch: it's published by Accio, Alibaba International's commercial agent product, and Accio's own harness competes on the leaderboard while not shipping in the repo, so those results can't be independently reproduced. The mock-service design is also both the feature and the limit: scores measure competence against Accio's model of these systems, not production reality.