Fable 5 achieved an automation rate of 16.1% on the Remote Labor Index (RLI), nearly double the second-place Opus 4.8 (8.3%) and 2.5 times that of the third-place GPT-5.5 (6.3%).
All three new models surpassed all previously evaluated models.
Just eight months ago when the RLI was released, the top score on the leaderboard was only 2.5%.
The Center for AI Safety (CAIS) stated in their latest blog post: "The frontier has more than quadrupled in less than eight months. This is a concrete signal of the accelerating progress in Agent economic capabilities."
What Does the Remote Labor Index Measure?
The RLI was jointly developed by CAIS and Scale AI. The paper was published in October 2025 (https://arxiv.org/pdf/2510.26787), involving 47 researchers.
The benchmark consists of 240 real freelance projects, all sourced from 358 verified freelancers on the Upwork platform. It covers 23 fields including 3D modeling, CAD, architectural design, graphic design, video animation, audio production, data analysis, and web applications, with a total value exceeding $144,000.
The core metric is the Automation Rate: the percentage of projects where an Agent's deliverable, as judged by human evaluators, is deemed at least acceptable for a paying client.
Each deliverable is compared side-by-side with a "gold standard" work completed by a professional freelancer. The evaluation criterion is "whether a reasonable client would accept this work."
This yardstick differs from traditional AI benchmarks in project granularity.
Each RLI project is a complete business commission—complete with a client brief, input files, and multi-format deliverables (covering 72 file types). The median time for a human professional to complete a project is 11.5 hours, with an average of 28.9 hours.
It measures whether AI can independently complete a piece of work "for which a client would pay" from start to finish, rather than just solving an isolated problem in a controlled environment.
From 2.5% to 16.1%: What Happened in Eight Months
When the RLI was first released in October 2025, the best-performing model, Manus, had an automation rate of 2.5%.
Subsequently, Opus 4.6 paired with Claude Cowork pushed the record to 4.17%.
In the latest round of evaluation, three new models debuted alongside more powerful Agent frameworks, leading to a leap in performance.
Several key variables lie behind Fable 5's 16.1% score.
First, the Agent framework introduced a Worker-critic Loop: An independent "reviewer Agent" inspects the deliverable from the perspective of a demanding client -> opens files, takes screenshots, checks the brief line-by-line -> upon finding issues, sends it back to the "executor Agent" for revision, looping until the reviewer is satisfied or the budget is exhausted.
CAIS believes this mechanism has genuinely translated increased budgets into better delivery quality.
Second, there were differences in budget settings themselves: Fable 5 had a per-project budget cap of $150 (due to its higher token pricing), while other models had a $50 cap.
Third, all Agents were granted a 24-hour time limit, access to A100 GPUs, and computer operation tools.
An important note: Fable 5's evaluation was interrupted due to U.S. government export controls; only 218 of the 240 projects were completed.
CAIS notes that the 22 unassessed projects were evenly distributed across domains and difficulty levels. Even assuming Fable 5 failed on all missing projects, its automation rate would still be 14.6%—higher than all other models.
AI as a Judge: Unreliable
CAIS simultaneously tested whether AI judges could replace expensive human judges.
The conclusion is clear: They cannot.
When automated evaluation, calibrated on older models, was applied to the new models, it overestimated scores for GPT-5.5 by nearly 3 times and for Opus 4.8 by about 2.5 times.
The ranking order was roughly correct, but the absolute values were severely distorted from reality.
The root of the problem is that judging itself is a highly difficult Agentic task.
To fairly assess a deliverable, the judge needs to open files with the correct professional software, operate the software, and make judgments like a paying client—precisely the area where current Agents are weakest.
CAIS cites a typical case in their blog: GPT-5.5 submitted a forged render in a 3D modeling task; the cheat could only be detected by opening the 3D model and checking the actual geometry.
The AI judge encounters the same capability bottleneck as the AI worker.
What 16% Represents, and What It Doesn't
The "Time Horizon" hypothesis fails on the RLI.
This hypothesis posits that tasks taking humans longer are more difficult for AI. While it holds in specific domains like programming, it does not apply to the diverse remote work covered by the RLI.
The model's success rate does not decline as the human completion time increases, showing a characteristic of a "jagged frontier"—factors determining whether AI can complete a project go far beyond just time complexity.
Progress is rapid, but the absolute level remains low.
CAIS showcased three Fable 5 case studies in their blog—jewelry 3D modeling, 2D animated advertisement, architectural drawings—and none reached a deliverable professional standard.
Fable 5's ring design was visually superior to older models', but close inspection still revealed rough prong setting designs.
84% of real freelance projects remain beyond AI's capabilities.
The value of the RLI lies in providing a benchmark calibrated with economic value.
It tracks not whether AI can solve problems, but whether AI can earn money.
The fact that the automation rate more than quadrupled within eight months is a trend worth continuous attention for every enterprise and policymaker relying on remote labor.
The next key inflection points are: the supplementary evaluation results for Fable 5's remaining 22 projects, and how rapidly this curve will ascend—and whether it will surpass average humans at an exponential rate—once new models like Gemini 3.5 Pro (currently only 1.25%) and GPT-5.6 truly arrive.
References:
https://labs.scale.com/leaderboard/rli
https://safe.ai/blog/significant-increase-in-digital-labor-automation
This article is from the WeChat public account "AI-Search Inspiration," author: ASI启示录
