Anthropic’s new AI leads freelance benchmarks, yet fails 58% of the tasks

Anthropic’s Fable 5.2 ranks first on the Remote Labor Index, a test of complete freelance projects, with 41.3% of its deliverables judged to meet a professional reference standard.

Scale’s Nov. 12 leaderboard lists Fable 5.2 at 41.3%, ahead of the next-highest model, Opus 5.5 at 21.25%. Anthropic reported the result in its Nov. 9 release announcement, labeling the RLI score “High effort” and indicating that safeguards were enabled.

The original RLI paper, submitted in October 2025 by researchers from the Center for AI Safety and Scale AI, describes a dataset of 240 projects across 23 freelance work categories. Each includes a brief, the files needed to complete the task, and a professional human-reference deliverable.

The selected projects include documents, audio, video, 3D models, and computer-aided design files. That range makes file production and finished-product quality part of the challenge, alongside understanding what the client requested.

The researchers’ original evaluation method uses trained human reviewers to inspect the AI’s work against the brief and reference deliverable.

Independent evaluations and majority voting determine outcomes, and a project passes when the AI’s deliverable satisfies the brief at least as well as the reference, at a quality a reasonable client would accept.

A useful prototype or a partial result can therefore fall below the pass threshold, even when it demonstrates relevant skills. Passing demands a deliverable that clears the professional reference standard.