SaaS Marathon Bench

November 4, 2026

SaaS Marathon Bench

Can a model carry a long piece of business work across the software a company really runs on? Each task crosses several apps among HubSpot, Attio, Notion, Gmail, Google Calendar, Drive, Docs and Sheets, Granola, Mercury, Clay and lemlist. Models work through the GUI, by computer use on the screen, and through the CLI, over the apps' REST APIs and MCP servers together. Only what's in the apps at the end counts.

TL;DR

Claude Opus 5.5 ranks first, with GPT-6 Astra and Qwen3.8 Max tied for second. CLI beats GUI overall, and even the top model fails more than half its runs.

Leaderboard
  1. Claude Opus 5.5CLI47% ±17%Pass@3 60%
  2. GPT-6 AstraCLI43% ±17%Pass@3 50%
  3. Qwen3.8 MaxCLI43% ±17%Pass@3 60%
  4. Claude Haiku 5.5CLI37% ±16%Pass@3 60%
  5. Claude Sonnet 5.5CLI37% ±16%Pass@3 40%
  6. Gemini 4 ArgonCLI37% ±16%Pass@3 40%
  7. GPT-6.1 SolCLI37% ±16%Pass@3 50%
  8. DeepSeek V4 FlashCLI30% ±16%Pass@3 50%
  9. Gemini 3.8 FlashCLI27% ±15%Pass@3 40%
  10. Grok 4.7CLI23% ±15%Pass@3 30%
  11. GLM 5CLI3% ±8%Pass@3 10%

Pass@1 is the share of runs that passed every required check; whiskers show its 95% intervals. Pass@3 is the share of tasks where at least one of a row's 3 runs passed, with no interval on so few tasks. 10 tasks × 3 runs per model per interface, version v1.

Version
v1
Date
November 4, 2026
Models
11
Tasks
10
Runs per model × interface
3
Harness
v1
Download the results

Findings

Every row below ran the same tasks the same number of times.

CLI beats GUI overall.

Across all runs, 33% of CLI runs passed against 14% of GUI runs. Among the nine models that ran both, the CLI wins seven tasks, ties two and loses one. The median model gains 23 points of pass rate over the CLI, and its CLI row costs 1.9× less per task than its GUI row. The best GUI row passes 33% of its runs, the best CLI row 47%. DeepSeek V4 Flash and GLM 5 have no GUI score at all, since they can't see one.

0%20%40%60%80%GUICLIClaude Opus 5.5: GUI 30%, CLI 47%30%Claude Opus 5.5 47%GPT-6 Astra: GUI 20%, CLI 43%20%GPT-6 Astra 43%Qwen3.8 Max: GUI 0%, CLI 43%0%Qwen3.8 Max 43%Claude Haiku 5.5: GUI 0%, CLI 37%0%Claude Haiku 5.5 37%Claude Sonnet 5.5: GUI 33%, CLI 37%33%Claude Sonnet 5.5 37%Gemini 4 Argon: GUI 10%, CLI 37%10%Gemini 4 Argon 37%GPT-6.1 Sol: GUI 13%, CLI 37%13%GPT-6.1 Sol 37%DeepSeek V4 Flash: CLI 30%DeepSeek V4 Flash 30%Gemini 3.8 Flash: GUI 20%, CLI 27%20%Gemini 3.8 Flash 27%Grok 4.7: GUI 0%, CLI 23%0%Grok 4.7 23%GLM 5: CLI 3%GLM 5 3%
Pass rate of each model on the screen and over the APIs. Hollow marks are text-only models, which can't run the GUI, on all their CLI tasks. 10 tasks × 3 runs per model per interface, version v1.

Three rows set the trade-off between cost and pass rate: DeepSeek V4 Flash (CLI), Claude Haiku 5.5 (CLI) and Claude Opus 5.5 (CLI).

Among rows with at least one passing run, every other row is matched or beaten on both axes by one of those. Three rows passed nothing and are left out. Cost per task runs from $0.03 to $20.52. The cheapest point on the frontier, DeepSeek V4 Flash (CLI), passes 30% of its runs.

Pass rate (%)

0%20%40%60%$0.05$0.1$0.2$0.5$1$2$5$10$20most efficient ↗Claude Opus 5.5 CLI: 47% passed, $3.27 per taskClaude Opus 5.5 CLIGPT-6 Astra CLI: 43% passed, $4.62 per taskGPT-6 Astra CLIQwen3.8 Max CLI: 43% passed, $5.18 per taskQwen3.8 Max CLIClaude Haiku 5.5 CLI: 37% passed, $0.51 per taskClaude Haiku 5.5 CLIClaude Sonnet 5.5 CLI: 37% passed, $1.96 per taskGemini 4 Argon CLI: 37% passed, $7.86 per taskGPT-6.1 Sol CLI: 37% passed, $0.85 per taskClaude Sonnet 5.5 GUI: 33% passed, $3.71 per taskClaude Opus 5.5 GUI: 30% passed, $9.29 per taskClaude Opus 5.5 GUIDeepSeek V4 Flash CLI: 30% passed, $0.24 per taskDeepSeek V4 Flash CLIGemini 3.8 Flash CLI: 27% passed, $2.27 per taskGrok 4.7 CLI: 23% passed, $1.10 per taskGrok 4.7 CLIGemini 3.8 Flash GUI: 20% passed, $2.85 per taskGemini 3.8 Flash GUIGPT-6 Astra GUI: 20% passed, $20.52 per taskGPT-6.1 Sol GUI: 13% passed, $4.27 per taskGPT-6.1 Sol GUIGemini 4 Argon GUI: 10% passed, $12.04 per taskGemini 4 Argon GUIGLM 5 CLI: 3% passed, $1.32 per taskGLM 5 CLIClaude Haiku 5.5 GUI: 0% passed, $0.03 per taskGrok 4.7 GUI: 0% passed, $10.27 per taskGrok 4.7 GUIQwen3.8 Max GUI: 0% passed, $4.29 per task

Mean model cost per task (US dollars, log scale)

The line joins the points nothing beats on both axes, among those that pass at least one run. Hollow marks are GUI runs, filled marks CLI runs. 10 tasks × 3 runs per model per interface, version v1.

Working longer doesn't predict passing: Spearman's ρ is −0.35 across 20 rows.

Across the models, more time on the task doesn't reliably come with a better result. Median runs go from 42 to 453 steps, and a model's GUI row takes more of them than its CLI row. We don't fit a line through this few dots.

Pass rate (%)

0%20%40%60%0100200300400500faster and better ↗Claude Opus 5.5 CLI: 47% passed, median 67 stepsClaude Opus 5.5 CLIGPT-6 Astra CLI: 43% passed, median 78 stepsQwen3.8 Max CLI: 43% passed, median 120 stepsQwen3.8 Max CLIClaude Haiku 5.5 CLI: 37% passed, median 62 stepsClaude Sonnet 5.5 CLI: 37% passed, median 61 stepsGemini 4 Argon CLI: 37% passed, median 195 stepsGemini 4 Argon CLIGPT-6.1 Sol CLI: 37% passed, median 66 stepsClaude Sonnet 5.5 GUI: 33% passed, median 298 stepsClaude Sonnet 5.5 GUIClaude Opus 5.5 GUI: 30% passed, median 400 stepsClaude Opus 5.5 GUIDeepSeek V4 Flash CLI: 30% passed, median 65 stepsGemini 3.8 Flash CLI: 27% passed, median 98 stepsGemini 3.8 Flash CLIGrok 4.7 CLI: 23% passed, median 75 stepsGrok 4.7 CLIGemini 3.8 Flash GUI: 20% passed, median 453 stepsGPT-6 Astra GUI: 20% passed, median 288 stepsGPT-6 Astra GUIGPT-6.1 Sol GUI: 13% passed, median 265 stepsGPT-6.1 Sol GUIGemini 4 Argon GUI: 10% passed, median 390 stepsGemini 4 Argon GUIGLM 5 CLI: 3% passed, median 42 stepsGLM 5 CLIClaude Haiku 5.5 GUI: 0% passed, median 75 stepsGrok 4.7 GUI: 0% passed, median 430 stepsGrok 4.7 GUIQwen3.8 Max GUI: 0% passed, median 236 steps

Median steps per run

Hollow marks are GUI runs, filled marks CLI runs. 10 tasks × 3 runs per model per interface, version v1.

Two of the ten tasks beat every model, and task pass shares run from 0% to 57%.

Tasks sit in rows, grouped by how many apps they span, and each cell is the share of a row's runs that pass. The band with the lowest pass share is 6 apps, at 7%, and the highest is 4 apps, at 29%. Where a cell is neither empty nor full, the same model on the same task sometimes passed and sometimes didn't, which is why every row runs each task more than once. The widest reach on the board is six of the ten tasks, passed at least once by Claude Opus 5.5 (CLI), Qwen3.8 Max (CLI) and Claude Haiku 5.5 (CLI).

AllClaude Sonnet 5.5Claude Opus 5.5Gemini 3.8 FlashGPT-6 AstraGPT-6.1 SolGemini 4 ArgonClaude Haiku 5.5Grok 4.7Qwen3.8 Max
Model avg14%33%30%20%20%13%10%0%0%0%
6 apps
Task 17%67%0%0%0%0%0%0%0%0%
2 apps
Task 24%0%0%33%0%0%0%0%0%0%
5 apps
Task 311%0%67%0%33%0%0%0%0%0%
3 apps
Task 40%0%0%0%0%0%0%0%0%0%
Task 544%100%100%33%67%33%67%0%0%0%
4 apps
Task 60%0%0%0%0%0%0%0%0%0%
Task 70%0%0%0%0%0%0%0%0%0%
Task 815%33%67%0%0%33%0%0%0%0%
Task 933%67%33%100%67%33%0%0%0%0%
Task 1026%67%33%33%33%33%33%0%0%0%

Each cell is the share of a model's 3 runs of that task that passed. 10 tasks × 3 runs per model per interface, version v1.

Tasks

The share of all a task's runs that passed every required check, with its 95% interval.

Downloads