SaaS Marathon Bench · Task 5
Untangle a client's duplicate records
Task pass rate57% ±12%
A client may be in the CRM more than once, with every copy saying something different. The agent works out what really happened: when both sides first actually spoke, what was signed and how it ended. It writes that history in a note on the company it keeps, adds a pointer to it on each duplicate and drafts the reply to a question that may never have been answered.
Results
ModelPass rate
Claude Opus 5.5GUI100%
Claude Sonnet 5.5GUI100%
Claude Sonnet 5.5CLI100%
Gemini 4 ArgonCLI100%
GPT-6 AstraCLI100%
GPT-6.1 SolCLI100%
Qwen3.8 MaxCLI100%
Claude Haiku 5.5CLI67%
Gemini 4 ArgonGUI67%
GPT-6 AstraGUI67%
Grok 4.7CLI67%
Claude Opus 5.5CLI33%
DeepSeek V4 FlashCLI33%
Gemini 3.8 FlashGUI33%
Gemini 3.8 FlashCLI33%
GPT-6.1 SolGUI33%
Claude Haiku 5.5GUI0%
GLM 5CLI0%
Grok 4.7GUI0%
Qwen3.8 MaxGUI0%