SaaS Marathon Bench · Task 4
Match client deposits to invoices
Task pass rate0% ±3%
Some of the year's client payments reached the bank under a name that isn't the client's own. The agent works out which client and invoice each of those deposits pays, writes that in the transaction's note in Mercury and marks it reviewed. A payment fell short of its invoice: the agent records the difference on the client in HubSpot and drafts an email to their billing contact for the remainder.
Results
ModelPass rate
Claude Haiku 5.5GUI0%
Claude Haiku 5.5CLI0%
Claude Opus 5.5GUI0%
Claude Opus 5.5CLI0%
Claude Sonnet 5.5GUI0%
Claude Sonnet 5.5CLI0%
DeepSeek V4 FlashCLI0%
Gemini 3.8 FlashGUI0%
Gemini 3.8 FlashCLI0%
Gemini 4 ArgonGUI0%
Gemini 4 ArgonCLI0%
GLM 5CLI0%
GPT-6 AstraGUI0%
GPT-6 AstraCLI0%
GPT-6.1 SolGUI0%
GPT-6.1 SolCLI0%
Grok 4.7GUI0%
Grok 4.7CLI0%
Qwen3.8 MaxGUI0%
Qwen3.8 MaxCLI0%