SaaS Marathon Bench · Task 7
Reconcile what a partner is owed
Task pass rate12% ±8%
A partner keeps asking where things stand. For every company the partner introduced, the agent works out what is owed under their agreement, what has already been paid and whether it was paid on time. It puts this in a new Google Sheet, adds a note with the amount owed to each company's record in Attio and drafts a note to the partner.
Results
ModelPass rate
Gemini 4 ArgonCLI100%
GPT-6 AstraCLI67%
Claude Opus 5.5CLI33%
GPT-6.1 SolCLI33%
Claude Haiku 5.5GUI0%
Claude Haiku 5.5CLI0%
Claude Opus 5.5GUI0%
Claude Sonnet 5.5GUI0%
Claude Sonnet 5.5CLI0%
DeepSeek V4 FlashCLI0%
Gemini 3.8 FlashGUI0%
Gemini 3.8 FlashCLI0%
Gemini 4 ArgonGUI0%
GLM 5CLI0%
GPT-6 AstraGUI0%
GPT-6.1 SolGUI0%
Grok 4.7GUI0%
Grok 4.7CLI0%
Qwen3.8 MaxGUI0%
Qwen3.8 MaxCLI0%