SaaS Marathon Bench · Task 10
Draft a client contract amendment
Task pass rate55% ±12%
An updated contract was promised to a client after a call about fees and never sent. The agent writes the amendment in a new Google Doc from what was actually agreed, flagging anything still ambiguous. It sets the fee on each of the client's open deals in HubSpot, with its percentage in a note on the deal, and drafts the cover email.
Results
ModelPass rate
Claude Haiku 5.5CLI100%- Claude Opus 5.5CLI100%
- Claude Sonnet 5.5CLI100%
Gemini 3.8 FlashCLI100%- Gemini 4 ArgonCLI100%
GPT-6 AstraCLI100%- GPT-6.1 SolCLI100%
Qwen3.8 MaxCLI100%- Claude Sonnet 5.5GUI67%
DeepSeek V4 FlashCLI67%- Claude Opus 5.5GUI33%
- Gemini 3.8 FlashGUI33%
- Gemini 4 ArgonGUI33%
- GPT-6 AstraGUI33%
- GPT-6.1 SolGUI33%
- Claude Haiku 5.5GUI0%
GLM 5CLI0%
Grok 4.7GUI0%- Grok 4.7CLI0%
- Qwen3.8 MaxGUI0%