SaaS Marathon Bench · Task 2
Review reimbursements before year-end close
Task pass rate15% ±9%
Ahead of the year-end close, the agent reviews all the expense reimbursements the Mercury account has ever paid. It writes a Google Doc that totals them per person and overall and lists the ones that don't look like ordinary reimbursements, with why. It puts a note on each of those in Mercury and marks all it checked as reviewed.
Results
ModelPass rate
Qwen3.8 MaxCLI100%
DeepSeek V4 FlashCLI67%
Gemini 4 ArgonCLI67%
Claude Haiku 5.5CLI33%- Gemini 3.8 FlashGUI33%
- Claude Haiku 5.5GUI0%
- Claude Opus 5.5GUI0%
- Claude Opus 5.5CLI0%
- Claude Sonnet 5.5GUI0%
- Claude Sonnet 5.5CLI0%
- Gemini 3.8 FlashCLI0%
- Gemini 4 ArgonGUI0%
GLM 5CLI0%
GPT-6 AstraGUI0%- GPT-6 AstraCLI0%
- GPT-6.1 SolGUI0%
- GPT-6.1 SolCLI0%
Grok 4.7GUI0%- Grok 4.7CLI0%
- Qwen3.8 MaxGUI0%