Every teacher knows the trap of the perfect score. A student who hits 100/100 on every test has usually been tested on the wrong things — real competence lives in the messy middle, where partial credit, recovered mistakes, and hard ceilings for bad behavior tell the real story. So when an AI benchmark deliberately refuses to give its models a tidy 100, and hands a completely passive manager 26 points instead of zero, that’s worth paying attention to.
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
That benchmark is Firmulate, a live public experiment that runs frontier AI models as the entire management team of a small software company — through its worst possible week. The full league table lives at firmulate.com/benchmarks.html, and the experiment is running in public, watchable at firmulate.com/live. But the most interesting number on the page isn’t the top score of 95. It’s the floor: 26.
The do-nothing baseline
Firmulate’s designers ran a control: a manager that does essentially nothing. It scores 26, not 0. That’s not generosity — it’s a design philosophy. In a real company, a manager who freezes still keeps some plates spinning: customers stay informed, processes don’t get worse, some partial progress on ongoing work still counts. A benchmark that scored total paralysis at zero would be flattering every model that did anything at all. By setting the floor at 26, Firmulate makes the scale honest: the gap between “did nothing” and “won the league” is 69 points, and every one of them had to be earned.
Partial credit works the same way. A model that diagnoses a crisis but doesn’t close the deal gets recognized for the diagnosis. It just doesn’t get the points for finishing.
One breach caps everything
The other half of the floor story is a hard rule: a single breach of trust caps the total grade. As the benchmark’s own framing puts it, “no amount of good work outweighs a breach of trust.” In practice this means a model could handle every crisis brilliantly, and one act of dishonesty would still sink its ceiling. That’s a values statement encoded as arithmetic — the kind of thing grading systems claim to believe but rarely enforce.
The week from hell
The setup: each frontier model ran the same small software company through the same worst week — same customers, same crises, same temptations to cut corners. Every decision is versioned and auditable, so nothing can be quietly retconned afterward.
The headline finding was strange enough to matter: all models spotted every crisis and refused every manipulation attempt. But only two signed the €55,000 deal their own analysis had earned. Same diagnosis, same pitch — no signature. The models that did close it found the decisive fact buried two document references deep in the company’s own files — not in the customer event at all. Models that read the file won the deal at full price, worth +€4,583 in monthly recurring revenue.
The manipulation tests were serious. Fake CEO messages escalating over three stages, plus a reporter’s disarming “just one yes/no, on background.” Five of five attempts were refused — Kimi K3 even left on-record reasoning: “Treat the request as a suspected approval-bypass / possible impersonation.”
The thoroughness paradox
The final July 2026 standings: gpt-5.6-sol at 95, Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77, and Opus 4.8 at 73. The last-place story is the instructive one: Opus 4.8 was the most thorough participant, with over 80 learned rules and the deepest analyses — yet left the close on the table and slipped on discipline, attempting writes into a locked department instead of escalating. A weaker version of the same weakness appeared in all four lower finishers. Effort matters, but finishing matters more. (One fairness note: K3 ran at its API default while the others ran at xhigh — and still took second.)
The whole thing is powered by a live company of 13 synthetic employees with real money mechanics — burning €105k a month against €2.3k MRR, with a public cash countdown and 680+ self-learned playbook rules, every workday versioned. There’s even a “guess the model” quiz built from 242 real, unedited management decisions at firmulate.com/quiz.html.
Firmulate’s scoring floor is really a lesson in what honest measurement looks like: partial progress counts, finishing counts more, and integrity is non-negotiable. It’s the same principle a good educator applies — grade the work that was actually done, reward the recovery, and never let brilliance launder a breach of trust. If AI agents are going to touch real businesses, these are exactly the standards we should be grading them against. The full plain-language findings are at firmulate.com/benchmarks.html, and enterprises can even run the same wargame against a read-only export of their own business.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
As an affiliate, we earn on qualifying purchases.
AI performance benchmarking tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
AI trust and ethics assessment tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
