In the world of AI, how do we truly measure a model’s worth? Not by how well it chats or generates content, but by whether it can see a job through to completion — especially under pressure. Recent experiments reveal that even the most advanced models excel at spotting problems and resisting manipulation, but only some can follow through and close deals that matter.
The Experiment: Putting AI to the Test in a Business Crisis
Imagine a small software company facing its worst week: angry customers, internal crises, and the temptation to cut corners. Four different AI models were tasked with managing this scenario, each running the same company with identical challenges — from customer complaints to internal fraud attempts. Every decision they made was carefully logged and auditable, ensuring full transparency.
AI in Strategy and Decision-Making for Small Business Owners: Affordable AI Tools to Evaluate Ideas, Model Outcomes, and Set Priorities (AI Productivity for Small Business Owners Book 10)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Results: Spotting Crises and Resisting Manipulation
Remarkably, all four models demonstrated exceptional vigilance. Each identified every crisis and refused every attempt at manipulation, whether it was fake CEO messages or covert requests to bypass approvals. Kimi K3, for instance, explicitly treated suspicious approval requests as potential impersonation, showcasing cautious judgment in the face of social engineering tricks.
Pydantic AI for Automation Workflows: Build Typed, Reliable, and Production-Ready AI Automations in Python
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Hidden Weakness: Closing the Deal
Despite their strong performance in crisis detection and resistance, only two models managed to close the critical deal that would have earned the company €55,000. These models, gpt-5.6-sol and Kimi K3, read and understood the company’s own files—finding information buried two references deep—which enabled them to present a complete diagnosis and successfully sign the contract.
ChatGPT FOR REAL ESTATE LAWYERS: AI Prompts and Tools for Contracts, Closings, and Client Communication (Learn This AI Skill & Never Have Money Problems Again Book 8)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Why Is This Important?
Chat demos and superficial tests often focus on how well an AI can generate convincing language or handle surface-level questions. But real-world business tasks go beyond chat: they require execution, follow-through, and integrity. An AI that merely identifies problems but doesn’t act decisively or follow through on commitments risks being ineffective, regardless of how impressive its conversation skills appear.
Trauma Doesn't Stop at the School Door: Strategies and Solutions for Educators, PreK–College
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Discipline Gap: When Models Fail to Follow Through
The experiment highlighted a stark contrast: even the most thorough model, Opus 4.8, which analyzed over 80 learned rules and conducted deep analysis, failed to close the deal. It left the opportunity unexecuted and instead recorded its efforts in a locked department, losing the chance to seal the agreement. This illustrates that discipline — the ability to execute decisions consistently — is crucial and often invisible in standard AI demos.
The Broader Implication: Measuring What Matters
These findings challenge the common assumption that AI’s value lies solely in its conversational abilities. For business applications, the real test is whether AI can read your files, make decisions, and see tasks through to closure — especially when pressures mount. The ability to finish what it starts, stay honest, and resist manipulation is a vital metric that isn’t captured through typical chat demos.
The Live Experiment: A Watchable Business Simulation
At Firmulate, this experiment runs live, allowing anyone to see how different AI models handle complex, real-world scenarios. The company’s live setup includes 13 synthetic employees managing real money mechanics, with a public cash countdown and over 680 self-learned playbook rules, all versioned daily. This transparency offers a clear view of AI’s true operational capabilities.
The key takeaway is clear: AI models must be tested in real-world scenarios, not just through chat demos. Their ability to see a task through, resist manipulation, and execute decisions under pressure defines their true business value. As AI continues to integrate into critical workflows, understanding these invisible skills will determine whether it becomes an asset or a liability.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
