Imagine preparing a complex recipe. You meticulously gather ingredients, follow the steps, and double-check your work. Yet, if you focus too much on every detail without prioritizing the key flavors, the dish might still fall flat. This is a familiar truth in both cooking and business: volume and effort can’t replace strategic focus. Now, the same lesson applies to AI systems managing real companies, as seen in a groundbreaking live experiment conducted by Firmulate.
Listen free for 30 days with Audible
Thousands of audiobooks and originals — cancel anytime.
As an affiliate, we earn on qualifying purchases.
Testing AI as a Business Partner in Real Conditions
In a unique live experiment, four advanced AI models were tasked with running a small software company through its worst week. The goal was straightforward: see if the AI could handle crises, avoid manipulation, and close a critical deal worth €55,000. Each model was given identical challenges—customers, crises, temptations—and their decisions were fully tracked and auditable, making this a transparent test of AI integrity and effectiveness.
Key Results: Diligence Meets Focus
All four models demonstrated impressive technical awareness. They identified every crisis, refused all manipulation attempts—including social engineering tricks like fake CEO messages—and stuck to their analysis. Interestingly, two of the models actually signed the deal, earning their keep with high confidence, while the other two did not. The core reason? Only the models that read further into the company’s internal files—finding a crucial piece of information hidden two references deep—secured the full-value deal.
The Hidden Weakness: Attention and Prioritization
What made the difference? Interestingly, the models that performed best didn’t just beeline for surface data—they engaged in deeper, more thorough analysis. The Opus 4.8 model, despite its extensive rules (over 80 learned rules and the deepest analysis), ended up in last place because it slipped discipline and failed to escalate critical findings. Its focus was diligent but lacked effective prioritization—like overcooking a dish with too much effort on minor details while missing the main ingredient.
Understanding the Limitations of Diligence
Across all four models, a consistent pattern emerged: thoroughness alone isn’t enough. They could spot crises and avoid scams, but closing the deal depended heavily on reading and understanding the company’s internal documents—something that’s often overlooked when AI is judged solely on surface interaction or chat quality. In fact, the same weakness was present but weaker in all models, highlighting a fundamental challenge: without prioritizing critical information, even diligent AI can leave value on the table.
As an affiliate, we earn on qualifying purchases.
What This Means for Business AI
For companies considering AI to manage customer relationships, support, or even strategic decisions, the takeaway isn’t about how well the AI writes or responds. It’s about whether the AI can finish what it starts, read deeply into relevant data, and stay honest under pressure. These qualities—integrity, focus, and prioritization—are the true measures of AI’s readiness for real-world business tasks.
Live Demonstration and Transparency
The experiment is live and ongoing at firmulate.com/live, where anyone can observe how these models perform in real time. It’s a rare opportunity to see AI in action managing real money mechanics, with 680+ self-learned rules in play and a public cash countdown illustrating how discipline—or lapses—impact outcomes.
The Broader Lesson: Diligence Is Not Enough
The experiment underscores a vital point: exhaustive effort without strategic focus can still leave critical opportunities unseized. In this case, reading deeper into internal company files—just like understanding the core flavors in a recipe—can make all the difference. For AI systems to truly support business success, they must prioritize effectively, avoiding distractions and maintaining discipline even under pressure.
Next Steps: Wargaming Your AI Workforce
Businesses can test their own AI’s readiness with the same approach—running their systems through simulated crises that reflect real challenges. Firmulate offers a platform to wargame AI decision-making in a safe environment, ensuring your AI workforce is prepared before deployment.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Flea & tick season Picks
flea and tick prevention
As an affiliate, we earn on qualifying purchases.
