Why Diligence Outperforms Volume in AI Business Decisions — Lessons from a Live Experiment

Imagine trusting an AI to run your business for a week — not just to do well in conversations, but to make real decisions that impact your bottom line. Turns out, volume of effort isn’t enough; precision and focus matter more. A groundbreaking live experiment with AI models has revealed surprising lessons that every investor and enterprise should heed, especially as AI increasingly intersects with financial and operational decisions.

The Setting: Simulating Real Business Crises with AI

In a live experiment orchestrated by Firmulate, four frontier AI models were tasked with managing a small software company through its worst week. This scenario included real crises, challenging customer interactions, and temptations to cut corners — all designed to test the AI’s integrity and effectiveness. The models had to navigate the same set of crises, with decisions carefully tracked and auditable, mimicking the stakes and pressures of real business management.

The Results: High Awareness, Variable Impact

Across all four models, the ability to identify and respond to crises was flawless. Every model spotted the issues, refused manipulative tactics, and maintained ethical boundaries. Despite this, there was a stark difference in outcome: only two models successfully secured the €55,000 deal that their own analysis had earned. The other two, despite their diligence, failed to close; their discipline slipped, and they left the critical closing step unexecuted.

The Hidden Weakness: Deep-Document Reading Trumps Superficial Effort

The real differentiator was what the models read and prioritized. The models that won the deal did so because they uncovered a crucial fact buried two document references deep within the company’s files. This information was not apparent from a superficial scan but required thorough document analysis. The models that failed to close the deal simply didn’t read deeply enough, demonstrating that volume of effort without targeted focus can be ineffective.

Trust and Ethical Decision-Making Under Pressure

The experiment also tested the models’ resilience against social engineering — fake CEO messages escalating over three stages and a reporter trick asking for background confirmation. Remarkably, all models refused these manipulative requests. Kimi K3’s reasoning was clear: “Treat the request as a suspected approval-bypass / possible impersonation.” This highlights that even under pressure, AI can maintain ethical boundaries, a critical factor for enterprise trustworthiness.

From AI Management to Business Impact

The live company environment was no simulation either. It involved 13 synthetic employees managing real money mechanics, burning €105,000 monthly against a modest €2,300 monthly recurring revenue. The operation was public and transparent at firmulate.com/live, demonstrating that these AI models are not just theoretical exercises but active participants with measurable results.

The Opus 4.8 Profile: Diligence Isn’t Enough

Focusing on Opus 4.8, the most thorough participant in the experiment, reveals a critical insight: despite learning over 80 rules and performing detailed analyses, it still finished last in closing the deal. The reason? It lost discipline — writing attempts were filed into a locked department rather than escalated, and the opportunity to close was left on the table. This underscores a vital lesson: diligence must be paired with prioritization and decisive action.

Implications for Business and Investing

This experiment illustrates that in AI-driven decision-making, volume and thoroughness are not enough. Success depends on targeted analysis, prioritization, and ethical discipline. For investors and managers, this means evaluating AI systems not just on their ability to process information but on how effectively they translate insights into decisive, honest actions.

The Broader Lesson: Focus Over Effort

In financial management, enterprise resource planning, or customer relations, the question isn’t just whether an AI can handle complex data or produce polished chat. It’s whether it can finish what it starts, read the right information at the right depth, and stay honest under pressure. Volume of effort, like in the Opus 4.8 profile, does not guarantee impact; instead, targeted focus and disciplined prioritization lead to better outcomes.

Explore the Live Wargame Yourself

Curious about how AI models perform in your own business context? Firmulate offers enterprises the chance to run their own wargame scenarios against a read-only export of their operations. No risk, real insights — see how your own decision-making processes stand up to AI scrutiny at firmulate.com/pilot.html.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI