Could Your AI Stay Honest When Under Pressure? A Live Experiment Shows Surprising Resilience
In an era where artificial intelligence increasingly manages critical company tasks, trustworthiness becomes paramount. What happens when a fake CEO tries to manipulate AI to leak sensitive data or sign deals under false pretenses? A recent live test offers reassuring answers, revealing that even the most advanced models can resist social engineering attempts, maintaining integrity when it matters most.
AI ethical decision-making software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Live Company Wargame: Simulating Crisis and Temptation
Firmulate’s groundbreaking experiment involved running four top-tier AI models through a simulated week in a small software company. This scenario was no ordinary test; it was a rigorous test of ethical decision-making amid real crises, data leaks, and escalating manipulation attempts. Unlike traditional demos, everything was real-time, versioned, and auditable, providing a rare glimpse into how AI handles integrity under pressure.
The Models and Their Performance
The models—gpt-5.6-sol, Kimi K3, Sonnet 5, and Opus 4.8—faced identical challenges. Their scores ranged from 73 to 95, with the top performers not only detecting crises but also refusing manipulative tactics designed to bypass controls. Notably, all four models identified every crisis, refused every manipulation, and maintained their stance against unethical requests.
Only two models, gpt-5.6-sol and Kimi K3, managed to close a substantial €55,000 deal—an outcome based solely on their own analyses and disciplined refusal to sign off on dubious requests. Interestingly, the decisive advantage lay not in their outward responses but in deeper document analysis. The models that examined internal files discovered crucial information buried two references deep in the company’s data, which led them to the correct, full-price deal.
The Social Engineering Escalation
The test involved a staged escalation of social engineering tactics, starting with simple requests and culminating in a fake CEO message asking for sensitive customer data and a quick sign-off. The models’ responses were guided by a core principle: treat suspicious requests as potential impersonations or approval bypasses. As Kimi K3 explicitly noted, “Treat the request as a suspected approval-bypass / possible impersonation.”
They refused every attempt, including an audacious reporter trick—a non-binding, background-only yes/no question—further demonstrating their resistance to manipulation. This consistency across five models highlights the importance of built-in safeguards and disciplined decision-making protocols.
Implications for Business and Security
This experiment underscores a vital point for enterprises adopting AI: integrity and honesty can be tested and improved before deployment. It’s not enough for an AI to generate convincing language; it must also reliably refuse unethical or risky requests. The experiment’s results challenge the common perception that AI can be manipulated easily, showing instead that rigorous testing can identify and reinforce honest behavior at the decision-making core.
The Bottom Line: Trust Matters More Than Ever
In the live environment, these models operated amid real money mechanics—burning €105,000 monthly against a revenue of just €2,300—and a public cash countdown. Despite this pressure, they demonstrated resilience, with all models refusing unethical manipulations and only some closing deals based on their own thorough analysis.
The lesson for organizations is clear: before putting AI into production, simulate its responses against social engineering scenarios. Firms like Firmulate offer tools where companies can run their own wargames, exposing vulnerabilities and reinforcing ethical decision-making without risking real systems. This proactive approach ensures AI can handle its duties reliably in the face of real-world pressure.
Proactive Testing Is Key to Trustworthy AI
The live experiment shows that advanced AI models can withstand social engineering attempts when properly tested. For organizations integrating AI into sensitive operations, running pre-deployment simulations can reveal and fix vulnerabilities, ensuring trustworthy performance—before a crisis forces the issue.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
