We Hired 18 Imaginary Users to Break Our Own Platform
By DigiEntropy Engineering · 2026-08-09 · 11 min read
Our tests all passed. They were written by people who already knew where to click. So we built 18 personas with goals instead of scripts, let them loose on the real site, and measured how it felt.
About this post — Authored by an AI assistant using DigiEntropy's polynomial corpus, the universal compressor predictor, and the project's chart-generation tooling. Charts are produced by Python scripts that read the same database the live site queries; tables and formulas are pulled from the same engineering modules. Findings reflect the project's current dataset and methodology — send corrections or deeper questions to admin@digientropy.com. Every test we had was green. We run four suites. Unit tests over the engineering invariants. A health check that proves the site is up. A crash sweep that opens every route and confirms nothing throws. And an evaluation harness that scores our assistant, Daisy, across 87 scenarios. All of them passed. And all of them shared one blind spot: every single test was written by somebody who already knew where to click. A test opens the Selection page, sees it render, and moves on. A purchasing manager who needs "something like a Bitzer 4VE-10Y but cheaper, available this week" opens the same page and gives up after 29 actions and ten minutes without an answer. The first outcome is a pass. The second is the business. Both happened on the same build. So we rebuilt the testing system around a different question. Not does it work — can a stranger with a goal actually get what they came for, and how did it feel? Goals, not scripts The new layer is called the User Simulation Lab. It runs 18 personas against the real platform in a real browser. Each one gets a goal, never a click path: A customer has asked you for something equivalent to a Bitzer 4VE-10Y, but cheaper and available sooner. Find them at least one alternative you would be willing to put in front of the customer. Then the agent loops: read the page, pick one human action, do it, look again. It has a patience budget and it is allowed to give up. Everything it does is instrumented and everything it says is kept. The success criteria are written the same way — as outcomes a person would recognise: - Identified at least one specific compressor model, from a brand other than Bitzer, presented by the site as an equivalent. - Saw a comparison of capacity, COP or both, at a stated operating point. - Found at least one commercial signal — a price, a supplier, or an availability indication. Not one of those mentions a route, a button, or a component. Move a button and the test still passes. Make the answer unreachable and it fails. That property is enforced by a test of its own: any success criterion containing a URL path or a component name is rejected. The roster is a…