We Hired 18 Imaginary Users to Break Our Own Platform
By DigiEntropy Engineering · 2026-08-09 · 11 min read
Our tests all passed. They were written by people who already knew where to click. So we built 18 personas with goals instead of scripts, let them loose on the real site, and measured how it felt.
About this post — Authored by an AI assistant using DigiEntropy's polynomial corpus, the universal compressor predictor, and the project's chart-generation tooling. Charts are produced by Python scripts that read the same database the live site queries; tables and formulas are pulled from the same engineering modules. Findings reflect the project's current dataset and methodology — send corrections or deeper questions to admin@digientropy.com.
Every test we had was green.
We run four suites. Unit tests over the engineering invariants. A health check that proves the site is up. A crash sweep that opens every route and confirms nothing throws. And an evaluation harness that scores our assistant, Daisy, across 87 scenarios.
All of them passed. And all of them shared one blind spot: every single test was written by somebody who already knew where to click.
A test opens the Selection page, sees it render, and moves on. A purchasing manager who needs "something like a Bitzer 4VE-10Y but cheaper, available this week" opens the same page and gives up after 29 actions and ten minutes without an answer. The first outcome is a pass. The second is the business. Both happened on the same build.
So we rebuilt the testing system around a different question. Not does it work — can a stranger with a goal actually get what they came for, and how did it feel?
18 - personas - against the real platform in a real browser
The new layer is called the User Simulation Lab. It runs 18 personas against the real platform in a real browser. Each one gets a goal, never a click path.
52 - goal-driven sessions - every persona against every mission that suits them
52 sessions. Every persona against every mission that suits them.
11 - got what they came for - 34 gave up, 7 ran out of patience
Eleven of 52 people got what they came for. Thirty-four gave up. Every one of those sessions ran against a platform whose entire automated test suite was green.
0.62 - would return, out of 5 - the lowest experience score
That is not a comfortable chart, and it is not meant to be. The lowest score is would return at 0.62 out of 5.
0.17 - gap between correctness and fit - on the same answers
Correctness 0.855. Fit 0.682. A 0.17 gap on the same answers. Daisy was technically right and humanly wrong.
0.727 - fit after the kept change - up from 0.682
Fit went 0.682 to 0.727, and the identical prompt reproduced 0.727 exactly on a repeat run. Correctness on the original 87-scenario suite stayed at 0.944 with zero fabrications.
69 s - to real specifications on production - Bilal, on a phone
…