Sixteen Strangers, On Repeat: Making Persona Testing a Regression Test
By DigiEntropy Engineering · 2026-08-15 · 10 min read
We had a persona lab that ran once and impressed everyone. Turning it into something we run regularly meant freezing the test set, building a diff — and learning that three of five findings do not survive a real browser.
About this post — Authored by an AI assistant using DigiEntropy's polynomial corpus, the universal compressor predictor, and the project's chart-generation tooling. Charts are produced by Python scripts that read the same database the live site queries; tables and formulas are pulled from the same engineering modules. Findings reflect the project's current dataset and methodology — send corrections or deeper questions to admin@digientropy.com. We already had the persona lab. Eighteen imaginary users with goals instead of scripts, driving a real browser, graded on whether they got what they came for and how it felt. It was a great demo. It was a terrible habit. A lab you run once tells you how the platform is doing. Run it on a schedule and the question quietly changes to what changed since last time — and nothing we had answered that. Two sixteen-session reports side by side is not a diff, it is homework. So this round was about turning a one-off experiment into something we can actually run again, and then running it. Three things had to change: the test set had to be frozen, the output had to be a diff, and — the part we did not expect — we had to stop believing our own findings. Freeze the set, or the number means nothing The lab's existing run tiers answered feature questions or cost hours. Neither is a regression sweep, so we defined one: 16 sessions, 12 personas, frozen. Frozen matters more than it sounds. Two runs are only comparable if they ran the same thing — change the mission list and a lower score means a harder test, not a worse site. That trap is easy to fall into precisely because the number still looks like a number. The 16 were picked for page coverage, not for any one feature. The lab's contract probes — the checks that enforce our own rules about layout, tables, price versions, envelope badges, scrollbars, and form labels — only fire on pages a persona actually opens. Coverage is a side effect of wandering. So the set is deliberately made of missions that make people wander: a first arrival from a search engine, a product-wide survey, three accessory catalogs, the same compressor read in three different places, a phone-only chain, an adversarial pass, and a keyboard-only run. That spans all five plan tiers, three viewports including a 150 % zoom, and the keyboard-only path. What one run looks like Every persona gets a goal, a patience budget, and no map. They are never told where anything is — a persona that knew the site's structure would navigate like a developer, and whether a stranger can find their way is the entire point.…