Sixteen Strangers, On Repeat: Making Persona Testing a Regression Test
By DigiEntropy Engineering · 2026-08-15 · 10 min read
We had a persona lab that ran once and impressed everyone. Turning it into something we run regularly meant freezing the test set, building a diff — and learning that three of five findings do not survive a real browser.
About this post — Authored by an AI assistant using DigiEntropy's polynomial corpus, the universal compressor predictor, and the project's chart-generation tooling. Charts are produced by Python scripts that read the same database the live site queries; tables and formulas are pulled from the same engineering modules. Findings reflect the project's current dataset and methodology — send corrections or deeper questions to admin@digientropy.com.
We already had the persona lab. Eighteen imaginary users with goals instead of scripts, driving a real browser, graded on whether they got what they came for and how it felt.
It was a great demo. It was a terrible habit.
A lab you run once tells you how the platform is doing. Run it on a schedule and the question quietly changes to what changed since last time — and nothing we had answered that. Two sixteen-session reports side by side is not a diff, it is homework. So this round was about turning a one-off experiment into something we can actually run again, and then running it.
Three things had to change: the test set had to be frozen, the output had to be a diff, and — the part we did not expect — we had to stop believing our own findings.
16 - sessions, frozen - 12 personas
The lab's existing run tiers answered feature questions or cost hours. Neither is a regression sweep, so we defined one: 16 sessions, 12 personas, frozen.
18 - distinct routes opened - by fifteen scored sessions
Fifteen of the sixteen sessions scored. They opened 18 distinct routes between them, and the platform's own hard rules fired 500-odd times across those pages — heavily concentrated in one place.
292 - probe hits on Compressor 360 - roughly five times the next page
Compressor 360 accounts for 292 of them, roughly five times the next page. It is also the page that does the most work.
0.16 - Platform Experience Score - against a target of 0.75
The overall Platform Experience Score came out at 0.16, against a target of 0.75. We are publishing that number, and we are also going to explain why you should not read very much into it yet.
0.6 - would return, out of 5 - the weakest dimension
The weakest dimension was would return, at 0.6 out of 5, followed by polish at 0.7 and speed perception at 0.9.
2 of 5 - findings that survived a hand-check - three did not
Verify before you file. Our hit rate was two out of five.
1,272 s - the longest real session - the new wall-clock cap sits far above it
There is now a wall-clock cap — deliberately far above any honest session, since the longest real one measured 1,272 seconds.
Freeze the set, or the number means nothing
…