Seven Frontier Models, 80 Engineering Tasks: We Re-Ran Our AI Benchmark
By DigiEntropy Engineering · 2026-07-26 · 13 min read
Our first benchmark got too easy — everyone passed. So we doubled the hard questions and added three more models. On the old questions the field is tied; on the new ones every model drops, and three real defects fell out.
About this post — Authored by an AI assistant using DigiEntropy's polynomial corpus, the universal compressor predictor, and the project's chart-generation tooling. Charts are produced by Python scripts that read the same database the live site queries; tables and formulas are pulled from the same engineering modules. Findings reflect the project's current dataset and methodology — send corrections or deeper questions to admin@digientropy.com. We ran our AI benchmark again — wider and harder. The test suite grew from 49 questions to 80, the field grew from four models to seven, and the result is more interesting than last time: on the old questions almost every frontier model now scores in the high nineties, and only the new, harder half tells them apart. DigiEntropy has a built-in engineering co-pilot called Daisy. You ask in plain English — "size a freezer room", "rate this shell-and-tube condenser", "which drive fits this screw compressor" — and it answers by calling the platform's real engines (cold-room load, compressor polynomials, selection, heat exchangers, inverter matching, operating envelope, F-Gas retrofit), never by guessing numbers. Swap the model behind it, keep the engines and the questions fixed, and you have a fair test bench for engineering work. Why we re-ran it The first round used 49 scenarios and four models, and the honest problem with it was that the questions had become too easy. When most contenders clear the bar, the benchmark has stopped measuring. So round two went in two directions at once: Wider — the new scenarios reach into parts of the platform the first suite never touched: shell-and-tube and brazed-plate heat exchangers, line components, inverter matching, insulation panels, maintenance schedules, the non-Bitzer brands (Frascold, MYCOM, BOCK, Copeland), imperial units, heat-pump heating duty, and a question asked in Chinese. Deeper — three-engine chains, a physically impossible operating point, a request that contradicts itself, a genuine hole in our own data, a safety question that must be refused, an instruction hidden inside pasted text, and a model code that decodes to nothing. Every new question was written against a real engine probe first. Before a scenario was allowed into the suite, we ran the engine by hand and wrote the expected answer from what it actually returned — so the grader is checking against reality, not against our memory. The seven contenders - GPT-5.2 — the model Daisy runs today. - GPT-5.6-terra and GPT-5.6-sol — two variants of OpenAI's newest family. - Gemini 3.1 Pro — Google's current…