What we believed, what broke it, what it cost

What broke it

A three-arm ablation. v2.0.0 measured a large decode gain after adding sched_setaffinity and credited it to core affinity - it was the project's headline. Adding a middle arm, the tuned thread count with affinity switched off, showed the thread count was carrying almost all of it. Affinity was a small residue, and on some devices a negative one.

What it cost

The headline claim of the first real release, and a rewrite of the benchmark to have three arms instead of two.

What replaced it

The naive / threads-only / optimized ablation that everything since has used, and a rule the project has kept: an optimization that has not been isolated has not been measured. Two configurations that differ in two ways cannot attribute anything.

What broke it

Measurement on the reference Dimensity 7300: PP 512 on Llama-3.2-1B-Q4_0 runs at 116 tok/s on 4 threads and 86 tok/s across all 8. An A55 is roughly a third of an A78, so the widened pool finished its share late and every GEMM waited on the stragglers. Textbook, and wrong here.

What it cost

A feature that had shipped and been described as an optimization was removed.

What replaced it

n_pp = n_gen. Which fixed the 4+4 case - and quietly created entry 5 below.

What broke it

Reading the ggml source rather than repeating the claim. Q4_K, Q5_K and Q6_K do have Arm dotprod repacking GEMM paths. The real gap is narrower and more specific: i8mm/SMMLA.

What it cost

A documentation claim that was flattering and false. It had propagated into several files.

What replaced it

KLEIDIAI-QUANTS.md, which states per quantization what actually reaches which kernel - and marks the i8mm question as untestable on the hardware available at the time rather than guessing.

What broke it

A Galaxy S26 Ultra export from a stranger. BenchmarkActivity.autoGenThreads() had restated the topology rule as availableProcessors() - 2 instead of calling it. When the clamp moved from 4 to 6 the two drifted: on a 2+6 flagship the native side derived 2 threads while the benchmark's copy returned 6 - so the threads-only arm ran six unpinned threads against an Auto arm running two. Two variables between arms: the exact attribution error the three-arm design exists to prevent.

What it cost

Every CSV that device produced recorded the wrong thread count for both arms.

What replaced it

One definition, called from both places. And a habit: a rule that is restated will drift; a rule that is called cannot.

What broke it

Contributed benchmarks from four SoCs nobody here owns. The derivation did the opposite of the written prediction. It counted cores within 10% of the fastest clock, and every modern flagship puts its prime core 17-20% above its own big cluster - so the count collapsed to 1, and only the minimum clamp pulled it back to 2. Because prefill inherited that number, prompt processing ran on two threads on every flagship: a Dimensity 7300 prefills at 139 tok/s while an SM8550 - stronger silicon, with i8mm - manages 111. Both development phones were immune: the CMF is a flat 4+4 and the OPPO's prime sits only 4.3% above its big cluster. The bug was invisible on every device the author owned.

What it cost

A published prediction had to be rewritten as a falsification, and prefill on every flagship had been degraded for several releases.

What replaced it

Decode and prefill now derive different widths from different signals - cpu_capacity for the performance cluster, clock for the narrow decode set. The decode rule was deliberately left alone, because it is the only one validated against devices actually measured.

What broke it

Isolating the right axis. The optimized arm showed 2.4x-3.4x better tokens per watt than naive, and pinning looked like the thing earning it - but optimized-vs-naive changes two things. On the axis that isolates pinning, threads-only vs optimized, the median across 15 rows is only +0.7% decode and +2.0% tokens per watt. The Pixel 10 shows the cost clearly: +29.3% faster, +33.5% more power, tok/W falls 3.2%. The wider dataset later corrected this correction too - a Dimensity 7300 gets +24.0% decode at 7.9% less power, so pinning can earn energy; it just cannot be assumed to.

What it cost

The energy story had to be re-attributed from pinning to thread count - the same correction as entry 1, on a different metric, weeks later.

What replaced it

Core placement became a user setting rather than an assumption, and the app now reports the measurement from the user's own phone instead of asserting a default.

What broke it

It was showing it. An OPPO CPH2737 reported 2.7 microwatts of decode and 11 million tokens per watt - not missing, absurd. And the cause was not where anyone would look: PowerMath's careful microamp/milliamp heuristic was fine. The voltage was wrong. EXTRA_VOLTAGE is documented in millivolts and that device reports whole volts, so both candidate wattages fell below the plausibility floor, the heuristic gave up, and returned the documented unit. Two independent 1000x errors compounding into 1e6.

What it cost

Six contributed rows with unusable power. Two of them had been marked power_valid = true, so the only-trustworthy-power view was serving nonsense.

What replaced it

normalizeVoltageMv() resolves the voltage unit before the current unit, and a rule: when a derived value is absurd, check every input's unit, not just the suspicious one. A plausibility heuristic is only as good as what you hand it, and it fails silently.

What broke it

Small, but it belongs here. The phone in entry 7 was reported as a Snapdragon. It reports mt6897 - a MediaTek Dimensity 8300. The bug it exposed is an OEM kernel unit quirk, so attributing it to the wrong vendor would have sent the next person looking in entirely the wrong place.

What it cost

Nothing but embarrassment - caught before it shipped anywhere that mattered.

What replaced it

Read the device's own report, not the spec sheet you remember.

What broke it

Reading the tensor table instead of the file_type label showed the shipped Llama-3.2-1B-Q4_0 is only 76% Q4_0 - token_embd.weight is Q6_K, and because Llama 3.2 ties its embeddings that tensor is also the output projection, the largest matmul in the model. Rebuilding it to Q8_0 took coverage to 97% for 8.2% more file. The prediction was written down first: prefill faster, decode slower. Measured on the reference phone, three reps, pinned: prefill 125.69 to 121.84 tok/s, decode 17.99 to 15.94. Slower on both.

What it cost

A day, and a recommendation that would have shipped a worse model to every user if the prediction had been trusted instead of tested.

What replaced it

llama.cpp computes logits only for a prompt's last position, so the output projection runs once per prefill rather than once per token - promoting it buys nothing there, and in decode a heavier tensor is a straight loss. Coverage is not throughput. The rebuilt file is published as a negative result and kept out of the catalog; the app now reports the measured fraction instead of asserting a boolean from a filename.

What broke it

Running the probe on the phone it was calibrated against. Two denominators had been filled in by derivation rather than measurement: memory bandwidth from what decode actually achieves (14 GB/s, discounted to 6.4), and integer throughput from what four Cortex-A78s ought to sustain (2.6 ops/ns). The probe reads 26.2 GB/s - a linear arraycopy sees near-peak DRAM while decode walks many separate tensors - and the true integer rate is 1.042, because a dependency-chained scalar MAC runs nowhere near a core's peak. Decode estimates came out 4.1x too fast, prefill 2.5x too slow. A 360M model was projected at 251 tok/s.

What it cost

Nothing shipped, but the feature was complete, reviewed, and passing 72 unit tests in that state - the tests assert internal consistency, and both wrong constants were perfectly consistent with themselves.

What replaced it

Both constants now carry the reading they came from, marked as the probe's own output on a known device rather than a datasheet or a derivation. The rule underneath: a ratio is only dimensionless if both sides were measured the same way. Substituting a physically-related quantity for one side rescales everything downstream, and no amount of self-consistency testing can see it.

What broke it

Preparing v3.7.0 on a different machine. The keystore that signed every release from v3.0.0 to v3.6.2 was not on it, and not anywhere that could be checked: no .jks on the whole drive, nothing in either repo's history, no CI secret holding it, nothing in the recycle bin or cloud storage. BUILD.md said the keystore lives outside the repo so it can be managed securely, and that sentence had been treated as a plan. Outside the repo turned out to mean not backed up at all - the .gitignore entry was doing the whole job, and a .gitignore entry is not a backup strategy.

What it cost

The signing identity, permanently. Every escape route fails for the same reason: a released APK carries only the public certificate, Android's key rotation requires the old key to sign the rotation proof, and distribution here is GitHub rather than Play, so Google never escrowed a copy. Anyone installing fresh is unaffected; anyone already on v3.6.2 must uninstall before upgrading, losing saved conversations.

What replaced it

A new keystore, and the rule the old sentence should have contained: outside the repo has to name where, in three places - a password manager, an encrypted cloud copy, and an offline copy. A secret that exists in one location on one machine has the durability of that machine, and no warning fires when it stops existing. The shape is the same as entries 7 and 10: the thing that failed was never measured. A backup nobody has restored from is not known to be a backup, in the same way a constant nobody read off the device is not known to be a measurement.