Contributed dataset · 15 devices · 13 SoCs

How much tuning earns, per chip

One row per device and quantization: the decode multiplier ENTITY earned over the naive 8-thread default on that silicon. It answers "how much does tuning earn on this chip", not "which phone is fastest". Anyone can add a row - run ENTITY Bench unplugged and switch on sharing.

snapshot · 26 submissions

Sort

Vendor Quant KleidiAI ISA valid power only

Contributed ENTITY Bench results, one row per device and quantization
Device	SoC & model	Naive tok/s	Tuned tok/s			Runs

Contributed ENTITY Bench results (static snapshot), sorted by tuning multiplier
Device	SoC & model	Naive tok/s	Tuned tok/s	Multiplier	tok/W	Runs
Samsung Galaxy S20 FE 5G 2026-07-24	Snapdragon 865 Q4_0	5.3	22.8	4.27x	5.00	3
Google Pixel 10 2026-07-22	Tensor G5 Q4_0	7.4	27.8	3.74x	5.01	3
vivo I2301 2026-07-24	MT6886 Q4_0	4.3	16.0	3.70x	3.46	1 (1 run)
Samsung Galaxy S22 Ultra 2026-07-23	Snapdragon 8 Gen 1 Q4_0	5.1	18.1	3.56x	3.81	3
Samsung Galaxy S23 2026-07-23	Snapdragon 8 Gen 2 Q4_0	6.7 (high sd)	23.8	3.54x	3.81	3
Samsung SM-A146B 2026-07-25	s5e8535 Q4_0	4.9	14.3	2.92x	5.31	1 (1 run)
OPPO CPH2737 2026-07-23	Dimensity 8300 Q8_0	24.2	52.3	2.16x	-	1 (1 run)
Nothing CMF Phone 1 2026-07-24	Dimensity 7300 Q4_0	8.9	18.1	2.02x	4.30	1 (1 run)
OPPO CPH2737 2026-07-24	Dimensity 8300 Q4_0	18.3	33.3	1.82x	-	1 (1 run)
OPPO CPH2553 2026-07-24	Dimensity 700 Q4_0	7.8	14.0	1.81x	4.36	1 (1 run)
Samsung SM-M366B 2026-07-27	s5e8835 Q4_0	9.4	16.6	1.76x	3.94	1 (1 run)
Nothing CMF Phone 1 2026-07-22	Dimensity 7300 Q8_0	15.4	25.9	1.68x	7.22	1 (1 run)
OPPO CPH2729 2026-07-25	SM6650 Q4_0	9.3	14.9	1.60x	4.14	1 (1 run)
Samsung SM-A235F 2026-07-25	SM6225 Q4_0	4.9	7.8	1.57x	2.61	1 (1 run)
TECNO Spark 10 2026-07-24	Helio G37 Q4_0	3.4	4.6	1.34x	1.77	1 (1 run)

How to read this board

Power and tok/W never render from a charging run - a charging phone reports the charger's current, not the workload's. Rows whose arms report physically inconsistent watts are flagged and excluded from power columns; the raw values stay in the detail panel.
n is always shown. A 1-run row is a single pass and provisional: two back-to-back single passes on the same phone, model and quant have disagreed by up to 19.6%.
Arms with decode relative SD above 25% are flagged and excluded from every aggregate. One Galaxy S23 naive arm measured 6.72 ± 5.95 tok/s, an 88.5% relative SD - noise, not a measurement.

One row per device + quantization, newest first, so a phone that submitted six times does not dominate. Older runs expand inside the row; byte-identical re-uploads are collapsed and counted.
Q8_0 rows are Qwen2.5-0.5B and Q4_0 rows are Llama-3.2-1B - never compare numbers across models; each row's multiplier is within-row and safe.
The multiplier is not pinning earning energy. On the axis that isolates pinning the median is +0.7% decode and +2.0% tok/W, positive on 9 of 15 rows - small in the middle, wide at the edges (-8.5% to +29.3% decode). The thread count is the lever that generalises.

What the dataset established, and what it falsified

Established

One binary spans a 7.3x range of silicon, and the ISA tiers are visible in the crowd data. The same APK and the same Llama-3.2-1B-Instruct-Q4_0 file run from 4.58 tok/s on a Helio G37 to 33.3 on a Dimensity 8300 - 19 contributed runs, no recompile, the backend variant chosen at startup. Split those runs by what the CPU reports:

Median throughput by reported Arm ISA tier

Reported ISA n Prompt Decode

no extensions 2 9.9 6.2

dotprod 9 90 16.6

dotprod + i8mm 8 106 21.3

Medians, tok/s. Crossing the dotprod line is worth 9.1x on prompt but only 2.7x on decode - and that asymmetry is the part that means something. A chip that were simply older and slower would drag both axes down together; prompt moving 3.4x harder than decode is the signature of a change in the integer matmul kernel specifically, which is what dotprod is. It is the same compute-bound / bandwidth-bound split the predictor is built on, arriving independently from devices this project does not own. Boundary: only two contributed devices report no extensions at all, so the 9.1x carries every other difference between those chips and the rest with it. The asymmetry survives that; the absolute multiplier should not be quoted alone.
Thread-count tuning pays on every SoC measured: 1.34x to 4.25x, and no device has regressed (22 rows, 9 SoCs). That range is the thread step alone - naive to threads-only. The multiplier column in the table above is the full naive-to-optimized ratio, so it reads slightly higher (4.27x on the same device); the gap between the two is what pinning contributed. It holds at both ends of the range - on SVE2 flagship silicon and on an all-Cortex-A53 Helio G37 reporting no Arm ISA extensions at all. The widest multiplier is a Snapdragon 865, which has no i8mm: the lever is scheduling, not a kernel.
Pinning is device-dependent in both speed and energy, and the sign is not predictable from the spec sheet. Median +0.7% decode and +2.0% tok/W, but the Pixel 10 buys +29.3% decode for +33.5% power (tok/W falls) while a Dimensity 7300 gets +24.0% decode for 7.9% less power (tok/W rises 34.4%). A setting, not a default, since v3.5.0.
The gain tracks core heterogeneity, but clock spread is the wrong predictor. A near-flat Dimensity 700 gets 1.79x and a wide-clocked Snapdragon 865 gets 4.25x - yet the Helio G37 has a flagship-like clock spread and gains only 1.34x, because all eight of its cores are Cortex-A53. What matters is relative core strength; cpu_capacities is the better candidate and is not yet tested against this table.

Median throughput by reported Arm ISA tier
Reported ISA	n	Prompt	Decode
no extensions	2	9.9	6.2
dotprod	9	90	16.6
dotprod + i8mm	8	106	21.3

Falsified

OPTIMIZATIONS.md predicted a flagship with more than four performance cores would derive 5 or 6 generation threads. The dataset showed the opposite: the rule counted cores within 10% of the fastest clock, and every modern flagship puts its prime core 17-20% above its own big cluster - so the count collapsed to 1 and was clamped to 2. A Dimensity 7300 prefilled at 139 tok/s while an SM8550 with i8mm managed 111. Fixed in v3.5.0 with cpu_capacity.

Cores passing the 10% clock window per SoC
SoC	top MHz	pass 10%	after clamp
Dimensity 7300	2500	4	4
Tensor G5	3782	1	2
SM8550	3360	1	2
SM8450	2995	1	2

A8 - Pinning is a speed lever with a power cost

Threads-only vs optimized run the same thread count; affinity is the only difference. Flip it per device: decode and watts move together, and tokens per watt often moves the other way.

measured

Pixel 10

Tensor G5 · 3 runs, contributed

decode: +29.3%
watts: +33.5%
tok/W: -3.2%

CMF Phone 1

Dimensity 7300 · 5 runs, 2026-07-18 export

decode: +20.7%
watts: -3.2%
tok/W: +25.4%

OPPO CPH2729

Snapdragon 6 Gen 4 · 5 runs, 2026-07-18 export

decode: +0.6%
watts: -29.4%
tok/W: +44.9%

Galaxy S23

Snapdragon 8 Gen 2 · 3 runs, contributed

decode: +1.7%
watts: +1.1%
tok/W: +0.2%

Data:Pixel 10 and Galaxy S23: contributed dataset, 3 runs per arm (benchmarks/CONTRIBUTED-DATA.md). CMF Phone 1 and OPPO CPH2729: five-run four-arm exports, 2026-07-18 (benchmarks/BENCHMARKS.md).

On the axis that isolates pinning (threads-only to optimized), the contributed dataset's median is +0.7% decode and +2.0% tok/W, positive on 9 of 15 rows - but the range runs -8.5% to +29.3% and both ends are real: one phone pays 33.5% more power to go 29.3% faster, another goes 24.0% faster on 7.9% less power. Nothing knowable before running it says which you have. That is why v3.5.0 made core placement a user setting instead of an assumption, and why the app reports the measurement from the user's own phone.

Privacy, in the app's own words

Contribution is off by default - nothing is sent until the toggle is on.
Summary numbers only. The per-pass 150 ms telemetry trace never leaves the phone.
No accounts, no identifier that links two runs from the same phone.
Every submission records whether the phone was charging, because power readings taken while charging are the charger's, not the workload's.
A "Show exactly what gets sent" screen renders the real JSON before the first upload.

This page reads that same public, non-identifying dataset over PostgREST with the publishable key. The key can only append rows and read them; the row-level-security policy is documented in benchmarks/CONTRIBUTE-BACKEND.md.

Contribute a row

Install ENTITY Bench (arm64, Android 13+).
Import or download a GGUF, unplug the phone, run the 3-arm benchmark.
Settings → Contribute results → enable sharing.

Get ENTITY Bench