Fully offline · arm64-v8a · Android 13+ · Apache-2.0
Merged into llama.cpp PR #25701,fb0e6b6 · the KleidiAI fallback is no
longer silent →
Your phone's AI is waiting on its slowest cores.
ENTITY reads the silicon, derives the thread width, and pins the fast cluster - 18.1 tok/s where llama.cpp's own default manages 10.8, on the same phone and the same model, for 42% less battery. Fully offline, and it proves which decision earned it.
A1 - The three-arm race
The same 128-token decode, run three ways on the same phone. The naive 8-thread default finishes last; the derived thread count earns most of the gap, pinning the rest.
Naive
8 threads, all cores
11.85 s · 128 tok · 10.8±1.3 tok/s
Threads only
4 threads, no pin
8.53 s · 128 tok · 15.0±0.5 tok/s
ENTITY Auto
4 threads, pinned
7.07 s · 128 tok · 18.1±0.4 tok/s
Naive to threads-only isolates the thread count (+39% decode); threads-only to Auto isolates the pinning (+21%, the two distributions do not overlap on this device). One tick inside a bar = 16 tokens. The 1-run "+102%" figure the bench screen reports is deliberately not used here; leaderboard rules mark single-pass results provisional.
100 tok 16.4 t/s TTFT 299 ms 33.0°C 4.34 W CPU 47% 1.2 GB live stats bar - ENTITY v3.6.2
Measured, not marketed
- 18.1
- tok/s decode, up from 10.8
- +68% over llama.cpp's own 8-thread default, CMF Phone 1, 5 runs per arm
- 50 J
- per 128 tokens, down from 86 J
- 42% less battery for the same work, unplugged, integrated from the power curve
- 3.9 s
- to first token, down from 13.4 s
- 3.4x lower TTFT, same phone and model
- 0
- network calls while it runs
- the opt-in model download is the only network touch
See it running
A real assistant, fully on the phone
Screen 01
Live metrics, not vibes
Every reply streams with a live stats bar - token rate, time to first token, temperature, watts and tokens per watt - so you can see the phone working, not just the words.
Screen 02
A catalog that knows your phone
On-device files plus a download catalog that is sized and ranked for this exact SoC, with GOOD FIT / FITS and KleidiAI badges per model.
Screen 03
Nothing hidden about the model
Per-model readout: quant, KleidiAI active or not, trained context, and the running context auto-fit to your free RAM so a 3B-class model loads instead of failing.
Screen 04
Benchmark the phone in-app
The same three-arm ablation lives inside the chat app, with a sustained thermal test and cooldown before every pass.
Screen 05
Auto by default, yours to override
ENTITY tunes itself, but core placement, efficiency mode and every sampler stay in your hands as a setting, never an assumption.
Head to head
Faster than Arm's own reference app, on Arm's own silicon
One session, 2026-07-20. Same phone, same Llama-3.2-1B-Instruct-Q4_0, same
PP 512 / TG 128 workload, each app's own benchmark screen, five runs each, 30 minute cooldown
between apps so nobody inherits another's heat.
| App | Prompt (tok/s) | Decode (tok/s) | Threads |
|---|---|---|---|
| PocketPal AI | 88.32 | 13.9 | 6 |
| Arm AI Chat (Arm's own app) | 121 ± 2.99 | 12.4 ± 0.08 | not reported |
| ENTITY | 128 | 18.2 | 4, pinned |
Against Arm AI Chat: +6% prompt, +47% token generation. Against PocketPal: +45% and +31%.
Decode is where the thread-count policy acts, so decode is where the margin is. The prompt column is close because all three run Q4_0 and reach the same KleidiAI kernels - that column mostly measures whether an app got the quantization right, and here everyone did. The decode gap has a named mechanism: PocketPal runs 6 threads on a 4+4 chip, so two land on Cortex-A55s at roughly a third of an A78's throughput and every step waits on them. Both sessions, screenshots and caveats - including the July run that disagrees with this one.
And the question underneath every number above: does going faster change what the model says? Thread count fixes the order of a floating-point reduction, so it could. Measured across four scheduling arms, 8 threads and the shipped 4 produce the same tokens byte for byte and the same twelve per-chunk perplexity values - with a control proving the comparison can still detect a difference when there is one. The equivalence measurement.
The control · iOS
Is this Arm the architecture, or Arm the platform?
Every optimization here works by reading and steering the hardware - which cores are fast, what they are clocked at, where a thread lands, what the battery draws. So a fair question is whether any of it is about arm64 at all, or only about Android exposing arm64. There is a control, and it is the best one available: Apple silicon is also arm64 - same architecture, same big.LITTLE shape - and it will not let an app see or steer any of it. ENTITY Bench was ported to iOS to find out which mechanisms survive.
4
cease to exist
Live per-core frequency, ADPF deadline hints, energy telemetry, and the multi-variant
v8.0-v9.2 backend dispatch. No /sys, no public API, no loading code from
outside the bundle.
2
survive degraded
Core placement becomes a QoS request the OS need not honour and will not confirm - Android reads the applied CPU mask back. Thermal detail collapses from a temperature to four coarse levels.
4
port intact, one improves
The KleidiAI advisor and the model-fit catalog are pure logic. And Apple reports its P-core count exactly, where Android has to infer it from per-core max frequency - the one mechanism the other platform does better.
Six of ten removed or crippled - and that is the answer. The gains on this site are not arm64 being fast. They are Android exposing arm64 well enough to be steered, on an architecture worth steering. Measured on two iPhones, where the runtime picks a single thread and forcing a second costs 16.8% and 11.4%.
And KleidiAI is the one mechanism that crosses the boundary intact - 1.089× on the A18, 1.045× on the A19 Pro. Both phones report SME and SME2, extensions none of the Android fleet has, and it changes nothing: there is one static binary and no variant to select.
Real int8 ONNX inference - ONNX Runtime 1.27.0, 285 MB of staged weights hashed per file, 30 iterations counterbalanced with 5 warmup rounds, p95 and p99 recorded. Tokens per watt stays an Android-only claim: iOS exposes no battery current to an app at all. The full mechanism table.
How it works
Three moves, per phone
- 01
Reads the silicon
Scores seven Arm CPU backends against the chip at startup and loads the fastest one it supports.
- 02
Derives the threads
Ranks cores by live frequency, picks the decode width from the top cluster, pins it to the performance cores.
- 03
Proves the gain
Ships the three-arm ablation - naive, threads-only, Auto - so you see which decision earned the speedup, in watts.
Upstream
The fix for this did not stay in this project
Arm's KleidiAI ships matmul kernels for Q4_0 and Q8_0 only. Every other quantization - the whole K-quant family included - falls back to generic ggml no matter which backend variant loaded, and the fallback is silent. That is how every benchmark this project published before v2.1.0 came to use Q3_K_L, so Arm's kernels never ran at all and nothing said so.
Finding that invalidated a release. Fixing it locally would have left the next person to find it the same way, so the warning went upstream instead: llama.cpp now prints once, per weight type, when a tensor reaches no KleidiAI kernel.
The measured cost of the mistake it prevents: Q3_K_L prefills at 43.4 tok/s where Q4_0 does 128.2 on the same phone and the same model - a 2.95× prompt-path penalty for a quantization choice with no warning attached. The gate figure and the full record.
ggml-org/llama.cpp
kleidiai : warn once when a weight type has no KleidiAI kernel
- pull
- #25701
- merged
- 2026-07-21, into master
- commit
fb0e6b6
Read it on GitHub →
Why watts
Built for where the network is not
Dead zones, grid-down, rural gaps - where a cloud assistant stops working and battery is survival. So the headline axis is tokens per watt, measured per pass from battery current and voltage, and hidden whenever charging would fake it.
The same 128 tokens cost 86 J naive and 50 J optimized - by finishing in 11.8 s instead of 19.9 s.
Live board
15 devices · 13 SoCsHow much tuning earns, per chip
- 1 Samsung Galaxy S20 FE 5G · Snapdragon 865 4.27x
- 2 Google Pixel 10 · Tensor G5 3.74x
- 3 vivo I2301 · MT6886 1 run 3.70x
13 distinct Arm SoCs, and I own two of them. Everything from a Cortex-A53 Helio G37 to a Tensor G5 arrived because somebody else ran the benchmark and switched sharing on - which is the only way a claim about Arm phones in general could be tested at all.
- Fully offline
- arm64-v8a
- Adaptive threads
- Core pinning
- Tokens per watt
- KleidiAI-aware
- Runtime CPU backend
- Thermal-aware
- Open source
Two apps, one runtime
Install ENTITY, or reproduce every number with ENTITY Bench
Direct APKs with SHA-256. arm64-v8a, Android 13+, Apache-2.0. No account, no telemetry you did not switch on.
This project's own ablation disproved its own v2.0.0 headline and published the correction instead of burying it - the withdrawn claims are why the surviving numbers can be trusted.
Who built this
Kamalesh VS
A solo build for the Arm Create: AI Optimization Challenge 2026. I build things that sense, move and fly - embedded systems, computer vision and on-device AI. ENTITY is that same instinct pointed at the CPU already in your pocket: measure it, tune to it, and prove every number.