ENTITY
Download

docs/OPTIMIZATIONS.md · the app's ABOUT screen

Every lever, with sources

What each runtime decision does, why it works, what it measured, and where it is in the source. Negative results are kept at equal weight - two of these levers were regressions and were removed or demoted, and that is stated here, not buried.

Runtime-selected CPU backend

The build compiles seven Arm CPU backend variants, armv8.0 through armv9.2, each with Arm KleidiAI micro-kernels, via stock llama.cpp GGML_CPU_ALL_VARIANTS=ON - a build decision, not hand-written kernels. At startup ggml_backend_load_all_from_path loads every variant in the APK and ggml scores them against the physical CPU; the best supported one runs. Old phones without dotprod get the armv8.0 fallback instead of SIGILL.

Source: lib/build.gradle.kts, ai_chat.cpp: init(). Cost: release APK ~9.8 MB against ~7 MB single-variant.

A3 - Backend selection at startup

Seven Arm CPU backend variants ship in every APK. At startup the CPU's feature flags are read, variants whose requirements the silicon cannot meet drop out, and the best survivor loads.

measured

CMF Phone 1 · Dimensity 7300

/proc/cpuinfo features

  • dotprod present
  • fp16 present
  • i8mm absent
  • sve absent
  • sve2 absent
  • sme absent

Shipped variants (best supported loads)

  1. android_armv8.0_1 baseline arm64 supported
  2. android_armv8.2_1 dotprod supported
  3. android_armv8.2_2 dotprod + fp16 loaded
  4. android_armv8.6_1 dotprod + fp16 + i8mm not supported
  5. android_armv9.0_1 dotprod + i8mm + fp16 + sve2 not supported
  6. android_armv9.2_1 dotprod + i8mm + fp16 + sve + sme not supported
  7. android_armv9.2_2 dotprod + i8mm + fp16 + sve + sve2 + sme not supported
Data:Variant list: llama.cpp ggml/src/CMakeLists.txt (GGML_CPU_ALL_VARIANTS, Android). Device flags: real cpu_flags rows from the contributed dataset. Eligibility shown from reported flags; ggml's startup scoring decides on-device.

Old phones without dotprod get the armv8.0 fallback instead of a crash - the Tecno Spark 10 row in the dataset is exactly that case. No device in the dataset reports SME yet, so the two armv9.2 variants have never been the loaded one. Each variant also carries Arm KleidiAI micro-kernels; whether they run is gated by the model's quantization (see A4).

Big-core affinity

build_fast_cpu_set() ranks every online core by cpuinfo_max_freq - live hardware detection, not a hardcoded mask - and pin_to_fast_cores() applies it with sched_setaffinity on every decode entry point, because ggml's worker pool spawns lazily and inherits the calling thread's mask. Each benchmark arm logs the mask the kernel actually applied, so a failed pin cannot pass as "pinning earns nothing".

Source: ai_chat.cpp: build_fast_cpu_set / pin_to_fast_cores; the ablation switch is g_pin_cores.

Measured: on the five-run 2026-07-18 exports, pinning over the same thread count earns +21% decode on the Dimensity 7300 (distributions non-overlapping) and +1% decode but median power 2.52 → 1.78 W on the Snapdragon 6 Gen 4. July's three-run sets read ~0%. Across the contributed dataset the median is +0.7% decode and +2.0% tok/W, over a range of -8.5% to +29.3% - so device-dependent that v3.5.0 made placement a user setting rather than ship any default as correct.

The mechanism figure is A2 on the overview; the full pinning story is A8 beside the leaderboard.

Adaptive thread count - two widths, two signals

Decode derives its width from the top frequency cluster: the count of cores within 10% of the fastest clock, clamped to 2-6. On the reference 4+4 that yields 4. Decode does not want width - an SM8550 in the dataset decodes 23.8 tok/s on 2 threads and 6.72 on 8 - because it is bandwidth-bound and two threads already saturate the bus.

Prefill (since v3.5.0) derives a separate, usually wider set from the kernel's own per-core cpu_capacity (1024 = strongest): every core strictly above the slowest tier, never narrower than decode. Frequency cannot rank cores - an A55 at 2.0 GHz and an A78 at 2.5 GHz are 25% apart in clock and roughly 3x apart in throughput.

Why two rules? The single clock rule was validated on the devices actually measured - and the contributed dataset then proved it wrong for prefill on prime-core flagships, where the 10% window admits only the prime core and the count collapsed to 2. The decode rule was deliberately left alone; the prefill rule moved to capacity. The full falsification is on the leaderboard page and in the journey.

Source: ai_chat.cpp: top_cluster_core_count / prompt_thread_count, mirrored by BenchmarkActivity.autoGenThreads() - one definition, called from both places, after a drift bug proved why (Journey section 4).

Adaptive context

Auto mode buckets on GGUF file size and current free RAM to choose a 2048, 4096 or 8192-token window before the model loads. Because weights are memory-mapped and paged on demand, the KV cache is the real RAM-pressure lever - which is what lets a 3B-class model run inside about 2 GB free instead of OOMing.

Source: MainActivity.adaptiveContext(), ai_chat.cpp: init_context() (reads back what llama.cpp actually allocated).

A5 - Adaptive context sizing

Drag model size and free RAM: the running context snaps between 2048 / 4096 / 8192 by the shipped rule, keeping the KV cache under the memory ceiling - and a fixed 8192 window shows the failure it prevents.

measured
2048 4096 8192

Auto: KV cache at the derived window fits under the ceiling

Fixed 8192 window (what a hardcoded default would do) -

red line = free-RAM ceiling. Weights are memory-mapped and paged on demand, so the KV cache - sized by this rule - is the actual RAM-pressure lever.

Anchor: Llama-3.2-1B-Instruct-Q4_0, 773 MB, trained context 131,072 → running context 4096 (auto-fit to free RAM) at 1.2 GB free - the shipped model-info card's exact state.

Data:Selection function: the shipped adaptiveContext() code, docs/OPTIMIZATIONS.md section 0. Anchor state: the model-info screenshot (773 MB model, trained context 131072, running context 4096 auto-fit, 1.2 GB free). Bar proportions are schematic.

In Auto mode only; manual mode uses the user's chosen context verbatim. The smaller window is an admission decision: it cuts KV-cache demand before a 3B-class model exhausts the remaining memory, which is what lets one run inside about 2 GB free.

Thermal-aware guard

Every eighth generated token, Auto mode asks Android's thermal status (cached for one second) and inserts a cooperative delay: 0 ms at NONE or LIGHT, 6 ms at MODERATE, 12 ms at SEVERE and above. Efficiency mode doubles the delay and caps inference at two threads. It is a soft back-off - no affinity change, no realtime request, no root-only thermal zones.

Source: ChatViewModel.ThermalGuard.delayMs().

A6 - The thermal guard

A small cooperative delay between tokens holds a flat sustained rate instead of running hot and hitting a hard throttle mid-answer; past the crossover, the guarded run has produced more total tokens.

schematic
0.50 0.75 1.00 0m1m2m3m4m5m crossover ~2.5 min cumulative tokens now favour the guard unguarded guarded hard throttle ahead 0-12 ms cooperative delay

y: decode rate relative to the cool-start rate · legend: solid bright = guarded, solid dim = unguarded · both lines schematic

Schematic:Schematic of the policy, not a measurement - no sustained-run export exists in benchmarks/results/ yet. The policy is shipped and real: 0 / 6 / 12 ms by Android thermal status, every 8th token, doubled in Efficiency mode. Run ENTITY Bench's SUSTAINED mode to measure it on your phone.

The guard is cooperative, not a scheduler control: it inserts the delay itself rather than changing affinity or clocks, giving the OS a recovery opportunity before a hard system throttle. Whether the crossover lands at this point on a given phone is exactly what the bench app's sustained test measures.

KV session cache

Each conversation's decoded state is rebuilt on demand: switching to or restoring a persisted chat primes the native KV through the same chat-template path as live turns ("Preparing conversation..."), so the next reply continues seamlessly instead of starting from a cold cache. Long conversations never hard-stop: on overflow the oldest half of the turns is trimmed and the system prompt always survives.

Source: ai_chat.cpp: primeHistoryNative / shift_context.

mmap weights

GGUF weights are memory-mapped, paged in on demand rather than fully resident. RAM pressure therefore comes from the KV cache, which the adaptive-context rule sizes - the pairing that lets a 773 MB model report 1.2 GB free while generating.

Source: llama.cpp mmap loading, used as shipped.

Energy telemetry

watts = |battery current| x voltage / 1e9, sampled every 150 ms during benchmark passes and once per render tick in chat. Both OEM current units are disambiguated by a plausibility window, and since v3.6.0 the voltage unit is normalised first - a contributed device reported whole volts where millivolts are documented and broke watts by a factor of a million. Power and tok/W hide entirely while charging.

Source: MainActivity.snapMetrics(), PowerMath, BenchmarkActivity.

Predicting a model before downloading it

The catalog used to rank models on RAM and ISA flags, which answers "will it fit" and not "will it be usable". A 4 GB phone could be told a 3B model fits and still find it decodes at 3 tok/s - after a 2 GB download. That is the worst possible way to learn it.

DeviceProbe answers it in about 60 ms with no model present, because the two phases of inference are bound by things that can be measured directly. Decode reads essentially every weight once per token, so tok/s ~= bandwidth / model_bytes - a physical relationship, not a fitted curve. It is measured with a 32 MB arraycopy, large enough that no phone's last-level cache can hold it, so the number is DRAM rather than SRAM. Prefill is a GEMM over the whole prompt and tracks integer throughput on the performance cores, measured with a dependency-chained multiply-accumulate loop - integer rather than float on purpose, since a float benchmark would rank a phone with strong FP and weak integer units far too highly.

Both readings are scaled against one calibration device measured properly with llama-bench, which is also what lets the estimate carry the Q4_0/Q8_0 trade instead of treating "reaches KleidiAI" as a single good thing.

Source: DeviceProbe.kt, ModelCatalog.assess. Anchor: CMF Phone 1, Dimensity 7300, Llama-3.2-1B-Instruct, 4 threads pinned.

Sized against free memory, not installed RAM. A 6 GB phone with a browser and a few background apps resident has well under 2 GB to give, and a rule written against 6 GB recommends models into memory that is already spoken for. The thresholds were re-derived rather than reused - fractions calibrated against total would reject models that genuinely run. A model is flagged tight above 70% of free memory rather than rejected, because weights are mmap'd page cache the kernel can evict; the KV cache is the anonymous memory that actually has to fit, and it is the smaller of the two. The cost is stated plainly on screen: the verdict moves with whatever else is running, because the phone really does have less room when it is busy.

This is an estimate and says so. It cannot see thermal headroom, memory pressure from other apps, or the vendor's scheduler. It exists to keep a user off a model their phone cannot serve, not to replace a benchmark - the real three-arm ablation still ships in both apps.

Two of its anchor constants were originally derived rather than measured, and were wrong by 4.1x and 2.5x. That is entry 10 on the journey page.

The quantization gate, and why decode does not care

A4 - The KleidiAI quantization gate

A GGUF's header names its quantization; that one field decides whether Arm's KleidiAI kernels run at all. Q4_0 and Q8_0 route to them - everything else drops to ggml's other CPU paths, and until PR #25701 nothing told you.

measured
gguf

model-Q4_0.gguf

header read

general.file_type

KleidiAI kernels Q4_0 · Q8_0 only

Arm-written matmul micro-kernels for the compute-bound prompt GEMM.

ggml CPU path - no KleidiAI

Not bare scalar code: K-quants still reach ggml's Arm dotprod repack kernels. What they miss is KleidiAI - and nothing said so until the upstream warning landed.

Q4_0 reaches Arm's fast path. The app's model card reports this state as "KLEIDIAI ACTIVE".

Measured cost of missing the fast path: Q3_K_L vs Q4_0, same phone and config
Metric Q3_K_L (733 MB, no fast path) Q4_0 (773 MB, fast path) change
Prompt throughput 42.7 tok/s 121 tok/s +183%
Derived TTFT (512-token prompt) 12,050 ms 4,299 ms -64%
Decode throughput 16.9 tok/s 14.7 tok/s -13%
Data:Kernel gate verified in ggml/src/ggml-cpu/kleidiai/kleidiai.cpp. Cost table: CMF Phone 1, same 512-token prompt, same 4-thread unpinned config, only the quantization differs - docs/KLEIDIAI-QUANTS.md.

The fallback used to be silent - no log line, nothing in logcat. This project contributed the one-time warning upstream: llama.cpp PR #25701, merged 2026-07-21 as commit fb0e6b6.

A7 - Prefill versus decode

Prompt processing is one big compute-bound GEMM - kernels matter. Decode streams the whole weight matrix from memory for every single token - bandwidth matters, kernels mostly cannot.

schematic

PREFILL - 512 tokens, one large GEMM

bottleneck: arithmetic - a better kernel (KleidiAI, i8mm/SMMLA) directly helps

DECODE - one token at a time

bottleneck: memory bandwidth - tracks bytes per weight; kernel quality cannot help a workload waiting on DRAM

Schematic:Mechanism panels are schematic. The conclusion is measured: moving Q3_K_L to Q4_0 on the CMF Phone 1 wins prompt +183% and loses decode -13% (docs/KLEIDIAI-QUANTS.md), because decode tracks bytes per weight, not kernel quality.

This split is why the Q4_0 move wins prompt throughput by +183% and TTFT by -64% while decode gives up 13% (Q4_0 carries ~6% more bytes than Q3_K_L) - and why decode gains come from thread placement and count, not from kernels.

A file named Q4_0 is only 76% Q4_0

The gate above is a claim about types. Checking it properly meant reading each file's tensor table instead of its general.file_type label — and the label is wrong about its own contents. In the exact Llama-3.2-1B Q4_0 the catalog ships, token_embd.weight is Q6_K and two ffn_down tensors are Q4_1. Llama 3.2 ties its embeddings, so that Q6_K tensor is also the output projection: the largest matmul in the model sits off the fast path, in a quantization chosen because it reaches the fast path.

Quality, size, KleidiAI coverage and measured speed by quantization
Quant Perplexity vs F16 Size On KleidiAI Prefill Decode
F16 reference14.2580 —2480 MB n/a——
Q8_014.2705 +0.09%1321 MB 100% 208.111.5
Q4_K_M14.7346 +3.34%808 MB 0%111.716.0
Q4_0 imatrix (shipped) 15.6159+9.52% 773 MB76% 128.218.2
Q3_K_L16.0927 +12.87%733 MB 0%43.417.4
Q4_0 no imatrix16.5272 +15.92%771 MB 79% 138.819.1

Perplexity: wikitext-2 test, 200 chunks, every quant built from one F16 source. Speed: CMF Phone 1, four threads pinned to the A78 cluster, three repetitions, 90 s cooldown between models. Coverage read from each file's tensor table, not its label.

Negative results, at equal weight

Two of the ideas above shipped, measured slower, and were removed or demoted. They stay on this page at the same size as the wins.