Lexo
← All writing

Running a 0.5B language model inside an Android keyboard

Last updated: 2026-07-26

Lexo (com.lexo.keyboard) is an Android keyboard. Its typing suggestions and its voice dictation both run on the phone, with no text or audio sent anywhere. This is a note on what it takes to fit that inside an input method, which is one of the more constrained places you can try to run a language model.

Why a keyboard is a hard place to put a model

An Android IME is not a normal app. It runs in a background-priority process that the system kills early under memory pressure, it has to paint a new suggestion strip within a couple of frames of each keystroke, and it is on screen during the most latency-sensitive thing a person does on a phone. Anything you load into it competes with the foreground app for RAM and CPU. So the interesting question was never "can a 0.5B model produce good suggestions," it was "can it do that without making the keyboard feel slow or getting the process killed."

Typing: Qwen2.5-0.5B through llama.cpp

The suggestion model is Qwen2.5-0.5B-Instruct, quantized to q8_0, a 531 MB GGUF file. It runs through llama.cpp, loaded over a custom JNI layer built for this, on CPU with a small context window (256 tokens) and four threads. q8_0 was the quant that held suggestion quality while keeping the file and the memory footprint manageable; smaller quants cost accuracy, and a larger context window costs latency for no measured benefit at this task.

The model does not run alone. Suggestions come from two tiers. A fast baseline tier (a population word-frequency prior plus a per-user personal-vocabulary store) is always available and is all you get without the unlock. The neural tier fuses the 0.5B model's next-token distribution with that baseline and with the user's own recurring vocabulary. Splitting it this way means the strip is never empty waiting on a decode, and the model's job is narrowed to the thing it is actually good at: ranking plausible next words in context.

One finding worth sharing because it surprised me: scaling the model barely moves suggestion quality. I measured Qwen2.5-1.5B-Instruct in the same harness against Qwen2.5-0.5B, and it bought about a point of accuracy on next-word top-1 (19.0% to 20.1%) and under half a point at top-5 (39.0% to 39.4%), for roughly 2.4x the decode cost. The ceiling on this task is next-word predictability of short informal text, not model size, so the 0.5B is close to the right point on the curve for a keyboard.

Voice: an offline transducer plus a learned cleanup tagger

Dictation is a separate pipeline. Audio at 16 kHz goes through a small voice-activity gate (silero VAD), then an offline NVIDIA Parakeet transducer (~0.6B, int8 ONNX) decoded through sherpa-onnx. The model is an offline one run in a pseudo-streaming loop: it re-decodes a trailing window on a roughly one-second cadence and commits text behind a short lag, which gives you live-feeling transcription without the accuracy hit of a true frame-synchronous streaming model. On a Pixel 9 Pro XL the decode runs at about a 0.05 real-time factor (a ten-second window decodes in roughly half a second), so the audio side has plenty of headroom on modern hardware.

Cleanup is where dictation usually feels magic or feels broken, and it is worth being precise about what Lexo does here: it is not a second language model rewriting your speech. A raw transcript goes through a small transformer tagger (an ELECTRA-small model with three heads, int8 ONNX, about 3 ms per turn) that tags each token for deletion (fillers, stutters, self-corrections), casing, and punctuation, followed by a deterministic rule pass (spoken punctuation, acronyms, number formatting, register detection so it does not force capital letters and periods into a terminal). The whole cleanup stage is fail-open: if it is slow or uncertain it leaves the raw transcript alone rather than risk mangling it. A generative-LLM cleanup pass is the obvious next step and is not shipped today.

The download tradeoff

All of this means a first-run download of about 1.2 GB (roughly 531 MB for the typing model, roughly 660 MB for the voice model, plus small tagger and VAD files). The models are not bundled in the app, they are pulled once from an object store on first setup over Wi-Fi, verified by md5, and stored in the app's private files directory. Bundling them made every Play Store update enormous and slow; a one-time download keeps updates small and, because the models live in the files directory, an app update never re-downloads them. For the audience that wants an on-device model, a 1.2 GB download is a fair price and a one-time one.

The privacy claim, and how to check it without trusting me

The design point is that nothing you type or say leaves the phone. You do not have to take that on faith.

The app declares exactly five permissions: RECORD_AUDIO (dictation), VIBRATE (haptics), POST_NOTIFICATIONS (download progress), INTERNET, and ACCESS_NETWORK_STATE.

INTERNET covers three things, and none of them is your text. The one-time model download. Google Play billing, for the paid unlock. And anonymous crash reporting through Firebase Crashlytics, which is off unless you turn it on at Settings > Advanced > Other > Send crash reports. Crash reports carry stack traces, the device model, and the app version. They never carry what you typed or what you said.

So point NetGuard or PCAPdroid at Lexo, and here is what you should actually expect to see. Past the model download, typing and dictating generate no traffic at all. You will see Crashlytics check in, because it is on until you turn it off. Turn it off and that stops too. I would rather tell you what the capture looks like than promise silence, have you catch a heartbeat, and leave you wondering what else I was wrong about.

The models are open-weight (Qwen is Apache-2.0, the Parakeet ASR model is NVIDIA NeMo under CC-BY-4.0), and the runtimes are llama.cpp (MIT) and sherpa-onnx (Apache-2.0), so there is nothing secret about the model side.

Honest limitations