Fitting a language model inside an Android keyboard: the 0.5B design, and what ships now
Lexo (com.lexo.keyboard) is an Android keyboard. Its typing suggestions and its voice dictation both run on the phone, with no text or audio sent anywhere. This is a note on what it takes to fit that inside an input method, which is one of the more constrained places you can try to run a language model.
Lexo is the keyboard described here. Thirty days free, then $9.99 once. Get it on Google Play
How to read this. The first half is the original design, built around a 0.5B model, and the reasoning that still holds. What runs today says what replaced it, language by language. I have kept the history because how it was built is the part worth reading.
Why a keyboard is a hard place to put a model
An Android IME is not a normal app. It runs in a background-priority process that the system kills early under memory pressure, it has to paint a new suggestion strip within a couple of frames of each keystroke, and it is on screen during the most latency-sensitive thing a person does on a phone. Anything you load into it competes with the foreground app for RAM and CPU. So the interesting question was never "can a 0.5B model produce good suggestions," it was "can it do that without making the keyboard feel slow or getting the process killed."
The first design: Qwen2.5-0.5B through llama.cpp
The first suggestion model was Qwen2.5-0.5B-Instruct, quantized to q8_0, a 531 MB GGUF file. It ran through llama.cpp, loaded over a custom JNI layer built for this, on CPU with a small context window (256 tokens) and four threads. q8_0 was the quant that held suggestion quality while keeping the file and the memory footprint manageable; smaller quants cost accuracy and a larger context window costs latency for no measured benefit at this task.
The model did not run alone. Suggestions came from two tiers. A fast baseline tier (a word-frequency prior plus a per-user personal-vocabulary store) filled the strip while a decode was in flight. The neural tier fused the neural model's next-token distribution with that baseline and with the user's own recurring vocabulary. Splitting it this way means the strip is never empty waiting on a decode, and the model's job is narrowed to the thing it is actually good at: ranking plausible next words in context. Both tiers sit behind the paid unlock: unpaid, there is no suggestion strip at all.
One finding worth sharing because it surprised me: scaling the model barely moved suggestion quality. I measured Qwen2.5-1.5B-Instruct in the same harness against Qwen2.5-0.5B, and it bought about a point of accuracy on next-word top-1 (19.0% to 20.1%) and under half a point at top-5 (39.0% to 39.4%), for roughly 2.4x the decode cost. The ceiling on this task is next-word predictability of short informal text, not model size, so a model near 0.5B is close to the right point on the curve for a keyboard. That finding is also why the English model later moved to a smaller trunk and a different readout rather than up to something larger.
What runs today
The shared tokenizer was not serving every language equally, so the single shared model gave way to one model per language, and the readout changed too: instead of a raw next-token distribution, the models score whole words with a word-level head. The strip is still two tiers, and everything still runs through llama.cpp on the CPU inside the keyboard process, one language resident at a time.
- English: a fine-tuned SmolLM2-360M with a word-scoring head. The setup download is about 545 MB. There is also a lighter English model, about 136 MB, which is the "Battery life" choice under Optimize for in Settings.
- The other eight languages: each has its own smaller word-scoring model, between 135 MB and 193 MB to download.
I am deliberately not listing the base model behind each of the other languages. They have changed more than once, and a list written here would be out of date before the next release.
Voice: an on-device recognizer plus a learned cleanup tagger
Dictation is a separate pipeline. Audio at 16 kHz goes through a small voice-activity gate (silero VAD), then an on-device recognizer run through sherpa-onnx. English, Spanish, Portuguese, German and French use NVIDIA Parakeet (about 0.6B parameters). Swedish uses KB-Whisper-small, and Polish, Turkish and Czech use Whisper large-v3 turbo. The Parakeet model is an offline one run in a pseudo-streaming loop: it re-decodes a trailing window on a roughly one-second cadence and commits text behind a short lag, which gives you live-feeling transcription without the accuracy hit of a true frame-synchronous streaming model. I measured the Parakeet decode at about a 0.05 real-time factor on a Pixel 9 Pro XL, a ten-second window in roughly half a second. That was on an early build, 1.0.3 (4199), and I have not re-measured it on the current one, so read it as a ballpark.
Cleanup is where dictation usually feels magic or feels broken, and two things are worth being precise about. First, Lexo ships it off by default: a fresh install types the transcript as recognized, and the toggle is Auto-clean voice typing in the Voice dictation settings, because it is not yet good enough to be the default. Second, it is not a second language model rewriting your speech. In English, a raw transcript goes through a small transformer tagger (an ELECTRA-small model with three heads, int8 ONNX) that tags each token for deletion (fillers, stutters, self-corrections), casing, and punctuation, followed by a deterministic rule pass (spoken punctuation, acronyms, number formatting, register detection so it does not force capital letters and periods into a terminal). The whole cleanup stage is fail-open: if it is unsure it leaves the raw transcript alone rather than risk mangling it. A generative-LLM cleanup pass is the obvious next step and is not shipped today.
The download tradeoff
All of this means a setup download of about 545 MB in English for the typing model, and between 135 MB and 193 MB for the other languages. Voice is opt-in and deferred to first use, so a keyboard-only install never pulls a speech model; when you do use the mic, it downloads separately. The models are not bundled in the app, they are pulled once from an object store on first setup over Wi-Fi, verified by md5, and stored in the app's private files directory. Bundling them made every Play Store update enormous and slow; a one-time download keeps updates small and, because the models live in the files directory, an app update does not re-download them unless the model itself changed. For the audience that wants an on-device model, a few hundred megabytes is a fair price and a one-time one.
The privacy claim, and how to check it without trusting me
The design point is that nothing you type or say leaves the phone. You do not have to take that on faith.
The app declares exactly five permissions: RECORD_AUDIO (dictation), VIBRATE (haptics), POST_NOTIFICATIONS (download progress), INTERNET, and ACCESS_NETWORK_STATE.
INTERNET covers five things, and none of them is your text: the one-time model download, Google Play billing, one anonymous request after install that carries only the Google Play install source tags and the app version, a daily tally of numbers-only usage counts (off unless you turn it on), and anonymous crash reporting through Firebase Crashlytics, which is off unless you turn it on at Settings > Advanced > Other > Send crash reports. If you do turn it on, crash reports carry stack traces, the device model, and the app version. They never carry what you typed or what you said.
So point NetGuard or PCAPdroid at Lexo, and here is what you should actually expect to see. Past the model download and the one-time install request, typing and dictating generate no traffic at all, and on a default install there is no crash-reporting traffic either, because collection stays off until you opt in. If you do opt in, you will see Crashlytics check in, and that is the one thing in a capture that is not the model download or the install request.
The models are open-weight and the runtimes are llama.cpp (MIT) and sherpa-onnx (Apache-2.0), so there is nothing secret about the model side. The per-model licence list is in the terms.
Honest limitations
- English is strongest. eight other languages also ship (Spanish, Portuguese, French, German, Swedish, Polish, Turkish and Czech), and closing the gap to English is the current focus.
- It is closed source, and the neural suggestion tier is a one-time paid unlock, Lexo Premium. The privacy story is verifiable regardless (previous section), but I understand that is a real objection for some people.
- Licensing and billing verification can be unreliable on de-Googled or custom ROMs, since that path leans on Play services.
- Voice cleanup is opt-in (off by default) and it is a tagger plus rules, not a generative rewrite: switched on, it fixes disfluencies and punctuation but will not restructure a rambling sentence the way a full LLM pass could.
If you want to use it
Everything in the "what runs today" parts above is what ships now, not a roadmap. It costs a one-time download of about 545 MB in English, plus the speech model if you want dictation, and $9.99 once after a 30-day trial, and if a language other than English is how you actually type, read the limitations again before you install. If you want your next-word ranking and your dictation to run on the phone and nowhere else, that is Lexo on Google Play.