Fitting a 0.5B LLM inside an Android keyboard
Lexo (com.lexo.keyboard) is an Android keyboard. Its typing suggestions and its voice dictation both run on the phone, with no text or audio sent anywhere. This is a note on what it takes to fit that inside an input method, which is one of the more constrained places you can try to run a language model.
Lexo is the keyboard described here. Thirty days free, then $9.99 once. Get it on Google Play
Since this was written. The 0.5B model below is no longer the whole answer. English now runs a fine-tuned SmolLM2-360M with a word-scoring head, Swedish, German and Polish run gemma-3-1b-it, and Turkish runs its own word-level pack. Spanish, Portuguese and French still run the 0.5B Qwen described here. What runs today has the current sizes. I have left the rest of the note as written, because how it was built is the part worth reading.
Why a keyboard is a hard place to put a model
An Android IME is not a normal app. It runs in a background-priority process that the system kills early under memory pressure, it has to paint a new suggestion strip within a couple of frames of each keystroke, and it is on screen during the most latency-sensitive thing a person does on a phone. Anything you load into it competes with the foreground app for RAM and CPU. So the interesting question was never "can a 0.5B model produce good suggestions," it was "can it do that without making the keyboard feel slow or getting the process killed."
Typing: Qwen2.5-0.5B through llama.cpp
The first suggestion model was Qwen2.5-0.5B-Instruct, quantized to q8_0, a 531 MB GGUF file, and it is still what Spanish, Portuguese and French run. It runs through llama.cpp, loaded over a custom JNI layer built for this, on CPU with a small context window (256 tokens) and four threads. q8_0 was the quant that held suggestion quality while keeping the file and the memory footprint manageable; smaller quants cost accuracy and a larger context window costs latency for no measured benefit at this task.
The model does not run alone. Suggestions come from two tiers. A fast baseline tier (a word-frequency prior plus a per-user personal-vocabulary store) fills the strip while a decode is in flight. The neural tier fuses the neural model's next-token distribution with that baseline and with the user's own recurring vocabulary. Splitting it this way means the strip is never empty waiting on a decode, and the model's job is narrowed to the thing it is actually good at: ranking plausible next words in context. Both tiers sit behind the paid unlock: unpaid, there is no suggestion strip at all.
One finding worth sharing because it surprised me: scaling the model barely moves suggestion quality. I measured Qwen2.5-1.5B-Instruct in the same harness against Qwen2.5-0.5B, and it bought about a point of accuracy on next-word top-1 (19.0% to 20.1%) and under half a point at top-5 (39.0% to 39.4%), for roughly 2.4x the decode cost. The ceiling on this task is next-word predictability of short informal text, not model size, so the 0.5B is close to the right point on the curve for a keyboard. That finding is also why English later moved down to a 360M trunk rather than up to something larger.
What runs today
The per-language split happened because one shared tokenizer was not serving every language equally. The languages the Qwen BPE fragmented worst got their own models. As of the 2.0.0 build:
- English: a fine-tuned SmolLM2-360M with a word-scoring head instead of a raw next-token readout. About 431 MB of GGUF and head weights together, so it is smaller than the 0.5B it replaced.
- Swedish, German, Polish: gemma-3-1b-it at Q4_K_M, about 806 MB each.
- Turkish: its own word-level pack, about 488 MB.
- Spanish, Portuguese, French: still Qwen2.5-0.5B-Instruct at q8_0, about 531 MB, exactly as described above.
Everything else in the typing path is unchanged: llama.cpp on CPU inside the keyboard process, one language resident at a time, the two-tier strip in front of it.
Voice: an offline transducer plus a learned cleanup tagger
Dictation is a separate pipeline. Audio at 16 kHz goes through a small voice-activity gate (silero VAD), then an offline NVIDIA Parakeet transducer (~0.6B, int8 ONNX) decoded through sherpa-onnx. The model is an offline one run in a pseudo-streaming loop: it re-decodes a trailing window on a roughly one-second cadence and commits text behind a short lag, which gives you live-feeling transcription without the accuracy hit of a true frame-synchronous streaming model. On a Pixel 9 Pro XL the decode runs at about a 0.05 real-time factor (a ten-second window decodes in roughly half a second), so the audio side has plenty of headroom on modern hardware.
Cleanup is where dictation usually feels magic or feels broken, and two things are worth being precise about. First, Lexo ships it off by default: a fresh install types the verbatim transcript, and the toggle is in Settings > Typing, because it is not yet good enough to be the default. Second, it is not a second language model rewriting your speech. A raw transcript goes through a small transformer tagger (an ELECTRA-small model with three heads, int8 ONNX, about 3 ms per turn) that tags each token for deletion (fillers, stutters, self-corrections), casing, and punctuation, followed by a deterministic rule pass (spoken punctuation, acronyms, number formatting, register detection so it does not force capital letters and periods into a terminal). The whole cleanup stage is fail-open: if it is slow or uncertain it leaves the raw transcript alone rather than risk mangling it. A generative-LLM cleanup pass is the obvious next step and is not shipped today.
The download tradeoff
All of this means a setup download of roughly 431 MB in English for the typing model, and then, only if you use the mic, roughly 661 MB for the voice model plus 14 MB for the cleanup tagger and small VAD files. Voice is opt-in and deferred to first use, so a keyboard-only install never pulls it. Other languages pull different models and land somewhere else. Swedish, German and Polish carry the larger gemma typing pack, and Turkish carries both its own typing pack and a Whisper-turbo recognizer of about 1.04 GB, so those installs are bigger. The models are not bundled in the app, they are pulled once from an object store on first setup over Wi-Fi, verified by md5, and stored in the app's private files directory. Bundling them made every Play Store update enormous and slow; a one-time download keeps updates small and, because the models live in the files directory, an app update never re-downloads them. For the audience that wants an on-device model, a few hundred megabytes per model is a fair price and a one-time one.
The privacy claim, and how to check it without trusting me
The design point is that nothing you type or say leaves the phone. You do not have to take that on faith.
The app declares exactly five permissions: RECORD_AUDIO (dictation), VIBRATE (haptics), POST_NOTIFICATIONS (download progress), INTERNET, and ACCESS_NETWORK_STATE.
INTERNET covers three things, and none of them is your text: the one-time model download, Google Play billing, and anonymous crash reporting through Firebase Crashlytics, which is off unless you turn it on at Settings > Advanced > Other > Send crash reports. If you do turn it on, crash reports carry stack traces, the device model, and the app version. They never carry what you typed or what you said.
So point NetGuard or PCAPdroid at Lexo, and here is what you should actually expect to see. Past the model download, typing and dictating generate no traffic at all, and on a default install there is no crash-reporting traffic either, because collection stays off until you opt in. If you do opt in, you will see Crashlytics check in, and that is the one thing in a capture that is not the model download.
The models are open-weight (SmolLM2 and Qwen are Apache-2.0, gemma-3-1b-it is under Google's Gemma Terms of Use, the Parakeet ASR model is NVIDIA NeMo under CC-BY-4.0), and the runtimes are llama.cpp (MIT) and sherpa-onnx (Apache-2.0), so there is nothing secret about the model side. The full per-model list is in the terms.
Honest limitations
- English is strongest. Spanish, Portuguese, French, German, Swedish, Polish and Turkish also ship. Languages the shared model's tokenizer covered badly now get their own dedicated models, and closing the gap to English is the current focus.
- It is closed source, and the neural suggestion tier is a one-time paid unlock. The privacy story is verifiable regardless (previous section), but I understand that is a real objection for some people.
- Licensing and billing verification can be unreliable on de-Googled or custom ROMs, since that path leans on Play services.
- Voice cleanup is opt-in (off by default; Settings > Typing) and it is a tagger plus rules, not a generative rewrite: switched on, it fixes disfluencies and punctuation but will not restructure a rambling sentence the way a full LLM pass could.
If you want to use it
Everything above is what ships today, not a roadmap. It costs a one-time download of about 431 MB in English, plus the speech model if you want dictation, and $9.99 once after a 30-day trial, and if a language other than English is how you actually type, read the limitations again before you install. If you want your next-word ranking and your dictation to run on the phone and nowhere else, that is Lexo on Google Play.
Related: what offline dictation on Android actually means, and private and offline Android keyboards compared, including where Lexo loses.