How Lexo works: taps, voice and grammar

A sentence, quietly fixed

0:00 Here's a sentence typed on a phone. Somewhere in the middle, a thumb missed a key, and the keyboard quietly fixed it.

0:06 This is a look at how Lexo does that. And it turns out to be a question about probability.

A tap is a point, keys are clouds

0:12 Every time your thumb hits the glass, Lexo receives two numbers: where you touched. Not a letter. A point.

0:18 The key under that point is a good guess, but only a guess. So Lexo treats each key as a little cloud of likely landing spots.

0:26 Then it asks how well each cloud explains the point. A tap near the edge of G still leaves a real chance you meant H.

0:33 And those clouds aren't fixed. Thumbs land low and off-center, each in their own way. So Lexo slowly moves each cloud toward where your thumb actually lands, learning only from words you typed and kept.

0:46 What it stores is a running average for each key. No words, no order. And it never leaves your phone.

The language model on your phone

0:52 The second question is harder. Given everything you've written so far, which word are you trying to say?

0:58 For that, Lexo runs a small language model, right on your phone. It reads the context, and boils it down to a single list of numbers: a vector that captures where the sentence seems to be going.

1:10 Then one more layer compares that vector against every word in its vocabulary, tens of thousands of them, all in one step.

1:18 Out comes a score for every word, and those scores become probabilities.

Putting the two together

1:22 Now put the two together. Say you typed T, E, H.

1:26 For each candidate word, Lexo weighs two things: how likely that word is here, and how likely your taps would be if that word is what you meant.

1:34 "The" fits the sentence, and fits the taps with a single slip. Taken as typed, "teh" fits the taps perfectly, and the sentence not at all.

1:43 But winning isn't enough. A correction has to beat leaving your text alone by a clear margin. If it clears that bar, the space bar fixes it.

1:51 If not, it just waits in the suggestion strip, and you decide. And when what you typed is already a real word, the bar goes up even higher.

Undo, and what Lexo learns

2:00 If Lexo gets one wrong, tap backspace right after the correction. That undoes it, and Lexo remembers.

2:07 Undo the same correction twice, and it stops correcting that word. Words you type often get learned too.

2:13 All of that stays on your device. The models themselves are never trained on what you type.

Glide typing

2:18 Glide typing is the same idea, with a continuous input. Your path across the keyboard gets resampled into evenly spaced points.

2:27 Lexo then searches its dictionary, growing candidate words letter by letter, and keeping the ones whose keys line up with the path, in order.

2:36 Then the language model weighs in on the survivors, so the word that fits both the shape and the sentence wins.

Dictation, on your phone

2:42 Lexo can also listen. Dictation runs on your phone too: an open speech model turns your voice into text, and no audio is ever uploaded.

2:52 First, a small voice detector marks where you're actually speaking, so the silence in between never reaches the model.

2:59 Speech models are most accurate when they hear a whole sentence. But you want to see words while you talk. So about once a second, Lexo re-reads the last few seconds of audio.

3:09 Each new reading can revise the newest words. A word only freezes once it's a second old and two readings in a row agree on it. That's why the end of a line sometimes fixes itself as you keep talking.

3:22 The recognizer also keeps four candidate transcripts alive at once, and gives a small boost to words in your personal vocabulary. So a name it's never heard can still come out your way.

3:33 And in English, saying period, comma, or new line gives you the mark itself.

Spelling and grammar (beta)

3:37 Newest of all: spelling and grammar checks, still in beta. Instead of the word you're typing, they look at what you've already written.

3:45 A spelling flag uses the same math as autocorrect. A word gets underlined if it isn't a word at all, or if it's a real word that the sentence says is almost certainly another one. Like form, where you meant from.

3:58 Grammar uses a different kind of model, called an edit tagger. It doesn't rewrite your sentence. It reads it and gives every word one label: keep it, delete it, or replace it with something specific.

4:10 Typos confuse the tagger, so Lexo first swaps in the spelling fixes, judges the cleaned-up sentence, and then maps any grammar fix back onto your own words.

4:20 And it only speaks up when it's confident. A second model has to propose the exact same edit before you ever see it. That trades catching everything for rarely being wrong.

4:31 A fix shows up as an underline right on the word. Tap it, and pick the fix. The grammar model is a separate download, English only for now, and your text never leaves your phone.

The whole picture

4:42 Points become key probabilities. Context becomes word probabilities. Speech becomes text, and finished text gets a second look. All of it, right on your phone.