Script Reader

Paste your script and read it one line at a time while you record.

This is the opening of the speculative decoding video, marked up as an example. Paste your own script over it.

Script

Every word your AI writes costs it a *full trip* through the entire model. Even the word "*the*."

Autoregressive language models generate -- or decode -- new tokens *one by one*, in order. Naturally, they are *unable* to decode a token without the most recent token present. Your hardware might have the capacity to process several tokens in parallel, / but for a single response, we don't know which tokens will occupy the earlier positions *yet*. Your model is generating tokens, just as this beaver is building his bridge -- one log at a time, [deadpan] with commendable precision. // But surely there is a way we could make this *parallel* somehow.

The natural solution would be to start *building ahead*. While one model works on one token, another works on the next, and the next, and the next. But that restriction is still holding us back -- we must decode prior tokens *before* the current one. You can't place a log with nothing to rest it on.

Autoregressive LLMs put the *full weight* of the model behind each token that is decoded. But not every prediction requires all of that intelligence. Often, a smaller, faster model can predict the same continuation. It won't always agree with the powerful model, but it doesn't *need* to. If it can give us a few likely tokens to work with, we can start putting that parallel capacity to use.

(Lossy Speculation)

Perhaps the powerful model gives 50% to one token and 48% to the token the fast model chose. Under the lossless rules, we would *replace* the fast model's token and throw away its proposed continuation.

18 lines · 264 words · about 1:46 at 150 words a minute

Marking Up a Script

*word*: Stress this word. Bold from Google Docs comes through as this, and Ctrl+B adds it here.
/: Short pause
//: Long pause, and the rest goes on a new line
[smile]: A note to yourself, shown small
(Outro): A line in parentheses is a section title, and longer ones show as notes. Starting a line with # also makes a title.