Why do chatbots miscount the R’s in “strawberry”?
In 28 seconds, you’ll see how language models read tokens instead of letters, and why that causes the mistake.
A language model doesn’t see a word as a row of letters. It splits text into chunks called tokens, and each chunk is stored as a number. Because the individual characters stay hidden inside those chunks, the model isn’t really counting. It is recalling spelling from patterns in its training data. If you ask it to spell the word out first, the letters appear in its own text, and the count usually comes out right. The split of “strawberry” shown in the video is illustrative: the exact chunks depend on the model.
Try it yourself: ask a chatbot to spell strawberry letter by letter, then count the R’s.
Sources:
1. OpenAI Tokenizer:
2. What are tokens and how to count them? (OpenAI Help Center):
3. tiktoken (OpenAI’s open-source tokenizer library):
4. Neural Machine Translation of Rare Words with Subword Units (Sennrich, Haddow, Birch):
How this video was made: researched and scripted with AI assistance, animated in code.
#AI #Tokens #HowAIWorks
source
