Ask an AI how many r’s are in “strawberry”. Not long ago, plenty of smart models answered: two 🍓
👓 It never sees the letters. AI chops text into blocks called tokens.
🧱 GPT-4’s tokenizer splits “strawberry” into str / aw / berry, and each becomes an ID: 496, 675, 15717. That’s all the model sees.
🤯 In a sentence, with a leading space, the whole of ” strawberry” is a single token (73700).
Why chop? Letter by letter is too long and slow; common chunks become one block, and ~100k blocks cover almost any text (GPT-4o: ~200k). “草履虫” is three blocks in GPT-4o but six byte fragments in GPT-4.
So knowing “berry” has two r’s is memorized, not seen. The fix: spell it out, then count — newer models that think before answering do exactly that, and most now get it right.
You see letters. It sees blocks.
Note: token splits and IDs from OpenAI’s open-source tiktoken (GPT-4: cl100k_base; GPT-4o: o200k_base). GPT-4o splits strawberry as st / raw / berry.
Paramecium Science: physics, chemistry, biology, computer science and AI, one “huh?” and one “oh!” at a time. Animation is illustrative, narration is AI-generated, music is original.
— Bluefin Tuna
#Shorts #AI #LLM #ChatGPT
source