EP 11 · AI that writes code famously couldn’t count the r’s in ‘strawberry’. And the same chatbot’s bill jumps when you switch to Hindi. One culprit.
LLMs don’t read letters. A tokenizer cuts text into sub-word pieces called tokens from a fixed vocabulary, and the model only sees their IDs. On GPT-4’s cl100k tokenizer, ‘strawberry’ becomes str | aw | berry (newer o200k splits it st | raw | berry), so the r’s are sealed inside the pieces and the model can only guess. Newer reasoning models mostly get it right by spelling the word out first.
The trap: pricing and the context window are counted in tokens, and replies are generated token by token. Devanagari takes 3 bytes per code point in UTF-8, and tokenizers learned fewer Hindi pieces, so the same meaning often needs several times more tokens on older tokenizers. Measure with your model’s real tokenizer. Need an exact letter count? Use a tool call.
Send this to a friend who has a GenAI interview coming up.
Follow @interviewtrap.ai for daily interview traps.
Animated emoji: Noto Emoji Animation by Google, CC BY 4.0 (creativecommons.org/licenses/by/4.0). Modified (resized/re-timed).
#GenerativeAI #LLM #Tokenization #ChatGPT #AIInterview #Freshers #InterviewTrapAI #Shorts
source