Why can an LLM miscount the R’s in strawberry? Tokenization in LLMs, explained.
It never sees letters. In GPT-4o’s tokenizer, ” strawberry” is ONE token (id 101830). Tokens come from byte pair encoding: count neighbouring pairs, merge the most frequent, repeat. Fix: spell it with spaces or have it run code. Senior follow-up: a 2024 study found models recognise letters but struggle to count them, worst when letters repeat.
Interview questions covered:
– Why can an LLM miscount letters in a word?
– How does byte pair encoding (BPE) build a tokenizer’s vocabulary?
– Is tokenization the only reason?
Sources:
– Sennrich et al. 2015, BPE for NMT:
– Fu et al. 2024, Why Do LLMs Struggle to Count Letters?:
– Our run: tiktoken o200k_base (GPT-4o), Oct 2026
Voice: AI voice clone (Cartesia)
Which answer would you give: 1, 2 or 3? Comment the number.
Follow for more explainers.
#shorts #aiinterview #llm #genai #tokenization
source