Ask an AI how many r’s are in “strawberry”, and it famously says two. The reason: LLMs never see letters. They see tokens.
A tokenizer chops text into chunks, and each chunk becomes a number. With GPT-4o’s tokenizer, “strawberry” on its own splits into st | raw | berry, and inside a sentence the whole word is a single token: 101830. Counting r’s in that is like spelling a word you’ve only ever heard.
Rule of thumb: 1 token ≈ 4 characters ≈ 0.75 words of English.
Why devs care: you pay per token, and the same sentence in Hindi takes 11 tokens vs 7 in English (GPT-4o tokenizer); with the older GPT-4 tokenizer it was 37.
(Newer reasoning models often get strawberry right by spelling the word out first, but the token limitation is still there.)
◀ Previous: What is an LLM, really? →
▶ Next: GenAI Basics #3 · Context window: why AI “forgets” →
📚 Code Cuts roadmap: Basics → Prompting → Embeddings → RAG → Agents → Shipping
#tokens #llm #chatgpt #generativeai #shorts
source
Try it yourself 👇 (pip install tiktoken)
import tiktoken
enc = tiktoken.get_encoding("o200k_base") # GPT-4o
ids = enc.encode("strawberry")
print(ids, [enc.decode([i]) for i in ids])
# [302, 1618, 19772] ['st', 'raw', 'berry']
Comment a word and I'll tell you how it gets tokenized 👇
It said 3 -_-