AI models don’t read letters. They read tokens: chunks of text turned into ID numbers. We ran GPT-4o’s tokenizer on “strawberry”.
Tokenizer: GPT-4o (o200k) via — “strawberry” → st | raw | berry (ids 302, 1618, 19772); ” strawberry” inside a sentence → id 101830.
Why AI Does That, part 1.
#AI #ChatGPT #LLM #Shorts
source