In 2024 ChatGPT kept saying strawberry has two r’s. A language model never sees letters: in GPT-4o’s tokenizer (tiktoken o200k_base), ” strawberry” with a leading space is a single token, ID 101830. Spelled out with spaces, each letter becomes its own token and can be counted.
Part 1 of 6 from episode 5 of our Transformer series: Tokenization.
Full video:
Token splits computed with tiktoken:
#Shorts #AI #ChatGPT #Tokenization
source