Tokenizers and BPE (Technical Deep Dive)
How an LLM splits text into tokens (BPE / WordPiece / SentencePiece), vocabulary, and why Korean uses more tokens
How an LLM splits text into tokens (BPE / WordPiece / SentencePiece), vocabulary, and why Korean uses more tokens