Skip to main content

One doc tagged with "tokenizer"

Tokenizers and BPE (Technical Deep Dive)

How an LLM splits text into tokens (BPE / WordPiece / SentencePiece), vocabulary, and why Korean uses more tokens