Byte-level BPE

What is Byte-level BPE?

Tokenization method that builds text units from byte sequences and frequent pair merges

In other languages

한국어바이트 수준 BPE

텍스트를 바이트 단위에서 반복적으로 병합해 하위 단어 토큰을 만드는 토큰화 방식이다

日本語バイトレベルBPE

テキストをバイト単位から頻出ペアの結合で分割するトークン化方式

Related Terms

• Byte Pair EncodingA subword tokenization algorithm that iteratively merges the most frequent pair of adjacent bytes or characters to create a compact vocabulary
• Parallel DecodingA text generation method that produces multiple tokens in a single step, reducing latency.
• Token spaceRepresentation of text as discrete units that language models process when generating or interpreting sequences
• TokenizationThe step of splitting text into smaller pieces (tokens) that a model can process.
• RetokenizationProcess of converting text into tokens again, sometimes producing different token boundaries or identifiers
• Constrained DecodingA technique that restricts a language model to select only tokens conforming to specific grammar or format rules when predicting the next word.
• BitextA text object consisting of two versions of the same document, typically in different languages, used as training data for machine translation systems.
• TrigramA contiguous sequence of three items (usually characters or words) from a given sequence of text