Large Language Models
159 views
SentencePiece
Quick Definition
Language-agnostic tokenizer operating on raw text bytes
Full Definition
A language-agnostic tokenizer treating text as raw bytes enabling tokenization without pre-segmentation.
Examples
multilingual models, Llama tokenization, open vocabulary
Related Terms
tokenization
bpe
wordpiece
More Large Language Models Terms
Jailbreaking
Circumventing AI safety restrictions through prompt techniques
Tokenization
Breaking text into tokens for language model processing
Beam Search
Search algorithm exploring multiple output sequences
Speculative Decoding
Using draft model candidates verified by large model
Inference LLM
Using a trained LLM to generate outputs from inputs
Training LLM
Adjusting parameters on large datasets to learn language