Large Language Models
303 views
SentencePiece
Quick Definition
Language-agnostic tokenizer operating on raw text bytes
Full Definition
A language-agnostic tokenizer treating text as raw bytes enabling tokenization without pre-segmentation.
Examples
multilingual models, Llama tokenization, open vocabulary
Related Terms
tokenization
bpe
wordpiece
More Large Language Models Terms
Flash Attention
Memory-efficient attention using GPU SRAM block computation
Sliding Window Attention
Attention limiting tokens to nearby window only
Fine-Tuning
Further training pre-trained models on specific domain data
Semantic Search
Search technique understanding meaning rather than keywords
Scaling Laws
Relationships between model size, data, compute, and performance
Next Token Prediction
Core LLM objective predicting the next token from context