Large Language Models
323 views
Tokenization
Quick Definition
Breaking text into tokens for language model processing
Full Definition
Breaking text into smaller units called tokens that language models can process.
Examples
BPE, SentencePiece, word embeddings, vocabulary building
Related Terms
large-language-model
embedding
bpe
More Large Language Models Terms
Beam Search
Search algorithm exploring multiple output sequences
Fine-Tuning
Further training pre-trained models on specific domain data
Temperature
Parameter controlling randomness in text generation
Linear Attention
Attention with linear complexity for long sequences
Pre-training
Initial training learning general language patterns from text
DPO
Alignment method bypassing reward model for preference learning