Large Language Models
324 views
Tokenization
Quick Definition
Breaking text into tokens for language model processing
Full Definition
Breaking text into smaller units called tokens that language models can process.
Examples
BPE, SentencePiece, word embeddings, vocabulary building
Related Terms
large-language-model
embedding
bpe
More Large Language Models Terms
System Prompt
Special prompt defining AI assistant behavior and constraints
Context Window
Maximum tokens a language model can process at once
Alignment Tax
Performance cost of aligning AI with human preferences
Mistral
Efficient open-source LLMs with strong performance
Sparse Attention
Attention attending to only a subset of tokens
Quantization LLM
Reducing LLM weight precision for efficient inference