Large Language Models
25 views
RoPE
Quick Definition
Position encoding using rotation matrices for relative positions
Full Definition
Rotary Position Embedding encoding relative positions using rotation matrices.
Examples
Llama, Mistral, length generalization
Related Terms
positional-encoding
transformer
More Large Language Models Terms
Temperature
Parameter controlling randomness in text generation
Attention Head
Individual attention mechanism learning specific patterns
Tokenizer Vocabulary
Complete set of tokens a language model can process
LoRA
Parameter-efficient fine-tuning using low-rank matrix decomposition
Tokenization
Breaking text into tokens for language model processing
Embeddings LLM
Dense vectors capturing semantic meaning of text