Large Language Models
155 views
RoPE
Quick Definition
Position encoding using rotation matrices for relative positions
Full Definition
Rotary Position Embedding encoding relative positions using rotation matrices.
Examples
Llama, Mistral, length generalization
Related Terms
positional-encoding
transformer
More Large Language Models Terms
Attention Head
Individual attention mechanism learning specific patterns
Instruction Tuning
Fine-tuning on instruction-response pairs for better following
Prompt Caching
Storing key-value states for repeated prompt prefixes
Prompt Template
Reusable structured format with dynamic input placeholders
Synthetic Data Generation
Using AI to create artificial training data replicating patterns
Context Window
Maximum tokens a language model can process at once