Large Language Models
286 views
RoPE
Quick Definition
Position encoding using rotation matrices for relative positions
Full Definition
Rotary Position Embedding encoding relative positions using rotation matrices.
Examples
Llama, Mistral, length generalization
Related Terms
positional-encoding
transformer
More Large Language Models Terms
Masked Language Model
Training objective predicting randomly masked input tokens
Temperature
Parameter controlling randomness in text generation
Top-K Sampling
Generation strategy considering top k most probable tokens
Structured Output
Generating LLM responses in predefined machine-readable formats
Nucleus Sampling
Generation using cumulative probability threshold for token selection
BERT
Google's bidirectional transformer for context understanding