Large Language Models
24 views
Multi-Head Attention
Quick Definition
Parallel attention operations concatenated for richer representations
Full Definition
A transformer mechanism running multiple attention operations in parallel and concatenating their outputs.
Examples
transformer layers, contextual learning, multi-perspective analysis
Related Terms
attention-mechanism
self-attention
transformer
More Large Language Models Terms
PEFT
Fine-tuning methods updating only small parameter subsets
Instruction Tuning
Fine-tuning on instruction-response pairs for better following
RoPE
Position encoding using rotation matrices for relative positions
Synthetic Data Generation
Using AI to create artificial training data replicating patterns
Autoregressive Model
Generative model predicting next token from previous tokens
Prompt Caching
Storing key-value states for repeated prompt prefixes