Large Language Models
135 views
Attention Head
Quick Definition
Individual attention mechanism learning specific patterns
Full Definition
Individual attention mechanisms within multi-head attention each learning different relationship patterns.
Examples
multi-head attention, self-attention, transformer layers
Related Terms
attention-mechanism
transformer
More Large Language Models Terms
Distillation LLM
Training smaller LLMs to replicate larger model behavior
Tool Use
Capability of LLMs to call external tools and APIs
DPO
Alignment method bypassing reward model for preference learning
RLHF
Aligning LLMs with human preferences using reinforcement learning
LoRA
Parameter-efficient fine-tuning using low-rank matrix decomposition
Prompt Caching
Storing key-value states for repeated prompt prefixes