Large Language Models
24 views
Attention Head
Quick Definition
Individual attention mechanism learning specific patterns
Full Definition
Individual attention mechanisms within multi-head attention each learning different relationship patterns.
Examples
multi-head attention, self-attention, transformer layers
Related Terms
attention-mechanism
transformer
More Large Language Models Terms
Nucleus Sampling
Generation using cumulative probability threshold for token selection
Positional Encoding
Incorporating token position information into transformers
RLHF
Aligning LLMs with human preferences using reinforcement learning
Sparse Attention
Attention attending to only a subset of tokens
Jailbreaking
Circumventing AI safety restrictions through prompt techniques
RAG
Enhancing LLMs by retrieving relevant external knowledge