Large Language Models
267 views
Attention Head
Quick Definition
Individual attention mechanism learning specific patterns
Full Definition
Individual attention mechanisms within multi-head attention each learning different relationship patterns.
Examples
multi-head attention, self-attention, transformer layers
Related Terms
attention-mechanism
transformer
More Large Language Models Terms
Synthetic Data Generation
Using AI to create artificial training data replicating patterns
Model Merging
Combining fine-tuned LLMs without additional training
Quantization LLM
Reducing LLM weight precision for efficient inference
Chain-of-Thought Prompting
Technique encouraging step-by-step reasoning in LLMs
Masked Language Model
Training objective predicting randomly masked input tokens
Tokenizer Vocabulary
Complete set of tokens a language model can process