Large Language Models
145 views
Sparse Attention
Quick Definition
Attention attending to only a subset of tokens
Full Definition
Attention where each token attends to only a subset of other tokens for efficiency.
Examples
Longformer, BigBird, efficient transformers
Related Terms
attention-mechanism
sliding-window-attention
More Large Language Models Terms
Distillation LLM
Training smaller LLMs to replicate larger model behavior
Multi-Head Attention
Parallel attention operations concatenated for richer representations
Tree-of-Thought
Framework exploring multiple reasoning paths for optimal solutions
Function Calling
Structured mechanism for LLMs to generate API function calls
RAG
Enhancing LLMs by retrieving relevant external knowledge
Grounded Generation
Generating outputs faithful to provided source documents