Large Language Models
278 views
Sparse Attention
Quick Definition
Attention attending to only a subset of tokens
Full Definition
Attention where each token attends to only a subset of other tokens for efficiency.
Examples
Longformer, BigBird, efficient transformers
Related Terms
attention-mechanism
sliding-window-attention
More Large Language Models Terms
Streaming
Sending LLM tokens to clients as they are generated
Synthetic Data Generation
Using AI to create artificial training data replicating patterns
Tokenization
Breaking text into tokens for language model processing
Alignment Tax
Performance cost of aligning AI with human preferences
Adapter Layer
Small trainable modules in frozen transformer layers
Attention Head
Individual attention mechanism learning specific patterns