Large Language Models
156 views
Sliding Window Attention
Quick Definition
Attention limiting tokens to nearby window only
Full Definition
Attention limiting each token to attend to a fixed-size local window of nearby tokens.
Examples
efficient attention, long sequences, Mistral architecture
Related Terms
attention-mechanism
sparse-attention
More Large Language Models Terms
Scaling Laws
Relationships between model size, data, compute, and performance
Embeddings LLM
Dense vectors capturing semantic meaning of text
Structured Output
Generating LLM responses in predefined machine-readable formats
Quantization LLM
Reducing LLM weight precision for efficient inference
Large Language Model
Massive AI model trained on text for language understanding and generation
Sparse Attention
Attention attending to only a subset of tokens