Large Language Models
28 views
Sliding Window Attention
Quick Definition
Attention limiting tokens to nearby window only
Full Definition
Attention limiting each token to attend to a fixed-size local window of nearby tokens.
Examples
efficient attention, long sequences, Mistral architecture
Related Terms
attention-mechanism
sparse-attention
More Large Language Models Terms
Multi-Head Attention
Parallel attention operations concatenated for richer representations
Tool Use
Capability of LLMs to call external tools and APIs
Model Merging
Combining fine-tuned LLMs without additional training
Training LLM
Adjusting parameters on large datasets to learn language
WordPiece
Subword tokenization splitting words into frequent units
Tokenization
Breaking text into tokens for language model processing