Large Language Models
311 views
Sliding Window Attention
Quick Definition
Attention limiting tokens to nearby window only
Full Definition
Attention limiting each token to attend to a fixed-size local window of nearby tokens.
Examples
efficient attention, long sequences, Mistral architecture
Related Terms
attention-mechanism
sparse-attention
More Large Language Models Terms
Grounded Generation
Generating outputs faithful to provided source documents
GGUF
File format for LLMs optimized for local inference
Tokenizer Vocabulary
Complete set of tokens a language model can process
Model Merging
Combining fine-tuned LLMs without additional training
Mixture of Experts
Architecture routing tokens to specialized expert sub-networks
KV Cache
Memory optimization storing key-values for faster generation