Large Language Models
27 views
Sparse Attention
Quick Definition
Attention attending to only a subset of tokens
Full Definition
Attention where each token attends to only a subset of other tokens for efficiency.
Examples
Longformer, BigBird, efficient transformers
Related Terms
attention-mechanism
sliding-window-attention
More Large Language Models Terms
RoPE
Position encoding using rotation matrices for relative positions
Hallucination LLM
LLMs generating plausible but incorrect information
Tool Use
Capability of LLMs to call external tools and APIs
Tokenizer Vocabulary
Complete set of tokens a language model can process
Llama
Meta AI's open-source large language models
KV Cache
Memory optimization storing key-values for faster generation