Large Language Models
143 views
KV Cache
Quick Definition
Memory optimization storing key-values for faster generation
Full Definition
A memory optimization storing computed key-value pairs to avoid redundant attention computation.
Examples
inference optimization, generation speedup, memory management
Related Terms
attention-mechanism
inference-optimization
More Large Language Models Terms
Positional Encoding
Incorporating token position information into transformers
Llama
Meta AI's open-source large language models
Alignment Tax
Performance cost of aligning AI with human preferences
Jailbreaking
Circumventing AI safety restrictions through prompt techniques
Nucleus Sampling
Generation using cumulative probability threshold for token selection
Streaming
Sending LLM tokens to clients as they are generated