Large Language Models
282 views
Prompt Caching
Quick Definition
Storing key-value states for repeated prompt prefixes
Full Definition
Storing computed key-value states for repeated prompt prefixes to reduce inference latency and cost.
Examples
reduced latency, cost optimization, system prompt reuse
Related Terms
kv-cache
inference-optimization
prompt-engineering
More Large Language Models Terms
Prompt Template
Reusable structured format with dynamic input placeholders
Beam Search
Search algorithm exploring multiple output sequences
Flash Attention
Memory-efficient attention using GPU SRAM block computation
GPT
OpenAI's family of autoregressive large language models
GGUF
File format for LLMs optimized for local inference
Mistral
Efficient open-source LLMs with strong performance