Large Language Models
31 views
Prompt Caching
Quick Definition
Storing key-value states for repeated prompt prefixes
Full Definition
Storing computed key-value states for repeated prompt prefixes to reduce inference latency and cost.
Examples
reduced latency, cost optimization, system prompt reuse
Related Terms
kv-cache
inference-optimization
prompt-engineering
More Large Language Models Terms
Grounding LLM
Connecting LLM outputs to verifiable external sources
WordPiece
Subword tokenization splitting words into frequent units
BERT
Google's bidirectional transformer for context understanding
Autoregressive Model
Generative model predicting next token from previous tokens
Structured Output
Generating LLM responses in predefined machine-readable formats
Next Token Prediction
Core LLM objective predicting the next token from context