Large Language Models
150 views
Prompt Caching
Quick Definition
Storing key-value states for repeated prompt prefixes
Full Definition
Storing computed key-value states for repeated prompt prefixes to reduce inference latency and cost.
Examples
reduced latency, cost optimization, system prompt reuse
Related Terms
kv-cache
inference-optimization
prompt-engineering
More Large Language Models Terms
RLHF
Aligning LLMs with human preferences using reinforcement learning
WordPiece
Subword tokenization splitting words into frequent units
SentencePiece
Language-agnostic tokenizer operating on raw text bytes
Nucleus Sampling
Generation using cumulative probability threshold for token selection
Quantization LLM
Reducing LLM weight precision for efficient inference
RAG
Enhancing LLMs by retrieving relevant external knowledge