Large Language Models
178 views
Speculative Decoding
Quick Definition
Using draft model candidates verified by large model
Full Definition
Using a small draft model to generate candidates verified in parallel by the large target model.
Examples
inference acceleration, LLM serving, throughput optimization
Related Terms
inference-optimization
kv-cache
More Large Language Models Terms
Alignment Tax
Performance cost of aligning AI with human preferences
Sparse Attention
Attention attending to only a subset of tokens
Guardrails LLM
Frameworks for monitoring and controlling LLM I/O
Pre-training
Initial training learning general language patterns from text
Quantization LLM
Reducing LLM weight precision for efficient inference
SentencePiece
Language-agnostic tokenizer operating on raw text bytes