Large Language Models
339 views
Speculative Decoding
Quick Definition
Using draft model candidates verified by large model
Full Definition
Using a small draft model to generate candidates verified in parallel by the large target model.
Examples
inference acceleration, LLM serving, throughput optimization
Related Terms
inference-optimization
kv-cache
More Large Language Models Terms
Next Token Prediction
Core LLM objective predicting the next token from context
Tokenization
Breaking text into tokens for language model processing
Instruction Tuning
Fine-tuning on instruction-response pairs for better following
Scaling Laws
Relationships between model size, data, compute, and performance
Beam Search
Search algorithm exploring multiple output sequences
Sparse Attention
Attention attending to only a subset of tokens