Large Language Models
143 views
Distillation LLM
Quick Definition
Training smaller LLMs to replicate larger model behavior
Full Definition
Training smaller student LLMs to replicate behavior of larger teacher models.
Examples
student-teacher training, model compression, edge deployment
Related Terms
model-distillation
quantization
More Large Language Models Terms
Mixture of Experts
Architecture routing tokens to specialized expert sub-networks
Linear Attention
Attention with linear complexity for long sequences
Grounding LLM
Connecting LLM outputs to verifiable external sources
Top-K Sampling
Generation strategy considering top k most probable tokens
Autoregressive Model
Generative model predicting next token from previous tokens
Attention Head
Individual attention mechanism learning specific patterns