Large Language Models
145 views
Mixture of Experts
Quick Definition
Architecture routing tokens to specialized expert sub-networks
Full Definition
An architecture routing tokens to specialized expert sub-networks via a gating mechanism.
Examples
Mixtral, Switch Transformer, sparse scaling
Related Terms
transformer
sparse-model
More Large Language Models Terms
Top-K Sampling
Generation strategy considering top k most probable tokens
Next Token Prediction
Core LLM objective predicting the next token from context
KV Cache
Memory optimization storing key-values for faster generation
Instruction Tuning
Fine-tuning on instruction-response pairs for better following
Model Merging
Combining fine-tuned LLMs without additional training
GPT
OpenAI's family of autoregressive large language models