Large Language Models
279 views
Mixture of Experts
Quick Definition
Architecture routing tokens to specialized expert sub-networks
Full Definition
An architecture routing tokens to specialized expert sub-networks via a gating mechanism.
Examples
Mixtral, Switch Transformer, sparse scaling
Related Terms
transformer
sparse-model
More Large Language Models Terms
GPT
OpenAI's family of autoregressive large language models
RoPE
Position encoding using rotation matrices for relative positions
Chain-of-Thought Prompting
Technique encouraging step-by-step reasoning in LLMs
Tree-of-Thought
Framework exploring multiple reasoning paths for optimal solutions
PEFT
Fine-tuning methods updating only small parameter subsets
Sparse Attention
Attention attending to only a subset of tokens