Large Language Models
280 views
Mixture of Experts
Quick Definition
Architecture routing tokens to specialized expert sub-networks
Full Definition
An architecture routing tokens to specialized expert sub-networks via a gating mechanism.
Examples
Mixtral, Switch Transformer, sparse scaling
Related Terms
transformer
sparse-model
More Large Language Models Terms
Grounded Generation
Generating outputs faithful to provided source documents
Next Token Prediction
Core LLM objective predicting the next token from context
Nucleus Sampling
Generation using cumulative probability threshold for token selection
Temperature
Parameter controlling randomness in text generation
Fine-Tuning
Further training pre-trained models on specific domain data
DPO
Alignment method bypassing reward model for preference learning