Large Language Models
24 views
Mixture of Experts
Quick Definition
Architecture routing tokens to specialized expert sub-networks
Full Definition
An architecture routing tokens to specialized expert sub-networks via a gating mechanism.
Examples
Mixtral, Switch Transformer, sparse scaling
Related Terms
transformer
sparse-model
More Large Language Models Terms
Flash Attention
Memory-efficient attention using GPU SRAM block computation
Function Calling
Structured mechanism for LLMs to generate API function calls
Positional Encoding
Incorporating token position information into transformers
Tokenizer Vocabulary
Complete set of tokens a language model can process
Autoregressive Model
Generative model predicting next token from previous tokens
Streaming
Sending LLM tokens to clients as they are generated