Large Language Models
223 views
Model Merging
Quick Definition
Combining fine-tuned LLMs without additional training
Full Definition
Combining multiple fine-tuned LLMs into a single model without additional training.
Examples
model soup, weight merging, multi-task models
Related Terms
fine-tuning
ensemble
More Large Language Models Terms
Function Calling
Structured mechanism for LLMs to generate API function calls
LoRA
Parameter-efficient fine-tuning using low-rank matrix decomposition
Mixture of Experts
Architecture routing tokens to specialized expert sub-networks
Flash Attention
Memory-efficient attention using GPU SRAM block computation
Inference LLM
Using a trained LLM to generate outputs from inputs
Sliding Window Attention
Attention limiting tokens to nearby window only