Flash Attention
Quick Definition
Memory-efficient attention using GPU SRAM block computation
Full Definition
An optimized attention algorithm computing in GPU SRAM blocks for reduced memory and increased speed.
Examples
fast training, memory optimization, transformer acceleration
Related Terms
attention-mechanism
gpu-optimization
More Large Language Models Terms
Temperature
Parameter controlling randomness in text generation
Distillation LLM
Training smaller LLMs to replicate larger model behavior
GPT
OpenAI's family of autoregressive large language models
Tool Use
Capability of LLMs to call external tools and APIs
Reward Model
Model scoring LLM outputs based on human preferences
Attention Head
Individual attention mechanism learning specific patterns