Flash Attention
Quick Definition
Memory-efficient attention using GPU SRAM block computation
Full Definition
An optimized attention algorithm computing in GPU SRAM blocks for reduced memory and increased speed.
Examples
fast training, memory optimization, transformer acceleration
Related Terms
attention-mechanism
gpu-optimization
More Large Language Models Terms
Llama
Meta AI's open-source large language models
Masked Language Model
Training objective predicting randomly masked input tokens
Large Language Model
Massive AI model trained on text for language understanding and generation
Sliding Window Attention
Attention limiting tokens to nearby window only
RAG
Enhancing LLMs by retrieving relevant external knowledge
QLoRA
Combining 4-bit quantization with LoRA for efficiency