Flash Attention
Quick Definition
Memory-efficient attention using GPU SRAM block computation
Full Definition
An optimized attention algorithm computing in GPU SRAM blocks for reduced memory and increased speed.
Examples
fast training, memory optimization, transformer acceleration
Related Terms
attention-mechanism
gpu-optimization
More Large Language Models Terms
Large Language Model
Massive AI model trained on text for language understanding and generation
Pre-training
Initial training learning general language patterns from text
Constitutional AI
Training AI to follow explicit principles for consistency
Embeddings LLM
Dense vectors capturing semantic meaning of text
Multi-Head Attention
Parallel attention operations concatenated for richer representations
Sliding Window Attention
Attention limiting tokens to nearby window only