Flash Attention
Quick Definition
Memory-efficient attention using GPU SRAM block computation
Full Definition
An optimized attention algorithm computing in GPU SRAM blocks for reduced memory and increased speed.
Examples
fast training, memory optimization, transformer acceleration
Related Terms
attention-mechanism
gpu-optimization
More Large Language Models Terms
WordPiece
Subword tokenization splitting words into frequent units
Llama
Meta AI's open-source large language models
Context Window
Maximum tokens a language model can process at once
Tokenizer Vocabulary
Complete set of tokens a language model can process
Training LLM
Adjusting parameters on large datasets to learn language
GGUF
File format for LLMs optimized for local inference