Artificial Intelligence
155 views
Attention Mechanism
Quick Definition
Component allowing models to focus on relevant input parts
Full Definition
A neural network component that allows models to focus on relevant parts of the input when producing output.
Examples
self-attention, cross-attention, multi-head attention
Related Terms
transformer
self-attention
More Artificial Intelligence Terms
Model Distillation
Creating smaller models that replicate larger model behavior
Quantization
Reducing model precision from 32-bit to lower bit representations
State Space Model
Architecture using linear attention for efficient sequence modeling
Transfer Learning
Adapting a pre-trained model for a different related task
Superintelligence
Hypothetical AI surpassing all human cognitive abilities
Federated Learning
Distributed ML training without sharing raw data