Large Language Models
18 views
Masked Language Model
Quick Definition
Training objective predicting randomly masked input tokens
Full Definition
A training objective where masked tokens are predicted from surrounding context.
Examples
BERT pre-training, fill-in-the-blank, contextual understanding
Related Terms
bert
pre-training
self-supervised-learning
More Large Language Models Terms
Tokenizer Vocabulary
Complete set of tokens a language model can process
Positional Encoding
Incorporating token position information into transformers
Grounded Generation
Generating outputs faithful to provided source documents
Sparse Attention
Attention attending to only a subset of tokens
Attention Head
Individual attention mechanism learning specific patterns
Prompt Caching
Storing key-value states for repeated prompt prefixes