Large Language Models
28 views
Pre-training
Quick Definition
Initial training learning general language patterns from text
Full Definition
Initial training phase where models learn general language patterns from massive text corpora.
Examples
GPT pre-training, BERT training, foundation model creation
Related Terms
self-supervised-learning
fine-tuning
More Large Language Models Terms
SentencePiece
Language-agnostic tokenizer operating on raw text bytes
WordPiece
Subword tokenization splitting words into frequent units
Multi-Head Attention
Parallel attention operations concatenated for richer representations
Sliding Window Attention
Attention limiting tokens to nearby window only
KV Cache
Memory optimization storing key-values for faster generation
Positional Encoding
Incorporating token position information into transformers