Large Language Models
132 views
Next Token Prediction
Quick Definition
Core LLM objective predicting the next token from context
Full Definition
The core training objective of autoregressive LLMs predicting the most probable next token given previous context.
Examples
GPT text generation, autocomplete, language modeling
Related Terms
autoregressive-model
large-language-model
pre-training
More Large Language Models Terms
Mistral
Efficient open-source LLMs with strong performance
Tokenization
Breaking text into tokens for language model processing
Alignment Tax
Performance cost of aligning AI with human preferences
Scaling Laws
Relationships between model size, data, compute, and performance
BPE
Subword tokenization algorithm merging frequent character pairs
Distillation LLM
Training smaller LLMs to replicate larger model behavior