Large Language Models
32 views
Tokenization
Quick Definition
Breaking text into tokens for language model processing
Full Definition
Breaking text into smaller units called tokens that language models can process.
Examples
BPE, SentencePiece, word embeddings, vocabulary building
Related Terms
large-language-model
embedding
bpe
More Large Language Models Terms
Scaling Laws
Relationships between model size, data, compute, and performance
Grounding LLM
Connecting LLM outputs to verifiable external sources
KV Cache
Memory optimization storing key-values for faster generation
Fine-Tuning
Further training pre-trained models on specific domain data
SentencePiece
Language-agnostic tokenizer operating on raw text bytes
Training LLM
Adjusting parameters on large datasets to learn language