Large Language Models
164 views
Tokenization
Quick Definition
Breaking text into tokens for language model processing
Full Definition
Breaking text into smaller units called tokens that language models can process.
Examples
BPE, SentencePiece, word embeddings, vocabulary building
Related Terms
large-language-model
embedding
bpe
More Large Language Models Terms
Vector Database
Specialized database for high-dimensional embedding queries
Mistral
Efficient open-source LLMs with strong performance
Instruction Tuning
Fine-tuning on instruction-response pairs for better following
SentencePiece
Language-agnostic tokenizer operating on raw text bytes
Hallucination LLM
LLMs generating plausible but incorrect information
Model Merging
Combining fine-tuned LLMs without additional training