Large Language Models
31 views
Tokenizer Vocabulary
Quick Definition
Complete set of tokens a language model can process
Full Definition
The complete set of tokens a language model can recognize and process.
Examples
word pieces, subwords, special tokens
Related Terms
tokenization
bpe
More Large Language Models Terms
Prompt Caching
Storing key-value states for repeated prompt prefixes
SentencePiece
Language-agnostic tokenizer operating on raw text bytes
Prompt Template
Reusable structured format with dynamic input placeholders
Tree-of-Thought
Framework exploring multiple reasoning paths for optimal solutions
Tool Use
Capability of LLMs to call external tools and APIs
Alignment Tax
Performance cost of aligning AI with human preferences