Large Language Models
286 views
Tokenizer Vocabulary
Quick Definition
Complete set of tokens a language model can process
Full Definition
The complete set of tokens a language model can recognize and process.
Examples
word pieces, subwords, special tokens
Related Terms
tokenization
bpe
More Large Language Models Terms
Hallucination LLM
LLMs generating plausible but incorrect information
Synthetic Data Generation
Using AI to create artificial training data replicating patterns
Tool Use
Capability of LLMs to call external tools and APIs
Semantic Search
Search technique understanding meaning rather than keywords
Constitutional AI
Training AI to follow explicit principles for consistency
Sliding Window Attention
Attention limiting tokens to nearby window only