Large Language Models
152 views
Tokenizer Vocabulary
Quick Definition
Complete set of tokens a language model can process
Full Definition
The complete set of tokens a language model can recognize and process.
Examples
word pieces, subwords, special tokens
Related Terms
tokenization
bpe
More Large Language Models Terms
Llama
Meta AI's open-source large language models
System Prompt
Special prompt defining AI assistant behavior and constraints
SentencePiece
Language-agnostic tokenizer operating on raw text bytes
Flash Attention
Memory-efficient attention using GPU SRAM block computation
QLoRA
Combining 4-bit quantization with LoRA for efficiency
Sliding Window Attention
Attention limiting tokens to nearby window only