Large Language Models
27 views
SentencePiece
Quick Definition
Language-agnostic tokenizer operating on raw text bytes
Full Definition
A language-agnostic tokenizer treating text as raw bytes enabling tokenization without pre-segmentation.
Examples
multilingual models, Llama tokenization, open vocabulary
Related Terms
tokenization
bpe
wordpiece
More Large Language Models Terms
KV Cache
Memory optimization storing key-values for faster generation
Jailbreaking
Circumventing AI safety restrictions through prompt techniques
Guardrails LLM
Frameworks for monitoring and controlling LLM I/O
Attention Head
Individual attention mechanism learning specific patterns
Embeddings LLM
Dense vectors capturing semantic meaning of text
Instruction Tuning
Fine-tuning on instruction-response pairs for better following