The Nuts and Bolts of Token Testing: Prompt Variations and Decoding in Practice
13 May 2025
Robust, repetitive prompts and UTF-8 understanding are key for accurately verifying and diagnosing under-trained tokens across language models.
Secret Tokens, Secret Trouble: The Hidden Flaws Lurking in Big-Name AIs
13 May 2025
Glitch tokens persist in closed and open models; aligning tokenizer and training data, plus targeted checks, are key to safer, more efficient LLMs.
How Tokenizer Choices Shape Hidden Risks in Popular Language Models
12 May 2025
Model-specific analysis reveals GPT-2, NeoX, and OLMo tokenizers each harbor unique under-trained tokens, influenced by training data and tokenizer choices.
Where Glitch Tokens Hide: Common Patterns in LLM Tokenizer Vocabularies
12 May 2025
Untrained tokens often stem from unused byte tokens, merged fragments, and special tokens-patterns found across major LLMs regardless of architecture.
How Many Glitch Tokens Hide in Popular LLMs? Revelations from Large-Scale Testing
12 May 2025
Under-trained token indicators efficiently flag risky tokens in LLMs, with cross-model results showing 0.1–1% of vocabularies consistently problematic.
Comprehensive Detection of Untrained Tokens in Language Model Tokenizers
12 May 2025
Presents new methods for detecting ‘glitch tokens’ in LLMs-untrained tokens that cause unwanted behavior-and tools for safer, more robust language models.
Supplementary Figures and Supplementary Tables
16 Dec 2024
LLMs Rely on Contextual Knowledge Over Background Knowledge
16 Dec 2024
LLMs rely on contextual knowledge over domain-specific facts, maintaining accuracy even with synthetic gene names in reasoning tasks.
How Extra Information Affects AI’s Ability to Think Logically
16 Dec 2024
LLMs demonstrate varying sensitivity to increasing distractors in reasoning tasks.