Skip to content
Skills Library
AI & agents · Learning & explanation

tokenization

/tokenization

Understand how text becomes tokens, and why token boundaries explain model behaviour on numbers, code, and non-English text. Use when a model behaves strangely on specific strings or costs more than expected.

Repository
AI-SKILLS ↗
Licence
MIT
Last repository change
05.09.2026

Assessment and security

90/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content25 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 05.09.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Understand how text becomes tokens, and why token boundaries explain model behaviour on numbers, code, and non-English text. Use when a model behaves strangely on specific strings or costs more than expected.

Original author description

Understand how text becomes tokens, and why token boundaries explain model behaviour on numbers, code, and non-English text. Use when a model behaves strangely on specific strings or costs more than expected.

How it works

  1. Method
  2. Boundaries
Original SKILL.md headings · 39 lines, 256 words
SKILL.md preview
---
name: tokenization
description: Understand how text becomes tokens, and why token boundaries explain model behaviour on numbers, code, and non-English text. Use when a model behaves strangely on specific strings or costs more than expected.
---

# Tokenization

Models do not see characters or words; they see tokens produced by a
learned vocabulary. A surprising amount of model behaviour, including
arithmetic errors and cost differences between languages, follows
directly from where the boundaries fall.

## Method

1. **Know that tokens are subwords, not words.** Common words are one
   token, rare words split into several, and the split is learned rather
   than linguistic.
2. **Expect worse behaviour on badly tokenized content.** Numbers split
   inconsistently, which is part of why arithmetic is unreliable, and
   unusual identifiers fragment into many tokens.
3. **Account for language cost differences.** Text in languages
   under-represented in the vocabulary uses several times more tokens
   for the same content, affecting both cost and effective context (see
   character-encoding).
4. **Measure context in tokens, not characters.** A character estimate
   is a rough approximation that fails exactly when the content is
   unusual (see context-window-management).
5. **Watch whitespace and leading spaces.** They are part of tokens, so
   a word with and without a preceding space are different tokens, which
   affects prompting and matching.
6. **Understand vocabulary boundaries in fine-tuning.** Domain terms
   that fragment badly can justify vocabulary extension, at the cost of
   retraining embeddings.
7. **Inspect tokenization when debugging.** Looking at the actual token
   split explains behaviour that reasoning about the text cannot.

## Boundaries

Tokenizers differ between model families, so token counts and behaviour
do not transfer. Tokenization is fixed at training time and cannot be
changed for an existing model. It explains some failures and is not the
cause of most reasoning errors.

Versions

The database has no other skills named tokenization.

Database snapshot: 13.11.2026 · package version 12dc3e9248c4 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100tokenization-bpe-wordpiece-tiktoken
    Use this skill when inspecting vocabulary compression, subword splitting, special tokens, and token counting across LLM architectures.
  • 100settings
    Use when the user asks what Repowise can be configured to do, how to turn a feature on or off, what it costs in tokens or model calls, why Repowise behaves a certain way in this repository, or asks you to change a Repowise option.

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.