Przejdź do treści
Biblioteka skilli
AI i agenci · Analiza

nlp

/nlp

Pomaga wybrać tokenizer i typ transformera do zadania NLP, diagnozuje przefragmentowanie tekstu nieangielskiego i dobiera metrykę, gdy wyniki klasyfikacji, NER lub streszczania wyglądają źle.

Repozytorium
rsc-harness ↗
Licencja
MIT
Ostatnia zmiana repozytorium
25.10.2026

Ocena i bezpieczeństwo

100/ 100Klasa: dobry
Format25 / 25
Frontmatter z nazwą i opisem, poprawna nazwa, opis do 1024 znaków, nazwa zgodna z folderem.
Odkrywalność25 / 25
Opis mówi, kiedy użyć skilla, ma sensowną długość i jest w 3. osobie.
Treść35 / 35
Co najmniej 150 słów i do 500 linii, nagłówki, listy kroków, przykłady albo kod.
Higiena15 / 15
Bez zaślepek typu TODO, bez ścieżek Windows, z licencją.
Ryzykobrak sygnałów w skanie
Pliki w paczce
5
Skrypty
0
Komentarze HTML
0
Adresy URL w SKILL.md
0
Przestarzałość

Brak sygnałów. Repozytorium zmienione 25.10.2026, bez odwołań do wycofanych modeli i API.

Licencja

MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.

Co robi i kiedy

Pomaga wybrać tokenizer i typ transformera do zadania NLP, diagnozuje przefragmentowanie tekstu nieangielskiego i dobiera metrykę, gdy wyniki klasyfikacji, NER lub streszczania wyglądają źle.

Oryginalny opis (EN)

Use when choosing how to tokenize text or which transformer type fits an NLP task, when a tokenizer over-fragments non-English text or inflates token cost, when picking a language metric, or when classification, NER or summarization output looks wrong and it is unclear whether the tokenizer, the architecture or the metric is at fault. Covers subword tokenizers, encoder versus decoder versus encoder-decoder choice, sentence embeddings, and the metric families. NOT retrieval or vector search (that is `embeddings-search`), NOT the RAG loop (that is `rag`), NOT prompt wording (that is `prompt-engi

Jak działa

  1. Route out first (loud — do not duplicate these)
  2. Decision: model type per task (get this right before anything else)
  3. 1. Tokenization
  4. 2. Tasks
  5. 3. Evaluation — pick the metric that matches the task
  6. 4. Multilingual pitfalls
  7. Guardrails / gotchas
  8. Related skills
  9. Checklist
  10. References
Sekcje z nagłówków SKILL.md, w oryginale · 219 linii, 1 851 słów
Podgląd SKILL.md
---
name: nlp
description: "Use when choosing how to tokenize text or which transformer type fits an NLP task, when a tokenizer over-fragments non-English text or inflates token cost, when picking a language metric, or when classification, NER or summarization output looks wrong and it is unclear whether the tokenizer, the architecture or the metric is at fault. Covers subword tokenizers, encoder versus decoder versus encoder-decoder choice, sentence embeddings, and the metric families. NOT retrieval or vector search (that is `embeddings-search`), NOT the RAG loop (that is `rag`), NOT prompt wording (that is `prompt-engineering`), NOT training the network (that is `finetuning`)."
tags: [nlp, tokenization, bpe, wordpiece, ner, text-classification, bleu, rouge, perplexity, transformers]
recommends: [deep-learning, embeddings-search, rag, training-data]
origin: risco
---

# nlp — tokenize the text, pick the model type, pick the metric

You own the **language-modeling discipline**: how raw text becomes tokens, which transformer
architecture fits a task, and which metric actually tells you whether it worked. When the
question is "which tokenizer," "BERT or GPT or T5 for this," "why does my Catalan text cost 3×
the tokens," or "is this BLEU score meaningful," this is the skill. You stop at retrieval, the
RAG loop, prompt wording, and the training step itself — those route out (below).

## Route out first (loud — do not duplicate these)

- **Retrieval embeddings + vector search** (which embedding model, chunking, recall@k, rerank)
  → [`../embeddings-search/SKILL.md`](../embeddings-search/SKILL.md). Sentence embeddings *live
  here* as a task; using them to *retrieve* is theirs.
- **The retrieve → prompt → generate → answer loop** and groundedness → [`../rag/SKILL.md`](../rag/SKILL.md).
- **Prompt wording / few-shot / system prompts** → [`../prompt-engineering/SKILL.md`](../prompt-engineering/SKILL.md).
- **Training the network** (LoRA/SFT, trainer loop, PyTorch) → [`../finetuning/SKILL.md`](../finetuning/SKILL.md)
  + [`../deep-learning/SKILL.md`](../deep-learning/SKILL.md).
- **The training corpus itself** (JSONL messages, label sets) → [`../training-data/SKILL.md`](../training-data/SKILL.md).

## Decision: model type per task (get this right before anything else)

Pick the architecture from the task's *shape*, not from what is trendy. A decoder LLM can
technically classify, but a fine-tuned encoder is smaller, faster, cheaper, and usually more
accurate on a fixed-label task.

| Task shape | Architecture | Why | Example families* |
|---|---|---|---|
| Understand / label a whole input (classification, NER, extractive QA, similarity) | **Encoder** (bidirectional) | Attends to the full sentence both directions; cheap to fine-tune and to serve | BERT, RoBERTa, DistilBERT, ModernBERT |
| Free-form generation, chat, few-shot | **Decoder** (autoregressive) | Attends only to prior tokens; predicts the next token | GPT-style, Llama, Gemma, Qwen |
| Transform input → new text (summarize, translate, generative QA) | **Encoder-decoder / seq2seq** | Encoder reads all of the source, decoder writes conditioned on it | T5 / FLAN-T5, BART, mT5 |

\* Architecture families are stable; specific checkpoints and their licenses are not — check the
HF model card before you commit (licenses change; see `open-weights`). ModernBERT (2024) is a
current long-context encoder; verify the latest at author time.

The two most common own-goals: reaching for a 7B decoder to do sentiment on 5 classes (an
encoder does it for a fraction of the cost), and forcing an encoder to *generate* (it cannot —
it has no decoder).

## 1. Tokenization

Every downstream number depends on this step, and its failures are **silent**. The single
load-bearing rule:

> **Load the tokenizer that shipped with the checkpoint, and use the same one at train and
> inference.** `AutoTokenizer.from_pretrained(same_checkpoint)`. A train/inference tokenizer
> mismatch — different vocab, different special tokens, different casing/normalization — maps
> text to token ids the model never saw and corrupts everything downstream with no error.

```python
from transformers import AutoTokenizer   # transformers current major ~v5 (verify at author time)

tok = AutoTokenizer.from_pretrained("bert-base-cased")
enc = tok("Tokenizers matter.", return_offsets_mapping=True)
tok.convert_ids_to_tokens(enc["input_ids"])
# ['[CLS]', 'Token', '##izers', 'matter', '.', '[SEP]']  — note WordPiece '##' continuation + added specials
```

**The four algorithms** (full mechanics in `references/tokenization.md`):

| Algorithm | Builds vocab by… | Applies by… | Used by |
|---|---|---|---|
| **BPE** | merging the most frequent adjacent pair, repeatedly | split to chars, replay learned merges | GPT-2 (byte-level), many |
| **WordPiece** | merging pairs that maximize a likelihood score | longest-match subword from the front (`##` continuations) | BERT family |
| **Unigram** (SentencePiece) | start large, *remove* tokens that least hurt corpus likelihood | most-probable segmentation | T5, ALBERT, mT5 |
| **Byte-level BPE** | BPE over the 256 raw **bytes**, not Unicode chars | same as BPE on bytes | GPT-2, RoBERTa |

- **Byte-level BPE has no `[UNK]`.** Base vocab is exactly 256 (all byte values), so every
  emoji, accent, and script maps to *some* byte sequence — nothing falls out as unknown
  (verified: HF NLP course ch.6). WordPiece/word-level tokenizers *do* have `[UNK]` and lose OOV
  content.
- **SentencePiece is reversible** — it treats space as a normal symbol (the `▁` meta-symbol), so
  `decode(encode(x)) == x` without language-specific detokenization rules. That is why it
  dominates multilingual models.

**Special tokens are not decoration.** `[CLS]`/`<s>` carries the pooled sentence
representation for classification; `[SEP]`/`</s>` marks segment/end; `[PAD]` fills a batch (and
must be masked out via `attention_mask`); `[MASK]` is the MLM target; `[UNK]` is the fallback.
Names differ by model (`[CLS]` in BERT vs `<s>` in RoBERTa) — another reason to never hand-roll
the tokenizer.

**Why it matters — three concrete costs:**

- **$ cost & context.** Token count *is* the bill and the context budget. Fewer tokens per
  sentence = cheaper calls and more room in the window.
- **OOV / information loss.** A tokenizer that emits `[UNK]` throws away content it can't
  represent; byte-level/SentencePiece degrade gracefully instead.
- **Fairness.** Vocab trained mostly on English fragments other scripts far harder — the same
  meaning costs more tokens, more money, and more latency (section 5).

## 2. Tasks

### Text classification (encoder + classification head)
```python
from transformers import pipeline
clf = pipeline("text-classification", model="distilbert-base-uncased-finetuned-sst-2-english")
clf("The service was slow but the food was incredible.")
# [{'label': 'POSITIVE', 'score': 0.99...}]
```
Metric: accuracy on balanced data; **macro-F1** the moment classes are imbalanced (accuracy
lies when 95% of rows are one class).

### Token classification / NER (encoder, per-token labels)
Labels are **B-/I-/O** spans aligned to subword tokens: the first subword of a word gets the
label, continuation subwords and special tokens get `-100` (ignored by the loss). Evaluate with
**seqeval** at the **entity** level, never per-token accuracy (per-token accuracy is inflated by
the flood of `O` tokens).
```python
from transformers import pipeline
ner = pipeline("token-classification", aggregation_strategy="simple")
ner("Ada Lovelace worked in London.")
# groups subwords back into entities: PER 'Ada Lovelace', LOC 'London'
```

### Seq2seq — summarization / translation (encoder-decoder)
```python
summ = pipeline("summarization", model="facebook/bart-large-cnn")
summ(long_article, max_length=130, min_length=30)
```
Metric: **ROUGE** for summarization, **BLEU/chrF** for translation — with the heavy caveat in
section 4.

### Sentence embeddings (SBERT — the *task*, not retrieval)
```python
from sentence_transformers import SentenceTransformer   # sentence-transformers ~v5 (verify)
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
emb = model.encode(["The weather is lovely today.", "It's so sunny outside!"])
model.similarity(emb, emb)   # semantic textual similarity / clustering / paraphrase mining
```
Producing/judging embeddings **for retrieval** (model choice, chunking, recall@k, rerankers) is
[`embeddings-search`](../embeddings-search/SKILL.md), not here.

## 3. Evaluation — pick the metric that matches the task

| Task | Primary metric | Catches | Trap |
|---|---|---|---|
| Classification | accuracy + **macro-F1** | wrong labels | accuracy hides minority-class failure |
| NER / token | **entity-level F1** (seqeval) | missed/partial spans | per-token accuracy is inflated by `O` |
| Translation | **BLEU / chrF** | n-gram overlap w/ reference | weak on meaning; chrF better for morphology |
| Summarization | **ROUGE** (1/2/L) | recall of reference n-grams | rewards copying; blind to faithfulness |
| Generation (LM) | **perplexity** | how well the model predicts held-out text | tokenizer-dependent — not comparable across tokenizers |
| Open-ended / chat | **LLM-as-judge** + human | quality overlap metrics miss | judge bias (position, verbosity, self-preference) |

**The caveat that governs this whole section:** BLEU, ROUGE, and chrF are n-gram/character
overlap metrics and **correlate weakly with human judgment on open-ended and creative
generation** — they reward matching the reference's exact phrasing, so a correct paraphrase
scores low and a fluent-but-wrong copy scores high (well documented; e.g. the summarization and
MT-evaluation literature). Use them for **regression tracking on a fixed reference set**, never
as the final verdict on quality. For open-ended output, use an **LLM-as-judge rubric plus a
human spot-check** — and know the judge has its own biases (position, verbosity, self-preference),
so pin the rubric and randomize order.

**Perplexity** = exp(mean token NLL): lower means the model predicts held-out text better. It is
**tokenizer-dependent**, so two models with different tokenizers have non-comparable perplexities
— only compare within the same tokenizer/vocab. Runnable snippets for seqeval, sacrebleu, ROUGE,
perplexity, and an LLM-judge harness are in `references/evaluation.md`.

## 4. Multilingual pitfalls

The English-centric trap: a tokenizer whose vocab was learned mostly on English **over-fragments
other scripts**. The same sentence in Ukrainian, Arabic, Hindi, or even accented Catalan can take
**2–15× more tokens** than its English equivalent (Petrov et al., *Language Model Tokenizers
Introduce Unfairness Between Languages*, NeurIPS 2023). That "fertility" (tokens per word)
inflation is a triple tax:

- **Money** — more tokens per identical meaning = a proportionally larger bill for the same work.
- **Context** — over-fragmented text eats the window faster, so fewer few-shot examples fit and
  long documents truncate sooner.
- **Quality** — sequences fragmented into byte-shards are harder to model, degrading accuracy for
  exactly the users the tool already serves worst.

Mitigations: prefer a **multilingual tokenizer/model** (mT5, XLM-R, a SentencePiece-based model)
whose vocab actually covers your languages; **measure fertility** on your own corpus (tokens per
word, per language) before you commit; and don't benchmark cost or latency only on English.

## Guardrails / gotchas

- **Tokenizer must match the checkpoint, at train and inference.** Mismatch corrupts silently,
  no error. The most expensive bug in this skill.
- **Encoders can't generate; don't classify with a giant decoder by default.** Match architecture
  to task shape.
- **BLEU/ROUGE/chrF ≠ quality on open-ended text.** Overlap metrics; weak human correlation. Track
  regressions with them; judge quality with an LLM-judge + human.
- **Entity-F1 (seqeval), not token accuracy, for NER.** `O` tokens inflate accuracy toward 1.0.
- **macro-F1, not accuracy, on imbalanced classes.**
- **Perplexity is tokenizer-relative** — never compare it across different tokenizers.
- **Don't benchmark tokenization/cost only in English** — fertility varies 2–15× across scripts.
- **Never assert a model's license from memory** — check the current model card; license classes
  shift (Llama = Meta Community license, not OSI-open; Gemma = custom terms; etc.).

## Related skills

- [`embeddings-search`](../embeddings-search/SKILL.md) — retrieval embeddings, chunking, recall@k,
  reranking. NLP owns *making/judging* sentence embeddings as a task; using them to search is theirs.
- [`rag`](../rag/SKILL.md) — the full retrieve→generate answer loop and groundedness.
- [`prompt-engineering`](../prompt-engineering/SKILL.md) — the wording of the prompt.
- [`finetuning`](../finetuning/SKILL.md) + [`deep-learning`](../deep-learning/SKILL.md) — actually
  training/adapting the network (this skill picks the type and metric; those move the weights).
- [`training-data`](../training-data/SKILL.md) — building the labeled corpus you train on.

## Checklist

- [ ] Architecture picked from task shape (encoder / decoder / enc-dec), not habit.
- [ ] Tokenizer loaded from the *same* checkpoint, used identically at train and inference.
- [ ] Tokenizer choice justified vs OOV, cost, and — if multilingual — measured fertility.
- [ ] Special tokens and `attention_mask` handled (padding masked, `-100` on ignored labels).
- [ ] Metric matches the task: macro-F1 (imbalanced), entity-F1/seqeval (NER), ROUGE/BLEU/chrF
      only as a regression signal, LLM-judge + human for open-ended.
- [ ] Perplexity compared only within one tokenizer.
- [ ] Retrieval / RAG / prompt / training concerns routed to the sibling skill, not re-solved here.

## References

- `references/tokenization.md` — BPE/WordPiece/Unigram/byte-level training mechanics, special
  tokens per family, offset mapping, and a fertility-measuring snippet.
- `references/evaluation.md` — runnable seqeval, sacrebleu (BLEU/chrF), ROUGE, perplexity, and an
  LLM-as-judge rubric, with when each lies.
Pliki w paczce (5)
  • SKILL.md 14 KB
  • evals/README.md 1 KB
  • evals/cases.yaml 4 KB
  • references/evaluation.md 5 KB
  • references/tokenization.md 5 KB

Wersje

W bazie nie ma innych skilli o nazwie nlp.

Migawka bazy z 13.11.2026 · wersja paczki 23847a70fc57 (skrót SHA-256 całej paczki)

Pobrania w ostatnich 30 dniach: 0 · ranking

Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.

Podobne skille

  • 100search
    Wyszukuje japońskie zasoby NLP, takie jak tokenizery, modele, zbiory danych i słowniki, z pytań w dowolnym języku.
  • 100tooluniverse-ml-inference-utility
    Uruchamia inferencję ML, NLP i wizji na modelach hostowanych w HuggingFace: klasyfikacja tekstu, embeddingi, NER, pytania i odpowiedzi, tłumaczenie, klasyfikacja obrazów i detekcja obiektów.
  • 100cohere-api
    Obsługuje API Cohere dla korporacyjnego NLP: embeddingi, reranking, RAG i generowanie tekstu, w tym modele Command A z użyciem narzędzi i cytowaniami.
  • 100hebrew-nlp-toolkit
    Prowadzi deweloperów przez modele i narzędzia NLP dla hebrajskiego (DictaLM, DictaBERT, AlephBERT, ivrit.ai): dobór modelu, preprocessing, tokenizację, NER, analizę sentymentu i rozpoznawanie mowy.
  • 100nlp-engineer
    Pomaga budować produkcyjne systemy NLP: potoki przetwarzania tekstu, modele językowe oraz zadania takie jak rozpoznawanie nazw, analiza sentymentu i tłumaczenie maszynowe.
  • 100multilingual-embeddings
    Opisuje wielojęzyczne i międzyjęzykowe embeddingi tekstu (multilingual-e5, LaBSE, BGE-M3, Cohere), tekst mieszany, pułapki tokenizacji i jakość wyszukiwania w poszczególnych językach.

Uwagi i zgłoszenia

Wiadomość trafi do Bartosza Sokolińskiego. Adres e-mail jest potrzebny tylko do odpowiedzi. Szczegóły w polityce prywatności.