Przejdź do treści
Biblioteka skilli
AI i agenci · Analiza

conversation-analysis

/conversation-analysis

Klasyfikuje rozmowy asystenta AI według zastosowania, rozwiązania i nastroju, maskując dane osobowe, i pokazuje, do czego ludzie go używają.

Repozytorium
ai-pm-skills ↗
Licencja
MIT
Ostatnia zmiana repozytorium
13.11.2026

Ocena i bezpieczeństwo

100/ 100Klasa: dobry
Format25 / 25
Frontmatter z nazwą i opisem, poprawna nazwa, opis do 1024 znaków, nazwa zgodna z folderem.
Odkrywalność25 / 25
Opis mówi, kiedy użyć skilla, ma sensowną długość i jest w 3. osobie.
Treść35 / 35
Co najmniej 150 słów i do 500 linii, nagłówki, listy kroków, przykłady albo kod.
Higiena15 / 15
Bez zaślepek typu TODO, bez ścieżek Windows, z licencją.
Ryzykobrak sygnałów w skanie
Pliki w paczce
59
Skrypty
0
Komentarze HTML
0
Adresy URL w SKILL.md
0
Przestarzałość

Brak sygnałów. Repozytorium zmienione 13.11.2026, bez odwołań do wycofanych modeli i API.

Licencja

MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.

Co robi i kiedy

Klasyfikuje rozmowy asystenta AI według zastosowania, rozwiązania i nastroju, maskując dane osobowe, i pokazuje, do czego ludzie go używają.

Oryginalny opis (EN)

Classify an AI assistant's conversations by use case, resolution and sentiment, PII masked. Use to build or check the taxonomy, pick the classifier, see what people use it for, or act on the results.

Jak działa

  1. When to use
  2. Inputs
  3. Method
  4. Output
  5. Rules and red lines
  6. Links
  7. Sources
Sekcje z nagłówków SKILL.md, w oryginale · 46 linii, 1 061 słów
Podgląd SKILL.md
---
name: conversation-analysis
description: Classify an AI assistant's conversations by use case, resolution and sentiment, PII masked. Use to build or check the taxonomy, pick the classifier, see what people use it for, or act on the results.
license: CC-BY-4.0
metadata:
  stage: "evaluate"
  version: "0.2.1"
  status: "tested"
  eval_threshold: "0.80"
  autonomy_relevant: "true"
  depends_on: "build-evals, ai-data-requirements"
  feeds: "assistant-kpis-cost-per-outcome, ai-roi-quantification, process-ai-redesign, build-evals, error-analysis, production-evals-monitoring"
  sources: "gdpr, own-conversation-classification, repo-chanmeng"
---

# conversation-analysis

## When to use
- "What do people use the assistant for?", a quarterly review, or knowledge base (KB) prioritisation by volume and resolution.
- A taxonomy of use cases or a labelling plan for an assistant's conversations needs building or checking before labelling starts.
- Choosing the model that classifies the conversations (cheap vs frontier, cost vs agreement), or checking one that has drifted.
- KPIs or ROI need volume by use case (`assistant-kpis-cost-per-outcome`, `ai-roi-quantification`).
- A classification is done and sound, and the question is what to do with the result: do not redo steps 1–7, go to the end of step 8 and answer with its hand-over.
- Do not use to build a classifier that acts in production (routing, triage, auto-reply): that is a feature, use `llm-feature-prd`.
- Do not use to grade whether the assistant's answers were correct: that is `build-evals`.

## Inputs
Mandatory: access to the transcripts (volume per month, period, channels), the personal-data rules (what counts as personal data here, who may read raw text), the decisions the analysis feeds (which KPI, which KB choice, which ROI line).
Ask if missing: a draft taxonomy (step 3 derives one if there is none), 150–200 hand-labelled conversations or labellers who know the domain, candidate classifiers with their price per conversation.
If transcripts contain personal data (names, emails, phone numbers, IBANs, employee or customer ids, health details), masking comes first: nothing else runs until it is done and checked.

## Method
1. **Mask personal data before any model call**, including the classifier candidates and any model used to draft the taxonomy. Deterministic patterns for structured data (IBAN, email, phone, ids), named-entity recognition for names and places, replaced by typed placeholders (`[NAME]`, `[IBAN]`, `[EMAIL]`). Check by hand a sample of masked conversations (100 or more) and report the residual leak rate. Until the check passes, no classification, labelling or taxonomy work starts. Raw text stays in its source system. Say why in the plan: GDPR art. 5(1)(c), data minimisation. Set a retention period for the masked text and the classifier logs and who deletes them: masked chat can still identify someone, so it does not stay forever (GDPR art. 5(1)(e), storage limitation).
2. **Fix the dimensions and what each one feeds**: use case (volume, ROI), domain or KB (quality by KB), resolution (resolved, escalated, abandoned, unresolved), sentiment, language. Drop a dimension that feeds no decision.
3. **Build the taxonomy and freeze it before labelling.** With no draft, or only a few examples, derive one from the data rather than from an org chart, which pushes real requests into "other": read about 50 masked conversations, write a free-form note on what each user wanted, cluster the notes into candidate classes, and keep reading until new conversations add no new class. Then, draft or not: 8–12 use cases, mutually exclusive. Each class has a definition, two examples and a counter-example. Overlapping classes are merged, or separated by a written rule (e.g. "a conversation with two requests takes the class of the first one the user states"). Keep "other" under 10% of conversations. Version the taxonomy: a change after freezing is a new version and the labelled set is relabelled.
4. **Write the annotation guide and label 150–200 conversations**, sampled across the period, channels and populations. Two labellers on at least 50 of them measure human agreement: it is the ceiling a classifier can reach.
5. **Compare the candidate classifiers on the labelled set**: agreement per dimension, never one average, the confusion between the closest classes, cost per conversation, latency. A candidate below the human ceiling on a dimension is not trusted on that dimension without review.
6. **Choose a setup with a confidence threshold.** When a cheap classifier is right most of the time and its confidence separates right from wrong, route by confidence: the cheap model keeps the cases above the threshold, a stronger model or a human takes the rest. Low-confidence cases go to review or to "unclassified", never forced into a class. Report the blended agreement and the blended cost per conversation at the chosen threshold. Recommend this routed setup as the target; one model alone is only an interim while the threshold is measured on the labelled set.
7. **Design the monthly drift check**: a fresh sample (50–100 conversations) labelled each month, agreement per dimension compared with the last run, and the shares of "other" and of low-confidence cases tracked. A drop in agreement, "other" above 10% or a new recurring request opens a taxonomy review. This check is the design behind the L5 gate "drift detection live" (references/autonomy-ladder.md): say which assistant step it gates.
8. **Join the output to the KPIs.** One row per conversation: `trace_id`, `use_case`, `domain`, `resolution`, `sentiment`, `language`, `confidence`, `classifier_version`, `taxonomy_version`. Volume by use case comes from these rows, never from declared usage (references/roi-model.md), and feeds `assistant-kpis-cost-per-outcome`. When a use case combines high volume with a high unresolved share, pull a masked sample of it and hand it to `build-evals` as candidate golden-set cases; scoring them stays there.

## Output
A classification plan following `templates/classification-plan.md`: masking step, its check and retention period, dimensions, frozen taxonomy with edge-case rules, annotation guide, candidate comparison (agreement per dimension, cost per conversation), chosen setup with threshold and routing, monthly drift check, output schema, unresolved clusters handed to `build-evals`. A section that cannot be filled says what is missing and who provides it.

## Rules and red lines
- Personal data is masked before any model call, including drafting the taxonomy. "It's internal" or "just a test" is not an exception.
- Never repeat a name, IBAN, email or id from the transcripts in the output: use the placeholders.
- The taxonomy is frozen before labelling: 8–12 use cases, overlaps merged or separated by a written rule.
- Agreement is reported per dimension, never as one average.
- Low-confidence cases are routed to review, never forced into a class.
- The classifier is re-measured every month on a fresh labelled sample.

## Links
references/roi-model.md · references/autonomy-ladder.md · build-evals · assistant-kpis-cost-per-outcome · ai-roi-quantification · llm-feature-prd

## Sources
- gdpr: data minimisation (art. 5) behind masking before any model call.
- own-conversation-classification: taxonomy size, freeze-then-label, per-dimension agreement and confidence routing, from classifying the conversations of internal assistants in production (method only).
- repo-chanmeng: bottom-up categories from free-form notes on traces, agreed retention for logs (ideas, own words).
Pliki w paczce (59)
  • SKILL.md 7 KB
  • evals/conversation-analysis-001/prompt.md 1 KB
  • evals/conversation-analysis-001/fixtures/bad.md 1 KB
  • evals/conversation-analysis-001/fixtures/good-run1.md 4 KB
  • evals/conversation-analysis-001/fixtures/good-run2.md 4 KB
  • evals/conversation-analysis-001/fixtures/good-run3.md 3 KB
  • evals/conversation-analysis-001/fixtures/good-run4.md 3 KB
  • evals/conversation-analysis-001/fixtures/good-run5.md 4 KB
  • evals/conversation-analysis-001/fixtures/good-run6.md 3 KB
  • evals/conversation-analysis-001/fixtures/good-run7.md 4 KB
  • evals/conversation-analysis-001/fixtures/good.md 1 KB
  • evals/conversation-analysis-001/graders/counter-example.md 1 KB
  • evals/conversation-analysis-001/graders/freeze-before-labelling.md 1 KB
  • evals/conversation-analysis-001/graders/other-cap.md 1 KB
  • evals/conversation-analysis-001/graders/redline-overlaps-resolved.md 1 KB
  • evals/conversation-analysis-001/graders/taxonomy-size.md 1 KB
  • evals/conversation-analysis-001/graders/triggered.md 1 KB
  • evals/conversation-analysis-001/graders/two-labellers.md 1 KB
  • evals/conversation-analysis-002/prompt.md 1 KB
  • evals/conversation-analysis-002/fixtures/bad-without-run1.md 2 KB
  • evals/conversation-analysis-002/fixtures/bad-without-run2.md 2 KB
  • evals/conversation-analysis-002/fixtures/bad-without-run3.md 2 KB
  • evals/conversation-analysis-002/fixtures/bad.md 1 KB
  • evals/conversation-analysis-002/fixtures/good-run1.md 2 KB
  • evals/conversation-analysis-002/fixtures/good-run2.md 2 KB
  • evals/conversation-analysis-002/fixtures/good-run3.md 3 KB
  • evals/conversation-analysis-002/fixtures/good-run4.md 3 KB
  • evals/conversation-analysis-002/fixtures/good-run5.md 3 KB
  • evals/conversation-analysis-002/fixtures/good-run6.md 2 KB
  • evals/conversation-analysis-002/fixtures/good.md 1 KB
  • evals/conversation-analysis-002/graders/blended-cost.md 1 KB
  • evals/conversation-analysis-002/graders/low-confidence-review.md 1 KB
  • evals/conversation-analysis-002/graders/monthly-remeasure.md 1 KB
  • evals/conversation-analysis-002/graders/per-dimension.md 1 KB
  • evals/conversation-analysis-002/graders/redline-route-by-confidence.md 1 KB
  • evals/conversation-analysis-002/graders/triggered.md 1 KB
  • evals/conversation-analysis-003/prompt.md 1 KB
  • evals/conversation-analysis-003/fixtures/bad.md 1 KB
  • evals/conversation-analysis-003/fixtures/good-run1.md 6 KB
  • evals/conversation-analysis-003/fixtures/good.md 1 KB
  • evals/conversation-analysis-003/graders/bottom-up-taxonomy.md 1 KB
  • evals/conversation-analysis-003/graders/gdpr.md 1 KB
  • evals/conversation-analysis-003/graders/leak-check.md 1 KB
  • evals/conversation-analysis-003/graders/placeholders.md 1 KB
  • evals/conversation-analysis-003/graders/redline-mask-first.md 1 KB
  • evals/conversation-analysis-003/graders/redline-no-pii-echo.md 1 KB
  • evals/conversation-analysis-003/graders/retention.md 1 KB
  • evals/conversation-analysis-003/graders/triggered.md 1 KB
  • evals/conversation-analysis-004/prompt.md 1 KB
  • evals/conversation-analysis-004/graders/not-triggered.md 1 KB
  • evals/conversation-analysis-005/prompt.md 1 KB
  • evals/conversation-analysis-005/fixtures/bad-with-1.md 1 KB
  • evals/conversation-analysis-005/fixtures/bad.md 1 KB
  • evals/conversation-analysis-005/fixtures/good.md 1 KB
  • evals/conversation-analysis-005/graders/failures-to-eval-set.md 1 KB
  • evals/conversation-analysis-005/graders/triggered.md 1 KB
  • references/autonomy-ladder.md 5 KB
  • references/roi-model.md 3 KB
  • templates/classification-plan.md 2 KB

Wersje

W bazie nie ma innych skilli o nazwie conversation-analysis.

Migawka bazy z 13.11.2026 · wersja paczki 6ad8ff560b81 (skrót SHA-256 całej paczki)

Pobrania w ostatnich 30 dniach: 0 · ranking

Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.

Podobne skille

  • 100check-messages
    Pokazuje wiadomości odebrane przez agenta Aicoo: o co pytali ludzie, historię rozmów i aktywność w skrzynce agenta.
  • 90classify
    Zgłasza propozycje pozycji pracy z rozmowy, wpisując je do pliku sugestii, bez zmian w dzienniku zdarzeń.
  • 100testing-assistant-conversations
    Testuje rozmowy asystenta, agenty i skille z AssistantTestLayer, Effect/vitest i zapisanymi odpowiedziami LLM w pamięci podręcznej.
  • 100azure-ai-textanalytics-py
    Analizuje tekst w Pythonie za pomocą Azure AI Text Analytics SDK: sentyment, rozpoznawanie encji, frazy kluczowe, wykrywanie języka, dane osobowe i NLP w ochronie zdrowia.

Uwagi i zgłoszenia

Wiadomość trafi do Bartosza Sokolińskiego. Adres e-mail jest potrzebny tylko do odpowiedzi. Szczegóły w polityce prywatności.