Skip to content
Skills Library
Data & analytics · Building & coding

data-quality

/data-quality

Add data quality checks to each layer of a pipeline and decide what a failure does (stop, quarantine or pass), so bad data stops before anyone reads it. Use it when you build or change a pipeline or model, add tests to tables, or numbers look wrong or disagree with the source.

Repository
yaap ↗
Licence
MIT
Last repository change
13.11.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 13.11.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Add data quality checks to each layer of a pipeline and decide what a failure does (stop, quarantine or pass), so bad data stops before anyone reads it. Use it when you build or change a pipeline or model, add tests to tables, or numbers look wrong or disagree with the source.

Original author description

Add data quality checks to each layer of a pipeline and decide what a failure does (stop, quarantine or pass), so bad data stops before anyone reads it. Use it when you build or change a pipeline or model, add tests to tables, or numbers look wrong or disagree with the source.

How it works

  1. What to check, per layer
  2. Writing a check
  3. Severity
  4. When a check fails
  5. Contracts between teams
  6. In YAAP
  7. Report
Original SKILL.md headings · 95 lines, 748 words
SKILL.md preview
---
name: data-quality
description: Add data quality checks to each layer of a pipeline and decide what a failure does (stop, quarantine or
  pass), so bad data stops before anyone reads it. Use it when you build or change a pipeline or model, add tests to
  tables, or numbers look wrong or disagree with the source.
license: MIT
metadata:
  sources: wshobson/plugins/data-engineering/skills/data-quality-frameworks
---

# Data quality

A check is only useful when it runs on every load and something happens when it fails. Write few checks on the
columns that matter, give each a severity, and decide in advance what a failure does.

## What to check, per layer

| Layer | Checks | Why |
|---|---|---|
| raw | The load arrived; row count within the usual range; the columns and types are the expected ones; the newest `_loaded_at` is recent | Catches a broken or partial delivery before it is transformed |
| clean | Keys unique and not null; required columns not null; values in their allowed set or range; foreign keys point to existing rows | Catches duplicates and broken joins, the most common cause of wrong numbers |
| served | Totals reconcile with clean or the source; business rules hold (a refund never exceeds its order); volume compared with the same day last week | Catches logic errors that every row on its own passes |

Start with the keys and the freshness of each table. Add a check when a problem was found, so it cannot return
unnoticed.

## Writing a check

A check is a query that returns the rows that break the rule; zero rows means it passed. Every tool (dbt tests,
Great Expectations, Soda, plain SQL in the pipeline) works this way underneath, so use the one the project has.

```sql
-- Error when it returns rows: orders whose customer does not exist.
select o.order_id
from clean.orders as o
left join clean.customers as c on c.customer_id = o.customer_id
where c.customer_id is null;
```

A volume check compares with history rather than a fixed number, which goes stale:

```sql
-- Warn when today's rows fall below half the average of the last 28 days.
with daily as (
    select order_date, count(*) as row_count
    from clean.orders
    where order_date >= current_date - 28
    group by order_date
)
select t.row_count, avg(h.row_count) as usual
from daily as t
cross join daily as h
where t.order_date = current_date
  and h.order_date < current_date
group by t.row_count
having t.row_count < 0.5 * avg(h.row_count);
```

When no rows arrived today at all, this query returns nothing; the freshness check on raw covers that case.

## Severity

- **error**: the data is wrong for its readers. Stop the downstream tasks.
- **warn**: unusual but possibly right (a quiet holiday). Report it and let the run continue.

Give a check a threshold when a few bad rows are acceptable ("at most 0.1% of rows without a country"), and say why
in the check's description.

## When a check fails

1. **Stop what reads from it.** Downstream tasks do not run; readers keep yesterday's correct data rather than
   today's wrong data.
2. **Tell the owner** what failed: the check, the table, the window, the number of failing rows and a few keys.
   Never put personal data from the rows in the alert.
3. **Find the cause.** Compare with the source: did the source change, was the delivery partial, or is the
   transformation wrong? Follow the `systematic-debugging` method when the cause is not obvious.
4. **Fix and rerun the window.** Because loads are idempotent, rerunning the same window after the fix is safe.
5. **Quarantine only on purpose.** Moving failing rows to a side table and loading the rest is right when a few bad
   rows must not hold back the rest; then the quarantine table gets its own alert and an owner.

Never make a failing check pass by loosening it, raising its threshold or deleting it without a reason in the
change. A check that fails for a good reason is a decision for the user.

## Contracts between teams

When another team produces the data, agree on a contract: the columns, types, key, freshness and who is told about a
change. Check the contract at the raw layer, so a broken delivery fails at the door with the producer named.

## In YAAP

- Run checks with `run_command` against a development database and quote the result.
- Use synthetic rows in tests and fixtures; when you show failing rows in a reply, show keys and counts.
- Record known data issues and their owners in the project memory with `update_memory`.

## Report

```
Checks: <added or changed, per table, with severity>
Result: <passed / failed, with counts>
Failure: <check, window, failing rows, cause, or "none">
Action: <stopped, quarantined, fixed and rerun>
Not verified: <for example checks that need production volumes>
```

Versions

RepositoryScoreLicence
This page · jelvodev/yaap100MIT
nimadorostkar/Claude-Skills-collection100MIT
SkillMedev/skills100MIT
nahid-sparktales/agent-dispatcher100MIT
yeaight7/agent-powerups100Apache-2.0
khuynh22/agent-dev-team100MIT
JoelLewis/finance_skills100MIT
SSameer20/data-analyst-toolkit90MIT
analyticalmid/data-workflow90MIT

Database snapshot: 13.11.2026 · package version 9a7d3548dd9d (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 90data-quality-checks
    Layer freshness, volume, schema, and distribution checks with quarantine and alert discipline. Use when adding quality gates to pipelines or when consumers keep finding bad data first.
  • 90clean-validate
    Design a data validation pipeline — schema checks, range validation, and quality metrics. Use when asked to "validate incoming data", "add schema checks", or "define data quality metrics".
  • 100data-pipeline-quality
    Automated data quality checks for pipelines. Testing pyramids, dbt test patterns, data contracts, circuit breakers, and monitoring. Use when implementing data quality checks, writing dbt tests, defining data contracts, setting up pipeline validation, building automated quality…
  • 100data-quality-checker
    Implement data quality checks, validation rules, and monitoring. Use when ensuring data quality, validating data pipelines, or implementing data governance.
  • 90data-quality-gates
    Use when deciding which data quality checks a model, source or pipeline needs and whether each one blocks publication or only warns: freshness, schema, keys, volume anomalies and business-rule tests.
  • 90ai-data-quality
    Sets data quality requirements for structured and operational data used by AI - accuracy, completeness, currency, consistency, uniqueness and meaning - with thresholds by use and risk tier, ways to measure them, owners and remediation, so agents and analytics do not confidently…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.