Skip to content
Skills Library
Data & analytics · Testing

implementing-data-quality-checks

/implementing-data-quality-checks

Add data quality checks to pipelines — freshness, volume/row-count anomalies, schema drift, null/uniqueness/referential integrity, and value distributions — using dbt tests, Great Expectations, or Soda, and deciding warn vs block. Use when adding data quality validation, catching bad data before it reaches consumers, setting up freshness/volume checks, or defining expectations.

Licence
Apache-2.0
Last repository change
30.08.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 30.08.2026, without references to retired models or APIs.

Licence

Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.

What it does and when to use it

Add data quality checks to pipelines — freshness, volume/row-count anomalies, schema drift, null/uniqueness/referential integrity, and value distributions — using dbt tests, Great Expectations, or Soda, and deciding warn vs block. Use when adding data quality validation, catching bad data before it reaches consumers, setting up freshness/volume checks, or defining expectations.

Original author description

Add data quality checks to pipelines — freshness, volume/row-count anomalies, schema drift, null/uniqueness/referential integrity, and value distributions — using dbt tests, Great Expectations, or Soda, and deciding warn vs block. Use when adding data quality validation, catching bad data before it reaches consumers, setting up freshness/volume checks, or defining expectations.

How it works

  1. When to use
  2. The six dimensions to cover
  3. Workflow
  4. Patterns
  5. Common pitfalls
Original SKILL.md headings · 79 lines, 442 words
SKILL.md preview
---
name: implementing-data-quality-checks
description: Add data quality checks to pipelines — freshness, volume/row-count anomalies, schema drift, null/uniqueness/referential integrity, and value distributions — using dbt tests, Great Expectations, or Soda, and deciding warn vs block. Use when adding data quality validation, catching bad data before it reaches consumers, setting up freshness/volume checks, or defining expectations.
---

# Implementing Data Quality Checks

## When to use

- Adding validation so bad data is caught before consumers see it.
- Setting up freshness, volume, schema-drift, or integrity checks.
- Choosing a tool (dbt tests, Great Expectations, Soda) and where to place checks.
- Do NOT use for dbt-specific test syntax only (use `testing-dbt-projects` for
  the dbt details).

## The six dimensions to cover

1. **Freshness** — did the data arrive on time? (max load timestamp vs SLA)
2. **Volume** — is the row count in the expected range? (catches partial loads and
   duplication)
3. **Schema** — did columns/types change unexpectedly? (drift)
4. **Completeness** — required fields not null.
5. **Uniqueness / integrity** — keys unique, foreign keys valid.
6. **Validity / distribution** — values in allowed ranges/sets; no sudden
   distribution shifts.

## Workflow

```
- [ ] Put blocking checks at the entry point (raw/staging) so bad data stops early
- [ ] Cover the six dimensions on critical tables
- [ ] Decide warn vs block per check by business impact
- [ ] Route failures to alerts + a quarantine/failure store
- [ ] Add reconciliation between source and target for critical flows
```

1. **Check early.** Validate at ingestion/staging so bad data fails fast rather
   than propagating into marts and dashboards.
2. **Cover the six dimensions** on tables that feed decisions/SLAs.
3. **Warn vs block.** Block (fail the run) on integrity-critical checks; warn on
   soft signals. Blocking everything causes alert fatigue and pipeline gridlock.
4. **Act on failures** — alert an owner and store failing rows for triage;
   optionally quarantine and continue.

## Patterns

**Great Expectations** — declarative expectations on a batch:

```python
validator.expect_column_values_to_not_be_null("order_id")
validator.expect_column_values_to_be_unique("order_id")
validator.expect_column_values_to_be_between("amount", min_value=0)
validator.expect_table_row_count_to_be_between(min_value=1000)
```

**Soda (SodaCL)** — freshness, volume, and integrity in YAML:

```yaml
checks for fct_orders:
  - freshness(ordered_at) < 24h
  - row_count between 1000 and 1000000
  - missing_count(order_id) = 0
  - duplicate_count(order_id) = 0
```

**Volume anomaly** — compare today's count to the trailing average and alert on a
large deviation, catching partial loads and accidental duplication.

## Common pitfalls

- **Checking only at the end** — bad data has already reached consumers; validate
  at entry.
- **Blocking on every soft check** — alert fatigue; reserve blocking for
  integrity-critical rules.
- **No owner/routing** — failures no one sees are worthless; route to a person and
  a failure store.
- **Freshness ignored** — stale data looks "correct" but is wrong; it is the most
  commonly missed dimension.
- **No reconciliation** — row-level checks pass while totals drift from source;
  add source-vs-target count/sum reconciliation for critical flows.
- **Static thresholds on seasonal data** — use trailing baselines, not fixed
  numbers, where volume varies.

Versions

The database has no other skills named implementing-data-quality-checks.

Database snapshot: 13.11.2026 · package version 81d9342b108e (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 90data-data-quality
    Use this skill when asked about data quality, data validation, data profiling, Great Expectations, dbt tests, data observability, Soda, Monte Carlo, data contracts, schema validation, or data SLAs. This skill enforces: data quality dimensions (completeness, accuracy, timeliness,…
  • 90data-quality-conventions
    Reference catalog of data-quality conventions - when to choose dbt-tests vs Great Expectations vs Soda, column-level vs table-level coverage, severity tiering, SLA and freshness conventions, and common anti-patterns to avoid. Use when designing coverage for a new data product or…
  • 100data-quality-frameworks
    Implement data quality validation with Great Expectations, dbt tests, and data contracts. Use when building data quality pipelines, implementing validation rules, or establishing data contracts.
  • 95data-quality-frameworks
    Implement data quality validation with Great Expectations, dbt tests, and data contracts. Use when building data quality pipelines, implementing validation rules, or establishing data contracts.
  • 95data-testing
    Use this skill when asked about data testing, dbt unit testing, data-diff, datafold, Soda, Great Expectations, data regression testing, data quality testing, data contract testing, or data validation automation. This skill enforces: dbt unit testing with dbt-unit-testing…
  • 100authoring-data-quality-checks
    Adds and runs data quality checks (dbt-test style assertions) on a project's warehouse tables and saved-query views, and HogQL catalog metrics: not-null, uniqueness, accepted values, referential integrity, row-count bounds, freshness, and custom HogQL. Metrics support custom SQL…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.