Skip to content
Skills Library
Data & analytics · Automation

implementing-data-cicd

/implementing-data-cicd

Set up CI/CD for data pipelines — SQL/dbt linting (SQLFluff), compilation and test gates, dbt Slim CI with state:modified and deferral, environment promotion (dev/staging/prod), and running only changed models on pull requests. Use when adding CI checks to a dbt or SQL project, automating pipeline tests on PRs, speeding up CI, or promoting changes across environments.

Licence
Apache-2.0
Last repository change
30.08.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 30.08.2026, without references to retired models or APIs.

Licence

Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.

What it does and when to use it

Set up CI/CD for data pipelines — SQL/dbt linting (SQLFluff), compilation and test gates, dbt Slim CI with state:modified and deferral, environment promotion (dev/staging/prod), and running only changed models on pull requests. Use when adding CI checks to a dbt or SQL project, automating pipeline tests on PRs, speeding up CI, or promoting changes across environments.

Original author description

Set up CI/CD for data pipelines — SQL/dbt linting (SQLFluff), compilation and test gates, dbt Slim CI with state:modified and deferral, environment promotion (dev/staging/prod), and running only changed models on pull requests. Use when adding CI checks to a dbt or SQL project, automating pipeline tests on PRs, speeding up CI, or promoting changes across environments.

How it works

  1. When to use
  2. Workflow
  3. Patterns
  4. Common pitfalls
Original SKILL.md headings · 77 lines, 421 words
SKILL.md preview
---
name: implementing-data-cicd
description: Set up CI/CD for data pipelines — SQL/dbt linting (SQLFluff), compilation and test gates, dbt Slim CI with state:modified and deferral, environment promotion (dev/staging/prod), and running only changed models on pull requests. Use when adding CI checks to a dbt or SQL project, automating pipeline tests on PRs, speeding up CI, or promoting changes across environments.
---

# Implementing Data CI/CD

## When to use

- Adding CI checks to a dbt/SQL/pipeline repo.
- Automating lint, compile, and tests on pull requests.
- Speeding up CI by building only changed models (Slim CI).
- Promoting changes across dev → staging → prod.
- Do NOT use for reviewing an individual PR's logic (use
  `reviewing-data-pipeline-code`).

## Workflow

```
- [ ] Lint SQL (SQLFluff) and check formatting on every PR
- [ ] Compile the project to catch ref/Jinja errors early
- [ ] Build + test only changed models against prod parents (Slim CI)
- [ ] Run in an isolated CI schema; tear it down after
- [ ] Promote across environments with the same code, different targets
```

1. **Lint first** — SQLFluff catches style and some correctness issues fast and
   cheap, before spinning up a warehouse.
2. **Compile** — `dbt compile` (or `parse`) fails on bad `ref()`/Jinja without
   running SQL.
3. **Slim CI** — build and test only `state:modified+` using a production
   `manifest.json` as state, and **defer** unchanged parents to prod so CI doesn't
   rebuild the whole project.
4. **Isolate** — run into a unique CI schema (e.g. per PR) and drop it afterward so
   runs don't collide.
5. **Promote** — the same code runs against dev/staging/prod via `--target`; never
   fork logic per environment.

## Patterns

**dbt Slim CI (GitHub Actions sketch):**

```yaml
- run: dbt deps
- run: dbt compile
- run: |
    dbt build --select state:modified+ \
      --defer --state ./prod-artifacts \
      --target ci
```

`--select state:modified+` builds changed models and their children; `--defer
--state` points unchanged parents at prod, so CI stays fast. A fresh/stale
`prod-artifacts/manifest.json` is required for correct selection.

**SQLFluff config** (`.sqlfluff`) — pin the dialect and templater so lint matches
your warehouse and dbt:

```ini
[sqlfluff]
dialect = snowflake
templater = dbt
```

**Environment promotion** — dev (developer schemas) → staging (full build on
merge) → prod (scheduled). Same repo, different `profiles.yml` targets and
credentials.

## Common pitfalls

- **Rebuilding the whole project on every PR** — slow and expensive; use Slim CI
  with `state:modified` + deferral.
- **Stale/missing state manifest** — `state:modified` selects the wrong nodes;
  refresh prod artifacts in CI.
- **Shared CI schema across PRs** — concurrent runs clobber each other; use a
  unique schema per PR and drop it.
- **Skipping `dbt deps` in CI** — macro/package-not-found failures.
- **Different logic per environment** — drift and "works in dev" bugs; vary only
  the target/config, not the code.
- **No teardown** — orphaned CI schemas accumulate cost.

Versions

The database has no other skills named implementing-data-cicd.

Database snapshot: 13.11.2026 · package version 476dc2f88949 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100data-pipeline-cicd
    Use this skill when asked about data pipeline CI/CD, dbt Cloud CI/CD, SQLFluff, SQL linting, Atlantis for SQL, data pipeline testing, environment promotion, schema change management, dbt test, data quality in CI, dataops, or database change management. This skill enforces: dbt…
  • 90devops-dataops
    Use this skill when implementing DataOps: CI/CD for data pipelines, dbt CI/CD, SQLFluff, Great Expectations, data pipeline versioning, data testing in CI, data environment management, data contract CI, data lineage in CI/CD. Do NOT use for: data warehouse schema design (use…
  • 100test-generator
    Generates dbt tests including schema tests, data quality tests, and freshness checks. Use when adding tests to dbt models or implementing data quality validation.
  • 100dbt-test-creator
    Create dbt test creator operations. Auto-activating skill for Data Pipelines. Triggers on: dbt test creator, dbt test creator Part of the Data Pipelines skill category. Use when writing or running tests. Trigger with phrases like "dbt test creator", "dbt creator", "dbt".
  • 95dbt-bigquery
    Expert guidance for creating, modifying, and optimizing dbt pipelines for BigQuery. Use this skill whenever user asks for generating or modifying a dbt model or project. Activate this skill when the user - Creates, modifies, or troubleshoots **dbt models or pipelines** - Needs…
  • 100dbt-model-generator
    Generate dbt model generator operations. Auto-activating skill for Data Pipelines. Triggers on: dbt model generator, dbt model generator Part of the Data Pipelines skill category. Use when working with dbt model generator functionality. Trigger with phrases like "dbt model…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.