implementing-data-cicd
/implementing-data-cicd
Set up CI/CD for data pipelines — SQL/dbt linting (SQLFluff), compilation and test gates, dbt Slim CI with state:modified and deferral, environment promotion (dev/staging/prod), and running only changed models on pull requests. Use when adding CI checks to a dbt or SQL project, automating pipeline tests on PRs, speeding up CI, or promoting changes across environments.
- Author
- unknown-333 ↗
- Repository
- awesome-data-engineering-skills ↗
- Licence
- Apache-2.0
- Last repository change
- 30.08.2026
Assessment and security
- Files in this package
- 1
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 30.08.2026, without references to retired models or APIs.
Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.
What it does and when to use it
Set up CI/CD for data pipelines — SQL/dbt linting (SQLFluff), compilation and test gates, dbt Slim CI with state:modified and deferral, environment promotion (dev/staging/prod), and running only changed models on pull requests. Use when adding CI checks to a dbt or SQL project, automating pipeline tests on PRs, speeding up CI, or promoting changes across environments.
Original author description
Set up CI/CD for data pipelines — SQL/dbt linting (SQLFluff), compilation and test gates, dbt Slim CI with state:modified and deferral, environment promotion (dev/staging/prod), and running only changed models on pull requests. Use when adding CI checks to a dbt or SQL project, automating pipeline tests on PRs, speeding up CI, or promoting changes across environments.
How it works
- When to use
- Workflow
- Patterns
- Common pitfalls
SKILL.md preview
---
name: implementing-data-cicd
description: Set up CI/CD for data pipelines — SQL/dbt linting (SQLFluff), compilation and test gates, dbt Slim CI with state:modified and deferral, environment promotion (dev/staging/prod), and running only changed models on pull requests. Use when adding CI checks to a dbt or SQL project, automating pipeline tests on PRs, speeding up CI, or promoting changes across environments.
---
# Implementing Data CI/CD
## When to use
- Adding CI checks to a dbt/SQL/pipeline repo.
- Automating lint, compile, and tests on pull requests.
- Speeding up CI by building only changed models (Slim CI).
- Promoting changes across dev → staging → prod.
- Do NOT use for reviewing an individual PR's logic (use
`reviewing-data-pipeline-code`).
## Workflow
```
- [ ] Lint SQL (SQLFluff) and check formatting on every PR
- [ ] Compile the project to catch ref/Jinja errors early
- [ ] Build + test only changed models against prod parents (Slim CI)
- [ ] Run in an isolated CI schema; tear it down after
- [ ] Promote across environments with the same code, different targets
```
1. **Lint first** — SQLFluff catches style and some correctness issues fast and
cheap, before spinning up a warehouse.
2. **Compile** — `dbt compile` (or `parse`) fails on bad `ref()`/Jinja without
running SQL.
3. **Slim CI** — build and test only `state:modified+` using a production
`manifest.json` as state, and **defer** unchanged parents to prod so CI doesn't
rebuild the whole project.
4. **Isolate** — run into a unique CI schema (e.g. per PR) and drop it afterward so
runs don't collide.
5. **Promote** — the same code runs against dev/staging/prod via `--target`; never
fork logic per environment.
## Patterns
**dbt Slim CI (GitHub Actions sketch):**
```yaml
- run: dbt deps
- run: dbt compile
- run: |
dbt build --select state:modified+ \
--defer --state ./prod-artifacts \
--target ci
```
`--select state:modified+` builds changed models and their children; `--defer
--state` points unchanged parents at prod, so CI stays fast. A fresh/stale
`prod-artifacts/manifest.json` is required for correct selection.
**SQLFluff config** (`.sqlfluff`) — pin the dialect and templater so lint matches
your warehouse and dbt:
```ini
[sqlfluff]
dialect = snowflake
templater = dbt
```
**Environment promotion** — dev (developer schemas) → staging (full build on
merge) → prod (scheduled). Same repo, different `profiles.yml` targets and
credentials.
## Common pitfalls
- **Rebuilding the whole project on every PR** — slow and expensive; use Slim CI
with `state:modified` + deferral.
- **Stale/missing state manifest** — `state:modified` selects the wrong nodes;
refresh prod artifacts in CI.
- **Shared CI schema across PRs** — concurrent runs clobber each other; use a
unique schema per PR and drop it.
- **Skipping `dbt deps` in CI** — macro/package-not-found failures.
- **Different logic per environment** — drift and "works in dev" bugs; vary only
the target/config, not the code.
- **No teardown** — orphaned CI schemas accumulate cost.
Versions
The database has no other skills named implementing-data-cicd.
Database snapshot: 13.11.2026 · package version 476dc2f88949 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100data-pipeline-cicd
Use this skill when asked about data pipeline CI/CD, dbt Cloud CI/CD, SQLFluff, SQL linting, Atlantis for SQL, data pipeline testing, environment promotion, schema change management, dbt test, data quality in CI, dataops, or database change management. This skill enforces: dbt… - 90devops-dataops
Use this skill when implementing DataOps: CI/CD for data pipelines, dbt CI/CD, SQLFluff, Great Expectations, data pipeline versioning, data testing in CI, data environment management, data contract CI, data lineage in CI/CD. Do NOT use for: data warehouse schema design (use… - 100test-generator
Generates dbt tests including schema tests, data quality tests, and freshness checks. Use when adding tests to dbt models or implementing data quality validation. - 100dbt-test-creator
Create dbt test creator operations. Auto-activating skill for Data Pipelines. Triggers on: dbt test creator, dbt test creator Part of the Data Pipelines skill category. Use when writing or running tests. Trigger with phrases like "dbt test creator", "dbt creator", "dbt". - 95dbt-bigquery
Expert guidance for creating, modifying, and optimizing dbt pipelines for BigQuery. Use this skill whenever user asks for generating or modifying a dbt model or project. Activate this skill when the user - Creates, modifies, or troubleshoots **dbt models or pipelines** - Needs… - 100dbt-model-generator
Generate dbt model generator operations. Auto-activating skill for Data Pipelines. Triggers on: dbt model generator, dbt model generator Part of the Data Pipelines skill category. Use when working with dbt model generator functionality. Trigger with phrases like "dbt model…