implementing-data-cicd
/implementing-data-cicd
Konfiguruje CI/CD dla potoków danych: linting SQL i dbt przez SQLFluff, bramki kompilacji i testów, dbt Slim CI ze state:modified, promocja między środowiskami i uruchamianie tylko zmienionych modeli.
- Autor
- unknown-333 ↗
- Repozytorium
- awesome-data-engineering-skills ↗
- Licencja
- Apache-2.0
- Ostatnia zmiana repozytorium
- 30.08.2026
Ocena i bezpieczeństwo
- Pliki w paczce
- 1
- Skrypty
- 0
- Komentarze HTML
- 0
- Adresy URL w SKILL.md
- 0
Brak sygnałów. Repozytorium zmienione 30.08.2026, bez odwołań do wycofanych modeli i API.
Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.
Co robi i kiedy
Konfiguruje CI/CD dla potoków danych: linting SQL i dbt przez SQLFluff, bramki kompilacji i testów, dbt Slim CI ze state:modified, promocja między środowiskami i uruchamianie tylko zmienionych modeli.
Oryginalny opis (EN)
Set up CI/CD for data pipelines — SQL/dbt linting (SQLFluff), compilation and test gates, dbt Slim CI with state:modified and deferral, environment promotion (dev/staging/prod), and running only changed models on pull requests. Use when adding CI checks to a dbt or SQL project, automating pipeline tests on PRs, speeding up CI, or promoting changes across environments.
Jak działa
- When to use
- Workflow
- Patterns
- Common pitfalls
Podgląd SKILL.md
---
name: implementing-data-cicd
description: Set up CI/CD for data pipelines — SQL/dbt linting (SQLFluff), compilation and test gates, dbt Slim CI with state:modified and deferral, environment promotion (dev/staging/prod), and running only changed models on pull requests. Use when adding CI checks to a dbt or SQL project, automating pipeline tests on PRs, speeding up CI, or promoting changes across environments.
---
# Implementing Data CI/CD
## When to use
- Adding CI checks to a dbt/SQL/pipeline repo.
- Automating lint, compile, and tests on pull requests.
- Speeding up CI by building only changed models (Slim CI).
- Promoting changes across dev → staging → prod.
- Do NOT use for reviewing an individual PR's logic (use
`reviewing-data-pipeline-code`).
## Workflow
```
- [ ] Lint SQL (SQLFluff) and check formatting on every PR
- [ ] Compile the project to catch ref/Jinja errors early
- [ ] Build + test only changed models against prod parents (Slim CI)
- [ ] Run in an isolated CI schema; tear it down after
- [ ] Promote across environments with the same code, different targets
```
1. **Lint first** — SQLFluff catches style and some correctness issues fast and
cheap, before spinning up a warehouse.
2. **Compile** — `dbt compile` (or `parse`) fails on bad `ref()`/Jinja without
running SQL.
3. **Slim CI** — build and test only `state:modified+` using a production
`manifest.json` as state, and **defer** unchanged parents to prod so CI doesn't
rebuild the whole project.
4. **Isolate** — run into a unique CI schema (e.g. per PR) and drop it afterward so
runs don't collide.
5. **Promote** — the same code runs against dev/staging/prod via `--target`; never
fork logic per environment.
## Patterns
**dbt Slim CI (GitHub Actions sketch):**
```yaml
- run: dbt deps
- run: dbt compile
- run: |
dbt build --select state:modified+ \
--defer --state ./prod-artifacts \
--target ci
```
`--select state:modified+` builds changed models and their children; `--defer
--state` points unchanged parents at prod, so CI stays fast. A fresh/stale
`prod-artifacts/manifest.json` is required for correct selection.
**SQLFluff config** (`.sqlfluff`) — pin the dialect and templater so lint matches
your warehouse and dbt:
```ini
[sqlfluff]
dialect = snowflake
templater = dbt
```
**Environment promotion** — dev (developer schemas) → staging (full build on
merge) → prod (scheduled). Same repo, different `profiles.yml` targets and
credentials.
## Common pitfalls
- **Rebuilding the whole project on every PR** — slow and expensive; use Slim CI
with `state:modified` + deferral.
- **Stale/missing state manifest** — `state:modified` selects the wrong nodes;
refresh prod artifacts in CI.
- **Shared CI schema across PRs** — concurrent runs clobber each other; use a
unique schema per PR and drop it.
- **Skipping `dbt deps` in CI** — macro/package-not-found failures.
- **Different logic per environment** — drift and "works in dev" bugs; vary only
the target/config, not the code.
- **No teardown** — orphaned CI schemas accumulate cost.
Wersje
W bazie nie ma innych skilli o nazwie implementing-data-cicd.
Migawka bazy z 13.11.2026 · wersja paczki 476dc2f88949 (skrót SHA-256 całej paczki)
Pobrania w ostatnich 30 dniach: 0 · ranking
Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.
Podobne skille
- 100data-pipeline-cicd
Wdraża CI/CD dla potoków danych: dbt Cloud z bramkami promocji środowisk, SQLFluff, zarządzanie zmianami schematu, testy potoków (dbt test, data diff) i jakość danych w CI. - 90devops-dataops
Wdraża DataOps: CI/CD dla pipeline'ów danych, testy dbt, SQLFluff, Great Expectations i kontrakty danych. - 100test-generator
Generuje testy dbt: schematyczne, jakości danych i świeżości, dla modeli dbt i walidacji danych. - 100dbt-test-creator
Tworzy testy dbt w ramach potoków danych i uruchamia się automatycznie na frazy związane z pisaniem lub uruchamianiem testów dbt. - 95dbt-bigquery
Tworzy i optymalizuje modele oraz potoki dbt dla BigQuery, poprawia SQL w projekcie i konfiguruje nowy projekt dbt. - 100dbt-model-generator
Generuje modele dbt w ramach potoków danych i uruchamia się automatycznie na frazy związane z generatorem modeli dbt.