Skip to content
Skills Library
Data & analytics · Building & coding

data-engineering

/data-engineering

Design reliable data pipelines: ingestion, ELT/dbt, orchestration, modeling, data quality, idempotency, lineage, cost. Use when building or reviewing ETL/ELT, analytics tables, streaming or batch jobs.

Licence
MIT
Last repository change
11.11.2026

Assessment and security

90/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content25 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 11.11.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Design reliable data pipelines: ingestion, ELT/dbt, orchestration, modeling, data quality, idempotency, lineage, cost. Use when building or reviewing ETL/ELT, analytics tables, streaming or batch jobs.

Original author description

Design reliable data pipelines: ingestion, ELT/dbt, orchestration, modeling, data quality, idempotency, lineage, cost. Use when building or reviewing ETL/ELT, analytics tables, streaming or batch jobs.

How it works

  1. Architecture
  2. Pipeline properties (all required)
  3. Modeling
  4. Data quality (tests are part of the pipeline)
  5. Governance & security
  6. Performance & cost
  7. Review checklist
Original SKILL.md headings · 32 lines, 394 words
SKILL.md preview
---
name: data-engineering
description: "Design reliable data pipelines: ingestion, ELT/dbt, orchestration, modeling, data quality, idempotency, lineage, cost. Use when building or reviewing ETL/ELT, analytics tables, streaming or batch jobs."
license: MIT
metadata:
  version: "1.0.0"
---

# Data engineering

Relational specifics: `postgres`. Observability of jobs: `observability`. Application-side schemas: `spring-data-jpa`.

## Architecture
- Prefer **ELT**: land raw data unchanged (immutable, partitioned by ingestion date), transform in the warehouse/lakehouse with SQL (dbt/SQLMesh). Layers: `raw/bronze` -> `staging/silver` (typed, deduped, renamed) -> `marts/gold` (business-ready, tested, documented).
- Batch by default; streaming (Kafka/Kinesis/Flink/Spark Structured Streaming) only when latency requirements (minutes or less) justify the operational cost. Define freshness SLAs per dataset.
- Storage formats: Parquet/Iceberg/Delta for lakes (columnar, schema evolution, time travel); partition by low-cardinality date columns; avoid tiny files (compaction).

## Pipeline properties (all required)
- **Idempotent**: re-running a window yields the same result (MERGE/upsert by business key, overwrite partitions, deterministic IDs). No append-only without dedupe.
- **Incremental**: process by watermark/CDC (Debezium) not full reload; support **backfill** for a date range with the same code.
- **Replayable**: keep raw inputs; transformations are pure functions of inputs + code version.
- **Observable**: run metadata (rows in/out, duration, freshness), alerts on failure and on late data; retries with backoff; dead-letter for bad records.
- Small, composable tasks in the orchestrator (Airflow/Dagster/Prefect); dependencies explicit; no business logic inside DAG definitions; secrets from a manager.

## Modeling
Dimensional modeling (facts at a declared grain + conformed dimensions) for analytics; surrogate keys; slowly changing dimensions type 2 where history matters; one definition per metric in a semantic/metrics layer; naming conventions (`dim_`, `fct_`, snake_case, `_at` timestamps UTC, `_id` keys).

## Data quality (tests are part of the pipeline)
Not-null, unique, accepted values, referential integrity, volume/freshness anomalies, distribution drift (dbt tests, Great Expectations, Soda). Fail the run or quarantine on critical breaks; publish a data-quality status per dataset. Contracts with producers (schema + semantics + SLA), versioned; breaking changes announced.

## Governance & security
PII classified and tagged; minimize, mask/tokenize, column- and row-level access policies; retention and deletion (GDPR/Loi 25 style requests must propagate to derived tables); audit access; lineage via OpenLineage/dbt docs; encrypt at rest and in transit; least-privilege service accounts.

## Performance & cost
Prune partitions, filter early, avoid `SELECT *` and cross joins, pre-aggregate heavy marts, choose clustering/sort keys by query patterns, right-size compute and auto-suspend, monitor cost per pipeline and per query, expire stale intermediates.

## Review checklist
Grain stated - idempotent and backfillable - tests on keys and freshness - raw preserved - PII handled - SLA and alerts defined - documented owners - cost estimated.

Versions

RepositoryScoreLicence
This page · swap-cameroon/kuro-antigravity-config90MIT
travisjneuman/.claude100MIT
fengshao1227/ccg-workflow100MIT
mhingan/agent-forge100MIT
albkue/my-skill100MIT
borhen68/skillengine100MIT
andreibesleaga/gabbe90CC-BY-4.0
cbrock84/headcount90MIT
magnus919/agent-skills85MIT

Database snapshot: 13.11.2026 · package version 7bf5451aa569 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100data-pipeline
    Data pipeline and analytics engineering patterns. Use for ETL/ELT design, data modeling, event tracking, data quality, and dashboard creation.
  • 100data-engineer
    Use this agent when you need to design, build, or optimize data pipelines, ETL/ELT processes, and data infrastructure. Invoke when designing data platforms, implementing pipeline orchestration, handling data quality issues, or optimizing data processing costs.
  • 100data-engineer
    Use this agent when you need to design, build, or optimize data pipelines, ETL/ELT processes, and data infrastructure. Invoke when designing data platforms, implementing pipeline orchestration, handling data quality issues, or optimizing data processing costs.
  • 100engineering-data-engineer
    Use when Codex should act as the Data Engineer specialist from Agency Agents. Expert data engineer specializing in building reliable data pipelines, lakehouse architectures, and scalable data infrastructure. Masters ETL/ELT, Apache Spark, dbt, streaming systems, and cloud data…
  • 95senior-data-engineer
    World-class data engineering skill for building scalable data pipelines, ETL/ELT systems, and data infrastructure. Expertise in Python, SQL, Spark, Airflow, dbt, Kafka, and modern data stack. Includes data modeling, pipeline orchestration, data quality, and DataOps. Use when…
  • 90data-engineer
    Data Engineer (/data) — designs and builds data pipelines: ETL/ELT, dbt transformations, warehouses/lakehouses (BigQuery, Snowflake, DuckDB, Postgres), streaming (Kafka, Flink), orchestration (Airflow, Dagster), and data quality. Use when building ingestion/transformation…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.