data-engineering
/data-engineering
Design reliable data pipelines: ingestion, ELT/dbt, orchestration, modeling, data quality, idempotency, lineage, cost. Use when building or reviewing ETL/ELT, analytics tables, streaming or batch jobs.
- Author
- swap-cameroon ↗
- Repository
- kuro-antigravity-config ↗
- Licence
- MIT
- Last repository change
- 11.11.2026
Assessment and security
- Files in this package
- 1
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 11.11.2026, without references to retired models or APIs.
MIT, open: use and modify while retaining author attribution.
What it does and when to use it
Design reliable data pipelines: ingestion, ELT/dbt, orchestration, modeling, data quality, idempotency, lineage, cost. Use when building or reviewing ETL/ELT, analytics tables, streaming or batch jobs.
Original author description
Design reliable data pipelines: ingestion, ELT/dbt, orchestration, modeling, data quality, idempotency, lineage, cost. Use when building or reviewing ETL/ELT, analytics tables, streaming or batch jobs.
How it works
- Architecture
- Pipeline properties (all required)
- Modeling
- Data quality (tests are part of the pipeline)
- Governance & security
- Performance & cost
- Review checklist
SKILL.md preview
--- name: data-engineering description: "Design reliable data pipelines: ingestion, ELT/dbt, orchestration, modeling, data quality, idempotency, lineage, cost. Use when building or reviewing ETL/ELT, analytics tables, streaming or batch jobs." license: MIT metadata: version: "1.0.0" --- # Data engineering Relational specifics: `postgres`. Observability of jobs: `observability`. Application-side schemas: `spring-data-jpa`. ## Architecture - Prefer **ELT**: land raw data unchanged (immutable, partitioned by ingestion date), transform in the warehouse/lakehouse with SQL (dbt/SQLMesh). Layers: `raw/bronze` -> `staging/silver` (typed, deduped, renamed) -> `marts/gold` (business-ready, tested, documented). - Batch by default; streaming (Kafka/Kinesis/Flink/Spark Structured Streaming) only when latency requirements (minutes or less) justify the operational cost. Define freshness SLAs per dataset. - Storage formats: Parquet/Iceberg/Delta for lakes (columnar, schema evolution, time travel); partition by low-cardinality date columns; avoid tiny files (compaction). ## Pipeline properties (all required) - **Idempotent**: re-running a window yields the same result (MERGE/upsert by business key, overwrite partitions, deterministic IDs). No append-only without dedupe. - **Incremental**: process by watermark/CDC (Debezium) not full reload; support **backfill** for a date range with the same code. - **Replayable**: keep raw inputs; transformations are pure functions of inputs + code version. - **Observable**: run metadata (rows in/out, duration, freshness), alerts on failure and on late data; retries with backoff; dead-letter for bad records. - Small, composable tasks in the orchestrator (Airflow/Dagster/Prefect); dependencies explicit; no business logic inside DAG definitions; secrets from a manager. ## Modeling Dimensional modeling (facts at a declared grain + conformed dimensions) for analytics; surrogate keys; slowly changing dimensions type 2 where history matters; one definition per metric in a semantic/metrics layer; naming conventions (`dim_`, `fct_`, snake_case, `_at` timestamps UTC, `_id` keys). ## Data quality (tests are part of the pipeline) Not-null, unique, accepted values, referential integrity, volume/freshness anomalies, distribution drift (dbt tests, Great Expectations, Soda). Fail the run or quarantine on critical breaks; publish a data-quality status per dataset. Contracts with producers (schema + semantics + SLA), versioned; breaking changes announced. ## Governance & security PII classified and tagged; minimize, mask/tokenize, column- and row-level access policies; retention and deletion (GDPR/Loi 25 style requests must propagate to derived tables); audit access; lineage via OpenLineage/dbt docs; encrypt at rest and in transit; least-privilege service accounts. ## Performance & cost Prune partitions, filter early, avoid `SELECT *` and cross joins, pre-aggregate heavy marts, choose clustering/sort keys by query patterns, right-size compute and auto-suspend, monitor cost per pipeline and per query, expire stale intermediates. ## Review checklist Grain stated - idempotent and backfillable - tests on keys and freshness - raw preserved - PII handled - SLA and alerts defined - documented owners - cost estimated.
Versions
| Repository | Score | Licence |
|---|---|---|
| This page · swap-cameroon/kuro-antigravity-config | 90 | MIT |
| travisjneuman/.claude | 100 | MIT |
| fengshao1227/ccg-workflow | 100 | MIT |
| mhingan/agent-forge | 100 | MIT |
| albkue/my-skill | 100 | MIT |
| borhen68/skillengine | 100 | MIT |
| andreibesleaga/gabbe | 90 | CC-BY-4.0 |
| cbrock84/headcount | 90 | MIT |
| magnus919/agent-skills | 85 | MIT |
Database snapshot: 13.11.2026 · package version 7bf5451aa569 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100data-pipeline
Data pipeline and analytics engineering patterns. Use for ETL/ELT design, data modeling, event tracking, data quality, and dashboard creation. - 100data-engineer
Use this agent when you need to design, build, or optimize data pipelines, ETL/ELT processes, and data infrastructure. Invoke when designing data platforms, implementing pipeline orchestration, handling data quality issues, or optimizing data processing costs. - 100data-engineer
Use this agent when you need to design, build, or optimize data pipelines, ETL/ELT processes, and data infrastructure. Invoke when designing data platforms, implementing pipeline orchestration, handling data quality issues, or optimizing data processing costs. - 100engineering-data-engineer
Use when Codex should act as the Data Engineer specialist from Agency Agents. Expert data engineer specializing in building reliable data pipelines, lakehouse architectures, and scalable data infrastructure. Masters ETL/ELT, Apache Spark, dbt, streaming systems, and cloud data… - 95senior-data-engineer
World-class data engineering skill for building scalable data pipelines, ETL/ELT systems, and data infrastructure. Expertise in Python, SQL, Spark, Airflow, dbt, Kafka, and modern data stack. Includes data modeling, pipeline orchestration, data quality, and DataOps. Use when… - 90data-engineer
Data Engineer (/data) — designs and builds data pipelines: ETL/ELT, dbt transformations, warehouses/lakehouses (BigQuery, Snowflake, DuckDB, Postgres), streaming (Kafka, Flink), orchestration (Airflow, Dagster), and data quality. Use when building ingestion/transformation…