Skip to content
Skills Library
Data & analytics · Building & coding

data-engineering

/data-engineering

Design and operate data infrastructure — database operations (vector, relational, graph, time-series), ETL/ELT pipeline design (dbt patterns, incremental loading), SQL analytical patterns, data quality monitoring, schema migration, and storage infrastructure management, including model-assisted transformation contracts, deterministic acceptance, bounded retries, and sink reconciliation. Do not use for statistical analysis or ML model development.

Licence
MIT
Last repository change
25.10.2026

Assessment and security

85/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content20 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
14
Scripts
0
HTML comments
0
URLs in SKILL.md
1
Deprecation

No signals detected. Repository changed on 25.10.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Design and operate data infrastructure — database operations (vector, relational, graph, time-series), ETL/ELT pipeline design (dbt patterns, incremental loading), SQL analytical patterns, data quality monitoring, schema migration, and storage infrastructure management, including model-assisted transformation contracts, deterministic acceptance, bounded retries, and sink reconciliation. Do not use for statistical analysis or ML model development.

Original author description

Design and operate data infrastructure — database operations (vector, relational, graph, time-series), ETL/ELT pipeline design (dbt patterns, incremental loading), SQL analytical patterns, data quality monitoring, schema migration, and storage infrastructure management, including model-assisted transformation contracts, deterministic acceptance, bounded retries, and sink reconciliation. Do not use for statistical analysis or ML model development.

How it works

  1. The Data Engineer's Domain
  2. Reference Files
  3. Related Skills
  4. Core Principles
Original SKILL.md headings · 52 lines, 644 words
SKILL.md preview
---
name: data-engineering
description: Design and operate data infrastructure — database operations (vector,
  relational, graph, time-series), ETL/ELT pipeline design (dbt patterns, incremental
  loading), SQL analytical patterns, data quality monitoring, schema migration, and
  storage infrastructure management, including model-assisted transformation contracts,
  deterministic acceptance, bounded retries, and sink reconciliation. Do not use for statistical analysis or ML model
  development.
license: MIT
metadata:
  tags: data-engineering, etl, dbt, sql, database, graph-db, time-series, vector-db,
    migration, data-quality, storage, influxdb, neo4j
  source_repo: https://github.com/magnus919/hermes-profiles
---

# Data Engineering Methodology

Data engineering is the operational backbone of data-driven systems. This methodology covers running, maintaining, and evolving data infrastructure — from relational databases and vector stores to graph databases, time-series stores, and the transformation pipelines that move data between them.

When a model proposes data transformations, load [the AI boundary workflow](references/ai-transformation-boundaries.md)
and use [the companion record](templates/ai-stage-contract.md).

## The Data Engineer's Domain

| You own | You don't own |
|---------|--------------|
| Database operations — schema management, indexing, backup/recovery, migration across relational, vector, graph, and time-series stores | Data modeling and schema design — that's the data architect |
| Data transformation pipelines — dbt models, ETL/ELT patterns, incremental loading, incremental strategies | Statistical analysis and experiments — that's the data scientist |
| Analytical SQL — window functions, CTEs, query optimization, execution plan analysis, star schema queries | Training infrastructure and model deployment — that's the ML engineer |
| Graph database operations — Neo4j data modeling, Cypher queries, graph algorithms, import/export | Application-level data access patterns — that's the developer |
| Time-series database operations — InfluxDB schema design, downsampling, retention policies, Telegraf | Infrastructure provisioning — that's the platform engineer |
| Data quality monitoring — integrity checks, deduplication, anomaly detection, freshness validation | Visual dashboard design — that's the analyst / product-design-and-ux |
| Storage infrastructure — capacity planning, performance tuning, archival strategies | 

## Reference Files

| Reference | When to load |
|-----------|-------------|
| `references/sql-analytical-patterns.md` | Writing analytical SQL — window functions, CTEs, execution plan reading, star schema queries, engine-specific optimization (PostgreSQL, DuckDB, ClickHouse, BigQuery, Snowflake) |
| `references/dbt-patterns.md` | Designing data transformation pipelines with dbt — project structure, modeling layers (staging/intermediate/facts/dimensions), materializations, tests, snapshots, Jinja macros, CI/CD, dbt Mesh |
| `references/etl-pipeline-design.md` | Building reliable data pipelines — extraction strategies (full, incremental, CDC), transformation layers, validation gates, error handling, idempotency |
| `references/data-quality.md` | Monitoring data integrity — quality dimensions, validation rule types, anomaly detection, deduplication strategies, pipeline health signals |
| `references/graph-databases.md` | Working with graph databases — Neo4j data modeling, Cypher query patterns (traversal, aggregation, pathfinding), import strategies, graph algorithms, pipeline integration |
| `references/time-series-databases.md` | Working with time-series databases — InfluxDB data model (measurements, tags, fields), schema design (cardinality), downsampling, retention, Telegraf ingest, comparison with TimescaleDB/QuestDB/Prometheus |
| `references/vector-db-operations.md` | Managing vector databases — Milvus, Qdrant, Chroma — index types, collection lifecycle, dimension migrations, backup strategies |
| `references/database-migrations.md` | Schema evolution — zero-downtime migration patterns, rollback planning, versioned schemas, test-first migrations |
| `references/backup-and-recovery.md` | Backup strategies per data store type, RPO/RTO planning, WAL archiving, snapshot management, recovery plan template |
| `references/ai-transformation-boundaries.md` | Defining acceptance, retry, cost, and publication boundaries for model-assisted transformations |

## Related Skills

- [postgres](../postgres/SKILL.md) — operating a PostgreSQL server itself: configuration review, index and query-plan diagnosis, vacuum/bloat management, WAL archiving and point-in-time recovery, replication and failover, upgrades. This skill owns the engine-specific runbooks; data-engineering owns the engine-neutral methodology.
- [supabase](../supabase/SKILL.md) — Supabase platform operations: migrations, RLS, Auth, Storage, Functions, and self-hosting. To measure an agent's Supabase task competence, use its [agent evals harness reference](../supabase/references/agent-evals.md).

## Core Principles

**Data without integrity is noise** — No pipeline, model, or dashboard is worth more than the quality of the data feeding it. Validate at every boundary.

**Design for operability** — Every database, pipeline, and store needs monitoring, backup, and recovery procedures defined before it goes to production. If you can't detect failure, you can't recover from it.

**Idempotency is a requirement** — Every pipeline should produce the same result whether it runs once or twice. Duplicate handling is not optional.

**Schema changes are code changes** — Every migration needs review, testing, and a rollback plan. Schema drift is technical debt with compounding interest.

**Know your storage characteristics** — Access patterns, retention requirements, growth rates, and consistency guarantees determine the right storage architecture. Choose based on data, not familiarity.
Files in this package (14)
  • README.md 2 KB
  • SKILL.md 6 KB
  • evals/evals.json 10 KB
  • references/ai-transformation-boundaries.md 4 KB
  • references/backup-and-recovery.md 4 KB
  • references/data-quality.md 3 KB
  • references/database-migrations.md 1 KB
  • references/dbt-patterns.md 36 KB
  • references/etl-pipeline-design.md 4 KB
  • references/graph-databases.md 38 KB
  • references/sql-analytical-patterns.md 42 KB
  • references/time-series-databases.md 35 KB
  • references/vector-db-operations.md 2 KB
  • templates/ai-stage-contract.md 1 KB

Versions

RepositoryScoreLicence
This page · magnus919/agent-skills85MIT
travisjneuman/.claude100MIT
fengshao1227/ccg-workflow100MIT
mhingan/agent-forge100MIT
albkue/my-skill100MIT
borhen68/skillengine100MIT
swap-cameroon/kuro-antigravity-config90MIT
andreibesleaga/gabbe90CC-BY-4.0
cbrock84/headcount90MIT

Database snapshot: 13.11.2026 · package version 5baa9aca9443 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 95senior-data-engineer
    World-class data engineering skill for building scalable data pipelines, ETL/ELT systems, and data infrastructure. Expertise in Python, SQL, Spark, Airflow, dbt, Kafka, and modern data stack. Includes data modeling, pipeline orchestration, data quality, and DataOps. Use when…
  • 100data-pipeline
    Data pipeline and analytics engineering patterns. Use for ETL/ELT design, data modeling, event tracking, data quality, and dashboard creation.
  • 100data-engineer
    Use this agent when you need to design, build, or optimize data pipelines, ETL/ELT processes, and data infrastructure. Invoke when designing data platforms, implementing pipeline orchestration, handling data quality issues, or optimizing data processing costs.
  • 90data-etl-pipeline
    Use this skill when asked about ETL, ELT, data pipeline, Airflow, dbt, data transformation, data ingestion, batch processing, or pipeline orchestration. This skill enforces: pipeline architecture with Airflow DAG design, dbt transformation with incremental loading, error…
  • 100data-engineer
    Use this agent when you need to design, build, or optimize data pipelines, ETL/ELT processes, and data infrastructure. Invoke when designing data platforms, implementing pipeline orchestration, handling data quality issues, or optimizing data processing costs.
  • 100senior-data-engineer
    Data engineering skill for building scalable data pipelines, ETL/ELT systems, and data infrastructure. Expertise in Python, SQL, Spark, Airflow, dbt, Kafka, and modern data stack. Includes data modeling, pipeline orchestration, data quality, and DataOps. Use when designing data…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.