Przejdź do treści
Biblioteka skilli
Edukacja · Nauka i wyjaśnianie

data-engineering

/data-engineering

Pomaga uczyć się i stosować data engineering: potoki, ETL i ELT, Kafka, CDC, Spark, Airflow, dbt, hurtownie i projekty ćwiczeniowe.

Repozytorium
my-skill ↗
Licencja
MIT
Ostatnia zmiana repozytorium
30.10.2026

Ocena i bezpieczeństwo

100/ 100Klasa: dobry
Format25 / 25
Frontmatter z nazwą i opisem, poprawna nazwa, opis do 1024 znaków, nazwa zgodna z folderem.
Odkrywalność25 / 25
Opis mówi, kiedy użyć skilla, ma sensowną długość i jest w 3. osobie.
Treść35 / 35
Co najmniej 150 słów i do 500 linii, nagłówki, listy kroków, przykłady albo kod.
Higiena15 / 15
Bez zaślepek typu TODO, bez ścieżek Windows, z licencją.
Ryzykobrak sygnałów w skanie
Pliki w paczce
1
Skrypty
0
Komentarze HTML
0
Adresy URL w SKILL.md
0
Przestarzałość

Brak sygnałów. Repozytorium zmienione 30.10.2026, bez odwołań do wycofanych modeli i API.

Licencja

MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.

Co robi i kiedy

Pomaga uczyć się i stosować data engineering: potoki, ETL i ELT, Kafka, CDC, Spark, Airflow, dbt, hurtownie i projekty ćwiczeniowe.

Oryginalny opis (EN)

Help Sothy Eng learn and apply data engineering. Use for data pipelines, ETL/ELT, batch and streaming processing, Kafka, CDC, Debezium, Spark, Airflow, dbt, PostgreSQL replication, data warehouses, lakehouses, data modeling, event-driven architecture, data quality and monitoring, replay/backfill design, and small practical data-engineering projects. Do NOT use for backend-job work (backend-job), thesis writing (thesis-helper), learning roadmaps (learning-roadmap), or interview prep (interview-prep), but this skill pairs well with those for cross-domain learning.

Jak działa

  1. Purpose
  2. When to use
  3. When not to use
  4. Context intake
  5. Workflow
  6. Adaptation rules
  7. Inputs needed
  8. Output contract
  9. Rules and constraints
  10. Quality checklist
Sekcje z nagłówków SKILL.md, w oryginale · 87 linii, 962 słów
Podgląd SKILL.md
---
name: data-engineering
description: Help Sothy Eng learn and apply data engineering. Use for data pipelines, ETL/ELT, batch and streaming processing, Kafka, CDC, Debezium, Spark, Airflow, dbt, PostgreSQL replication, data warehouses, lakehouses, data modeling, event-driven architecture, data quality and monitoring, replay/backfill design, and small practical data-engineering projects. Do NOT use for backend-job work (backend-job), thesis writing (thesis-helper), learning roadmaps (learning-roadmap), or interview prep (interview-prep), but this skill pairs well with those for cross-domain learning.
version: 1.0.0
author: Sothy Eng
---

# Data Engineering

## Purpose
Guide Sothy from backend developer to competent data engineer: teach concepts with reasoning, design real pipelines, and build small end-to-end projects that go on the CV — with a bias toward simple, cheap, practical stacks.

## When to use
- Learning a concept (ETL vs. ELT, CDC, watermarking, partitioning) or building with any tool in that space: Kafka, Debezium, Airflow, dbt, Spark, PostgreSQL logical replication, warehouse/lakehouse.
- Designing an ingestion pipeline, choosing source→destination architecture, deciding batch vs. streaming.
- Implementing data quality checks, monitoring, idempotent writes, schema evolution handling, replay/backfill.
- Scoping a small data-engineering project (e.g., analyzing events from another project).

## When not to use
- Backend-job work → `backend-job`.
- Car-part retrieval internals (embedding choices, evaluation) → `car-part-retrieval` — though cross-instead-of-parallel learning is fine (e.g., "events from my retrieval system" → data-engineering is right).
- Combined plans → `learning-roadmap`.
- Mock interview DE questions → `interview-prep`.

## Context intake
- Goal type (teach vs build vs debug) and the volume/latency band; tooling and budget read from `docs/workflow-profile.md`.
Detect or confirm before starting (ask at most what is genuinely needed):
- Which context: one of the contexts defined in `docs/workflow-profile.md` ("Contexts")? Read `docs/active-context.md` first: if `Current` names a context set within the last day, use it, apply its rules, and do not ask. Otherwise fall back to the active roles and projects in `docs/workflow-profile.md`; ask when it could be two different contexts.
- What is the desired output: a plan, code, a review, tests, documentation, a deployment plan, or research notes? Confirm if ambiguous.

## Workflow
1. Read the my-skill plugin's `docs/workflow-profile.md`. If it is missing, contradicts this request, or was last updated more than ~3 months ago, confirm or update it with Sothy before applying any role, project, or stack assumptions.
2. Clarify the goal type first — teaching depth differs from scaffolding an implementation.
3. Identify source, destination, volume, latency requirement, and transformation needs; if missing, state sensible assumptions and label them.
4. Propose a **simple architecture first** (often: Kafka topic → dbt/SQL → Postgres/warehouse tables scheduled by Airflow), then show the scaled-up version only if needed.
5. Define end-to-end concerns: ingestion, raw storage, transformation, orchestration, quality checks, monitoring, and replay/backfill strategy — even briefly for toy projects.
6. Provide code/configuration in the stated stack (Docker Compose for local multi-tool setups; SQL DDL; Airflow DAG skeletons; dbt models with tests).
7. Teach while building: 2–3 sentences per non-obvious choice, batch-vs-streaming trade-offs to make learning stick.
8. Relate examples to the car-part retrieval project when useful (e.g., pipeline analyzing search events), keeping both skills' outputs consistent.
9. Suggest a concrete next exercise: something Sothy can build in his available hours.

## Adaptation rules

- The pipeline shape (source -> raw storage -> transform -> orchestration -> quality checks) is constant; tool choice scales with volume and budget, not trend.
- Stack anchors on what Sothy actually knows (Postgres/Spring) and cross-pollinates with current projects when useful.
- Local/free-first always; cloud spend only when the profile justifies it.

## Inputs needed
- Goal type: learning, real implementation, debugging an existing pipeline, architecture review, or interview prep.
- Source system(s) and destination(s); volume (rows/day, GB/day); latency requirement (minutes, seconds, daily); load types (append/merge).
- Current stack and constraints: free-tier/local, cloud budget if any, existing Postgres experience.
- For debugging: pipeline error, logs, where data stopped flowing.

## Output contract
- **Goal** — one sentence restating intent.
- **Architecture diagram** — Mermaid or ASCII.
- **Data flow** — numbered step-by-step source → storage.
- **Technology choices and trade-offs** — table with why-this-not-that.
- **Implementation steps** — ordered, Docker Compose snippets where useful.
- **Sample code/config** — runnable starting points.
- **Data-quality and monitoring checks** — freshness, volume anomalies, schema, duplicates, nulls.
- **Next learning task** — the next small thing to build after this one.

## Rules and constraints
- Do not recommend expensive or overly complex tools when simple ones work; justify tool additions (e.g., "you don't need Spark under 5 GB").
- Always explain batch versus streaming trade-offs when the choice arises — don't pick silently.
- Cover idempotency, schema evolution, late-arriving data, and failure recovery whenever the design is real (not just a sketch).
- Use local/free options (Docker Compose, LocalStack equivalents, community editions) by default; ask before assuming cloud spend.
- Don't invent miracles: state known sharp edges of tools (Debezium config fiddliness, Airflow scheduler quirks) rather than pretending setup is trivial.
- Keep ties to the car-part retrieval project optional and consistent — never silently merge backend-job context into examples.

## Quality checklist

- [ ] Batch-vs-streaming decision reasoned, not silent.
- [ ] Idempotency, schema evolution, and failure recovery addressed for real builds.
- [ ] Simple architecture shown before scaled-up version.
- [ ] Quality checks (freshness/volume/schema/dedupe) defined.
- [ ] A next hands-on task specified.

## Examples
1. **Request:** "Teach me Kafka fundamentals — I know Spring Boot consumes topics at work."
   **Behavior:** Goal = learning; leverage known producer/consumer experience; explain partitions, offsets, keys, consumer groups, delivery semantics with diagrams; propose a hands-on task (produce and consume a small test topic with console tools); output format applied; next learning task = Dockerized broker + small Python producer.
2. **Request:** "Create a beginner-friendly Kafka, CDC, and warehouse project to analyze car-part search events."
   **Behavior:** Design: Postgres source + Debezium → Kafka topic → simple sink + dbt models → warehouse tables (daily searches, top categories); Docker Compose setup; quality checks (row counts, freshness, dedupe); 4-step build plan with milestones; Mermaid diagram; notes on CDC quirks; next task = add Airflow orchestration.

## Lessons learned

Record improvements after real usage (maintained via ``continuous-improvement``). Format:

- ``<date>`` -- ``<what happened>`` -> ``<change made to this SKILL.md>``

(empty -- entries appear after real usage)

Wersje

RepozytoriumWynikLicencja
Ta strona · albkue/my-skill100MIT
travisjneuman/.claude100MIT
fengshao1227/ccg-workflow100MIT
mhingan/agent-forge100MIT
borhen68/skillengine100MIT
swap-cameroon/kuro-antigravity-config90MIT
andreibesleaga/gabbe90CC-BY-4.0
cbrock84/headcount90MIT
magnus919/agent-skills85MIT

Migawka bazy z 13.11.2026 · wersja paczki da9b501a1675 (skrót SHA-256 całej paczki)

Pobrania w ostatnich 30 dniach: 0 · ranking

Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.

Podobne skille

  • 100senior-data-engineer
    Buduje pipeline'y wsadowe i strumieniowe w Airflow, dbt, Spark i Kafka: architektury danych, kontrole jakości, optymalizacja ETL i ELT oraz diagnostyka awarii.
  • 95senior-data-engineer
    Projektuje i buduje potoki danych ETL i ELT oraz infrastrukturę danych w Pythonie, SQL, Sparku, Airflow i dbt.
  • 100learning-roadmap
    Układa realistyczne plany nauki i projektów: tygodniowe i miesięczne plany, mapy dla backendu, MLOps, system design i rozmowy kwalifikacyjne.
  • 91senior-data-engineer
    Wspiera budowę skalowalnych potoków danych, systemów ETL i ELT oraz infrastruktury danych w Pythonie, SQL, Spark, Airflow, dbt i Kafka, z jakością danych i DataOps.
  • 100senior-data-engineer
    Buduje skalowalne pipeline'y danych, systemy ETL i ELT oraz infrastrukturę danych w Pythonie, SQL, Spark, Airflow, dbt i Kafka, z modelowaniem, jakością danych i DataOps.
  • 100senior-data-engineer
    Buduje skalowalne pipeline'y danych, systemy ETL i ELT oraz infrastrukturę danych w Pythonie, SQL, Spark, Airflow, dbt i Kafka, z modelowaniem, jakością danych i DataOps.

Uwagi i zgłoszenia

Wiadomość trafi do Bartosza Sokolińskiego. Adres e-mail jest potrzebny tylko do odpowiedzi. Szczegóły w polityce prywatności.