gcp-data-engineering
/gcp-data-engineering
Doradza w architekturze potoków danych w Google Cloud: hurtownia i jezioro danych, wsad kontra strumień, orkiestracja i koszty BigQuery.
- Autor
- jggomez ↗
- Repozytorium
- expert-ai-developer-skills ↗
- Licencja
- Apache-2.0
- Ostatnia zmiana repozytorium
- 10.11.2026
Ocena i bezpieczeństwo
- Pliki w paczce
- 2
- Skrypty
- 0
- Komentarze HTML
- 0
- Adresy URL w SKILL.md
- 0
Brak sygnałów. Repozytorium zmienione 10.11.2026, bez odwołań do wycofanych modeli i API.
Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.
Co robi i kiedy
Doradza w architekturze potoków danych w Google Cloud: hurtownia i jezioro danych, wsad kontra strumień, orkiestracja i koszty BigQuery.
Oryginalny opis (EN)
Guides architecture decisions for data pipelines on Google Cloud — lake/warehouse/lakehouse design, batch vs. streaming, orchestration tool selection (Dataform, Cloud Composer/Airflow, Dataflow/Beam), and BigQuery cost/performance tuning (partitioning, clustering, slots). Use when designing a new pipeline, choosing a GCP data service, or optimizing BigQuery cost/performance.
Jak działa
- Overview
- When to Use
- Process
- Usage
- Red Flags
- Verification
- References
Podgląd SKILL.md
--- name: gcp-data-engineering description: Guides architecture decisions for data pipelines on Google Cloud — lake/warehouse/lakehouse design, batch vs. streaming, orchestration tool selection (Dataform, Cloud Composer/Airflow, Dataflow/Beam), and BigQuery cost/performance tuning (partitioning, clustering, slots). Use when designing a new pipeline, choosing a GCP data service, or optimizing BigQuery cost/performance. --- # GCP Data Engineering Skill ## Overview Guides architectural decisions for data pipelines, Medallion lakehouses, and analytical storage on Google Cloud Platform (GCP). Balances data volume, SLA freshness, operational simplicity, and cloud cost—prioritizing warehouse-native ELT (Dataform, BigQuery) over custom streaming (Dataflow/Beam) unless sub-minute SLAs apply, while enforcing capability-gated artifacts and fail-honest CLI verification. ## When to Use ### Trigger Scenarios - Designing GCP lakehouse platforms (Cloud Storage, BigQuery, Datastream, Pub/Sub, Dataform, Composer). - Selecting batch vs. log-based CDC vs. streaming ingestion and Dataform vs. Composer vs. Dataflow orchestration. - Designing BigQuery partitioning, clustering, and Medallion layers (`Bronze` → `Silver` → `Gold`). ### When NOT to Use - **CDC & SCD Type 2 merge patterns**: Route to `cdc-scd-patterns`. - **Slow/expensive BigQuery SQL plan tuning**: Route to `bigquery-query-optimization`. - **Single-node RDBMS SQL tuning**: Route to `sql-query-optimization`. - **Application backend code**: Route to `python-expert`. ## Process ### Phase 1: Capability-Gated Planning & Target `AGENTS.md` 1. Maintain root `AGENTS.md` (`<= 90 lines`) linking `dataform`/`bq` commands and `context/memory/MEMORY.md`. 2. Author `PLAN.md` (`<= 50 lines`) gating artifacts individually: - `docs/<feature-name>/data-contracts.md` (`<= 60 lines`): **ONLY IF** new/modified schemas or SLAs are introduced; **SKIP** if schemas exist. - `docs/<feature-name>/pipeline-lineage.md` (`<= 45 lines`): **ONLY IF** multi-layer Bronze/Silver/Gold flows change; **SKIP** for single-mart edits. - Delegate `sql-query-optimizer` **ONLY IF** large partitioned scans or multi-table JOIN/window shuffle risks exist; invoke Tier 3 `Data Challenger` **ONLY IF** CDC/SCD-2 merges, late-arriving dedup, or destructive/backfill logic changes. ### Phase 2: Medallion Storage, Ingestion & Orchestration 1. **Storage Layers**: GCS immutable raw landing zone + BigQuery Medallion tables: `Bronze` (append-only raw/CDC), `Silver` (deduplicated, typed, SCD-2), `Gold` (BI star marts). 2. **Ingestion (Default Batch)**: Scheduled loads (GCS/Dataform/Composer); **Datastream** for operational DB CDC; **Pub/Sub → BigQuery** (or **Dataflow** only when windowed stream joins/non-SQL code are required). 3. **Orchestration**: **Dataform** for warehouse-native SQLX & assertions; **Cloud Composer** only for cross-service DAGs. ### Phase 3: BigQuery Cost Governance & Fail-Honest Verification 1. **Partition & Cluster**: Partition by date/timestamp or ingestion time (`require_partition_filter = true` on large tables); cluster up to 4 selective filter/join columns; prune columns (never `SELECT *`). 2. **Fail-Honest CLI Gate**: Run `dataform compile` and `bq query --dry_run`. If CLI binaries or GCP ADC credentials fail/missing, **STOP** and report `BLOCKED: Not executed — <exact CLI error>`—never fabricate byte counts or compile success. 3. **Standardized Test Matrix (`docs/<feature-name>/data-test-matrix.md <= 75 lines`)**: Record IEEE/ISTQB test cases (`TC-01`...) with `ID`, `Ref`, `Level`, `Objective`, `Steps`, `Expected Result`, `Assertion / Query`, and `Status`. ## Usage ```bash # Estimate scanned bytes via BigQuery dry-run bq query --use_legacy_sql=false --dry_run "SELECT id, created_at FROM \`my_project.analytics.events\` WHERE event_date = '2026-09-01'" # Compile Dataform dependency graph (scoped or full) dataform compile --actions silver_orders dataform compile ``` - **Host Execution**: Run `bq` and `dataform` via shell (`Bash` in Claude Code; `Bash` in Antigravity) and persist contracts/SLAs to `context/memory/MEMORY.md`. ## Red Flags - Fabricating `bq query --dry_run` byte counts or pretending `dataform compile` ran when CLI/ADC credentials fail instead of reporting `BLOCKED: Not executed — <exact CLI error>`. - Using Dataflow/Kafka when batch/Datastream meets SLAs, or Composer for SQL-only transforms Dataform handles natively. - Querying `SELECT *` or raw CDC tables without partition filters or a Silver deduplication layer. ## Verification - [ ] Root `AGENTS.md` (`<= 90 lines`), `PLAN.md` (`<= 50 lines`), and capability-gated `docs/<feature-name>/` artifacts respect line caps. - [ ] Tables define explicit partitioning (`require_partition_filter = true`) and clustering keys across `Bronze`/`Silver`/`Gold`. - [ ] `dataform compile` and `bq query --dry_run` exit `0`, OR execution halts honestly with `BLOCKED: Not executed — <exact CLI error>`. - [ ] `docs/<feature-name>/data-test-matrix.md` (`<= 75 lines`) documents `TC-01`... with `ID`, `Ref`, `Level`, `Objective`, `Steps`, `Expected Result`, `Assertion / Query`, and `Status`. ## References - [GCP Data Stack Reference](references/gcp-data-stack.md)
Pliki w paczce (2)
- SKILL.md 5 KB
- references/gcp-data-stack.md 3 KB
Wersje
| Repozytorium | Wynik | Licencja |
|---|---|---|
| Ta strona · jggomez/expert-ai-developer-skills | 100 | Apache-2.0 |
| jggomez/expert-ai-developer-skills | 100 | Apache-2.0 |
Migawka bazy z 13.11.2026 · wersja paczki ba190f65a87b (skrót SHA-256 całej paczki)
Pobrania w ostatnich 30 dniach: 0 · ranking
Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.
Podobne skille
- 90bigquery-and-dataform-platform-engineering
Prowadzi przez inżynierię danych wokół BigQuery i Dataform: modele fizyczne, granice ingestu, przepływy transformacji, kontrola slotów i kosztów oraz wybór między BigQuery, Dataflow i Dataproc. - 95cloud-native-gcp-bigquery
Wyjaśnia architekturę BigQuery i stosuje ją do partycjonowania, klastrowania, kosztów i tuningu zapytań. - 100bigquery
Wspiera pracę z Google Cloud BigQuery: standardowy SQL, partycjonowanie, klastrowanie, optymalizację kosztów i ML, przy analizach na ogromnych zbiorach i projektowaniu hurtowni danych. - 85gcp-dataflow
Pomaga pisać, pakować i uruchamiać potoki Apache Beam w Cloud Dataflow na GCP, także jako szablony Flex Template. - 100bigquery
Odpytuje i zarządza danymi w Google BigQuery: zbiory danych i tabele, IAM, partycjonowanie, klastrowanie i wzorce bq CLI, przy zapytaniach zawsze z oszacowaniem kosztu przed wykonaniem. - 100gcp-dataflow
Prowadzi pisanie, pakowanie, uruchamianie i diagnozowanie potoków Apache Beam na Dataflow: konfiguracja Java, Python i Go, Flex Templates, Cloud Build oraz analiza kondycji zadań i autoskalowania.