gcp-dataflow
/gcp-dataflow
Pomaga pisać, pakować i uruchamiać potoki Apache Beam w Cloud Dataflow na GCP, także jako szablony Flex Template.
- Autor
- 420company ↗
- Repozytorium
- artemis ↗
- Licencja
- MIT
- Ostatnia zmiana repozytorium
- 25.10.2026
Ocena i bezpieczeństwo
- Pliki w paczce
- 3
- Skrypty
- 0
- Komentarze HTML
- 0
- Adresy URL w SKILL.md
- 0
Brak sygnałów. Repozytorium zmienione 25.10.2026, bez odwołań do wycofanych modeli i API.
MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.
Co robi i kiedy
Pomaga pisać, pakować i uruchamiać potoki Apache Beam w Cloud Dataflow na GCP, także jako szablony Flex Template.
Oryginalny opis (EN)
Provides guidance for writing, packaging and executing Apache Beam pipelines on GCP using Cloud Dataflow. Use when: - Creating an Apache Beam Dataflow pipeline. - Creating a Google Flex Template.
Jak działa
- Creating a new project
- Flex Templates
- Launching Apache Beam Pipelines with Dataflow Runner
Podgląd SKILL.md
---
name: gcp-dataflow
description: 'Provides guidance for writing, packaging and executing Apache Beam pipelines
on GCP using Cloud Dataflow. Use when: - Creating an Apache Beam Dataflow pipeline.
- Creating a Google Flex Template.
'
license: Apache-2.0
metadata:
version: v1
publisher: google
---
# Apache Beam Pipelines on Cloud Dataflow
Expert guidance for writing and packaging Apache Beam pipelines to run on Google
Cloud Dataflow.
## Creating a new project
Use this section when creating a new project for a Dataflow pipeline.
- If the user doesn't say explicitly which language (Java, Python, Go) shall
be used to write the pipeline, you MUST confirm the language.
- Determine which version of Beam SDK should be used by searching for the most
recently released version of Apache Beam, unless the user already uses a
particular version.
- **Action**: Run a web search for the latest Apache Beam SDK release.
- YOU MUST use same version of Apache Beam consistently throughout the project
in Dockerfiles, `requirements.txt`, and other similar files where versions
are specified.
### Java projects using Gradle
Use this section when configuring a Dataflow Java pipeline project using gradle.
- **Shadow Jars (Fat Jars)**: Do NOT propose to use the Shadow plugin
(`com.github.johnrengelman.shadow`) unless the user explicitly requests a
Fat Jar.
- **Passing command-line parameters**: Use the `application` plugin for
passing command-line parameters.
- **SLF4J Logging Dependency Alignment**:
- Verify the `slf4j-api` version pulled transitively by Apache Beam.
- You MUST configure the application logging backend (`slf4j-simple`,
`logback-classic`, etc.) to exactly match the major/minor version of the
resolved `slf4j-api`.
### Structure the pipeline as a Dataflow Flex Template
When creating new Dataflow pipeline projects, configure them as a Flex template.
Flex Templates offer a hermetic and reproducible launch environment, and are
easy to launch with `gcloud` or with orchestrators like Cloud Composer.
Follow the Flex Templates section below.
## Flex Templates
- **Provide Instructions**: Provide instructions on rebuilding and running
Flex Templates to the user in walkthrough.
- **Use Single Docker Image for Python pipelines**: For Python Flex Templates,
it is better to use a single image for the template launcher image and for
the worker runtime environment (`--sdk_container_image`). Whenever
configuring or suggesting a Dataflow Flex Template for a Python pipeline
that requires extra dependencies (e.g., using `--requirements_file`,
`--setup_file`, or `--extra_package`), **YOU MUST recommend the Single
Docker Image Configuration** as detailed in
[python_flex_template_reference.md](resources/python_flex_template_reference.md).
- **Prefer Cloud Build over Local Docker**:
- Do NOT assume local Docker availability on the workspace machine.
- **Action**: Suggest and provide `cloudbuild.yaml` out-of-the-box for
building and pushing images unless local setup is explicitly requested.
- When building images with Cloud Build in the background you MUST provide
the link where the user can monitor the long-running operation.
## Launching Apache Beam Pipelines with Dataflow Runner
- When launching Python Pipelines without a Flex Template with
`DataflowRunner`, you MUST scan the pipeline project directory for the
following files:
- **`requirements.txt`**:
- If found, you MUST include `--requirements_file` pipeline option.
- **`setup.py`**:
- If found, you MUST include `--setup_file` pipeline option. This is
critical if the pipeline uses local modules or packages.
- When launching Python Pipelines with a Flex Template, if the Flex Template
image is also the SDK Container image (Single Docker Image Configuration),
then you MUST supply the image in the `sdk_container_image` parameter.
- Confirm the launch command with the user.
### Lookup environment resources instead of using placeholder values
- Avoid using generic placeholders (e.g., `your-gcp-project-id`) for GCP
resources when drafting run scripts or configs. **Action**: If values are
unknown, proactively run commands like `gcloud config get-value project` to
find active resources to pre-fill scripts for the user. Confirm the values
with the user before proceeding.
Pliki w paczce (3)
- SKILL.json 1 KB
- SKILL.md 4 KB
- resources/python_flex_template_reference.md 2 KB
Wersje
| Repozytorium | Wynik | Licencja |
|---|---|---|
| Ta strona · 420company/artemis | 85 | MIT |
| gemini-cli-extensions/data-agent-kit-starter-pack | 100 | Apache-2.0 |
Migawka bazy z 13.11.2026 · wersja paczki 90ab56562a43 (skrót SHA-256 całej paczki)
Pobrania w ostatnich 30 dniach: 0 · ranking
Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.
Podobne skille
- 100apache-beam-unified-stream-batch-pipelines
Buduje przenośne potoki danych strumieniowych i wsadowych w Apache Beam, uruchamiane na Direct, Flink lub Dataflow. - 100beam-pipeline-builder
Prawdopodobnie pomaga budować potoki danych w Apache Beam i uruchamia się automatycznie przy pracy nad tego typu potokami w obszarze Data Pipelines. - 100gcp-data-engineering
Doradza w architekturze potoków danych w Google Cloud: jezioro, hurtownia i lakehouse, batch czy streaming, dobór Dataform, Composer lub Dataflow oraz koszty i wydajność BigQuery (partycjonowanie, klastrowanie, sloty). - 90apache-beam-unified-batch-and-stream
Prowadzi przez potoki Apache Beam łączące logikę wsadową i strumieniową: transformacje, okna, runnery, powtórki i przenośność między backendami. - 100gcp-data-engineering
Doradza w architekturze potoków danych w Google Cloud: hurtownia i jezioro danych, wsad kontra strumień, orkiestracja i koszty BigQuery. - 90dataplex-and-bigquery-governance
Prowadzi przez zarządzanie danymi w GCP z Dataplex i BigQuery: jeziora, strefy, policy tags, jakość metadanych, lineage, discovery i kontrolowana publikacja w Cloud Storage, BigQuery i Dataflow.