Przejdź do treści
Biblioteka skilli
Dane i analityka · Konwersja i formaty

python-airflow-to-orchestra

/python-airflow-to-orchestra

Przepisuje zadania Airflow z PythonOperator, PythonVirtualenvOperator i @task na równoważne zadania Python w pipeline Orchestra.

Licencja
MIT
Ostatnia zmiana repozytorium
27.10.2026

Ocena i bezpieczeństwo

100/ 100Klasa: dobry
Format25 / 25
Frontmatter z nazwą i opisem, poprawna nazwa, opis do 1024 znaków, nazwa zgodna z folderem.
Odkrywalność25 / 25
Opis mówi, kiedy użyć skilla, ma sensowną długość i jest w 3. osobie.
Treść35 / 35
Co najmniej 150 słów i do 500 linii, nagłówki, listy kroków, przykłady albo kod.
Higiena15 / 15
Bez zaślepek typu TODO, bez ścieżek Windows, z licencją.
Ryzykobrak sygnałów w skanie
Pliki w paczce
1
Skrypty
0
Komentarze HTML
0
Adresy URL w SKILL.md
4
Przestarzałość

Brak sygnałów. Repozytorium zmienione 27.10.2026, bez odwołań do wycofanych modeli i API.

Licencja

MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.

Co robi i kiedy

Przepisuje zadania Airflow z PythonOperator, PythonVirtualenvOperator i @task na równoważne zadania Python w pipeline Orchestra.

Oryginalny opis (EN)

Use this skill when the user wants to convert an Airflow task that uses PythonOperator or PythonVirtualenvOperator into an equivalent Orchestra Python pipeline task. Triggers: any mention of migrating or rewriting PythonOperator tasks to Orchestra; Airflow DAG code using PythonOperator, PythonVirtualenvOperator, ExternalPythonOperator, or @task decorator tasks.

Jak działa

  1. Overview
  2. Parameter Mapping
  3. Orchestra YAML Structure
  4. Conversion Steps
  5. Credentials the Code Needs: Link, Don't Copy
  6. Before / After Example
  7. Gotchas
  8. References
  9. Before converting: check it isn't actually a Slack task
  10. Adding Alerts
Sekcje z nagłówków SKILL.md, w oryginale · 188 linii, 1 416 słów
Podgląd SKILL.md
---
name: python-airflow-to-orchestra
description: "Use this skill when the user wants to convert an Airflow task that uses PythonOperator or PythonVirtualenvOperator into an equivalent Orchestra Python pipeline task. Triggers: any mention of migrating or rewriting PythonOperator tasks to Orchestra; Airflow DAG code using PythonOperator, PythonVirtualenvOperator, ExternalPythonOperator, or @task decorator tasks."
---

# Python: Airflow → Orchestra Conversion

## Overview

Airflow's `PythonOperator` runs a callable inline in the worker — the code lives right there in the DAG file, not in some separate repo the DAG checks out at runtime. Orchestra's Python integration (**Execute Script**) has two modes: `source: INLINE` runs code pasted directly into the task's `parameters.code`, and `source: GIT` runs a file checked out from a Git repo via `parameters.command`. **Default to `INLINE`** — it's the direct match for what Airflow already does (code inline in the DAG), needs no Git repo or connection wiring, and skips a whole conversion step. Only reach for `GIT` when the callable itself checks out and runs a script that already lives in a separate repo (rare for `PythonOperator` — more of a `BashOperator` pattern).

## Parameter Mapping

| Airflow concept | Orchestra YAML field | Notes |
|---|---|---|
| `python_callable` (the function) | `parameters.code` (inline) | Copy the callable body verbatim — no extraction to a separate file/repo needed |
| top-level `import`s beyond the stdlib | `parameters.build_command` | e.g. `build_command: 'pip install pandas boto3'` — installs before the code runs |
| `op_kwargs` / `op_args` | Inline into `code` as literals, or `parameters.environment_variables` + `os.environ.get(...)` in the code | |
| `requirements` (PythonVirtualenvOperator) | `parameters.build_command` | `pip install -r requirements` list as a `pip install ...` command |
| Python version | `parameters.python_version` | e.g. `'3.12'` |
| `task_id` | `name:` | Human-readable task name |
| `provide_context=True` / `**context` | Not applicable | Orchestra context is available via environment variables |
| `pool` / `queue` | Not applicable | Managed by Orchestra |
| upstream `>>` chains | `depends_on:` | |

## Orchestra YAML Structure

```yaml
version: v1
name: <pipeline-name>
pipeline:
  <stage-uuid>:
    tasks:
      <task-uuid>:
        integration: PYTHON
        integration_job: PYTHON_EXECUTE_SCRIPT
        name: <task_id value from Airflow>
        connection: null                          # usually null for INLINE — no repo/creds needed unless the code itself needs a specific connection's secrets
        parameters:
          source: INLINE                           # default — code already lives in the DAG, no Git repo involved
          code: |
            <callable body, copied verbatim>
          build_command: 'pip install pandas'      # optional — only for non-stdlib imports
          python_version: '3.12'
        depends_on: []
        condition: null
        tags: []
```

Use `source: GIT` + `command:` (see `python-dagster-to-orchestra`/`python-prefect-to-orchestra` for the field shape) only when the Airflow task's callable genuinely runs a script that's already checked into a separate Git repo — not just because Orchestra supports it.

## Conversion Steps

1. **Copy the callable body verbatim** into `parameters.code` — no need to extract it into a standalone file or commit anything to Git.
2. **List non-stdlib imports** (`pandas`, `boto3`, etc.) and set `parameters.build_command: 'pip install <package> ...'`.
3. **Handle `op_kwargs`** — either inline the values as literals in the code, or read them via `os.environ.get(...)` and set `parameters.environment_variables`.
4. **Set `connection`** — leave `null` unless the code needs credentials/secrets that belong on a specific Orchestra connection (e.g. it instantiates a boto3/Snowflake client using connection-injected env vars).
5. **Wire dependencies**.

If instead the Airflow task genuinely checks out and runs a script from a separate Git repo (not the DAG repo itself), use `source: GIT`, commit/reference the script there, and create/verify an Orchestra Python connection pointing at that repo (URL, branch, credentials) as described for `source: GIT` tasks.

## Credentials the Code Needs: Link, Don't Copy

A callable that read a credential through `BaseHook.get_connection(...)` or a hook's `conn_id` still needs those credentials once it's inline in `parameters.code`. If that credential already exists as its own Orchestra connection, **link** it on the Python connection (**Linked connections** on the connection's form) rather than copying its user/password into that connection's **Secrets** — see `airflow-connections-to-orchestra`.

Linked credentials are exported as environment variables named `<INTEGRATION>__<CONNECTION_ID>__<FIELD>` (every part upper-cased, non-alphanumeric runs collapsed to a single `_`, parts joined by a **double** underscore, `<FIELD>` being the field's ID from the integration's **Authentication parameters** table) — e.g. `SNOWFLAKE__PROD_12345__SNOWFLAKE_PASSWORD`. The pre-installed `orchestra-sdk` builds those names for you:

```python
import os
import snowflake.connector
from orchestra_sdk.orchestra import OrchestraSDK

orchestra = OrchestraSDK(api_key=os.environ["ORCHESTRA_API_KEY"])
creds = orchestra.get_linked_connection("snowflake", connection_id="prod_12345")

conn = snowflake.connector.connect(
    account=creds.get("snowflake_account"),
    user=creds.get("snowflake_user"),
    password=creds.get("snowflake_password"),
)
```

Drop `connection_id=` only when a single connection of that integration is linked. A linked connection that has been deleted, or whose credentials can't be decrypted, is **skipped silently** — its variables are simply absent, so don't write code that assumes `os.environ["..."]` is set. Full reference: [Linked connections](https://docs.getorchestra.io/docs/integrations/python#linked-connections).

## Before / After Example

### Airflow DAG (before)

```python
from airflow.operators.python import PythonOperator

def compute_metrics(start_date: str, end_date: str) -> None:
    import pandas as pd
    df = pd.read_csv("s3://my-bucket/data.csv")
    # ... processing ...
    df.to_parquet("s3://my-bucket/output.parquet")

metrics_task = PythonOperator(
    task_id="compute_metrics",
    python_callable=compute_metrics,
    op_kwargs={"start_date": "{{ ds }}", "end_date": "{{ tomorrow_ds }}"},
)
```

### Orchestra YAML (after)

The callable body moves into `parameters.code` as-is — no separate script file, no Git repo, no connection needed here since nothing credentialed is involved:

```yaml
version: v1
name: my-pipeline
pipeline:
  stage-001:
    tasks:
      task-001:
        integration: PYTHON
        integration_job: PYTHON_EXECUTE_SCRIPT
        name: compute_metrics
        connection: null
        parameters:
          source: INLINE
          code: |
            import os
            import pandas as pd

            start_date = os.environ["START_DATE"]
            end_date = os.environ["END_DATE"]

            df = pd.read_csv("s3://my-bucket/data.csv")
            # ... processing ...
            df.to_parquet("s3://my-bucket/output.parquet")
          build_command: 'pip install pandas'
          python_version: '3.12'
          environment_variables: '{"START_DATE": "2024-01-01", "END_DATE": "2024-01-02"}'
        depends_on: []
        condition: null
        tags: []
```

## Gotchas

- **Default to `source: INLINE`, not `GIT`** — the callable's code already lives in the DAG file; pasting it into `parameters.code` is a direct match. Don't reach for `source: GIT` (extracting to a script, committing it, wiring a connection) just because that's the other mode Orchestra supports — it's extra work with no source-fidelity benefit unless the callable genuinely runs a script from an external repo.
- **Consuming a JSON-shaped upstream output (converted XCom) in `code`?** Wrap the `${{ ...OUTPUTS[...] }}` substitution in Python triple-quotes for `json.loads()`, not single/double — see `airflow-xcoms-to-orchestra`'s Gotchas for why (raw text substitution, no escaping).
- **Airflow context / Jinja templates**: Airflow macros like `{{ ds }}` are not available in Orchestra. Replace with static values, pipeline variables, or read from environment variables set by Orchestra.
- **Inline lambdas / closures**: if the callable uses closures or imports from elsewhere in the same DAG file, inline those into `code` too so it's self-contained.
- **`@task` decorator (TaskFlow API)**: same approach — copy the decorated function body into `code` verbatim.
- **`PythonVirtualenvOperator` requirements**: list them in `build_command` as a `pip install ...` command.
- **`environment_variables` is a JSON string, not a nested map** — it's a single string field on the parameters model (e.g. `'{"KEY": "value"}'`), not a YAML mapping.
- **No specific connection mapped**: `connection: null` is the norm for `INLINE` tasks with no external credentials. Only set a specific connection if the code needs secrets injected from one (e.g. a boto3/Snowflake client reading connection-provided env vars) — never invent a placeholder like `${{ ENV.PYTHON_CONNECTION }}` just to fill the field.
- **Secrets**: do not hardcode credentials in `code`. If secrets are needed, use an Orchestra connection's injected env vars — and when the credential is already its own Orchestra connection, link it rather than copying its secrets (see above).
- **`source: GIT` still exists for real Git-backed scripts** — if the Airflow task checks out and runs a script from a separate repo, use `GIT` + `parameters.command`, and create/verify a Python connection pointing at that repo (URL, branch, credentials). Orchestra supports sparse checkout for large monorepos, configurable on the connection.

## References

- Orchestra docs: https://docs.getorchestra.io/docs/integrations/python
- Orchestra Execute Script: https://docs.getorchestra.io/docs/integrations/python/python_execute_script
- Orchestra linked connections: https://docs.getorchestra.io/docs/integrations/python#linked-connections

## Before converting: check it isn't actually a Slack task

A `PythonOperator`/`@task` whose entire function body posts a message to Slack — even one that calls `slack_sdk.WebClient(...).chat_postMessage(...)` directly instead of a dedicated Slack operator — is **not** generic Python. It belongs on `integration: SLACK` / `integration_job: SEND_SLACK_MESSAGE`, not `PYTHON_EXECUTE_SCRIPT`. See `slack-airflow-to-orchestra` before converting any task whose main job is formatting and sending a Slack message, even when it's a normal mid-DAG step rather than a callback.

## Adding Alerts

If the Airflow DAG uses `on_failure_callback` or `on_success_callback` for Slack/email notifications, replace those with an `alerts` block in the pipeline YAML. Alerts fire based on overall pipeline status and support Slack, Email, PagerDuty, Microsoft Teams, and Webhook destinations.

```yaml
version: v1
name: my-pipeline

alerts:
  - name: on-failure
    statuses:
      - FAILED
    destinations:
      - integration: SLACK
        destination: '#data-alerts'
    custom_message: 'Optional context message.'

  - name: on-success
    statuses:
      - SUCCEEDED
    destinations:
      - integration: SLACK
        destination: '#data-alerts'

pipeline:
  # ... tasks unchanged
```

Valid statuses: `FAILED`, `SUCCEEDED`, `CANCELLED`, `WARNING`, `SKIPPED`. Multiple alerts with different destinations are supported — each needs a unique `name`. See the `slack-airflow-to-orchestra` skill for full schema details.

Wersje

W bazie nie ma innych skilli o nazwie python-airflow-to-orchestra.

Migawka bazy z 13.11.2026 · wersja paczki c101c8aca2ac (skrót SHA-256 całej paczki)

Pobrania w ostatnich 30 dniach: 0 · ranking

Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.

Podobne skille

  • 100fivetran-airflow-to-orchestra
    Przekształca zadanie Airflow używające FivetranOperator lub FivetranSensor w równoważne zadanie potoku Orchestra.
  • 100slack-airflow-to-orchestra
    Zamienia alerty Slack z zadań Airflow na równoważny alert lub zadanie w pipeline Orchestra.
  • 100airflow-dag-generator
    Prawdopodobnie generuje DAG-i Airflow dla potoków danych, na podstawie nazwy skilla i kategorii Data Pipelines.
  • 100domino-airflow-kubeflow
    Uruchamia zadania Domino Jobs z zewnętrznego orkiestratora, np. Airflow lub Kubeflow, wraz z poświadczeniami, dostępnością sieci i śledzeniem statusu.
  • 100migrating-airflow-2-to-3
    Prowadzi migrację projektów Apache Airflow 2.x do Airflow 3.x: zmiany łamiące, problemy ze zgodnością i modernizacja bazy kodu.
  • 86airflow
    Pomaga tworzyć DAG-i Airflow, układać zależności zadań i automatyzować procesy ETL w potokach danych.

Uwagi i zgłoszenia

Wiadomość trafi do Bartosza Sokolińskiego. Adres e-mail jest potrzebny tylko do odpowiedzi. Szczegóły w polityce prywatności.