debugging-airflow-pipelines
/debugging-airflow-pipelines
Diagnose failing or stuck Apache Airflow pipelines — task failures and retries, scheduler/executor problems, XCom errors, zombie/queued tasks, dependency deadlocks, and pools/concurrency limits. Use when an Airflow task fails or is stuck queued, the scheduler is not running tasks, a DAG will not trigger, or tasks become zombies.
- Author
- unknown-333 ↗
- Repository
- awesome-data-engineering-skills ↗
- Licence
- Apache-2.0
- Last repository change
- 30.08.2026
Assessment and security
- Files in this package
- 1
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 30.08.2026, without references to retired models or APIs.
Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.
What it does and when to use it
Diagnose failing or stuck Apache Airflow pipelines — task failures and retries, scheduler/executor problems, XCom errors, zombie/queued tasks, dependency deadlocks, and pools/concurrency limits. Use when an Airflow task fails or is stuck queued, the scheduler is not running tasks, a DAG will not trigger, or tasks become zombies.
Original author description
Diagnose failing or stuck Apache Airflow pipelines — task failures and retries, scheduler/executor problems, XCom errors, zombie/queued tasks, dependency deadlocks, and pools/concurrency limits. Use when an Airflow task fails or is stuck queued, the scheduler is not running tasks, a DAG will not trigger, or tasks become zombies.
How it works
- When to use
- Workflow
- Patterns
- Common pitfalls
SKILL.md preview
--- name: debugging-airflow-pipelines description: Diagnose failing or stuck Apache Airflow pipelines — task failures and retries, scheduler/executor problems, XCom errors, zombie/queued tasks, dependency deadlocks, and pools/concurrency limits. Use when an Airflow task fails or is stuck queued, the scheduler is not running tasks, a DAG will not trigger, or tasks become zombies. --- # Debugging Airflow Pipelines ## When to use - A task failed, is stuck in `queued`/`scheduled`, or became a zombie. - The scheduler isn't launching runs, or a DAG won't trigger. - XCom pull errors, dependency deadlocks, or pool/concurrency starvation. - Do NOT use for writing new DAGs (use `authoring-airflow-dags`). ## Workflow ``` - [ ] Read the task log first (Grid view -> task -> Logs) - [ ] Check task state and why: failed, up_for_retry, queued, or none - [ ] Localize: task-level bug vs scheduler/executor vs resource limit - [ ] Fix root cause, then clear the task to re-run ``` 1. **Read the task log.** The Grid/Graph view → failed task → Logs shows the real exception nearly every time. 2. **Check the state and reason.** `queued` for a long time is usually a resource/executor issue, not a code bug. 3. **Localize** using the table below. 4. **Re-run** by clearing the task instance (and downstream if needed) rather than re-triggering the whole DAG. ## Patterns **Task keeps failing** — read the log; fix the exception; confirm `retries` are set so transient errors self-heal. Use `on_failure_callback` for alerting. **Task stuck in `queued`/`scheduled`:** - Worker capacity exhausted, or a `pool` is full → check pool slots and `max_active_tasks`/parallelism. - Celery/Kubernetes executor not picking up → check worker health and the message broker/queue. **Zombie tasks** (process died, heartbeat lost) → often OOM or a killed worker. Check worker memory/logs; reduce task memory or raise limits; Airflow marks it failed and retries. **Scheduler not creating runs:** - DAG parse error → `airflow dags list-import-errors` (top-level code exceptions). - DAG paused, or `start_date` in the future, or `catchup=False` with no new interval yet. - `max_active_runs` reached → older runs not completing block new ones. **Dependency deadlock / "no status"** — an upstream is `skipped` with the default trigger rule; adjust `trigger_rule` (e.g. `all_done`, `none_failed_min_one_success`) for branch/cleanup tasks. **XCom errors** — pulling a key a task never pushed, or a payload too large for the metadata DB. Push explicitly and pass storage references for big data. ## Common pitfalls - **Re-triggering the whole DAG** instead of clearing the failed task — reruns work already done and can duplicate non-idempotent side effects. - **Blaming code for `queued` tasks** — check pools, parallelism, and workers first. - **Ignoring import errors** — one bad DAG file can stall parsing/scheduling. - **No retries on flaky external calls** — every transient blip pages someone. - **Clearing a non-idempotent task** — confirm the task is safe to re-run, or you duplicate data (see `writing-idempotent-transformations`).
Versions
The database has no other skills named debugging-airflow-pipelines.
Database snapshot: 13.11.2026 · package version 3a3f879997a7 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100airflow-scheduler-and-dag-troubleshooting
Diagnoses a stuck or failed Airflow DAG run, scheduler-health problems (slow parsing, stuck queued tasks), and plans safe task-retry/backfill strategy. Use when the user reports "Airflow DAG is stuck," "task stuck in queued state," "scheduler seems unhealthy," "DAG run failed… - 95airflow
Queries, manages, and troubleshoots Apache Airflow using the af CLI. Covers listing DAGs, triggering runs, reading task logs, diagnosing failures, debugging DAG import errors, checking connections, variables, pools, and monitoring health. Also routes to sub-skills for writing… - 100airflow-dag-generator
Generate airflow dag generator operations. Auto-activating skill for Data Pipelines. Triggers on: airflow dag generator, airflow dag generator Part of the Data Pipelines skill category. Use when working with airflow dag generator functionality. Trigger with phrases like "airflow… - 100airflow
Queries, manages, and troubleshoots Apache Airflow using the `af` CLI. Use when working with anything related to Airflow - a DAG, a DAG run, a task log, an import or parse error, a broken DAG, or any Airflow operation. Covers listing and triggering DAGs, retrying runs, reading… - 100airflow-expert
Expert-level Apache Airflow orchestration, DAGs, operators, sensors, XComs, task dependencies, and scheduling. Use when the user mentions orchestration, DAGs, workflow, scheduling, or data pipeline, or when the task involves DAG Fundamentals, Task Dependencies and Branching,… - 100airflow-dag-authoring-and-validation
Authors Apache Airflow DAGs — task dependencies, operator choice, sensors, idempotency — and validates them with `airflow dags test`, import-time checks, and DAG linting before they reach a production scheduler. Use when the user asks to "write an Airflow DAG," "add a task…