Skip to content
Skills Library
Data & analytics · Analysis

debugging-airflow-pipelines

/debugging-airflow-pipelines

Diagnose failing or stuck Apache Airflow pipelines — task failures and retries, scheduler/executor problems, XCom errors, zombie/queued tasks, dependency deadlocks, and pools/concurrency limits. Use when an Airflow task fails or is stuck queued, the scheduler is not running tasks, a DAG will not trigger, or tasks become zombies.

Licence
Apache-2.0
Last repository change
30.08.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 30.08.2026, without references to retired models or APIs.

Licence

Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.

What it does and when to use it

Diagnose failing or stuck Apache Airflow pipelines — task failures and retries, scheduler/executor problems, XCom errors, zombie/queued tasks, dependency deadlocks, and pools/concurrency limits. Use when an Airflow task fails or is stuck queued, the scheduler is not running tasks, a DAG will not trigger, or tasks become zombies.

Original author description

Diagnose failing or stuck Apache Airflow pipelines — task failures and retries, scheduler/executor problems, XCom errors, zombie/queued tasks, dependency deadlocks, and pools/concurrency limits. Use when an Airflow task fails or is stuck queued, the scheduler is not running tasks, a DAG will not trigger, or tasks become zombies.

How it works

  1. When to use
  2. Workflow
  3. Patterns
  4. Common pitfalls
Original SKILL.md headings · 67 lines, 426 words
SKILL.md preview
---
name: debugging-airflow-pipelines
description: Diagnose failing or stuck Apache Airflow pipelines — task failures and retries, scheduler/executor problems, XCom errors, zombie/queued tasks, dependency deadlocks, and pools/concurrency limits. Use when an Airflow task fails or is stuck queued, the scheduler is not running tasks, a DAG will not trigger, or tasks become zombies.
---

# Debugging Airflow Pipelines

## When to use

- A task failed, is stuck in `queued`/`scheduled`, or became a zombie.
- The scheduler isn't launching runs, or a DAG won't trigger.
- XCom pull errors, dependency deadlocks, or pool/concurrency starvation.
- Do NOT use for writing new DAGs (use `authoring-airflow-dags`).

## Workflow

```
- [ ] Read the task log first (Grid view -> task -> Logs)
- [ ] Check task state and why: failed, up_for_retry, queued, or none
- [ ] Localize: task-level bug vs scheduler/executor vs resource limit
- [ ] Fix root cause, then clear the task to re-run
```

1. **Read the task log.** The Grid/Graph view → failed task → Logs shows the real
   exception nearly every time.
2. **Check the state and reason.** `queued` for a long time is usually a
   resource/executor issue, not a code bug.
3. **Localize** using the table below.
4. **Re-run** by clearing the task instance (and downstream if needed) rather than
   re-triggering the whole DAG.

## Patterns

**Task keeps failing** — read the log; fix the exception; confirm `retries` are
set so transient errors self-heal. Use `on_failure_callback` for alerting.

**Task stuck in `queued`/`scheduled`:**

- Worker capacity exhausted, or a `pool` is full → check pool slots and
  `max_active_tasks`/parallelism.
- Celery/Kubernetes executor not picking up → check worker health and the message
  broker/queue.

**Zombie tasks** (process died, heartbeat lost) → often OOM or a killed worker.
Check worker memory/logs; reduce task memory or raise limits; Airflow marks it
failed and retries.

**Scheduler not creating runs:**

- DAG parse error → `airflow dags list-import-errors` (top-level code exceptions).
- DAG paused, or `start_date` in the future, or `catchup=False` with no new
  interval yet.
- `max_active_runs` reached → older runs not completing block new ones.

**Dependency deadlock / "no status"** — an upstream is `skipped` with the default
trigger rule; adjust `trigger_rule` (e.g. `all_done`, `none_failed_min_one_success`)
for branch/cleanup tasks.

**XCom errors** — pulling a key a task never pushed, or a payload too large for the
metadata DB. Push explicitly and pass storage references for big data.

## Common pitfalls

- **Re-triggering the whole DAG** instead of clearing the failed task — reruns
  work already done and can duplicate non-idempotent side effects.
- **Blaming code for `queued` tasks** — check pools, parallelism, and workers first.
- **Ignoring import errors** — one bad DAG file can stall parsing/scheduling.
- **No retries on flaky external calls** — every transient blip pages someone.
- **Clearing a non-idempotent task** — confirm the task is safe to re-run, or you
  duplicate data (see `writing-idempotent-transformations`).

Versions

The database has no other skills named debugging-airflow-pipelines.

Database snapshot: 13.11.2026 · package version 3a3f879997a7 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100airflow-scheduler-and-dag-troubleshooting
    Diagnoses a stuck or failed Airflow DAG run, scheduler-health problems (slow parsing, stuck queued tasks), and plans safe task-retry/backfill strategy. Use when the user reports "Airflow DAG is stuck," "task stuck in queued state," "scheduler seems unhealthy," "DAG run failed…
  • 95airflow
    Queries, manages, and troubleshoots Apache Airflow using the af CLI. Covers listing DAGs, triggering runs, reading task logs, diagnosing failures, debugging DAG import errors, checking connections, variables, pools, and monitoring health. Also routes to sub-skills for writing…
  • 100airflow-dag-generator
    Generate airflow dag generator operations. Auto-activating skill for Data Pipelines. Triggers on: airflow dag generator, airflow dag generator Part of the Data Pipelines skill category. Use when working with airflow dag generator functionality. Trigger with phrases like "airflow…
  • 100airflow
    Queries, manages, and troubleshoots Apache Airflow using the `af` CLI. Use when working with anything related to Airflow - a DAG, a DAG run, a task log, an import or parse error, a broken DAG, or any Airflow operation. Covers listing and triggering DAGs, retrying runs, reading…
  • 100airflow-expert
    Expert-level Apache Airflow orchestration, DAGs, operators, sensors, XComs, task dependencies, and scheduling. Use when the user mentions orchestration, DAGs, workflow, scheduling, or data pipeline, or when the task involves DAG Fundamentals, Task Dependencies and Branching,…
  • 100airflow-dag-authoring-and-validation
    Authors Apache Airflow DAGs — task dependencies, operator choice, sensors, idempotency — and validates them with `airflow dags test`, import-time checks, and DAG linting before they reach a production scheduler. Use when the user asks to "write an Airflow DAG," "add a task…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.