Skip to content
Skills Library
Programming · Building & coding

celery-expert

/celery-expert

This skill should be used when the user asks to "configure Celery", "set up Celery with Redis", "write Celery tasks", "add background tasks with Celery", "celery beat schedule", "celery task retry", "celery performance tuning", "celery TDD", "test Celery tasks", "celery worker config", "celery monitoring", "celery security", or when code imports celery or the project uses a celeryconfig/celery.py module.

Repository
zweihander ↗
Licence
Apache-2.0
Last repository change
19.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
8
Scripts
3
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 19.10.2026, without references to retired models or APIs.

Licence

Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.

What to check before installing
  • 3 scripts in this package. Review them before installing, as Claude may execute them.

Scripts: examples/celery_app.py, examples/celery_beat_schedule.py, examples/tasks.py

What it does and when to use it

This skill should be used when the user asks to "configure Celery", "set up Celery with Redis", "write Celery tasks", "add background tasks with Celery", "celery beat schedule", "celery task retry", "celery performance tuning", "celery TDD", "test Celery tasks", "celery worker config", "celery monitoring", "celery security", or when code imports celery or the project uses a celeryconfig/celery.py module.

Triggers from the description

configure Celeryset up Celery with Rediswrite Celery tasksadd background tasks with Celerycelery beat schedulecelery task retrycelery performance tuningcelery TDD
Original author description

This skill should be used when the user asks to "configure Celery", "set up Celery with Redis", "write Celery tasks", "add background tasks with Celery", "celery beat schedule", "celery task retry", "celery performance tuning", "celery TDD", "test Celery tasks", "celery worker config", "celery monitoring", "celery security", or when code imports celery or the project uses a celeryconfig/celery.py module.

How it works

  1. Overview
  2. When to Apply
  3. Task Design Principles
  4. Redis Broker Quick Config
  5. Retry Strategy Summary
  6. Workflow Patterns
  7. Pre-Implementation Checklist
  8. Critical Reminders
  9. Additional Resources
Original SKILL.md headings · 145 lines, 642 words
SKILL.md preview
---
name: celery-expert
description: This skill should be used when the user asks to "configure Celery", "set up Celery with Redis", "write Celery tasks", "add background tasks with Celery", "celery beat schedule", "celery task retry", "celery performance tuning", "celery TDD", "test Celery tasks", "celery worker config", "celery monitoring", "celery security", or when code imports celery or the project uses a celeryconfig/celery.py module.
metadata:
  skills:
    "python-dev-workflow:tests-manager": "Use for Celery test planning, pytest fixtures, eager-mode decisions, live-worker integration tests, and test isolation."
---

# Celery Expert (Redis Broker)

## Overview

Guide production Celery with Redis as broker/result backend, covering task
design, workflows, Beat scheduling, performance, monitoring, and security.

**Core Principles:**

1. **Reliability** — task retries, acknowledgment strategies, no task loss
2. **Performance Aware** — optimize throughput with chunking, pooling, and proper prefetch
3. **Security** — safe serialization (JSON only, never pickle), broker authentication
4. **Observable** — monitoring, metrics, tracing, alerting

**Pydantic models in tasks:** Do not manually serialize Pydantic models (`.dict()`, `.model_dump()`, `**kwargs`). Use `celery-pydantic` so tasks can accept models directly.

## When to Apply

Activate when any of the following appear:

- `from celery import Celery` or `@app.task` / `@shared_task` in code
- `celeryconfig.py`, `celery.py`, or `beat_schedule` in project files
- User asks about background tasks, task queues, async processing, or distributed workers
- User mentions Flower, Celery Beat, task retries, or task routing

## Task Design Principles

Every task should follow these rules:

- **Idempotent** — safe to retry without side effects. Set values, don't increment them.
- **`bind=True`** — access `self.request.id`, `self.retry()`, and task metadata
- **JSON serializer** — never use pickle (remote code execution risk)
- **Time limits** — always set both `soft_time_limit` and `time_limit`
- **`acks_late=True`** for critical tasks — acknowledge after completion, not before
- **Small results** — store large data externally, return only references

```python
@app.task(
    bind=True,
    max_retries=3,
    acks_late=True,
    reject_on_worker_lost=True,
    time_limit=300,
    soft_time_limit=240,
    rate_limit='100/m',
)
def process_order(self, order_id: int):
    ...
```

## Redis Broker Quick Config

```python
from celery import Celery

app = Celery('myapp')
app.conf.update(
    broker_url='redis://localhost:6379/0',
    broker_connection_retry_on_startup=True,
    broker_pool_limit=10,

    result_backend='redis://localhost:6379/1',
    result_expires=3600,

    task_serializer='json',
    result_serializer='json',
    accept_content=['json'],

    task_acks_late=True,
    task_reject_on_worker_lost=True,
    task_time_limit=300,
    task_soft_time_limit=240,

    worker_prefetch_multiplier=4,
    worker_max_tasks_per_child=1000,
)
```

See `references/production-config.md` for complete production setup with TLS, beat, and deployment.

## Retry Strategy Summary

| Pattern | Use When | Config |
|---------|----------|--------|
| Manual retry | Custom backoff logic | `self.retry(exc=exc, countdown=2 ** self.request.retries)` |
| Auto-retry | Standard HTTP/network errors | `autoretry_for=(RequestException,), retry_backoff=True` |
| Auto-retry + jitter | High-concurrency retries | Add `retry_jitter=True, retry_backoff_max=600` |
| Reject (no retry) | Permanent failures, poison messages | `raise Reject(reason, requeue=False)` |

## Workflow Patterns

```python
from celery import chain, group, chord

# CHAIN: Sequential (A -> B -> C)
workflow = chain(fetch_data.s(url), process_item.s(), send_notification.s())

# GROUP: Parallel execution
job = group(fetch_data.s(url) for url in urls)

# CHORD: Map-Reduce (parallel + callback)
workflow = chord(group(process_item.s(item) for item in items))(aggregate_results.s())
```

## Pre-Implementation Checklist

**Before writing code:**
- [ ] Write a failing test for task behavior (use `python-dev-workflow:tests-manager`)
- [ ] Define idempotency strategy
- [ ] Choose queue routing for task priority
- [ ] Plan retry strategy and error handling

**Before deploying:**
- [ ] Time limits set (soft + hard)
- [ ] `acks_late=True` on critical tasks
- [ ] JSON serialization confirmed (not pickle)
- [ ] Redis broker authentication configured
- [ ] Result expiration set
- [ ] Monitoring active (Flower/Prometheus)
- [ ] Dead letter queue handling implemented

## Critical Reminders

**NEVER:** use pickle serialization, run without time limits, store large data in results, create non-idempotent tasks, run without broker auth, expose Flower without auth, manually serialize Pydantic models (use `celery-pydantic`)

**ALWAYS:** use JSON serialization, set time limits (soft + hard), make tasks idempotent, use `acks_late=True` for critical tasks, set result expiration, implement retry with backoff, monitor with Flower/Prometheus, log with correlation IDs

## Additional Resources

### Reference Files

- **`references/performance-patterns.md`** — Task chunking, prefetch tuning, result backend optimization, connection pooling, queue routing
- **`references/production-config.md`** — Complete Redis production config, beat scheduling, worker deployment, health checks
- **`references/security-standards.md`** — JSON serializer enforcement, Redis TLS/auth, input validation, Flower security
- **`references/common-mistakes.md`** — Anti-pattern table with fixes: pickle, idempotency, time limits, large results, pydantic serialization

### Example Files

Working code examples in `examples/`:

- **`examples/celery_app.py`** — App factory with Redis broker and recommended defaults
- **`examples/tasks.py`** — Task definitions: retry, chunking, idempotency patterns
- **`examples/celery_beat_schedule.py`** — Crontab and interval periodic task config
Files in this package (8)
  • SKILL.md 6 KB
  • examples/celery_app.py 1 KB
  • examples/celery_beat_schedule.py 1 KB
  • examples/tasks.py 3 KB
  • references/common-mistakes.md 4 KB
  • references/performance-patterns.md 4 KB
  • references/production-config.md 6 KB
  • references/security-standards.md 3 KB

Versions

The database has no other skills named celery-expert.

Database snapshot: 13.11.2026 · package version c32d2a4d41e6 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100python-celery-task-queues
    Use this skill when executing distributed asynchronous background tasks and scheduled workflows in Python using Celery.
  • 100django-celery
    Django + Celery async task patterns — configuration, task design, beat scheduling, retries, canvas workflows, monitoring, and testing. Use when adding background jobs, scheduled tasks, or async processing to a Django app.
  • 100django-celery
    Django + Celery async task patterns — configuration, task design, beat scheduling, retries, canvas workflows, monitoring, and testing. Use when adding background jobs, scheduled tasks, or async processing to a Django app.
  • 100django-celery
    Django + Celery async task patterns — configuration, task design, beat scheduling, retries, canvas workflows, monitoring, and testing. Use when adding background jobs, scheduled tasks, or async processing to a Django app.
  • 100django-celery
    Django + Celery 异步任务模式 — 配置、任务设计、Beat 调度、重试、Canvas 工作流、监控和测试。当向 Django 应用添加后台任务、定时任务或异步处理时使用。
  • 100using-celery
    Celery 5.3+ distributed task queue with Beat scheduler, Redis/RabbitMQ brokers, workflow patterns, and FastAPI integration. Use for background jobs, periodic tasks, and async processing.

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.