celery-expert
/celery-expert
This skill should be used when the user asks to "configure Celery", "set up Celery with Redis", "write Celery tasks", "add background tasks with Celery", "celery beat schedule", "celery task retry", "celery performance tuning", "celery TDD", "test Celery tasks", "celery worker config", "celery monitoring", "celery security", or when code imports celery or the project uses a celeryconfig/celery.py module.
- Author
- alex-kopylov ↗
- Repository
- zweihander ↗
- Licence
- Apache-2.0
- Last repository change
- 19.10.2026
Assessment and security
- Files in this package
- 8
- Scripts
- 3
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 19.10.2026, without references to retired models or APIs.
Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.
- 3 scripts in this package. Review them before installing, as Claude may execute them.
Scripts: examples/celery_app.py, examples/celery_beat_schedule.py, examples/tasks.py
What it does and when to use it
This skill should be used when the user asks to "configure Celery", "set up Celery with Redis", "write Celery tasks", "add background tasks with Celery", "celery beat schedule", "celery task retry", "celery performance tuning", "celery TDD", "test Celery tasks", "celery worker config", "celery monitoring", "celery security", or when code imports celery or the project uses a celeryconfig/celery.py module.
Triggers from the description
Original author description
This skill should be used when the user asks to "configure Celery", "set up Celery with Redis", "write Celery tasks", "add background tasks with Celery", "celery beat schedule", "celery task retry", "celery performance tuning", "celery TDD", "test Celery tasks", "celery worker config", "celery monitoring", "celery security", or when code imports celery or the project uses a celeryconfig/celery.py module.
How it works
- Overview
- When to Apply
- Task Design Principles
- Redis Broker Quick Config
- Retry Strategy Summary
- Workflow Patterns
- Pre-Implementation Checklist
- Critical Reminders
- Additional Resources
SKILL.md preview
---
name: celery-expert
description: This skill should be used when the user asks to "configure Celery", "set up Celery with Redis", "write Celery tasks", "add background tasks with Celery", "celery beat schedule", "celery task retry", "celery performance tuning", "celery TDD", "test Celery tasks", "celery worker config", "celery monitoring", "celery security", or when code imports celery or the project uses a celeryconfig/celery.py module.
metadata:
skills:
"python-dev-workflow:tests-manager": "Use for Celery test planning, pytest fixtures, eager-mode decisions, live-worker integration tests, and test isolation."
---
# Celery Expert (Redis Broker)
## Overview
Guide production Celery with Redis as broker/result backend, covering task
design, workflows, Beat scheduling, performance, monitoring, and security.
**Core Principles:**
1. **Reliability** — task retries, acknowledgment strategies, no task loss
2. **Performance Aware** — optimize throughput with chunking, pooling, and proper prefetch
3. **Security** — safe serialization (JSON only, never pickle), broker authentication
4. **Observable** — monitoring, metrics, tracing, alerting
**Pydantic models in tasks:** Do not manually serialize Pydantic models (`.dict()`, `.model_dump()`, `**kwargs`). Use `celery-pydantic` so tasks can accept models directly.
## When to Apply
Activate when any of the following appear:
- `from celery import Celery` or `@app.task` / `@shared_task` in code
- `celeryconfig.py`, `celery.py`, or `beat_schedule` in project files
- User asks about background tasks, task queues, async processing, or distributed workers
- User mentions Flower, Celery Beat, task retries, or task routing
## Task Design Principles
Every task should follow these rules:
- **Idempotent** — safe to retry without side effects. Set values, don't increment them.
- **`bind=True`** — access `self.request.id`, `self.retry()`, and task metadata
- **JSON serializer** — never use pickle (remote code execution risk)
- **Time limits** — always set both `soft_time_limit` and `time_limit`
- **`acks_late=True`** for critical tasks — acknowledge after completion, not before
- **Small results** — store large data externally, return only references
```python
@app.task(
bind=True,
max_retries=3,
acks_late=True,
reject_on_worker_lost=True,
time_limit=300,
soft_time_limit=240,
rate_limit='100/m',
)
def process_order(self, order_id: int):
...
```
## Redis Broker Quick Config
```python
from celery import Celery
app = Celery('myapp')
app.conf.update(
broker_url='redis://localhost:6379/0',
broker_connection_retry_on_startup=True,
broker_pool_limit=10,
result_backend='redis://localhost:6379/1',
result_expires=3600,
task_serializer='json',
result_serializer='json',
accept_content=['json'],
task_acks_late=True,
task_reject_on_worker_lost=True,
task_time_limit=300,
task_soft_time_limit=240,
worker_prefetch_multiplier=4,
worker_max_tasks_per_child=1000,
)
```
See `references/production-config.md` for complete production setup with TLS, beat, and deployment.
## Retry Strategy Summary
| Pattern | Use When | Config |
|---------|----------|--------|
| Manual retry | Custom backoff logic | `self.retry(exc=exc, countdown=2 ** self.request.retries)` |
| Auto-retry | Standard HTTP/network errors | `autoretry_for=(RequestException,), retry_backoff=True` |
| Auto-retry + jitter | High-concurrency retries | Add `retry_jitter=True, retry_backoff_max=600` |
| Reject (no retry) | Permanent failures, poison messages | `raise Reject(reason, requeue=False)` |
## Workflow Patterns
```python
from celery import chain, group, chord
# CHAIN: Sequential (A -> B -> C)
workflow = chain(fetch_data.s(url), process_item.s(), send_notification.s())
# GROUP: Parallel execution
job = group(fetch_data.s(url) for url in urls)
# CHORD: Map-Reduce (parallel + callback)
workflow = chord(group(process_item.s(item) for item in items))(aggregate_results.s())
```
## Pre-Implementation Checklist
**Before writing code:**
- [ ] Write a failing test for task behavior (use `python-dev-workflow:tests-manager`)
- [ ] Define idempotency strategy
- [ ] Choose queue routing for task priority
- [ ] Plan retry strategy and error handling
**Before deploying:**
- [ ] Time limits set (soft + hard)
- [ ] `acks_late=True` on critical tasks
- [ ] JSON serialization confirmed (not pickle)
- [ ] Redis broker authentication configured
- [ ] Result expiration set
- [ ] Monitoring active (Flower/Prometheus)
- [ ] Dead letter queue handling implemented
## Critical Reminders
**NEVER:** use pickle serialization, run without time limits, store large data in results, create non-idempotent tasks, run without broker auth, expose Flower without auth, manually serialize Pydantic models (use `celery-pydantic`)
**ALWAYS:** use JSON serialization, set time limits (soft + hard), make tasks idempotent, use `acks_late=True` for critical tasks, set result expiration, implement retry with backoff, monitor with Flower/Prometheus, log with correlation IDs
## Additional Resources
### Reference Files
- **`references/performance-patterns.md`** — Task chunking, prefetch tuning, result backend optimization, connection pooling, queue routing
- **`references/production-config.md`** — Complete Redis production config, beat scheduling, worker deployment, health checks
- **`references/security-standards.md`** — JSON serializer enforcement, Redis TLS/auth, input validation, Flower security
- **`references/common-mistakes.md`** — Anti-pattern table with fixes: pickle, idempotency, time limits, large results, pydantic serialization
### Example Files
Working code examples in `examples/`:
- **`examples/celery_app.py`** — App factory with Redis broker and recommended defaults
- **`examples/tasks.py`** — Task definitions: retry, chunking, idempotency patterns
- **`examples/celery_beat_schedule.py`** — Crontab and interval periodic task config
Files in this package (8)
- SKILL.md 6 KB
- examples/celery_app.py 1 KB
- examples/celery_beat_schedule.py 1 KB
- examples/tasks.py 3 KB
- references/common-mistakes.md 4 KB
- references/performance-patterns.md 4 KB
- references/production-config.md 6 KB
- references/security-standards.md 3 KB
Versions
The database has no other skills named celery-expert.
Database snapshot: 13.11.2026 · package version c32d2a4d41e6 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100python-celery-task-queues
Use this skill when executing distributed asynchronous background tasks and scheduled workflows in Python using Celery. - 100django-celery
Django + Celery async task patterns — configuration, task design, beat scheduling, retries, canvas workflows, monitoring, and testing. Use when adding background jobs, scheduled tasks, or async processing to a Django app. - 100django-celery
Django + Celery async task patterns — configuration, task design, beat scheduling, retries, canvas workflows, monitoring, and testing. Use when adding background jobs, scheduled tasks, or async processing to a Django app. - 100django-celery
Django + Celery async task patterns — configuration, task design, beat scheduling, retries, canvas workflows, monitoring, and testing. Use when adding background jobs, scheduled tasks, or async processing to a Django app. - 100django-celery
Django + Celery 异步任务模式 — 配置、任务设计、Beat 调度、重试、Canvas 工作流、监控和测试。当向 Django 应用添加后台任务、定时任务或异步处理时使用。 - 100using-celery
Celery 5.3+ distributed task queue with Beat scheduler, Redis/RabbitMQ brokers, workflow patterns, and FastAPI integration. Use for background jobs, periodic tasks, and async processing.