processing-streaming-data
/processing-streaming-data
Build reliable streaming data pipelines with Kafka, Spark Structured Streaming, or Flink — delivery semantics (at-least-once vs exactly-once), idempotent consumers, event-time windowing and watermarks, handling late/out-of-order data, and checkpointing. Use when building or debugging streaming pipelines, configuring consumer groups, choosing delivery guarantees, or handling late events.
- Author
- unknown-333 ↗
- Repository
- awesome-data-engineering-skills ↗
- Licence
- Apache-2.0
- Last repository change
- 30.08.2026
Assessment and security
- Files in this package
- 1
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 30.08.2026, without references to retired models or APIs.
Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.
What it does and when to use it
Build reliable streaming data pipelines with Kafka, Spark Structured Streaming, or Flink — delivery semantics (at-least-once vs exactly-once), idempotent consumers, event-time windowing and watermarks, handling late/out-of-order data, and checkpointing. Use when building or debugging streaming pipelines, configuring consumer groups, choosing delivery guarantees, or handling late events.
Original author description
Build reliable streaming data pipelines with Kafka, Spark Structured Streaming, or Flink — delivery semantics (at-least-once vs exactly-once), idempotent consumers, event-time windowing and watermarks, handling late/out-of-order data, and checkpointing. Use when building or debugging streaming pipelines, configuring consumer groups, choosing delivery guarantees, or handling late events.
How it works
- When to use
- Workflow
- Patterns
- Common pitfalls
SKILL.md preview
---
name: processing-streaming-data
description: Build reliable streaming data pipelines with Kafka, Spark Structured Streaming, or Flink — delivery semantics (at-least-once vs exactly-once), idempotent consumers, event-time windowing and watermarks, handling late/out-of-order data, and checkpointing. Use when building or debugging streaming pipelines, configuring consumer groups, choosing delivery guarantees, or handling late events.
---
# Processing Streaming Data
## When to use
- Building or debugging Kafka / Spark Structured Streaming / Flink pipelines.
- Choosing delivery semantics and making consumers idempotent.
- Windowing on event time and handling late/out-of-order events.
- Do NOT use for batch ELT (use `building-ingestion-pipelines`).
## Workflow
```
- [ ] Choose delivery semantics; make effects idempotent regardless
- [ ] Use event time (not processing time) with watermarks
- [ ] Configure checkpointing for recovery
- [ ] Decide late-data policy (allowed lateness -> update, drop, or side output)
- [ ] Size partitions/parallelism to the throughput
```
1. **Delivery semantics.** At-least-once is the common default (duplicates
possible on retry). Exactly-once needs transactional sinks/offsets. Either way,
**make the downstream effect idempotent** (upsert by key) so duplicates don't
corrupt state — this is more robust than relying on exactly-once alone.
2. **Event time + watermarks.** Window by when the event happened, not when it was
processed; a watermark bounds how long to wait for stragglers.
3. **Checkpoint** so a failed job resumes from the last committed offset/state
instead of reprocessing everything or losing data.
4. **Late data policy** — allowed lateness updates windows; beyond it, drop or
route to a side output/dead-letter for reconciliation.
## Patterns
**Idempotent consumer** — key the sink write on a stable event id so replays
upsert rather than duplicate:
```python
# Structured Streaming: exactly-once-ish via idempotent upsert in foreachBatch
def upsert(batch_df, batch_id):
(delta_table.alias("t")
.merge(batch_df.dropDuplicates(["event_id"]).alias("s"), "t.event_id = s.event_id")
.whenNotMatchedInsertAll().execute())
(stream.writeStream.foreachBatch(upsert)
.option("checkpointLocation", "/chk/events").start())
```
**Event-time windowing with watermark:**
```python
(events
.withWatermark("event_time", "10 minutes")
.groupBy(window("event_time", "5 minutes"), "user_id")
.count())
```
**Kafka consumer basics** — consumer group for scale-out; commit offsets **after**
processing (not before) to avoid data loss; partition count caps parallelism.
## Common pitfalls
- **Relying on exactly-once instead of idempotent sinks** — any at-least-once hop
reintroduces duplicates; upsert by key.
- **Processing-time windows** — produce wrong results when events arrive late or
the job lags; use event time + watermark.
- **Committing offsets before processing** — a crash then loses those messages.
- **No checkpointing** — recovery either reprocesses everything or loses state.
- **Unbounded state** (no watermark/TTL) — keyed state grows until the job OOMs.
- **Too few partitions** — caps consumer parallelism; you can't scale past
partition count.
- **Ignoring dead-letter/late paths** — bad or very late events silently vanish.
Versions
The database has no other skills named processing-streaming-data.
Database snapshot: 13.11.2026 · package version e21bd533d320 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100stream-processing-expert
Build continuous data pipelines over event streams: windowing, watermarks, exactly-once semantics, stateful processing and change data capture. Use when the user mentions stream processing, Flink, Spark Structured Streaming, CDC or Debezium, event time versus processing time,… - 90streaming-and-messaging-systems
Guides agents through event streaming and real-time data pipeline design. Use when working with Kafka, Kinesis, Flink, stream processing, windowing, stateful consumers, or near-real-time publish flows. - 95backend-data-streaming
Use this skill when the user says 'data streaming', 'kafka architecture', 'stream processing', 'flink job', 'kafka streams', 'exactly once', 'watermarks', 'windowed aggregation', 'consumer lag', 'log compaction'. Designs backend streaming pipelines on Kafka (or compatible logs)… - 90data-streaming
Use this skill when asked about streaming, Kafka, Flink, Kinesis, stream processing, event stream, real-time, CDC, change data capture, message queue, or stream architecture. This skill enforces: Kafka topic design with partitioning strategy, Flink/ksqlDB stream processing with… - 100realtime-data-agent
Activates RealtimeDataAgent for streaming data and event-driven architecture. Use when you need Kafka or Kinesis topic design with partition strategy, Flink or Spark Streaming transformation logic, WebSocket server implementation for real-time clients, real-time threshold… - 100data-pipelines
Builds batch and streaming data pipelines that are idempotent, re-runnable and safe to backfill, with incremental loads, late-data handling, schema contracts and data quality gates. Use when writing ETL or ELT jobs, Airflow, Dagster or Prefect DAGs, dbt models, Kafka, Spark or…