Skip to content
Skills Library
Data & analytics · Building & coding

processing-streaming-data

/processing-streaming-data

Build reliable streaming data pipelines with Kafka, Spark Structured Streaming, or Flink — delivery semantics (at-least-once vs exactly-once), idempotent consumers, event-time windowing and watermarks, handling late/out-of-order data, and checkpointing. Use when building or debugging streaming pipelines, configuring consumer groups, choosing delivery guarantees, or handling late events.

Licence
Apache-2.0
Last repository change
30.08.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 30.08.2026, without references to retired models or APIs.

Licence

Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.

What it does and when to use it

Build reliable streaming data pipelines with Kafka, Spark Structured Streaming, or Flink — delivery semantics (at-least-once vs exactly-once), idempotent consumers, event-time windowing and watermarks, handling late/out-of-order data, and checkpointing. Use when building or debugging streaming pipelines, configuring consumer groups, choosing delivery guarantees, or handling late events.

Original author description

Build reliable streaming data pipelines with Kafka, Spark Structured Streaming, or Flink — delivery semantics (at-least-once vs exactly-once), idempotent consumers, event-time windowing and watermarks, handling late/out-of-order data, and checkpointing. Use when building or debugging streaming pipelines, configuring consumer groups, choosing delivery guarantees, or handling late events.

How it works

  1. When to use
  2. Workflow
  3. Patterns
  4. Common pitfalls
Original SKILL.md headings · 72 lines, 387 words
SKILL.md preview
---
name: processing-streaming-data
description: Build reliable streaming data pipelines with Kafka, Spark Structured Streaming, or Flink — delivery semantics (at-least-once vs exactly-once), idempotent consumers, event-time windowing and watermarks, handling late/out-of-order data, and checkpointing. Use when building or debugging streaming pipelines, configuring consumer groups, choosing delivery guarantees, or handling late events.
---

# Processing Streaming Data

## When to use

- Building or debugging Kafka / Spark Structured Streaming / Flink pipelines.
- Choosing delivery semantics and making consumers idempotent.
- Windowing on event time and handling late/out-of-order events.
- Do NOT use for batch ELT (use `building-ingestion-pipelines`).

## Workflow

```
- [ ] Choose delivery semantics; make effects idempotent regardless
- [ ] Use event time (not processing time) with watermarks
- [ ] Configure checkpointing for recovery
- [ ] Decide late-data policy (allowed lateness -> update, drop, or side output)
- [ ] Size partitions/parallelism to the throughput
```

1. **Delivery semantics.** At-least-once is the common default (duplicates
   possible on retry). Exactly-once needs transactional sinks/offsets. Either way,
   **make the downstream effect idempotent** (upsert by key) so duplicates don't
   corrupt state — this is more robust than relying on exactly-once alone.
2. **Event time + watermarks.** Window by when the event happened, not when it was
   processed; a watermark bounds how long to wait for stragglers.
3. **Checkpoint** so a failed job resumes from the last committed offset/state
   instead of reprocessing everything or losing data.
4. **Late data policy** — allowed lateness updates windows; beyond it, drop or
   route to a side output/dead-letter for reconciliation.

## Patterns

**Idempotent consumer** — key the sink write on a stable event id so replays
upsert rather than duplicate:

```python
# Structured Streaming: exactly-once-ish via idempotent upsert in foreachBatch
def upsert(batch_df, batch_id):
    (delta_table.alias("t")
       .merge(batch_df.dropDuplicates(["event_id"]).alias("s"), "t.event_id = s.event_id")
       .whenNotMatchedInsertAll().execute())

(stream.writeStream.foreachBatch(upsert)
   .option("checkpointLocation", "/chk/events").start())
```

**Event-time windowing with watermark:**

```python
(events
  .withWatermark("event_time", "10 minutes")
  .groupBy(window("event_time", "5 minutes"), "user_id")
  .count())
```

**Kafka consumer basics** — consumer group for scale-out; commit offsets **after**
processing (not before) to avoid data loss; partition count caps parallelism.

## Common pitfalls

- **Relying on exactly-once instead of idempotent sinks** — any at-least-once hop
  reintroduces duplicates; upsert by key.
- **Processing-time windows** — produce wrong results when events arrive late or
  the job lags; use event time + watermark.
- **Committing offsets before processing** — a crash then loses those messages.
- **No checkpointing** — recovery either reprocesses everything or loses state.
- **Unbounded state** (no watermark/TTL) — keyed state grows until the job OOMs.
- **Too few partitions** — caps consumer parallelism; you can't scale past
  partition count.
- **Ignoring dead-letter/late paths** — bad or very late events silently vanish.

Versions

The database has no other skills named processing-streaming-data.

Database snapshot: 13.11.2026 · package version e21bd533d320 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100stream-processing-expert
    Build continuous data pipelines over event streams: windowing, watermarks, exactly-once semantics, stateful processing and change data capture. Use when the user mentions stream processing, Flink, Spark Structured Streaming, CDC or Debezium, event time versus processing time,…
  • 90streaming-and-messaging-systems
    Guides agents through event streaming and real-time data pipeline design. Use when working with Kafka, Kinesis, Flink, stream processing, windowing, stateful consumers, or near-real-time publish flows.
  • 95backend-data-streaming
    Use this skill when the user says 'data streaming', 'kafka architecture', 'stream processing', 'flink job', 'kafka streams', 'exactly once', 'watermarks', 'windowed aggregation', 'consumer lag', 'log compaction'. Designs backend streaming pipelines on Kafka (or compatible logs)…
  • 90data-streaming
    Use this skill when asked about streaming, Kafka, Flink, Kinesis, stream processing, event stream, real-time, CDC, change data capture, message queue, or stream architecture. This skill enforces: Kafka topic design with partitioning strategy, Flink/ksqlDB stream processing with…
  • 100realtime-data-agent
    Activates RealtimeDataAgent for streaming data and event-driven architecture. Use when you need Kafka or Kinesis topic design with partition strategy, Flink or Spark Streaming transformation logic, WebSocket server implementation for real-time clients, real-time threshold…
  • 100data-pipelines
    Builds batch and streaming data pipelines that are idempotent, re-runnable and safe to backfill, with incremental loads, late-data handling, schema contracts and data quality gates. Use when writing ETL or ELT jobs, Airflow, Dagster or Prefect DAGs, dbt models, Kafka, Spark or…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.