Przejdź do treści
Biblioteka skilli
Dane i analityka · Budowanie i kod

databricks-spark

/databricks-spark

Pomaga pisać i sprawdzać kod Spark i Databricks: wydajność, układ tabel Delta, Unity Catalog, koszty, strumienie i testy.

Repozytorium
data-workflow ↗
Licencja
MIT
Ostatnia zmiana repozytorium
27.10.2026

Ocena i bezpieczeństwo

90/ 100Klasa: dobry
Format25 / 25
Frontmatter z nazwą i opisem, poprawna nazwa, opis do 1024 znaków, nazwa zgodna z folderem.
Odkrywalność25 / 25
Opis mówi, kiedy użyć skilla, ma sensowną długość i jest w 3. osobie.
Treść25 / 35
Co najmniej 150 słów i do 500 linii, nagłówki, listy kroków, przykłady albo kod.
Higiena15 / 15
Bez zaślepek typu TODO, bez ścieżek Windows, z licencją.
Ryzykobrak sygnałów w skanie
Pliki w paczce
1
Skrypty
0
Komentarze HTML
0
Adresy URL w SKILL.md
20
Przestarzałość

Brak sygnałów. Repozytorium zmienione 27.10.2026, bez odwołań do wycofanych modeli i API.

Licencja

MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.

Co robi i kiedy

Pomaga pisać i sprawdzać kod Spark i Databricks: wydajność, układ tabel Delta, Unity Catalog, koszty, strumienie i testy.

Oryginalny opis (EN)

Apache Spark (PySpark, Spark SQL, Structured Streaming) and the Databricks platform — execution and shuffles, UDFs vs built-ins, caching, Delta layout (OPTIMIZE, VACUUM, liquid clustering, deletion vectors), Unity Catalog, compute and cost, Lakeflow Jobs and pipelines, bundles, secrets, streaming checkpoints, and testing. Use when writing or reviewing Spark code or notebooks, when a job is slow, skewed, out of memory, or costly, or when designing tables and jobs on Databricks.

Jak działa

  1. Steps
  2. Code
  3. Tables
  4. Platform
  5. Streaming
Sekcje z nagłówków SKILL.md, w oryginale · 82 linii, 800 słów
Podgląd SKILL.md
---
name: databricks-spark
description: Apache Spark (PySpark, Spark SQL, Structured Streaming) and the Databricks platform — execution and shuffles, UDFs vs built-ins, caching, Delta layout (OPTIMIZE, VACUUM, liquid clustering, deletion vectors), Unity Catalog, compute and cost, Lakeflow Jobs and pipelines, bundles, secrets, streaming checkpoints, and testing. Use when writing or reviewing Spark code or notebooks, when a job is slow, skewed, out of memory, or costly, or when designing tables and jobs on Databricks.
---

# Databricks and Spark

Spark work is judged at the scale it runs at, so every change is read against the plan and the Spark UI, never
against a sample's timing. Query-level SQL rules (ANSI mode, `EXPLAIN`, join hints, `MERGE`, time travel) live in the
`sql` skill's [`spark-sql.md`](../sql/references/spark-sql.md).

## Steps

1. **Pin the runtime.** Read the Databricks Runtime (or Spark) version from the cluster, job, or bundle config: the
   current LTS is DBR 18 (Spark 4.1); Spark 4 turns ANSI mode on. Done when the version is named.
2. **Find where the time goes** before changing code: the job's stages in the Spark UI, the shuffle read and write
   per stage, spill, the slowest task against the median (skew), and the physical plan. Done when the bottleneck is
   named with the metric that shows it.
3. **Change one thing**, from [Code](#code) or [Tables](#tables), and rerun at the same scale. Done when the same
   metric moved, and the output is unchanged (row count and control totals, per the contract).

## Code

- **Transformations are lazy; actions run them.** Each action starts a job; a shuffle splits it into stages. A
  shuffle (joins, `groupBy`, `repartition`, `distinct`) is the expensive part: count them in the plan.
- **The driver is one machine.** `collect()` and `toPandas()` bring every row to it: use `take(n)`, `limit`, or
  write the result to a table and read a summary.
- **Built-in functions first.** They run in the engine (and on Photon). A Python UDF serializes every row; a pandas
  UDF moves Arrow batches and is up to 100x faster than a Python UDF, but still slower than a built-in. Loops over
  rows in the driver are replaced by column expressions or window functions.
- **Partitions**: `spark.sql.shuffle.partitions` is 200 by default and AQE coalesces small ones. `repartition(n)`
  shuffles to a balanced count; `coalesce(n)` merges without a shuffle, and `coalesce(1)` can pull the upstream work
  onto one task: use `repartition` there.
- **Skew**: AQE splits a partition above 5x the median and above 256 MB in sort-merge joins; for skew it cannot see,
  find the hot keys with a `groupBy(key).count()` and handle them apart.
- **Caching**: on Databricks, avoid `cache()` as a habit and rely on the disk cache; cache only a result reused
  several times in one job, and `unpersist()` it.
- **Testing**: `pyspark.testing.utils.assertDataFrameEqual` and `assertSchemaEqual`, with pytest fixtures for the
  session, on small DataFrames built in the test. https://spark.apache.org/docs/latest/api/python/getting_started/testing_pyspark.html
- https://spark.apache.org/docs/latest/rdd-programming-guide.html ·
  https://spark.apache.org/docs/latest/sql-performance-tuning.html · https://docs.databricks.com/aws/en/udf/ ·
  https://docs.databricks.com/aws/en/lakehouse-architecture/performance-efficiency/best-practices

## Tables

- **Unity Catalog**: names are `catalog.schema.table`; managed tables are the default and recommended (a dropped
  one is recoverable for 7 days); files that are not tables live in volumes (`/Volumes/<catalog>/<schema>/<volume>/…`).
  https://docs.databricks.com/aws/en/data-governance/unity-catalog/ · https://docs.databricks.com/aws/en/volumes/
- **Partitioning**: none under 1 TB; liquid clustering from 1 TB to 100 TB and first above that; a partition holds at
  least 1 GB and partitions on a low-cardinality column only. https://docs.databricks.com/aws/en/tables/partitions
- **Small files**: `OPTIMIZE` bin-packs them and is idempotent; optimized writes are on by default for `MERGE`;
  auto compaction tunes file size. Predictive optimization runs `OPTIMIZE`, `VACUUM`, and `ANALYZE` on managed tables
  by itself. https://docs.databricks.com/aws/en/delta/optimize ·
  https://docs.databricks.com/aws/en/optimizations/predictive-optimization
- **`VACUUM`** keeps 7 days by default and ends time travel past its retention; a shorter retention can delete files
  of jobs still running, so preview with `DRY RUN`. https://docs.databricks.com/aws/en/delta/vacuum
- **Deletion vectors** make `DELETE`, `UPDATE`, and `MERGE` soft deletes: removing personal data for good takes
  `REORG TABLE … APPLY (PURGE)` and then `VACUUM`. https://docs.databricks.com/aws/en/delta/deletion-vectors

## Platform

- **Compute**: serverless for most new work; job compute for scheduled work, which costs much less than all-purpose
  compute; a SQL warehouse for interactive SQL; auto termination on everything interactive. Photon is on by default
  and does not accelerate UDFs or RDD code. https://docs.databricks.com/aws/en/lakehouse-architecture/cost-optimization/best-practices ·
  https://docs.databricks.com/aws/en/compute/photon
- **Lakeflow Jobs** orchestrate tasks as a DAG with time or event triggers. **Lakeflow pipelines** (formerly DLT;
  Python module `pyspark.pipelines` imported as `dp`) declare streaming tables, materialized views, flows, and
  expectations (see the `data-quality` skill). https://docs.databricks.com/aws/en/jobs/ ·
  https://docs.databricks.com/aws/en/ldp/concepts/
- **Declarative Automation Bundles** (formerly Databricks Asset Bundles) keep jobs and pipelines as YAML in the repo
  (`databricks.yml`) and deploy them with `databricks bundle validate | deploy | run`.
  https://docs.databricks.com/aws/en/dev-tools/bundles/
- **Secrets**: `dbutils.secrets.get(scope=…, key=…)`; output shows `[REDACTED]`, but redaction covers only the literal
  value, so a secret is never transformed or printed. https://docs.databricks.com/aws/en/security/secrets/

## Streaming

- Exactly-once needs a replayable source, a checkpoint, and an idempotent sink. The checkpoint survives restarts only
  while sources, the sink path, and stateful operations stay the same.
- A watermark bounds lateness: rows later than it are dropped, and state is cleaned only in append or update mode.
- `trigger(availableNow=True)` processes what exists and stops; it replaces the once trigger.
- `foreachBatch` is at-least-once; a `MERGE` inside it is made idempotent with `txnAppId` and `txnVersion = batchId`,
  and a new checkpoint needs a new `txnAppId`.
- https://spark.apache.org/docs/latest/streaming/apis-on-dataframes-and-datasets.html ·
  https://docs.databricks.com/aws/en/structured-streaming/delta-lake

Wersje

W bazie nie ma innych skilli o nazwie databricks-spark.

Migawka bazy z 13.11.2026 · wersja paczki 18a779c47286 (skrót SHA-256 całej paczki)

Pobrania w ostatnich 30 dniach: 0 · ranking

Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.

Podobne skille

  • 100databricks-workspace
    Odpytuje Databricks przez Spark SQL i Unity Catalog oraz przenosi pliki, notatniki i zadania między projektem lokalnym a obszarem roboczym.
  • 90databricks-lakeflow-pipeline-engineering
    Projektuje potoki Lakeflow Spark Declarative Pipelines: warstwy medallion, orkiestrację Lakeflow Jobs, układ tabel Delta, Auto Loader, ewolucję schematu, widoki zmaterializowane i strategię backfill, bez uruchamiania potoków.
  • 100databricks-expert
    Pomaga w platformie Databricks, Apache Spark, Delta Lake, MLflow, notatnikach i zarządzaniu klastrami, w tym przetwarzaniu danych w PySpark.
  • 100databricks-migration
    Przenosi notatniki i zadania Databricks do Fabric: dbutils na notebookutils, sekrety na Key Vault, DBFS na OneLake, Unity Catalog na Lakehouse i Jobs oraz DLT na Spark Job Definitions i Pipelines.
  • 100matlab-connect-databricks
    Łączy MATLAB z Databricks przez Spark (Databricks Connect) lub JDBC: uwierzytelnianie, sesje Spark, odczyt tabel Unity Catalog z filtrowaniem po stronie serwera oraz połączenia z SQL Warehouse.
  • 100engineering-databricks-pipelines
    Buduje niezawodne potoki Databricks na lakehouse: tabele Delta Lake, MERGE, Delta Live Tables, Unity Catalog, Auto Loader oraz dobór rozmiaru klastrów i zadań.

Uwagi i zgłoszenia

Wiadomość trafi do Bartosza Sokolińskiego. Adres e-mail jest potrzebny tylko do odpowiedzi. Szczegóły w polityce prywatności.