Skip to content
Skills Library
Data & analytics · Automation

executing-spark

/executing-spark

Execute arbitrary Python or PySpark code on Fabric Spark compute without creating a notebook artifact; ephemeral Livy sessions with full Delta table access. Automatically invoke when the user asks to "run PySpark in Fabric", "create a Livy session", "execute Python on Fabric compute", "run Spark without a notebook", "submit code to Fabric", "ephemeral Spark execution", "run ETL in Fabric".

Licence
GPL-3.0
Last repository change
27.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
3
Scripts
0
HTML comments
0
URLs in SKILL.md
3
Deprecation

No signals detected. Repository changed on 27.10.2026, without references to retired models or APIs.

Licence

GPL-3.0, otwarta.

What it does and when to use it

Execute arbitrary Python or PySpark code on Fabric Spark compute without creating a notebook artifact; ephemeral Livy sessions with full Delta table access. Automatically invoke when the user asks to "run PySpark in Fabric", "create a Livy session", "execute Python on Fabric compute", "run Spark without a notebook", "submit code to Fabric", "ephemeral Spark execution", "run ETL in Fabric".

Triggers from the description

run PySpark in Fabriccreate a Livy sessionexecute Python on Fabric computerun Spark without a notebooksubmit code to Fabricephemeral Spark executionrun ETL in Fabric
Original author description

Execute arbitrary Python or PySpark code on Fabric Spark compute without creating a notebook artifact; ephemeral Livy sessions with full Delta table access. Automatically invoke when the user asks to "run PySpark in Fabric", "create a Livy session", "execute Python on Fabric compute", "run Spark without a notebook", "submit code to Fabric", "ephemeral Spark execution", "run ETL in Fabric".

How it works

  1. Prerequisites
  2. Critical: Authentication
  3. Lifecycle
  4. Getting IDs
  5. Submitting Code
  6. What Works
  7. What Does Not Work
  8. When to Use This vs Alternatives
  9. References
Original SKILL.md headings · 93 lines, 513 words
SKILL.md preview
---
name: executing-spark
version: 26.40.3
description: Execute arbitrary Python or PySpark code on Fabric Spark compute without creating a notebook artifact; ephemeral Livy sessions with full Delta table access. Automatically invoke when the user asks to "run PySpark in Fabric", "create a Livy session", "execute Python on Fabric compute", "run Spark without a notebook", "submit code to Fabric", "ephemeral Spark execution", "run ETL in Fabric".
---

# Executing Spark Code in Fabric (No Notebook)

Run arbitrary PySpark or Python code on Fabric Spark compute via the Livy API. No notebook artifact is created or persisted; sessions are ephemeral. Full read/write access to lakehouse Delta tables via Spark SQL.

## Prerequisites

- Azure CLI authenticated (`az login`)
- A lakehouse in the target workspace (the Livy session runs against it)
- Fabric capacity (F or trial)

## Critical: Authentication

The Livy API requires a token from `az account get-access-token --resource https://api.fabric.microsoft.com`. Tokens from `fab auth` do **not** work for OneLake storage access inside the Spark session.

```python
import subprocess, json

# On Windows, pass the whole command as ONE string with shell=True; list-form args get
# mangled. The list form is correct on Linux and macOS.
result = subprocess.run(
    "az account get-access-token --resource https://api.fabric.microsoft.com",
    shell=True, capture_output=True, text=True
)
token = json.loads(result.stdout)["accessToken"]
```

Do not output or log the token. Pass it directly to the API call.

## Lifecycle

```
1. Create session   POST .../sessions              {"kind": "pyspark"}
2. Wait for idle    GET  .../sessions/{id}          poll until state: "idle" (~30-90s)
3. Submit code      POST .../sessions/{id}/statements   {"code": "...", "kind": "pyspark"}
4. Get result       GET  .../sessions/{id}/statements/{n}   poll until state: "available"
5. Delete session   DELETE .../sessions/{id}        ALWAYS do this
```

Base URL: `https://api.fabric.microsoft.com/v1/workspaces/{wsId}/lakehouses/{lhId}/livyapi/versions/2023-12-01`

**CRITICAL: Always delete sessions when done.** Idle sessions consume Fabric capacity units (CUs). A forgotten session burns compute until it times out (default: 20 minutes). In automation, wrap cleanup in a `finally` block.

## Getting IDs

```bash
WS_ID=$(fab get "Workspace.Workspace" -q "id" | tr -d '"')
LH_ID=$(fab get "Workspace.Workspace/Lakehouse.Lakehouse" -q "id" | tr -d '"')
```

## Submitting Code

Submit PySpark or pure Python as statements. The `spark` object is available automatically.

```python
# Statement payload
{"code": "df = spark.sql('SELECT * FROM products LIMIT 10')\ndf.show()", "kind": "pyspark"}
```

Results are in `output.data["text/plain"]` when `state: "available"` and `output.status: "ok"`.

## What Works

- `spark.sql("SELECT ...")` ; full Spark SQL against lakehouse tables
- `spark.sql("SHOW TABLES")` ; metastore access
- `df.write.mode("overwrite").saveAsTable(...)` ; write Delta tables
- Pure Python (pandas, numpy, pyarrow); runs on Spark container
- In-memory Spark DataFrames and transformations
- Multiple sequential statements in one session

## What Does Not Work

- `deltalake` (delta-rs) is not pre-installed; use Spark SQL instead
- `notebookutils` has limited functionality (no FUSE mount at `/lakehouse/default/`)
- Tokens from `fab auth` ; must use `az` CLI token
- Tokens expire after ~60 minutes; long sessions need token refresh

## When to Use This vs Alternatives

| Scenario | Approach |
|----------|----------|
| Quick read-only exploration | DuckDB locally (fastest; see `using-duckdb` skill) |
| Write data back to lakehouse | Livy session or notebook |
| Ephemeral transform; no artifact | Livy session (this skill) |
| Complex multi-cell workflow | Notebook (`nb exec` or portal) |
| Scheduled ETL | Notebook via `fab job run` |
| Agent-driven compute (Dagster, orchestrators) | Livy session |

## References

- **`references/livy-api.md`** -- Full API reference with endpoints, request/response formats, and error handling
- **`references/example-script.md`** -- Complete working script that creates a session, queries data, writes results, and cleans up
Files in this package (3)
  • SKILL.md 4 KB
  • references/example-script.md 4 KB
  • references/livy-api.md 4 KB

Versions

RepositoryScoreLicence
This page · insightfulanalytics/pbi_agentic_dev100GPL-3.0
data-goblin/power-bi-agentic-development100GPL-3.0

Database snapshot: 13.11.2026 · package version c18daee9f45d (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100gcp-spark
    Develops, optimizes and runs PySpark/Spark code on Managed Spark (Dataproc clusters and Serverless) on Google Cloud. Use when: - Authoring or running Spark/PySpark notebooks (any kernel), incl. data analysis, reports and visualizations. - Writing Spark ETL pipelines or preparing…
  • 100apache-spark
    Apache Spark is a distributed engine for batch, SQL and streaming data processing, used from Python through PySpark. Use when a user asks to process big data, run distributed computations, build ETL pipelines, analyse data at scale, read Parquet from S3, or stream from Kafka with PySpark.
  • 100pyspark-etl
    Best practices for building performant, testable PySpark ETL pipelines with Spark SQL and Apache Iceberg. Use when writing or reviewing PySpark jobs, designing joins and window functions, working with map/array higher-order functions, or building idempotent cumulative/snapshot table merges.
  • 100spark
    Expert Apache Spark assistance covering PySpark, Spark SQL, DataFrame transformations, broadcast joins, and cluster optimization. Use when processing petabyte distributed data and running big data ETL pipelines.
  • 100gcp-spark
    Authors, optimizes, and runs PySpark and Spark code on Google Cloud Managed Spark (Dataproc Serverless, clusters, and Spark Connect). Use when: - Authoring or running Spark/PySpark notebooks in any kernel (Local Kernel with spark-remote templates, Remote Spark / Remote Kernel,…
  • 100e2e-medallion-architecture
    Plans and builds end-to-end Fabric data platforms on the Bronze, Silver, and Gold medallion pattern: ingestion to analytics, PySpark and Delta layers, pipeline orchestration, per-layer Spark tuning, Materialized Lake View versus notebook choices for Silver and Gold, and Direct…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.