Skip to content
Skills Library
Data & analytics · Building & coding

zarr-xarray-integration

/zarr-xarray-integration

Integrates Zarr with xarray and Dask for labeled, multi-dimensional scientific data workflows. Covers reading and writing Zarr stores with xarray, append and region-write operations, multi-file virtual datasets, Dask chunk alignment with Zarr chunks, encoding configuration, consolidated metadata, and performance optimization. Use when the user asks about reading or writing Zarr stores with xarray, aligning Dask chunks with Zarr arrays, or optimizing large-scale multi-dimensional data pipelines.

Repository
rse-plugins ↗
Licence
BSD-3-Clause
Last repository change
14.10.2026

Assessment and security

95/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content30 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
5
Scripts
1
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 14.10.2026, without references to retired models or APIs.

Licence

BSD-3-Clause, otwarta.

What to check before installing
  • 1 scripts in this package. Review them before installing, as Claude may execute them.

Scripts: assets/xarray-zarr-roundtrip.py

What it does and when to use it

Integrates Zarr with xarray and Dask for labeled, multi-dimensional scientific data workflows. Covers reading and writing Zarr stores with xarray, append and region-write operations, multi-file virtual datasets, Dask chunk alignment with Zarr chunks, encoding configuration, consolidated metadata, and performance optimization. Use when the user asks about reading or writing Zarr stores with xarray, aligning Dask chunks with Zarr arrays, or optimizing large-scale multi-dimensional data pipelines.

Original author description

Integrates Zarr with xarray and Dask for labeled, multi-dimensional scientific data workflows. Covers reading and writing Zarr stores with xarray, append and region-write operations, multi-file virtual datasets, Dask chunk alignment with Zarr chunks, encoding configuration, consolidated metadata, and performance optimization. Use when the user asks about reading or writing Zarr stores with xarray, aligning Dask chunks with Zarr arrays, or optimizing large-scale multi-dimensional data pipelines.

How it works

  1. Resources
  2. Installation
  3. Read
  4. Write
  5. Append
  6. Region Writes (Parallel-Safe)
  7. Dask Chunk Alignment
  8. Encoding Fields
  9. Performance Quick Reference
Original SKILL.md headings · 138 lines, 624 words
SKILL.md preview
---
name: zarr-xarray-integration
description: Integrates Zarr with xarray and Dask for labeled, multi-dimensional scientific data workflows. Covers reading and writing Zarr stores with xarray, append and region-write operations, multi-file virtual datasets, Dask chunk alignment with Zarr chunks, encoding configuration, consolidated metadata, and performance optimization. Use when the user asks about reading or writing Zarr stores with xarray, aligning Dask chunks with Zarr arrays, or optimizing large-scale multi-dimensional data pipelines.
metadata:
  references:
    - references/patterns.md
    - references/examples.md
    - references/common-issues.md
---

# Zarr + xarray Integration

Use xarray as the high-level interface for reading, writing, and analyzing Zarr datasets, with Dask for parallel out-of-core computation.

## Resources

| Resource | Purpose |
|----------|---------|
| [references/patterns.md](references/patterns.md) | Full patterns: cloud reads, encoding, append, region writes, chunk alignment, consolidated metadata |
| [references/examples.md](references/examples.md) | Real-world workflows |
| [references/common-issues.md](references/common-issues.md) | Chunk mismatch, append errors, memory OOM, stale consolidated metadata |
| [assets/xarray-zarr-roundtrip.py](assets/xarray-zarr-roundtrip.py) | Runnable roundtrip demo |

## Installation

```bash
pixi add xarray zarr dask numpy     # pixi
pip install xarray[complete] zarr dask[complete]  # pip
pip install zarr[remote]             # adds s3fs/gcsfs for cloud
```

## Read

```python
import xarray as xr

ds = xr.open_zarr("data.zarr")                                    # local, lazy (Dask)
ds = xr.open_zarr("s3://bucket/data.zarr", storage_options={"anon": True},
                  consolidated=True)                               # cloud, public
ds = xr.open_zarr("data.zarr", chunks={"time": 30, "lat": 90})   # explicit Dask chunks
ds = xr.open_dataset("data.zarr", engine="zarr", chunks={})       # alternative form
```

| `open_zarr` parameter | Default | Description |
|-----------------------|---------|-------------|
| `chunks` | `"auto"` | `{}` = match Zarr chunks exactly; `None` = load eagerly |
| `consolidated` | `None` | Read consolidated metadata (faster cloud opens) |
| `storage_options` | `None` | fsspec kwargs (e.g. `{"anon": True}` for public S3) |
| `decode_cf` | `True` | Decode CF conventions (times, units, masks) |
| `group` | `None` | Open a specific group within the store |

## Write

```python
# Basic write
ds.to_zarr("output.zarr", mode="w")

# With per-variable encoding (recommended)
encoding = {
    "temperature":   {"chunks": {"time": 30, "lat": 90, "lon": 180}, "dtype": "float32"},
    "precipitation": {"chunks": {"time": 30, "lat": 90, "lon": 180}, "dtype": "float32"},
}
ds.to_zarr("output.zarr", mode="w", encoding=encoding, consolidated=True)

# To cloud
ds.to_zarr("s3://bucket/output.zarr", storage_options={"key": "...", "secret": "..."}, mode="w")
```

## Append

**Step 1 — Create initial store:**
```python
ds_initial.to_zarr("timeseries.zarr", mode="w")
```
**Step 2 — Append subsequent batches:**
```python
ds_new.to_zarr("timeseries.zarr", append_dim="time")
```
**Step 3 — Validate:**
```python
ds_check = xr.open_zarr("timeseries.zarr")
expected_len = len(ds_initial.time) + len(ds_new.time)
assert ds_check.dims["time"] == expected_len, f"Expected {expected_len}, got {ds_check.dims['time']}"
```

Note: non-appended dimensions must match exactly. See [common-issues.md](references/common-issues.md) Issue 2 for conflict errors. After appending, re-consolidate metadata if `consolidated=True` was used on initial write.

## Region Writes (Parallel-Safe)

**Step 1 — Pre-allocate with `compute=False`:**
```python
ds_full.to_zarr("parallel_output.zarr", mode="w", compute=False)
```
**Step 2 — Each worker writes its own non-overlapping region:**
```python
ds_chunk.to_zarr("parallel_output.zarr", region={"time": slice(day_start, day_end)})
```
**Step 3 — Validate after all workers complete:**
```python
ds_out = xr.open_zarr("parallel_output.zarr")
assert not ds_out["temperature"].isnull().all(), "Some regions were not written"
assert ds_out.dims["time"] == 365, f"Expected 365 time steps, got {ds_out.dims['time']}"
```

Region writes require a pre-existing store (step 1). Regions must not overlap between workers.

## Dask Chunk Alignment

Dask chunks must be exact multiples of Zarr chunks to avoid redundant reads:

```python
# Best: use Zarr's native chunks
ds = xr.open_zarr("data.zarr", chunks={})

# Check alignment
for var in ds.data_vars:
    zarr_chunks = ds[var].encoding.get("chunks")
    dask_chunks = ds[var].data.chunksize
    print(f"{var}: zarr={zarr_chunks}, dask={dask_chunks}")

# Misaligned example to avoid:
# Zarr chunks = (30, 90, 180); Dask chunks = (45, ...) — 45 is not a multiple of 30
```

See [common-issues.md](references/common-issues.md) Issue 1 for performance symptoms of misaligned chunks.

## Encoding Fields

| Field | Purpose |
|-------|---------|
| `chunks` | Zarr chunk sizes (dict or tuple) |
| `dtype` | On-disk data type |
| `compressor` | Compression codec (numcodecs object or `None` for default) |
| `_FillValue` | Fill value for missing data |
| `scale_factor` / `add_offset` | CF packing parameters |

See [patterns.md Pattern 2](references/patterns.md) for complete encoding examples including CF packing.

## Performance Quick Reference

| Symptom | Fix |
|---------|-----|
| Slow cloud open | Use `consolidated=True` on write and read |
| Memory OOM on `to_zarr` | Pass Dask-backed (lazy) Dataset; do not call `.compute()` first |
| Slow reads | Align Dask chunks with Zarr chunks (`chunks={}`) |
| Stale metadata after append | Call `zarr.consolidate_metadata("store.zarr")` after each append |
Files in this package (5)
  • SKILL.md 6 KB
  • assets/xarray-zarr-roundtrip.py 5 KB
  • references/common-issues.md 10 KB
  • references/examples.md 9 KB
  • references/patterns.md 11 KB

Versions

The database has no other skills named zarr-xarray-integration.

Database snapshot: 13.11.2026 · package version 9244cde10622 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100alterlab-zarr
    Chunked, compressed N-dimensional arrays for cloud storage with Zarr — parallel I/O, S3/GCS integration, and NumPy/Dask/Xarray compatibility. Use when storing or reading large N-D scientific arrays, streaming chunked data to/from cloud object stores, or building large-scale…
  • 100zarr-python
    Stores and queries chunked N-D scientific arrays with Zarr-Python 3, including codecs, sharding, S3/GCS storage, and NumPy/Dask/Xarray integration. Use for array layout, bounded I/O, format migration, or scientific metadata preservation.
  • 100xarray-for-multidimensional-data
    Use when the user asks about Xarray, xarray datasets, NetCDF files, multidimensional labeled arrays, or scientific/climate data analysis in Python. Work with labeled multidimensional arrays for scientific data analysis using Xarray. Covers NetCDF/HDF5/Zarr I/O, Dask integration…
  • 95data-migration
    Migrates data between formats with a focus on converting HDF5 and NetCDF datasets to Zarr. Covers xarray-based conversion, direct zarr.copy operations, VirtualiZarr for reference-based virtual Zarr stores, kerchunk for legacy workflows, validation strategies, and batch migration…
  • 100xarray
    N-dimensional labeled arrays for geoscience data. Read/write NetCDF, work with climate and oceanographic datasets, perform multi-dimensional analysis with labeled coordinates. Use when the agent needs to: (1) Read/write NetCDF or Zarr files, (2) Work with multidimensional arrays…
  • 100geozarr
    GeoZarr conventions for geospatial Zarr stores: CF grid_mapping, coordinate arrays, Zarr v2/v3 dimension declarations, multiscales, proj:/spatial: namespaces, and xarray/rioxarray/GDAL interop. Use when authoring, patching, validating, or ingesting Zarr stores intended to…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.