Skip to content
Skills Library
Data & analytics · Conversion & formats

data-migration

/data-migration

Migrates data between formats with a focus on converting HDF5 and NetCDF datasets to Zarr. Covers xarray-based conversion, direct zarr.copy operations, VirtualiZarr for reference-based virtual Zarr stores, kerchunk for legacy workflows, validation strategies, and batch migration pipelines. Use when the user asks about converting HDF5 or NetCDF files to Zarr, migrating scientific datasets, or working with VirtualiZarr or kerchunk for virtual stores.

Repository
rse-plugins ↗
Licence
BSD-3-Clause
Last repository change
14.10.2026

Assessment and security

95/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content30 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
5
Scripts
1
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 14.10.2026, without references to retired models or APIs.

Licence

BSD-3-Clause, otwarta.

What to check before installing
  • 1 scripts in this package. Review them before installing, as Claude may execute them.

Scripts: assets/hdf5-to-zarr-migration.py

What it does and when to use it

Migrates data between formats with a focus on converting HDF5 and NetCDF datasets to Zarr. Covers xarray-based conversion, direct zarr.copy operations, VirtualiZarr for reference-based virtual Zarr stores, kerchunk for legacy workflows, validation strategies, and batch migration pipelines. Use when the user asks about converting HDF5 or NetCDF files to Zarr, migrating scientific datasets, or working with VirtualiZarr or kerchunk for virtual stores.

Original author description

Migrates data between formats with a focus on converting HDF5 and NetCDF datasets to Zarr. Covers xarray-based conversion, direct zarr.copy operations, VirtualiZarr for reference-based virtual Zarr stores, kerchunk for legacy workflows, validation strategies, and batch migration pipelines. Use when the user asks about converting HDF5 or NetCDF files to Zarr, migrating scientific datasets, or working with VirtualiZarr or kerchunk for virtual stores.

How it works

  1. Resources
  2. Strategy Selection
  3. HDF5 to Zarr
  4. NetCDF to Zarr
  5. VirtualiZarr (No Data Copy)
  6. Kerchunk (Legacy)
  7. Validation
Original SKILL.md headings · 161 lines, 590 words
SKILL.md preview
---
name: data-migration
description: Migrates data between formats with a focus on converting HDF5 and NetCDF datasets to Zarr. Covers xarray-based conversion, direct zarr.copy operations, VirtualiZarr for reference-based virtual Zarr stores, kerchunk for legacy workflows, validation strategies, and batch migration pipelines. Use when the user asks about converting HDF5 or NetCDF files to Zarr, migrating scientific datasets, or working with VirtualiZarr or kerchunk for virtual stores.
metadata:
  references:
    - references/patterns.md
    - references/examples.md
    - references/common-issues.md
---

# Data Migration to Zarr

Migrate HDF5, NetCDF, and other formats to Zarr using xarray, h5py, VirtualiZarr, or kerchunk.

## Resources

| Resource | Purpose |
|----------|---------|
| [references/patterns.md](references/patterns.md) | Full patterns: single-file, multi-file, virtual refs, incremental batch, post-migration validation |
| [references/examples.md](references/examples.md) | Real-world migration examples |
| [references/common-issues.md](references/common-issues.md) | HDF5 features unsupported in Zarr, metadata loss, memory errors, fill-value mismatches |
| [assets/hdf5-to-zarr-migration.py](assets/hdf5-to-zarr-migration.py) | Runnable migration script |

## Strategy Selection

| Strategy | Data Copy | Best For |
|----------|-----------|----------|
| xarray conversion | Full copy | Simple files, rechunking needed, CF metadata |
| h5py + zarr.copy | Full copy | Complex HDF5 group hierarchies |
| VirtualiZarr | No copy | Large archives, cloud-hosted sources |
| kerchunk | No copy | Legacy workflows, existing JSON refs |

```
Migrating to Zarr?
├── Single file, simple structure → xr.open_dataset() → ds.to_zarr()
├── Many files, same structure   → xr.open_mfdataset() → ds.to_zarr()
├── No data copy wanted          → VirtualiZarr (modern) or kerchunk (legacy)
├── Complex HDF5 group hierarchy → h5py + zarr group copy
└── Need rechunking              → xarray with chunks= and encoding=
```

## HDF5 to Zarr

**Step 1 — Open and inspect:**
```python
import xarray as xr
ds = xr.open_dataset("simulation_output.h5", engine="h5netcdf")
print(ds)
```

**Step 2 — Convert with encoding:**
```python
encoding = {var: {"chunks": {"time": 30, "x": 100, "y": 100}} for var in ds.data_vars}
ds.to_zarr("simulation_output.zarr", encoding=encoding, consolidated=True)
```

**Step 3 — Validate:**
```python
ds_zarr = xr.open_zarr("simulation_output.zarr")
xr.testing.assert_allclose(ds, ds_zarr)
print("Validation passed")
```

For complex HDF5 group hierarchies, use h5py recursion — see [patterns.md Pattern 4](references/patterns.md).

## NetCDF to Zarr

**Single file:**
```python
ds = xr.open_dataset("climate_model_output.nc", chunks={"time": 30})
encoding = {
    "temperature":   {"chunks": {"time": 30, "lat": 90, "lon": 180}, "dtype": "float32"},
    "precipitation": {"chunks": {"time": 30, "lat": 90, "lon": 180}, "dtype": "float32"},
}
ds.to_zarr("climate_output.zarr", encoding=encoding, consolidated=True)
# Validate
ds_zarr = xr.open_zarr("climate_output.zarr")
assert dict(ds.dims) == dict(ds_zarr.dims), "Dimension mismatch"
xr.testing.assert_allclose(ds, ds_zarr)
```

**Multiple NetCDFs → single store:**
```python
from pathlib import Path
ds = xr.open_mfdataset(sorted(Path("netcdf_archive/").glob("*.nc")),
                       combine="nested", concat_dim="time", chunks={"time": 30})
ds.to_zarr("combined_archive.zarr", encoding=encoding, consolidated=True)
```

See [patterns.md Pattern 5](references/patterns.md) for incremental migration with checkpointing for large archives.

## VirtualiZarr (No Data Copy)

```python
from virtualizarr import open_virtual_dataset
import xarray as xr
from pathlib import Path

# Single file
vds = open_virtual_dataset("large_data.nc")

# Multiple files
vds_list = [open_virtual_dataset(str(p)) for p in sorted(Path("archive/").glob("*.nc"))]
combined = xr.concat(vds_list, dim="time")

# Write virtual store (only references, no data copy)
combined.virtualize.to_zarr("virtual_archive.zarr")

# Read back normally
ds = xr.open_zarr("virtual_archive.zarr")
```

Note: reads resolve to the original source files — keep source files accessible.

## Kerchunk (Legacy)

```python
import json
from kerchunk.hdf import SingleHdf5ToZarr
from kerchunk.combine import MultiZarrToZarr
import fsspec

# Scan each file
url = "s3://bucket/file_001.nc"
with fsspec.open(url, mode="rb", anon=True) as f:
    refs = SingleHdf5ToZarr(f, url).translate()
with open("refs_001.json", "w") as f:
    json.dump(refs, f)

# Combine reference files
refs_list = [json.load(open(rf)) for rf in ["refs_001.json", "refs_002.json"]]
combined_refs = MultiZarrToZarr(refs_list, concat_dims=["time"]).translate()
with open("combined_refs.json", "w") as f:
    json.dump(combined_refs, f)

# Open virtual dataset
mapper = fsspec.get_mapper("reference://", fo="combined_refs.json")
ds = xr.open_zarr(mapper)
```

## Validation

Run after every migration before deleting the source:

```python
import xarray as xr
import numpy as np

def validate_migration(source_path, zarr_path, engine="netcdf4"):
    ds_src = xr.open_dataset(source_path, engine=engine)
    ds_dst = xr.open_zarr(zarr_path)
    errors = []
    if dict(ds_src.dims) != dict(ds_dst.dims):
        errors.append(f"Dims: {dict(ds_src.dims)} vs {dict(ds_dst.dims)}")
    missing = set(ds_src.data_vars) - set(ds_dst.data_vars)
    if missing:
        errors.append(f"Missing variables: {missing}")
    for var in set(ds_src.data_vars) & set(ds_dst.data_vars):
        src, dst = ds_src[var].values, ds_dst[var].values
        if not np.allclose(src.astype(dst.dtype), dst, equal_nan=True, rtol=1e-5):
            errors.append(f"{var}: max diff = {np.nanmax(np.abs(src.astype(dst.dtype) - dst))}")
    if errors:
        for e in errors: print(f"  ✗ {e}")
        return False
    print("Validation passed")
    return True
```

If validation fails: check encoding dtype mismatches → see [common-issues.md](references/common-issues.md).
Files in this package (5)
  • SKILL.md 6 KB
  • assets/hdf5-to-zarr-migration.py 5 KB
  • references/common-issues.md 13 KB
  • references/examples.md 11 KB
  • references/patterns.md 13 KB

Versions

RepositoryScoreLicence
This page · uw-ssec/rse-plugins95BSD-3-Clause
TerminalSkills/skills100Apache-2.0
muon-m2/magento2-tools100MIT
cbrock84/headcount90MIT
arjunprabhulal/devops-skills90MIT

Database snapshot: 13.11.2026 · package version 9a656ec71851 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100xarray-for-multidimensional-data
    Use when the user asks about Xarray, xarray datasets, NetCDF files, multidimensional labeled arrays, or scientific/climate data analysis in Python. Work with labeled multidimensional arrays for scientific data analysis using Xarray. Covers NetCDF/HDF5/Zarr I/O, Dask integration…
  • 95zarr-xarray-integration
    Integrates Zarr with xarray and Dask for labeled, multi-dimensional scientific data workflows. Covers reading and writing Zarr stores with xarray, append and region-write operations, multi-file virtual datasets, Dask chunk alignment with Zarr chunks, encoding configuration,…
  • 100netcdf-io
    Export a weather-skills Zarr to a NetCDF (.nc) file, keeping its attributes and processing history, or read one or more NetCDF files (NetCDF-3 or NetCDF-4, such as CDS, CHIRPS or model downloads) into a weather-skills Zarr. Use when someone wants data as .nc for another tool, or…
  • 100zarr-python
    Stores and queries chunked N-D scientific arrays with Zarr-Python 3, including codecs, sharding, S3/GCS storage, and NumPy/Dask/Xarray integration. Use for array layout, bounded I/O, format migration, or scientific metadata preservation.
  • 100xarray
    N-dimensional labeled arrays for geoscience data. Read/write NetCDF, work with climate and oceanographic datasets, perform multi-dimensional analysis with labeled coordinates. Use when the agent needs to: (1) Read/write NetCDF or Zarr files, (2) Work with multidimensional arrays…
  • 90rseng-scientific-file-formats
    Covers choosing and handling scientific data formats in code: HDF5 and NetCDF for array data, CF conventions and standard metadata, Parquet for tabular data, domain standards (NeXus and similar), self-describing files, chunking and compression choices, and migrating away from…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.