Skip to content
Skills Library
Data & analytics · Building & coding

download-script-dev

/download-script-dev

Use this skill when users ask to "develop a download script", "debug data download", "fix download error", "create data pipeline template", "download template", "GAIA data pipeline", "download from S3", "access Zarr store", "cloud data access", or mention sources like CONUS404, HRRR, WRF, PRISM, Stage IV, USGS, ORNL, DEM, Synoptic, or IRIS. Generates CONFIG-at-top Python download scripts, validates source-specific configuration (auth, endpoint/path, date range, variables, AOI/CRS, and output format), and diagnoses common failures (403/auth, timeout/retry, CRS mismatch, missing binaries, and pa

Repository
rse-plugins ↗
Licence
BSD-3-Clause
Last repository change
14.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
14
Scripts
0
HTML comments
0
URLs in SKILL.md
5
Deprecation

No signals detected. Repository changed on 14.10.2026, without references to retired models or APIs.

Licence

BSD-3-Clause, otwarta.

What it does and when to use it

Use this skill when users ask to "develop a download script", "debug data download", "fix download error", "create data pipeline template", "download template", "GAIA data pipeline", "download from S3", "access Zarr store", "cloud data access", or mention sources like CONUS404, HRRR, WRF, PRISM, Stage IV, USGS, ORNL, DEM, Synoptic, or IRIS. Generates CONFIG-at-top Python download scripts, validates source-specific configuration (auth, endpoint/path, date range, variables, AOI/CRS, and output format), and diagnoses common failures (403/auth, timeout/retry, CRS mismatch, missing binaries, and pa

Triggers from the description

develop a download scriptdebug data downloadfix download errorcreate data pipeline templatedownload templateGAIA data pipelinedownload from S3access Zarr store
Original author description

Use this skill when users ask to "develop a download script", "debug data download", "fix download error", "create data pipeline template", "download template", "GAIA data pipeline", "download from S3", "access Zarr store", "cloud data access", or mention sources like CONUS404, HRRR, WRF, PRISM, Stage IV, USGS, ORNL, DEM, Synoptic, or IRIS. Generates CONFIG-at-top Python download scripts, validates source-specific configuration (auth, endpoint/path, date range, variables, AOI/CRS, and output format), and diagnoses common failures (403/auth, timeout/retry, CRS mismatch, missing binaries, and pa

How it works

  1. Requirements
  2. Script Structure Pattern
  3. Four Data Access Patterns
  4. Spatial Subsetting
  5. Output Formats
  6. Common Issues and Debugging
  7. Reference Files
Original SKILL.md headings · 287 lines, 1,076 words
SKILL.md preview
---
name: download-script-dev
description: Use this skill when users ask to "develop a download script", "debug data download", "fix download error", "create data pipeline template", "download template", "GAIA data pipeline", "download from S3", "access Zarr store", "cloud data access", or mention sources like CONUS404, HRRR, WRF, PRISM, Stage IV, USGS, ORNL, DEM, Synoptic, or IRIS. Generates CONFIG-at-top Python download scripts, validates source-specific configuration (auth, endpoint/path, date range, variables, AOI/CRS, and output format), and diagnoses common failures (403/auth, timeout/retry, CRS mismatch, missing binaries, and partial downloads).
metadata:
  references:
    - references/DOWNLOAD_PATTERNS.md
    - references/CONFIGURATION.md
    - references/DATA_SOURCES.md
---

# Download Script Development Skill

## Requirements

- **Python 3.9+** with `xarray`, `geopandas`, `rioxarray`
- **Source-specific libraries:** `herbie-data` (HRRR), `pyPRISMClimate` (PRISM), `obspy` (IRIS), `boto3` (WRF/S3), `elevation` (DEM), `s3fs` (CONUS404)
- **System dependencies:** `wgrib2` for HRRR (`conda install -c conda-forge wgrib2`, not pip)

## Script Structure Pattern

All download scripts follow CONFIG-at-top separating parameters from logic. Complete templates for each source are in [`references/DOWNLOAD_PATTERNS.md`](references/DOWNLOAD_PATTERNS.md).

```python
import xarray as xr
import geopandas as gpd
import logging
from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path

# ============================================================
# Configuration — modify these parameters before running
# ============================================================
CONFIG = {
    "source": "SOURCE_NAME",
    "date_range": ("2024-01-01", "2024-01-31"),
    "variables": ["var1", "var2"],
    "aoi_path": "../data/GIS/boundary.json",
    "output_path": "../data/output.zarr",
    "output_format": "zarr",   # "zarr" | "netcdf" | "csv"
    "max_workers": 8,
}
# ============================================================
# Download logic — generally no need to modify below this line
# ============================================================

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger(__name__)


def validate_config(cfg):
    required = ["source", "date_range", "variables", "output_path", "output_format"]
    missing = [k for k in required if not cfg.get(k)]
    if missing:
        raise ValueError(f"Missing config keys: {missing}")


def validate_aoi(aoi):
    if aoi.empty:
        raise ValueError("AOI GeoDataFrame is empty — check aoi_path")
    if aoi.crs is None:
        raise ValueError("AOI CRS is None — set CRS before subsetting")


def download_with_retry(item, download_fn, retries=3, backoff=2):
    """Run download_fn(item) with exponential backoff on failure."""
    import time
    for attempt in range(retries):
        try:
            return download_fn(item)
        except Exception as e:
            if attempt == retries - 1:
                raise
            wait = backoff ** attempt
            log.warning(f"Attempt {attempt+1} failed for {item}: {e}. Retrying in {wait}s")
            time.sleep(wait)


def main():
    # Step 0: Validate configuration
    validate_config(CONFIG)
    log.info(f"Starting {CONFIG['source']} download: {CONFIG['date_range']}")

    # Step 1: Load and validate AOI
    aoi = gpd.read_file(CONFIG["aoi_path"])
    validate_aoi(aoi)
    log.info(f"AOI loaded: {len(aoi)} feature(s), CRS={aoi.crs}")

    # Step 2: Build item list and download with retry + failure capture
    items = []  # populate with dates, URLs, or identifiers for this source
    failed = []
    results = []
    with ThreadPoolExecutor(max_workers=CONFIG["max_workers"]) as executor:
        futures = {executor.submit(download_with_retry, item, _download_item): item
                   for item in items}
        for i, future in enumerate(as_completed(futures), 1):
            item = futures[future]
            try:
                results.append(future.result())
                log.info(f"[{i}/{len(items)}] Downloaded {item}")
            except Exception as e:
                log.error(f"FAILED {item}: {e}")
                failed.append((item, str(e)))

    # Step 3: Validate completeness — fail hard if any items are missing
    if failed:
        log.error(f"{len(failed)} items failed after retries: {[f[0] for f in failed]}")
        raise RuntimeError(
            f"Download incomplete: {len(failed)}/{len(items)} failed. "
            "Fix CONFIG or network issues, then re-run."
        )
    expected = len(items)
    assert len(results) == expected, f"Expected {expected} results, got {len(results)}"

    # Step 4: Combine results and validate dimensions
    ds = xr.open_mfdataset(results, combine="by_coords") if results else None
    assert ds is not None and ds.dims.get("time", 0) > 0, "Combined dataset has no time steps"

    # Step 5: Reproject AOI if needed, spatial subset, validate bounds
    if aoi.crs != ds.rio.crs:
        aoi = aoi.to_crs(ds.rio.crs)
    ds_sub = ds.rio.clip(aoi.geometry)
    assert not ds_sub.isnull().all().values.any(), "Spatial subset is entirely NaN — check AOI alignment"

    # Step 6: Derive variables and validate units/ranges
    # (source-specific; see references/DOWNLOAD_PATTERNS.md)

    # Step 7: Save output and verify artifact
    out = Path(CONFIG["output_path"])
    if CONFIG["output_format"] == "zarr":
        ds_sub.to_zarr(out, mode="w")
        xr.open_zarr(out)   # verify reopens cleanly
    elif CONFIG["output_format"] == "netcdf":
        ds_sub.to_netcdf(out)
        xr.open_dataset(out).close()
    log.info(f"Saved to {out}")

    # Step 8: QC summary
    log.info(f"QC | time={ds_sub.dims.get('time')} | "
             f"bounds={ds_sub.rio.bounds()} | failed={len(failed)}")


def _download_item(item):
    """Source-specific download — replace with actual implementation."""
    raise NotImplementedError("Replace with source-specific download logic")


if __name__ == "__main__":
    main()
```

**Error recovery:** If Step 3 validation fails, check the error log for failed items, fix `CONFIG` (wrong date range, missing credentials, bad URL), and re-run `main()`. The `results` list is rebuilt fresh each run — no partial-state issues.

## Four Data Access Patterns

### 1. Direct HTTP Download (PRISM, Stage IV, DEM)

```python
import requests
from pathlib import Path

def download_file(url: str, dest_dir: str, retries=3) -> Path:
    dest = Path(dest_dir) / Path(url).name
    if dest.exists():
        return dest  # resume support
    session = requests.Session()
    from requests.adapters import HTTPAdapter
    from urllib3.util.retry import Retry
    session.mount("https://", HTTPAdapter(max_retries=Retry(total=retries, backoff_factor=2)))
    r = session.get(url, timeout=60)
    r.raise_for_status()
    dest.write_bytes(r.content)
    return dest
```

### 2. REST API Query (USGS, Synoptic)

```python
import requests, os

def query_synoptic(station_ids: list[str], start: str, end: str) -> dict:
    token = os.environ["SYNOPTIC_API_TOKEN"]
    params = {
        "stid": ",".join(station_ids),
        "start": start.replace("-", "") + "0000",
        "end":   end.replace("-", "") + "2359",
        "token": token,
        "output": "json",
    }
    r = requests.get("https://api.synopticdata.com/v2/stations/timeseries",
                     params=params, timeout=60)
    r.raise_for_status()
    return r.json()
```

### 3. Cloud Object Storage / S3 (CONUS404, HRRR, WRF-CMIP6)

```python
import s3fs, xarray as xr

# CONUS404 — anonymous public bucket
fs = s3fs.S3FileSystem(anon=True, client_kwargs={"endpoint_url": "https://usgs.osn.mghpcc.org"})
store = s3fs.S3Map("s3://mdmf/gdp/CONUS404/hourly.zarr", s3=fs)
ds = xr.open_zarr(store, consolidated=True)

# WRF-CMIP6 — boto3 with unsigned config
import boto3
from botocore import UNSIGNED
from botocore.config import Config
s3 = boto3.client("s3", config=Config(signature_version=UNSIGNED))
```

### 4. Specialized Libraries (Herbie/HRRR, pyPRISMClimate, obspy/IRIS)

```python
# HRRR via Herbie
from herbie import Herbie
H = Herbie("2024-01-15 12:00", model="hrrr", product="sfc", fxx=0)
ds = H.xarray("TMP:2 m")  # 2-metre temperature

# IRIS via obspy
from obspy.clients.fdsn import Client
client = Client("IRIS")
inv = client.get_stations(network="UW", station="*", channel="BHZ",
                          starttime="2024-01-01", endtime="2024-02-01")
```

See [`references/DOWNLOAD_PATTERNS.md`](references/DOWNLOAD_PATTERNS.md) for complete pipeline implementations of each pattern.

## Spatial Subsetting

| Grid Type | Method | Sources |
|-----------|--------|---------|
| Regular (lat/lon) | `ds.rio.clip(aoi.geometry)` | PRISM, Stage IV, DEM |
| Curvilinear (model) | `regionmask` | CONUS404, WRF-CMIP6 |
| Irregular (points) | `shapely.contains()` | USGS station data |

Always match CRS before subsetting: `aoi = aoi.to_crs(ds.rio.crs)`.

## Output Formats

| Format | Use When |
|--------|----------|
| **Zarr** (preferred) | Large gridded datasets, cloud workflows |
| **NetCDF** | Sharing with traditional tools, small datasets |
| **CSV** | Tabular station data (USGS) |

## Common Issues and Debugging

### wgrib2 Not Found (HRRR)
```python
import shutil
assert shutil.which("wgrib2"), "Install: conda install -c conda-forge wgrib2"
```

### S3 Auth Errors (CONUS404, WRF)
```python
# CONUS404
fs = s3fs.S3FileSystem(anon=True, client_kwargs={"endpoint_url": "https://usgs.osn.mghpcc.org"})
# WRF
s3 = boto3.client("s3", config=Config(signature_version=UNSIGNED))
```

### CRS Mismatch
```python
aoi = aoi.to_crs(ds.rio.crs)
ds_clipped = ds.rio.clip(aoi.geometry)
# If ds has no CRS: ds.rio.write_crs("EPSG:4326", inplace=True)
```

### Network Timeouts
```python
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
session = requests.Session()
session.mount("https://", HTTPAdapter(max_retries=Retry(total=3, backoff_factor=2)))
```

### USGS RDB Parsing
```python
lines = [l for l in response.text.splitlines() if not l.startswith("#")]
df = pd.read_csv(io.StringIO("\n".join([lines[0]] + lines[2:])), sep="\t")
# Note: USGS returns local time — convert to UTC using station timezone metadata
```

### Memory Issues
```python
ds = xr.open_dataset(path, chunks={"time": 100})  # lazy
print(f"Dataset size: {ds.nbytes / 1e9:.1f} GB")   # check before .compute()
```

## Reference Files

- **`references/sources/`** — Per-source files (`hrrr.md`, `conus404.md`, etc.): endpoints, auth, response formats. Load only the relevant source.
- **`references/DOWNLOAD_PATTERNS.md`** — Complete pipeline code for HTTP, S3, REST API, and library patterns.
- **`references/CONFIGURATION.md`** — Per-source parameter tables with types, defaults, and validation rules.
Files in this package (14)
  • SKILL.md 11 KB
  • references/CONFIGURATION.md 8 KB
  • references/DATA_SOURCES.md 1 KB
  • references/DOWNLOAD_PATTERNS.md 15 KB
  • references/sources/conus404.md 1 KB
  • references/sources/dem-srtm.md 1 KB
  • references/sources/hrrr.md 1 KB
  • references/sources/iris.md 1 KB
  • references/sources/ornl-daymet.md 1 KB
  • references/sources/prism.md 1 KB
  • references/sources/stage-iv.md 1 KB
  • references/sources/synoptic.md 1 KB
  • references/sources/usgs.md 2 KB
  • references/sources/wrf-cmip6.md 1 KB

Versions

The database has no other skills named download-script-dev.

Database snapshot: 13.11.2026 · package version f8e599375656 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100usgs-data-download
    Download water level data from USGS using the dataretrieval package. Use when accessing real-time or historical streamflow data, downloading gage height or discharge measurements, or working with USGS station IDs.

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.