Przejdź do treści
Biblioteka skilli
Dane i analityka · Budowanie i kod

download-script-dev

/download-script-dev

Tworzy i debuguje skrypty Pythona do pobierania danych z S3, Zarr i źródeł takich jak CONUS404, HRRR, PRISM czy USGS, z konfiguracją na górze pliku.

Repozytorium
rse-plugins ↗
Licencja
BSD-3-Clause
Ostatnia zmiana repozytorium
14.10.2026

Ocena i bezpieczeństwo

100/ 100Klasa: dobry
Format25 / 25
Frontmatter z nazwą i opisem, poprawna nazwa, opis do 1024 znaków, nazwa zgodna z folderem.
Odkrywalność25 / 25
Opis mówi, kiedy użyć skilla, ma sensowną długość i jest w 3. osobie.
Treść35 / 35
Co najmniej 150 słów i do 500 linii, nagłówki, listy kroków, przykłady albo kod.
Higiena15 / 15
Bez zaślepek typu TODO, bez ścieżek Windows, z licencją.
Ryzykobrak sygnałów w skanie
Pliki w paczce
14
Skrypty
0
Komentarze HTML
0
Adresy URL w SKILL.md
5
Przestarzałość

Brak sygnałów. Repozytorium zmienione 14.10.2026, bez odwołań do wycofanych modeli i API.

Licencja

BSD-3-Clause, otwarta.

Co robi i kiedy

Tworzy i debuguje skrypty Pythona do pobierania danych z S3, Zarr i źródeł takich jak CONUS404, HRRR, PRISM czy USGS, z konfiguracją na górze pliku.

Wyzwalacze z opisu

develop a download scriptdebug data downloadfix download errorcreate data pipeline templatedownload templateGAIA data pipelinedownload from S3access Zarr store
Oryginalny opis (EN)

Use this skill when users ask to "develop a download script", "debug data download", "fix download error", "create data pipeline template", "download template", "GAIA data pipeline", "download from S3", "access Zarr store", "cloud data access", or mention sources like CONUS404, HRRR, WRF, PRISM, Stage IV, USGS, ORNL, DEM, Synoptic, or IRIS. Generates CONFIG-at-top Python download scripts, validates source-specific configuration (auth, endpoint/path, date range, variables, AOI/CRS, and output format), and diagnoses common failures (403/auth, timeout/retry, CRS mismatch, missing binaries, and pa

Jak działa

  1. Requirements
  2. Script Structure Pattern
  3. Four Data Access Patterns
  4. Spatial Subsetting
  5. Output Formats
  6. Common Issues and Debugging
  7. Reference Files
Sekcje z nagłówków SKILL.md, w oryginale · 287 linii, 1 076 słów
Podgląd SKILL.md
---
name: download-script-dev
description: Use this skill when users ask to "develop a download script", "debug data download", "fix download error", "create data pipeline template", "download template", "GAIA data pipeline", "download from S3", "access Zarr store", "cloud data access", or mention sources like CONUS404, HRRR, WRF, PRISM, Stage IV, USGS, ORNL, DEM, Synoptic, or IRIS. Generates CONFIG-at-top Python download scripts, validates source-specific configuration (auth, endpoint/path, date range, variables, AOI/CRS, and output format), and diagnoses common failures (403/auth, timeout/retry, CRS mismatch, missing binaries, and partial downloads).
metadata:
  references:
    - references/DOWNLOAD_PATTERNS.md
    - references/CONFIGURATION.md
    - references/DATA_SOURCES.md
---

# Download Script Development Skill

## Requirements

- **Python 3.9+** with `xarray`, `geopandas`, `rioxarray`
- **Source-specific libraries:** `herbie-data` (HRRR), `pyPRISMClimate` (PRISM), `obspy` (IRIS), `boto3` (WRF/S3), `elevation` (DEM), `s3fs` (CONUS404)
- **System dependencies:** `wgrib2` for HRRR (`conda install -c conda-forge wgrib2`, not pip)

## Script Structure Pattern

All download scripts follow CONFIG-at-top separating parameters from logic. Complete templates for each source are in [`references/DOWNLOAD_PATTERNS.md`](references/DOWNLOAD_PATTERNS.md).

```python
import xarray as xr
import geopandas as gpd
import logging
from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path

# ============================================================
# Configuration — modify these parameters before running
# ============================================================
CONFIG = {
    "source": "SOURCE_NAME",
    "date_range": ("2024-01-01", "2024-01-31"),
    "variables": ["var1", "var2"],
    "aoi_path": "../data/GIS/boundary.json",
    "output_path": "../data/output.zarr",
    "output_format": "zarr",   # "zarr" | "netcdf" | "csv"
    "max_workers": 8,
}
# ============================================================
# Download logic — generally no need to modify below this line
# ============================================================

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger(__name__)


def validate_config(cfg):
    required = ["source", "date_range", "variables", "output_path", "output_format"]
    missing = [k for k in required if not cfg.get(k)]
    if missing:
        raise ValueError(f"Missing config keys: {missing}")


def validate_aoi(aoi):
    if aoi.empty:
        raise ValueError("AOI GeoDataFrame is empty — check aoi_path")
    if aoi.crs is None:
        raise ValueError("AOI CRS is None — set CRS before subsetting")


def download_with_retry(item, download_fn, retries=3, backoff=2):
    """Run download_fn(item) with exponential backoff on failure."""
    import time
    for attempt in range(retries):
        try:
            return download_fn(item)
        except Exception as e:
            if attempt == retries - 1:
                raise
            wait = backoff ** attempt
            log.warning(f"Attempt {attempt+1} failed for {item}: {e}. Retrying in {wait}s")
            time.sleep(wait)


def main():
    # Step 0: Validate configuration
    validate_config(CONFIG)
    log.info(f"Starting {CONFIG['source']} download: {CONFIG['date_range']}")

    # Step 1: Load and validate AOI
    aoi = gpd.read_file(CONFIG["aoi_path"])
    validate_aoi(aoi)
    log.info(f"AOI loaded: {len(aoi)} feature(s), CRS={aoi.crs}")

    # Step 2: Build item list and download with retry + failure capture
    items = []  # populate with dates, URLs, or identifiers for this source
    failed = []
    results = []
    with ThreadPoolExecutor(max_workers=CONFIG["max_workers"]) as executor:
        futures = {executor.submit(download_with_retry, item, _download_item): item
                   for item in items}
        for i, future in enumerate(as_completed(futures), 1):
            item = futures[future]
            try:
                results.append(future.result())
                log.info(f"[{i}/{len(items)}] Downloaded {item}")
            except Exception as e:
                log.error(f"FAILED {item}: {e}")
                failed.append((item, str(e)))

    # Step 3: Validate completeness — fail hard if any items are missing
    if failed:
        log.error(f"{len(failed)} items failed after retries: {[f[0] for f in failed]}")
        raise RuntimeError(
            f"Download incomplete: {len(failed)}/{len(items)} failed. "
            "Fix CONFIG or network issues, then re-run."
        )
    expected = len(items)
    assert len(results) == expected, f"Expected {expected} results, got {len(results)}"

    # Step 4: Combine results and validate dimensions
    ds = xr.open_mfdataset(results, combine="by_coords") if results else None
    assert ds is not None and ds.dims.get("time", 0) > 0, "Combined dataset has no time steps"

    # Step 5: Reproject AOI if needed, spatial subset, validate bounds
    if aoi.crs != ds.rio.crs:
        aoi = aoi.to_crs(ds.rio.crs)
    ds_sub = ds.rio.clip(aoi.geometry)
    assert not ds_sub.isnull().all().values.any(), "Spatial subset is entirely NaN — check AOI alignment"

    # Step 6: Derive variables and validate units/ranges
    # (source-specific; see references/DOWNLOAD_PATTERNS.md)

    # Step 7: Save output and verify artifact
    out = Path(CONFIG["output_path"])
    if CONFIG["output_format"] == "zarr":
        ds_sub.to_zarr(out, mode="w")
        xr.open_zarr(out)   # verify reopens cleanly
    elif CONFIG["output_format"] == "netcdf":
        ds_sub.to_netcdf(out)
        xr.open_dataset(out).close()
    log.info(f"Saved to {out}")

    # Step 8: QC summary
    log.info(f"QC | time={ds_sub.dims.get('time')} | "
             f"bounds={ds_sub.rio.bounds()} | failed={len(failed)}")


def _download_item(item):
    """Source-specific download — replace with actual implementation."""
    raise NotImplementedError("Replace with source-specific download logic")


if __name__ == "__main__":
    main()
```

**Error recovery:** If Step 3 validation fails, check the error log for failed items, fix `CONFIG` (wrong date range, missing credentials, bad URL), and re-run `main()`. The `results` list is rebuilt fresh each run — no partial-state issues.

## Four Data Access Patterns

### 1. Direct HTTP Download (PRISM, Stage IV, DEM)

```python
import requests
from pathlib import Path

def download_file(url: str, dest_dir: str, retries=3) -> Path:
    dest = Path(dest_dir) / Path(url).name
    if dest.exists():
        return dest  # resume support
    session = requests.Session()
    from requests.adapters import HTTPAdapter
    from urllib3.util.retry import Retry
    session.mount("https://", HTTPAdapter(max_retries=Retry(total=retries, backoff_factor=2)))
    r = session.get(url, timeout=60)
    r.raise_for_status()
    dest.write_bytes(r.content)
    return dest
```

### 2. REST API Query (USGS, Synoptic)

```python
import requests, os

def query_synoptic(station_ids: list[str], start: str, end: str) -> dict:
    token = os.environ["SYNOPTIC_API_TOKEN"]
    params = {
        "stid": ",".join(station_ids),
        "start": start.replace("-", "") + "0000",
        "end":   end.replace("-", "") + "2359",
        "token": token,
        "output": "json",
    }
    r = requests.get("https://api.synopticdata.com/v2/stations/timeseries",
                     params=params, timeout=60)
    r.raise_for_status()
    return r.json()
```

### 3. Cloud Object Storage / S3 (CONUS404, HRRR, WRF-CMIP6)

```python
import s3fs, xarray as xr

# CONUS404 — anonymous public bucket
fs = s3fs.S3FileSystem(anon=True, client_kwargs={"endpoint_url": "https://usgs.osn.mghpcc.org"})
store = s3fs.S3Map("s3://mdmf/gdp/CONUS404/hourly.zarr", s3=fs)
ds = xr.open_zarr(store, consolidated=True)

# WRF-CMIP6 — boto3 with unsigned config
import boto3
from botocore import UNSIGNED
from botocore.config import Config
s3 = boto3.client("s3", config=Config(signature_version=UNSIGNED))
```

### 4. Specialized Libraries (Herbie/HRRR, pyPRISMClimate, obspy/IRIS)

```python
# HRRR via Herbie
from herbie import Herbie
H = Herbie("2024-01-15 12:00", model="hrrr", product="sfc", fxx=0)
ds = H.xarray("TMP:2 m")  # 2-metre temperature

# IRIS via obspy
from obspy.clients.fdsn import Client
client = Client("IRIS")
inv = client.get_stations(network="UW", station="*", channel="BHZ",
                          starttime="2024-01-01", endtime="2024-02-01")
```

See [`references/DOWNLOAD_PATTERNS.md`](references/DOWNLOAD_PATTERNS.md) for complete pipeline implementations of each pattern.

## Spatial Subsetting

| Grid Type | Method | Sources |
|-----------|--------|---------|
| Regular (lat/lon) | `ds.rio.clip(aoi.geometry)` | PRISM, Stage IV, DEM |
| Curvilinear (model) | `regionmask` | CONUS404, WRF-CMIP6 |
| Irregular (points) | `shapely.contains()` | USGS station data |

Always match CRS before subsetting: `aoi = aoi.to_crs(ds.rio.crs)`.

## Output Formats

| Format | Use When |
|--------|----------|
| **Zarr** (preferred) | Large gridded datasets, cloud workflows |
| **NetCDF** | Sharing with traditional tools, small datasets |
| **CSV** | Tabular station data (USGS) |

## Common Issues and Debugging

### wgrib2 Not Found (HRRR)
```python
import shutil
assert shutil.which("wgrib2"), "Install: conda install -c conda-forge wgrib2"
```

### S3 Auth Errors (CONUS404, WRF)
```python
# CONUS404
fs = s3fs.S3FileSystem(anon=True, client_kwargs={"endpoint_url": "https://usgs.osn.mghpcc.org"})
# WRF
s3 = boto3.client("s3", config=Config(signature_version=UNSIGNED))
```

### CRS Mismatch
```python
aoi = aoi.to_crs(ds.rio.crs)
ds_clipped = ds.rio.clip(aoi.geometry)
# If ds has no CRS: ds.rio.write_crs("EPSG:4326", inplace=True)
```

### Network Timeouts
```python
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
session = requests.Session()
session.mount("https://", HTTPAdapter(max_retries=Retry(total=3, backoff_factor=2)))
```

### USGS RDB Parsing
```python
lines = [l for l in response.text.splitlines() if not l.startswith("#")]
df = pd.read_csv(io.StringIO("\n".join([lines[0]] + lines[2:])), sep="\t")
# Note: USGS returns local time — convert to UTC using station timezone metadata
```

### Memory Issues
```python
ds = xr.open_dataset(path, chunks={"time": 100})  # lazy
print(f"Dataset size: {ds.nbytes / 1e9:.1f} GB")   # check before .compute()
```

## Reference Files

- **`references/sources/`** — Per-source files (`hrrr.md`, `conus404.md`, etc.): endpoints, auth, response formats. Load only the relevant source.
- **`references/DOWNLOAD_PATTERNS.md`** — Complete pipeline code for HTTP, S3, REST API, and library patterns.
- **`references/CONFIGURATION.md`** — Per-source parameter tables with types, defaults, and validation rules.
Pliki w paczce (14)
  • SKILL.md 11 KB
  • references/CONFIGURATION.md 8 KB
  • references/DATA_SOURCES.md 1 KB
  • references/DOWNLOAD_PATTERNS.md 15 KB
  • references/sources/conus404.md 1 KB
  • references/sources/dem-srtm.md 1 KB
  • references/sources/hrrr.md 1 KB
  • references/sources/iris.md 1 KB
  • references/sources/ornl-daymet.md 1 KB
  • references/sources/prism.md 1 KB
  • references/sources/stage-iv.md 1 KB
  • references/sources/synoptic.md 1 KB
  • references/sources/usgs.md 2 KB
  • references/sources/wrf-cmip6.md 1 KB

Wersje

W bazie nie ma innych skilli o nazwie download-script-dev.

Migawka bazy z 13.11.2026 · wersja paczki f8e599375656 (skrót SHA-256 całej paczki)

Pobrania w ostatnich 30 dniach: 0 · ranking

Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.

Podobne skille

  • 100usgs-data-download
    Pobiera dane o poziomie wód z USGS pakietem dataretrieval: bieżące i historyczne dane o przepływie, wysokość wody i natężenie przepływu według identyfikatorów stacji.

Uwagi i zgłoszenia

Wiadomość trafi do Bartosza Sokolińskiego. Adres e-mail jest potrzebny tylko do odpowiedzi. Szczegóły w polityce prywatności.