Skip to content
Skills Library
Science & research · Building & coding

earth2studio-create-datasource

/earth2studio-create-datasource

Create and validate Earth2Studio data source wrappers (DataSource, ForecastSource, DataFrameSource, ForecastFrameSource) from remote stores. Do NOT use for fetching data with existing sources, model inference, or installation tasks.

Repository
skills ↗
Licence
Apache-2.0
Last repository change
12.11.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
19
Scripts
11
HTML comments
1
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 12.11.2026, without references to retired models or APIs.

Licence

Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.

What to check before installing
  • 11 scripts in this package. Review them before installing, as Claude may execute them.
  • SKILL.md zawiera komentarze HTML (1). Nie widać ich w podglądzie Markdown, a model je czyta.

Scripts: evals/environment/setup/bootstrap.sh, evals/targets/eval_1_target.py, evals/targets/eval_2_target.py, evals/targets/eval_3_target.py, evals/targets/eval_4_target.py, evals/targets/test/eval_1_test_target.py, evals/targets/test/eval_2_test_target.py, evals/targets/test/eval_3_test_target.py, evals/targets/test/eval_4_test_target.py, references/implementation-guide.py, references/testing-guide.py

What it does and when to use it

Create and validate Earth2Studio data source wrappers (DataSource, ForecastSource, DataFrameSource, ForecastFrameSource) from remote stores. Do NOT use for fetching data with existing sources, model inference, or installation tasks.

Arguments

/earth2studio-create-datasource URL or description of remote data store (optional)
Original author description

Create and validate Earth2Studio data source wrappers (DataSource, ForecastSource, DataFrameSource, ForecastFrameSource) from remote stores. Do NOT use for fetching data with existing sources, model inference, or installation tasks.

How it works

  1. Purpose
  2. Prerequisites
  3. Workspace
  4. Instructions
  5. Examples
  6. Limitations
  7. Reminders
Original SKILL.md headings · 328 lines, 1,418 words
SKILL.md preview
---
name: earth2studio-create-datasource
version: 0.16.0
license: Apache-2.0
metadata:
  author: NVIDIA Earth-2 Team <agent-skills@nvidia.com>
  tags:
    - earth2studio
    - earth2
    - python
    - data-source
    - forecast-source
    - integration
description: >
  Create and validate Earth2Studio data source wrappers (DataSource,
  ForecastSource, DataFrameSource, ForecastFrameSource) from remote stores.
  Do NOT use for fetching data with existing sources, model inference, or
  installation tasks.
argument-hint: URL or description of remote data store (optional)
---

# Create and Validate Data Source

## Purpose

End-to-end workflow for implementing a new Earth2Studio data source wrapper
that connects a remote data store (S3, GCS, Azure, HTTP, HuggingFace) to
Earth2Studio's async data fetching infrastructure — from analysis through
implementation, testing, validation, and PR submission.

## Prerequisites

- Earth2Studio dev environment with `uv` (`uv run python` must work)
- Git configured with fork (`origin`) and upstream (`upstream`) remotes
- Access to the target remote data store (credentials if private)
- Python 3.10+

## Workspace

Use the directory containing `pyproject.toml`. For Harbor evals, write to
`/workspace/output/` preserving paths. Never read `evals/targets/`.

## Instructions

> **Python Environment:** Always use `uv run python` or the local `.venv`.
> Never use the system Python directly.

Follow every step in order.

> **[CONFIRM] gates:** Only Step 1 (Source Type) and Step 12 (Sanity-Check
> Plots) require explicit user approval. All other `[CONFIRM]` markers are
> advisory — present decisions inline and proceed without blocking.
>
> **Deliverables first:** Write the source file and test file (Steps 6–7)
> before extended exploration, documentation, registration, CHANGELOG, or PR
> work. Skip Steps 8–14 when the user asks for implementation only.
>
> **Before you finish:** Run verification commands in the repo root so results
> appear in the session log:
>
> ```bash
> uv run pytest test/data/test_<source>.py -x
> make format && make lint
> ```
>
> **Be concise:** Avoid long architecture reports; summarize decisions in a
> few sentences and move on to file writes.
>
> **Hangs or User Feedback** If agent becomes stuck or user provides a
> correction during this skills use, conservatively review relevant part of
> the skill and improve. Be concise.
>
> **One source type per invocation.** Invoke again for companion types.

### Reference Files

Load these on demand during the relevant steps:

| File | Content | Load at |
|---|---|---|
| `references/implementation-guide.py` | Skeleton source with FILL comments | Steps 3–10 |
| `references/testing-guide.py` | Test skeleton with FILL comments | Step 11 |
| `references/validation-guide.md` | Plot templates, PR body template, Greptile handling | Steps 12–14 (optional, for templates) |

---

### Workflow Overview

```text
Step 0: Obtain reference → Step 1: Determine type → Step 2: Dependencies
→ Step 3: Add deps → Step 4: Create lexicon → Step 5: Update vocab/schema
→ Step 6: Create skeleton → Step 7: Implement source → Step 8: Register
→ Step 9: Documentation → Step 10: CHANGELOG → Step 11: Tests
→ Step 12: Validate & plots (user confirms) → Step 13: PR + sanity comment
→ Step 14: Greptile review
```

---

### Step 0 — Obtain Remote Data Store Reference

If `$ARGUMENTS` is provided, use it (URL → WebFetch; file path → read).

If empty, ask:

> Please provide a URL, API documentation link, or description of the
> remote data store. This will be used to understand storage format,
> access pattern, variable inventory, temporal/spatial resolution.

---

### Step 1 — Determine Source Type

| Protocol | Returns | Has `lead_time`? | Use |
|---|---|---|---|
| **DataSource** | `xr.DataArray` | No | Gridded analysis/reanalysis |
| **ForecastSource** | `xr.DataArray` | Yes | Gridded forecast |
| **DataFrameSource** | `pd.DataFrame` | No | Sparse/station obs |
| **ForecastFrameSource** | `pd.DataFrame` | Yes | Sparse forecast obs |

Key factors: gridded vs sparse → DataArray vs DataFrame; analysis vs forecast → Source vs ForecastSource.

#### [CONFIRM — Source Type]

Present recommended type with justification. Ask for confirmation.

---

### Step 2 — Examine Remote Store & Propose Dependencies

**Analyze:** storage backend, file format, authentication, access pattern,
temporal/spatial resolution, variable inventory.

**Prefer fsspec:**

| Backend | Preferred | Avoid |
|---|---|---|
| AWS S3 | `s3fs` (core dep) | `boto3` directly |
| GCS | `gcsfs` (core dep) | `google-cloud-storage` |
| Azure | `adlfs` | `azure-storage-blob` |
| HTTP | `fsspec` (core dep) | `requests` |
| HuggingFace | `huggingface_hub` (core dep) | custom scripts |

Only fall back to dedicated libraries when fsspec cannot access the store.

Check `pyproject.toml` — only propose packages not already present.
Core deps include: `s3fs`, `gcsfs`, `fsspec`, `zarr`, `netCDF4`, `h5py`,
`pygrib`, `huggingface-hub`, `pandas`, `pyarrow`.

#### [CONFIRM — Dependencies & Access Pattern]

Present: backend, fsspec filesystem, new packages (with license), auth method.

---

### Step 3 — Add Dependencies

> **Load `references/implementation-guide.py` from here through Step 10.**

If new packages needed:

1. `uv add --extra data <package>`
2. `uv lock`
3. Add optional dependency imports using `OptionalDependencyFailure` pattern

---

### Step 4 — Create Lexicon Class

Create `earth2studio/lexicon/<source_name>.py` with:

- `metaclass=LexiconType`
- `VOCAB: dict[str, str]` mapping E2S names → remote keys
- `get_item(cls, val)` returning `tuple[str, Callable]`
- Use `::` separator for structured keys

Map remote variables against `E2STUDIO_VOCAB` (282 entries in
`earth2studio/lexicon/base.py`).

#### [CONFIRM — Lexicon & Variable Mapping]

Present: class name, key format, full mapping table, modifiers, reference URL.

---

### Step 5 — Update E2STUDIO_VOCAB / SCHEMA (if needed)

- New vocab: surface = descriptive abbrev; pressure = `{name}{level}`
- New schema fields: DataFrame sources only, check `E2STUDIO_SCHEMA`

#### [CONFIRM — Vocabulary & Schema Updates]

Skip if no updates needed.

---

### Step 6 — Create Skeleton Data Source File

Follow canonical method ordering:

1. Class constants
2. SCHEMA
3. `__init__`
4. `_async_init`
5. `__call__`
6. `fetch`
7. `_create_tasks`
8. `fetch_wrapper`
9. `fetch_array`
10. `_validate_time`
11. Helpers
12. `cache` property
13. `available` classmethod

Use async task dataclass pattern for parallel execution.

#### [CONFIRM — Skeleton]

Present: class name, file path, skeleton code, task dataclass.

---

### Step 7 — Implement the Data Source & Tests

> **The test file is a co-equal deliverable.** Create `test/data/test_<filename>.py`
> alongside the source. Add `test_<source>_call_mock` for async sources.

**Sync sources:** Use `prep_data_inputs`/`prep_forecast_inputs`, direct `__call__`.

**Async sources:** See `references/implementation-guide.py` for required patterns:
`_sync_async`, `managed_session`, `gather_with_concurrency`, `async_retry`,
pure async I/O, `try/finally` cleanup. Constructor params: `cache=True`,
`verbose=True`, `async_timeout=600`, `async_workers=16`, `retries=3`.
DataFrame sources add `time_tolerance`.

---

### Step 8 — Register the Source

- `earth2studio/data/__init__.py` — alphabetical import
- `earth2studio/lexicon/__init__.py` — alphabetical import
- Verify `pyproject.toml` deps

---

### Step 9 — Update Documentation

- Add to correct RST file (`datasources_analysis.rst` / `_forecast.rst` / `_dataframe.rst`)
- Class docstring: Parameters, Warning (download size), Note (reference URLs), Badges (last)
- All public methods: NumPy-style docstrings

---

### Step 10 — Update CHANGELOG.md

Add entry under the current unreleased version. See
`references/implementation-guide.py` REGISTRATION CHECKLIST for the format.

One line per source. Do NOT add separate lexicon entries.

---

### Step 11 — Verify Style & Expand Tests

Run `make format && make lint && make license`. Load `references/testing-guide.py`
for test skeletons. Required tests: `test_<source>_fetch` (slow), `_cache` (slow),
`_call_mock`, `_exceptions`, `_available`. Target 90%+ coverage with `--slow`.

#### [CONFIRM — Tests]

Present test file, functions, coverage.

---

### Step 12 — Validate Variables & Sanity-Check

1. Validate all lexicon vars against real data (run script, do NOT commit)
2. Remove variables with < 10% valid data
3. Create sanity-check plot (gridded or sparse template)
4. Tell user the plot path and ask for visual confirmation

#### [CONFIRM — Sanity-Check Plots]

User MUST visually inspect plots. Do not proceed without confirmation.

---

### Step 13 — Branch, Commit & Open PR

1. Create branch `feat/data-source-<name>`
2. Commit (do NOT add sanity-check script/images)
3. Push to fork
4. `gh pr create --repo NVIDIA/earth2studio`
5. **Immediately post sanity-check validation as PR comment** with:
   - Variable coverage table (name, count, range, unit)
   - Data validation summary (regions, storms/stations, time range)
   - Key findings (physically reasonable values, conversions verified)
   - Full validation script in `<details>` block
   - Image placeholder: `<!-- Drag and drop sanity-check image here -->`

#### [CONFIRM — Ready to Submit]

Verify all steps complete before creating PR.

---

### Step 14 — Automated Code Review

1. Poll for Greptile review (5 min timeout)
2. Categorize feedback (bug/style/perf/docs/suggestion/false-positive)
3. Present triage table to user
4. Implement accepted fixes
5. Respond to PR comments
6. Push

#### [CONFIRM — Review Triage]

User approves which comments to address.

---

## Examples

```text
User: Add a data source for the NOAA GFS analysis on S3
Agent: [loads skill, proceeds through Steps 0–14]
```

---

## Limitations

- One source type per invocation
- Requires network access for validation (Step 12)
- SPDX license headers required in all files

---

## Reminders

**DO:** `uv run python`, `loguru.logger`, alphabetical order in `__init__.py`/RST/CHANGELOG,
canonical method ordering, async utilities (`managed_session`, `gather_with_concurrency`,
`async_retry`), pure async I/O, reference URLs in docstrings, `try/finally` cleanup.

**AVOID:** `asyncio.to_thread`, bare `tqdm.gather`, xarray for loading, full file downloads.

**NEVER:** `loop.set_default_executor()`, commit secrets, commit sanity-check scripts/images.
Files in this package (19)
  • BENCHMARK.md 2 KB
  • SKILL.md 10 KB
  • skill-card.md 3 KB
  • skill.oms.sig 8 KB
  • evals/config.yml 1 KB
  • evals/evals.json 6 KB
  • evals/environment/Dockerfile 1 KB
  • evals/environment/setup/bootstrap.sh 1 KB
  • evals/targets/eval_1_target.py 2 KB
  • evals/targets/eval_2_target.py 3 KB
  • evals/targets/eval_3_target.py 4 KB
  • evals/targets/eval_4_target.py 12 KB
  • evals/targets/test/eval_1_test_target.py 2 KB
  • evals/targets/test/eval_2_test_target.py 3 KB
  • evals/targets/test/eval_3_test_target.py 4 KB
  • evals/targets/test/eval_4_test_target.py 4 KB
  • references/implementation-guide.py 17 KB
  • references/testing-guide.py 10 KB
  • references/validation-guide.md 10 KB

Versions

RepositoryScoreLicence
This page · nvidia/skills100Apache-2.0
nvidia/earth2studio100Apache-2.0

Database snapshot: 13.11.2026 · package version 0057d837be1b (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100earth2studio-create-prognostic
    Create Earth2Studio prognostic (time-stepping forecast) model wrappers. Do NOT use for diagnostic models, data sources, or installation.
  • 100earth2studio-create-diagnostic
    Create Earth2Studio diagnostic model wrappers for single-step data transformations, including simple derived diagnostics, packaged AutoModel diagnostics, and generative or diffusion diagnostics. Do NOT use for prognostic time-stepping models, data sources, or installation.
  • 100devextreme-datasource
    Help developers configure the DevExtreme data layer across Angular, React, Vue, jQuery, and ASP.NET Core Controls: DataSource, ArrayStore, LocalStore, ODataStore, and CustomStore. Use when someone asks about binding components to data, loading data from a REST API, configuring…
  • 90idmp-datasource
    IDMP datasource skill. Use it to list and inspect connections, verify connectivity before deeper metadata reads, and diagnose database or table visibility issues instead of stopping at the connection list.
  • 100add-datasource
    Use when adding an unspecified data source to an Expo/React Native Power Apps mobile app; routes to Dataverse, SharePoint, or another connector.
  • 100earth2studio-create-prognostic
    Create Earth2Studio prognostic (time-stepping forecast) model wrappers. Do NOT use for diagnostic models, data sources, or installation. The user may optionally supply a URL or a local path to a reference inference script.

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.