datahub-api
/datahub-api
Reference and coding guide for DataHub integration in backend development. Use when implementing or debugging code that touches DataHub — entities (datasets, dashboards, charts), aspects (ownership, tags, domains, glossary terms, structured properties), lineage, assertions, URNs, ingestion/emission, GraphQL, REST, or the acryl-datahub Python SDK. Also answers data-modeling questions and helps choose native DataHub features vs custom extensions. Trigger this skill whenever a task involves DataHub metadata, lineage, ingestion, or any DataHub API surface, even without explicit mention of "API".
- Author
- selhorys ↗
- Repository
- dataspoke-baseline ↗
- Licence
- Apache-2.0
- Last repository change
- 05.10.2026
Assessment and security
- Files in this package
- 2
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 7
No signals detected. Repository changed on 05.10.2026, without references to retired models or APIs.
Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.
What it does and when to use it
Reference and coding guide for DataHub integration in backend development. Use when implementing or debugging code that touches DataHub — entities (datasets, dashboards, charts), aspects (ownership, tags, domains, glossary terms, structured properties), lineage, assertions, URNs, ingestion/emission, GraphQL, REST, or the acryl-datahub Python SDK. Also answers data-modeling questions and helps choose native DataHub features vs custom extensions. Trigger this skill whenever a task involves DataHub metadata, lineage, ingestion, or any DataHub API surface, even without explicit mention of "API".
Triggers from the description
Arguments
/datahub-api [task]
Original author description
Reference and coding guide for DataHub integration in backend development. Use when implementing or debugging code that touches DataHub — entities (datasets, dashboards, charts), aspects (ownership, tags, domains, glossary terms, structured properties), lineage, assertions, URNs, ingestion/emission, GraphQL, REST, or the acryl-datahub Python SDK. Also answers data-modeling questions and helps choose native DataHub features vs custom extensions. Trigger this skill whenever a task involves DataHub metadata, lineage, ingestion, or any DataHub API surface, even without explicit mention of "API".
How it works
- Phase 1 — Understand the Task & Select Mode
- Phase 2 — Reference Navigation
- Phase 3 — Check Prerequisites
- Phase 4 — Explore Live API Documentation (optional)
- Phase 5 — Write Code, Execute, Verify
- Constraints
SKILL.md preview
---
name: datahub-api
description: Reference and coding guide for DataHub integration in backend development. Use when implementing or debugging code that touches DataHub — entities (datasets, dashboards, charts), aspects (ownership, tags, domains, glossary terms, structured properties), lineage, assertions, URNs, ingestion/emission, GraphQL, REST, or the acryl-datahub Python SDK. Also answers data-modeling questions and helps choose native DataHub features vs custom extensions. Trigger this skill whenever a task involves DataHub metadata, lineage, ingestion, or any DataHub API surface, even without explicit mention of "API".
argument-hint: "[task]"
allowed-tools: Read, Grep, Glob, Bash(python3 *), Bash(pip3 *), Bash(curl *)
---
## Phase 1 — Understand the Task & Select Mode
**Step 0 — Check reference materials:**
```bash
[ -d "ref/github/datahub" ] && echo "ref present" || echo "ref MISSING"
```
If `ref/github/datahub` is **missing**, stop immediately and tell the user:
> `ref/github/datahub` is not present. Run `/ref-setup` to download
> the DataHub v1.6.0 source (shallow clone, ~1-3 min). Retry this task after it completes.
**Step 1 — Select operating mode:**
| If the task looks like... | Mode |
|---|---|
| "Does DataHub have X?", "What's the best way to model Y?", "Should I use Z or build custom?" | **Q&A** — research and answer, no execution |
| "Write a Python module to do X", "Build a utility for Y", "Test whether Z works" | **Code Writer** — write, execute, verify |
| Mixed ("explain and write the code") | Q&A first, then Code Writer |
**Step 2 — Identify the API layer** (both modes):
| Task | API to use |
|---|---|
| Search, entity reads, lineage queries (UI-facing) | GraphQL |
| Emit lineage, custom aspects, bulk ingestion | Python SDK (`DatahubRestEmitter`) |
| Token management, soft delete, batch operations | GraphQL mutation |
| Advanced admin, index management | OpenAPI REST |
**Step 3 — Identify entity type** (both modes): dataset, dashboard, chart, tag, lineage, assertion, structured property, etc.
After Phase 1:
- **Q&A** routes to Phase 2 → answer → done.
- **Code Writer** continues through Phases 2–5.
---
## Phase 2 — Reference Navigation
The full DataHub source (v1.6.0) lives in `ref/github/datahub/`. Use the reference lookup table in [reference.md](reference.md) §Reference Lookup Table to find the right files.
**For Q&A mode**: read the relevant reference files, then write a clear answer with:
- Exact field names, aspect names, URN formats
- Source file citations from `ref/`
- Recommendation and rationale
- If no native solution exists: propose the minimal custom approach (Structured Properties before Glossary before fully custom aspects)
Follow the **Decision Protocol** in [reference.md](reference.md) §Decision Protocol to determine whether DataHub supports the concept natively.
**For Code Writer mode**: continue to Phase 3.
---
## Phase 3 — Check Prerequisites
Run these checks before executing any code:
```bash
# 1. Check acryl-datahub is installed
python3 -c "import datahub; print('acryl-datahub', datahub.__version__)" 2>/dev/null \
|| pip3 install acryl-datahub --quiet
# 2. Read the GMS URL from .env.dev (GMS has its own ingress host)
source helm-charts/.env.dev 2>/dev/null || true
DATAHUB_GMS_URL="${DATASPOKE_DEV_DATAHUB_GMS_URL:-http://datahub-gms.${DATASPOKE_KUBE_INGRESS_DOMAIN}}"
# 3. Check GMS is reachable via ingress
curl -s "${DATAHUB_GMS_URL}/config" \
| python3 -c "import sys,json; d=json.load(sys.stdin); print('GMS ok, version:', d['versions']['acryldata/datahub']['version'])" \
|| echo "ERROR: GMS not reachable at ${DATAHUB_GMS_URL}. Run: ./helm-charts/bin/health-check.sh"
# 4. Check token
if [ -z "${DATASPOKE_DEV_DATAHUB_TOKEN:-}" ]; then
echo "DATASPOKE_DEV_DATAHUB_TOKEN not set — generate via: http://datahub.${DATASPOKE_KUBE_INGRESS_DOMAIN} → Settings → Access Tokens"
fi
```
If any prerequisite fails, stop and inform the user with the fix instructions.
---
## Phase 4 — Explore Live API Documentation (optional)
Only use this if the static `ref/` files don't answer the question.
Derive the base URLs from `helm-charts/.env.dev`: `source helm-charts/.env.dev`, then:
- GMS base: `${DATASPOKE_DEV_DATAHUB_GMS_URL}` (i.e. `http://datahub-gms.${DATASPOKE_KUBE_INGRESS_DOMAIN}`)
- DataHub UI: `http://datahub.${DATASPOKE_KUBE_INGRESS_DOMAIN}`
| Resource | URL pattern | Notes |
|---|---|---|
| Swagger UI (REST/OpenAPI) | `<GMS_BASE>/openapi/swagger-ui/index.html` | Set Bearer token in Authorize dialog |
| Raw OpenAPI spec | `curl -s -H "Authorization: Bearer $DATASPOKE_DEV_DATAHUB_TOKEN" <GMS_BASE>/openapi/v3/api-docs` | JSON, pipe to `python3 -m json.tool` |
| GraphiQL | `http://datahub.<DOMAIN>/api/graphiql` | Browser only, uses session cookie |
| Unauthenticated health | `<GMS_BASE>/config`, `<GMS_BASE>/health` | No token needed |
---
## Phase 5 — Write Code, Execute, Verify
### 5.1 SDK Setup Pattern
Read the GMS URL from the environment (set by `source helm-charts/.env.dev`):
```python
from datahub.ingestion.graph.client import DataHubGraph, DatahubClientConfig
import os
# GMS is accessible via its own ingress host: http://datahub-gms.<INGRESS_DOMAIN>
gms_url = os.environ["DATASPOKE_DEV_DATAHUB_GMS_URL"] # set in helm-charts/.env
graph = DataHubGraph(DatahubClientConfig(
server=gms_url,
token=os.environ.get("DATASPOKE_DEV_DATAHUB_TOKEN", ""),
))
```
For bulk emission:
```python
from datahub.emitter.rest_emitter import DatahubRestEmitter
import os
emitter = DatahubRestEmitter(
os.environ["DATASPOKE_DEV_DATAHUB_GMS_URL"],
token=os.environ.get("DATASPOKE_DEV_DATAHUB_TOKEN", ""),
)
```
### 5.2 Execution Loop
For each operation:
1. **Read the relevant tutorial** from `ref/github/datahub/docs/api/tutorials/`
2. **Find the matching SDK example** in `ref/github/datahub/metadata-ingestion/examples/library/`
3. **Write the code** to a temp file
4. **Execute**: `python3 /tmp/datahub_test_script.py`
5. **Verify** by reading back the written entity:
```python
# Read back to confirm the write succeeded
aspect = graph.get_aspect(entity_urn=urn, aspect_type=AspectClass)
assert aspect is not None, "Entity not found after write"
print("Verified:", aspect)
```
6. **Iterate** on failure — diagnose error, fix, re-run. Stop after 3 consecutive failures and report the blocker.
7. **Report**: final code + what was written + verification output
### 5.3 Common URN Builders
```python
from datahub.emitter.mce_builder import (
make_dataset_urn, # urn:li:dataset:(urn:li:dataPlatform:<p>,<name>,<env>)
make_data_platform_urn, # urn:li:dataPlatform:<name>
make_schema_field_urn, # urn:li:schemaField:(<dataset_urn>,<field>)
make_tag_urn, # urn:li:tag:<name>
make_user_urn, # urn:li:corpuser:<username>
make_group_urn, # urn:li:corpGroup:<name>
make_domain_urn, # urn:li:domain:<uuid>
)
```
### 5.4 Emitter Behavior
- **Retry**: 3 attempts on `[429, 500, 502, 503, 504]` with exponential backoff
- **Payload limit**: ~15 MB per request; split large batches
- **Rate limiting**: exponential back-off on 429 with 2x multiplier
---
## Constraints
1. **Never use `kubectl exec`** to interact with DataHub — it bypasses the API surface and doesn't reflect production behavior.
2. **Never run ad-hoc `kubectl port-forward`** — DataHub GMS is accessible via ingress at `http://datahub-gms.<INGRESS_DOMAIN>` (`$DATASPOKE_DEV_DATAHUB_GMS_URL`). If the ingress is unreachable, run `./helm-charts/bin/health-check.sh` to diagnose.
3. **Always read the matching tutorial/example first** before writing API code.
4. **Prefer static `ref/` lookup over live API exploration** for speed — only fall back to Swagger/GraphiQL when the static ref is ambiguous.
---
See [reference.md](reference.md) for the reference lookup table, decision protocol, and known patterns.
Files in this package (2)
- SKILL.md 8 KB
- reference.md 17 KB
Versions
The database has no other skills named datahub-api.
Database snapshot: 13.11.2026 · package version 89339c586f2f (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100datahub-search
Use this skill when the user wants to search the DataHub catalog, discover entities, answer ad-hoc questions about their data, find datasets, or browse by platform or domain. Triggers on: "search DataHub", "find datasets", "who owns X", "what tables contain PII", "what columns… - 100datahub-connector-planning
Plans new DataHub connectors by classifying the source system, researching it using a dedicated agent or inline research, and generating a _PLANNING.md blueprint with entity mapping and architecture decisions. Use when building a new connector, researching a source system for… - 90data-data-catalog
Use this skill when asked about data catalog, DataHub, Amundsen, Apache Atlas, OpenMetadata, metadata management, data discovery, business glossary, data lineage, data ownership, or column-level lineage. This skill enforces: metadata ingestion pipelines, column-level lineage… - 90openmetadata-datahub-and-openlineage
Guides agents through metadata platform and lineage workflows using OpenMetadata, DataHub, or OpenLineage-compatible systems. Use when improving discovery, lineage quality, metadata governance, or producer-to-catalog integration. - 100test-manual-api-wired
Walk a single api-wired UC scenario manually: read the source test file, print each REST request, ask for approval, fire it, print the response, and probe side effects (DB rows, DataHub aspects, k8s secrets). Optional argument `scope` selects the case (free-form: "UC1", "UC1… - 100data-catalog-and-lineage-governance-datahub-amundsen
Guides deploying and operating enterprise data catalog and governance tooling — DataHub or Amundsen — for discovering, tagging, classifying, and owning datasets, features, and models across an organization, including deprecation workflows that must check downstream consumers…