Skip to content
Skills Library
Programming · Building & coding

datahub-api

/datahub-api

Reference and coding guide for DataHub integration in backend development. Use when implementing or debugging code that touches DataHub — entities (datasets, dashboards, charts), aspects (ownership, tags, domains, glossary terms, structured properties), lineage, assertions, URNs, ingestion/emission, GraphQL, REST, or the acryl-datahub Python SDK. Also answers data-modeling questions and helps choose native DataHub features vs custom extensions. Trigger this skill whenever a task involves DataHub metadata, lineage, ingestion, or any DataHub API surface, even without explicit mention of "API".

Licence
Apache-2.0
Last repository change
05.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
2
Scripts
0
HTML comments
0
URLs in SKILL.md
7
Deprecation

No signals detected. Repository changed on 05.10.2026, without references to retired models or APIs.

Licence

Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.

What it does and when to use it

Reference and coding guide for DataHub integration in backend development. Use when implementing or debugging code that touches DataHub — entities (datasets, dashboards, charts), aspects (ownership, tags, domains, glossary terms, structured properties), lineage, assertions, URNs, ingestion/emission, GraphQL, REST, or the acryl-datahub Python SDK. Also answers data-modeling questions and helps choose native DataHub features vs custom extensions. Trigger this skill whenever a task involves DataHub metadata, lineage, ingestion, or any DataHub API surface, even without explicit mention of "API".

Triggers from the description

API

Arguments

/datahub-api [task]
Original author description

Reference and coding guide for DataHub integration in backend development. Use when implementing or debugging code that touches DataHub — entities (datasets, dashboards, charts), aspects (ownership, tags, domains, glossary terms, structured properties), lineage, assertions, URNs, ingestion/emission, GraphQL, REST, or the acryl-datahub Python SDK. Also answers data-modeling questions and helps choose native DataHub features vs custom extensions. Trigger this skill whenever a task involves DataHub metadata, lineage, ingestion, or any DataHub API surface, even without explicit mention of "API".

How it works

  1. Phase 1 — Understand the Task & Select Mode
  2. Phase 2 — Reference Navigation
  3. Phase 3 — Check Prerequisites
  4. Phase 4 — Explore Live API Documentation (optional)
  5. Phase 5 — Write Code, Execute, Verify
  6. Constraints
Original SKILL.md headings · 182 lines, 917 words
SKILL.md preview
---
name: datahub-api
description: Reference and coding guide for DataHub integration in backend development. Use when implementing or debugging code that touches DataHub — entities (datasets, dashboards, charts), aspects (ownership, tags, domains, glossary terms, structured properties), lineage, assertions, URNs, ingestion/emission, GraphQL, REST, or the acryl-datahub Python SDK. Also answers data-modeling questions and helps choose native DataHub features vs custom extensions. Trigger this skill whenever a task involves DataHub metadata, lineage, ingestion, or any DataHub API surface, even without explicit mention of "API".
argument-hint: "[task]"
allowed-tools: Read, Grep, Glob, Bash(python3 *), Bash(pip3 *), Bash(curl *)
---

## Phase 1 — Understand the Task & Select Mode

**Step 0 — Check reference materials:**

```bash
[ -d "ref/github/datahub" ] && echo "ref present" || echo "ref MISSING"
```

If `ref/github/datahub` is **missing**, stop immediately and tell the user:

> `ref/github/datahub` is not present. Run `/ref-setup` to download
> the DataHub v1.6.0 source (shallow clone, ~1-3 min). Retry this task after it completes.

**Step 1 — Select operating mode:**

| If the task looks like... | Mode |
|---|---|
| "Does DataHub have X?", "What's the best way to model Y?", "Should I use Z or build custom?" | **Q&A** — research and answer, no execution |
| "Write a Python module to do X", "Build a utility for Y", "Test whether Z works" | **Code Writer** — write, execute, verify |
| Mixed ("explain and write the code") | Q&A first, then Code Writer |

**Step 2 — Identify the API layer** (both modes):

| Task | API to use |
|---|---|
| Search, entity reads, lineage queries (UI-facing) | GraphQL |
| Emit lineage, custom aspects, bulk ingestion | Python SDK (`DatahubRestEmitter`) |
| Token management, soft delete, batch operations | GraphQL mutation |
| Advanced admin, index management | OpenAPI REST |

**Step 3 — Identify entity type** (both modes): dataset, dashboard, chart, tag, lineage, assertion, structured property, etc.

After Phase 1:
- **Q&A** routes to Phase 2 → answer → done.
- **Code Writer** continues through Phases 2–5.

---

## Phase 2 — Reference Navigation

The full DataHub source (v1.6.0) lives in `ref/github/datahub/`. Use the reference lookup table in [reference.md](reference.md) §Reference Lookup Table to find the right files.

**For Q&A mode**: read the relevant reference files, then write a clear answer with:
- Exact field names, aspect names, URN formats
- Source file citations from `ref/`
- Recommendation and rationale
- If no native solution exists: propose the minimal custom approach (Structured Properties before Glossary before fully custom aspects)

Follow the **Decision Protocol** in [reference.md](reference.md) §Decision Protocol to determine whether DataHub supports the concept natively.

**For Code Writer mode**: continue to Phase 3.

---

## Phase 3 — Check Prerequisites

Run these checks before executing any code:

```bash
# 1. Check acryl-datahub is installed
python3 -c "import datahub; print('acryl-datahub', datahub.__version__)" 2>/dev/null \
  || pip3 install acryl-datahub --quiet

# 2. Read the GMS URL from .env.dev (GMS has its own ingress host)
source helm-charts/.env.dev 2>/dev/null || true
DATAHUB_GMS_URL="${DATASPOKE_DEV_DATAHUB_GMS_URL:-http://datahub-gms.${DATASPOKE_KUBE_INGRESS_DOMAIN}}"

# 3. Check GMS is reachable via ingress
curl -s "${DATAHUB_GMS_URL}/config" \
  | python3 -c "import sys,json; d=json.load(sys.stdin); print('GMS ok, version:', d['versions']['acryldata/datahub']['version'])" \
  || echo "ERROR: GMS not reachable at ${DATAHUB_GMS_URL}. Run: ./helm-charts/bin/health-check.sh"

# 4. Check token
if [ -z "${DATASPOKE_DEV_DATAHUB_TOKEN:-}" ]; then
  echo "DATASPOKE_DEV_DATAHUB_TOKEN not set — generate via: http://datahub.${DATASPOKE_KUBE_INGRESS_DOMAIN} → Settings → Access Tokens"
fi
```

If any prerequisite fails, stop and inform the user with the fix instructions.

---

## Phase 4 — Explore Live API Documentation (optional)

Only use this if the static `ref/` files don't answer the question.

Derive the base URLs from `helm-charts/.env.dev`: `source helm-charts/.env.dev`, then:
- GMS base: `${DATASPOKE_DEV_DATAHUB_GMS_URL}` (i.e. `http://datahub-gms.${DATASPOKE_KUBE_INGRESS_DOMAIN}`)
- DataHub UI: `http://datahub.${DATASPOKE_KUBE_INGRESS_DOMAIN}`

| Resource | URL pattern | Notes |
|---|---|---|
| Swagger UI (REST/OpenAPI) | `<GMS_BASE>/openapi/swagger-ui/index.html` | Set Bearer token in Authorize dialog |
| Raw OpenAPI spec | `curl -s -H "Authorization: Bearer $DATASPOKE_DEV_DATAHUB_TOKEN" <GMS_BASE>/openapi/v3/api-docs` | JSON, pipe to `python3 -m json.tool` |
| GraphiQL | `http://datahub.<DOMAIN>/api/graphiql` | Browser only, uses session cookie |
| Unauthenticated health | `<GMS_BASE>/config`, `<GMS_BASE>/health` | No token needed |

---

## Phase 5 — Write Code, Execute, Verify

### 5.1 SDK Setup Pattern

Read the GMS URL from the environment (set by `source helm-charts/.env.dev`):
```python
from datahub.ingestion.graph.client import DataHubGraph, DatahubClientConfig
import os

# GMS is accessible via its own ingress host: http://datahub-gms.<INGRESS_DOMAIN>
gms_url = os.environ["DATASPOKE_DEV_DATAHUB_GMS_URL"]  # set in helm-charts/.env

graph = DataHubGraph(DatahubClientConfig(
    server=gms_url,
    token=os.environ.get("DATASPOKE_DEV_DATAHUB_TOKEN", ""),
))
```

For bulk emission:
```python
from datahub.emitter.rest_emitter import DatahubRestEmitter
import os

emitter = DatahubRestEmitter(
    os.environ["DATASPOKE_DEV_DATAHUB_GMS_URL"],
    token=os.environ.get("DATASPOKE_DEV_DATAHUB_TOKEN", ""),
)
```

### 5.2 Execution Loop

For each operation:

1. **Read the relevant tutorial** from `ref/github/datahub/docs/api/tutorials/`
2. **Find the matching SDK example** in `ref/github/datahub/metadata-ingestion/examples/library/`
3. **Write the code** to a temp file
4. **Execute**: `python3 /tmp/datahub_test_script.py`
5. **Verify** by reading back the written entity:

```python
# Read back to confirm the write succeeded
aspect = graph.get_aspect(entity_urn=urn, aspect_type=AspectClass)
assert aspect is not None, "Entity not found after write"
print("Verified:", aspect)
```

6. **Iterate** on failure — diagnose error, fix, re-run. Stop after 3 consecutive failures and report the blocker.
7. **Report**: final code + what was written + verification output

### 5.3 Common URN Builders

```python
from datahub.emitter.mce_builder import (
    make_dataset_urn,        # urn:li:dataset:(urn:li:dataPlatform:<p>,<name>,<env>)
    make_data_platform_urn,  # urn:li:dataPlatform:<name>
    make_schema_field_urn,   # urn:li:schemaField:(<dataset_urn>,<field>)
    make_tag_urn,            # urn:li:tag:<name>
    make_user_urn,           # urn:li:corpuser:<username>
    make_group_urn,          # urn:li:corpGroup:<name>
    make_domain_urn,         # urn:li:domain:<uuid>
)
```

### 5.4 Emitter Behavior

- **Retry**: 3 attempts on `[429, 500, 502, 503, 504]` with exponential backoff
- **Payload limit**: ~15 MB per request; split large batches
- **Rate limiting**: exponential back-off on 429 with 2x multiplier

---

## Constraints

1. **Never use `kubectl exec`** to interact with DataHub — it bypasses the API surface and doesn't reflect production behavior.
2. **Never run ad-hoc `kubectl port-forward`** — DataHub GMS is accessible via ingress at `http://datahub-gms.<INGRESS_DOMAIN>` (`$DATASPOKE_DEV_DATAHUB_GMS_URL`). If the ingress is unreachable, run `./helm-charts/bin/health-check.sh` to diagnose.
3. **Always read the matching tutorial/example first** before writing API code.
4. **Prefer static `ref/` lookup over live API exploration** for speed — only fall back to Swagger/GraphiQL when the static ref is ambiguous.

---

See [reference.md](reference.md) for the reference lookup table, decision protocol, and known patterns.
Files in this package (2)
  • SKILL.md 8 KB
  • reference.md 17 KB

Versions

The database has no other skills named datahub-api.

Database snapshot: 13.11.2026 · package version 89339c586f2f (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100datahub-search
    Use this skill when the user wants to search the DataHub catalog, discover entities, answer ad-hoc questions about their data, find datasets, or browse by platform or domain. Triggers on: "search DataHub", "find datasets", "who owns X", "what tables contain PII", "what columns…
  • 100datahub-connector-planning
    Plans new DataHub connectors by classifying the source system, researching it using a dedicated agent or inline research, and generating a _PLANNING.md blueprint with entity mapping and architecture decisions. Use when building a new connector, researching a source system for…
  • 90data-data-catalog
    Use this skill when asked about data catalog, DataHub, Amundsen, Apache Atlas, OpenMetadata, metadata management, data discovery, business glossary, data lineage, data ownership, or column-level lineage. This skill enforces: metadata ingestion pipelines, column-level lineage…
  • 90openmetadata-datahub-and-openlineage
    Guides agents through metadata platform and lineage workflows using OpenMetadata, DataHub, or OpenLineage-compatible systems. Use when improving discovery, lineage quality, metadata governance, or producer-to-catalog integration.
  • 100test-manual-api-wired
    Walk a single api-wired UC scenario manually: read the source test file, print each REST request, ask for approval, fire it, print the response, and probe side effects (DB rows, DataHub aspects, k8s secrets). Optional argument `scope` selects the case (free-form: "UC1", "UC1…
  • 100data-catalog-and-lineage-governance-datahub-amundsen
    Guides deploying and operating enterprise data catalog and governance tooling — DataHub or Amundsen — for discovering, tagging, classifying, and owning datasets, features, and models across an organization, including deprecation workflows that must check downstream consumers…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.