darkhunt-telemetry-python-integration
/darkhunt-telemetry-python-integration
Use this skill when integrating `darkhunt-telemetry` (the Darkhunt trace-hub PYTHON SDK, published on PyPI) into a Python service. Covers: install (published on PyPI), singleton client setup, trace + generation + span emission via the `with`-based active-context helpers, backdated `start_time`, graceful shutdown, routing-field discipline (tenant_id / workspace_id / application_id), creating an OBSERVABILITY application via the Darkhunt MCP, in-cluster vs public ingest, and — the big part — multi-agent topology + agent handoffs across every Python transport (an in-process contextvars carrier, o
- Author
- darkhunt-security ↗
- Repository
- darkhunt-telemetry-python ↗
- Licence
- Apache-2.0
- Last repository change
- 06.10.2026
Assessment and security
- Files in this package
- 1
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 4
No signals detected. Repository changed on 06.10.2026, without references to retired models or APIs.
Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.
What it does and when to use it
Use this skill when integrating `darkhunt-telemetry` (the Darkhunt trace-hub PYTHON SDK, published on PyPI) into a Python service. Covers: install (published on PyPI), singleton client setup, trace + generation + span emission via the `with`-based active-context helpers, backdated `start_time`, graceful shutdown, routing-field discipline (tenant_id / workspace_id / application_id), creating an OBSERVABILITY application via the Darkhunt MCP, in-cluster vs public ingest, and — the big part — multi-agent topology + agent handoffs across every Python transport (an in-process contextvars carrier, o
Original author description
Use this skill when integrating `darkhunt-telemetry` (the Darkhunt trace-hub PYTHON SDK, published on PyPI) into a Python service. Covers: install (published on PyPI), singleton client setup, trace + generation + span emission via the `with`-based active-context helpers, backdated `start_time`, graceful shutdown, routing-field discipline (tenant_id / workspace_id / application_id), creating an OBSERVABILITY application via the Darkhunt MCP, in-cluster vs public ingest, and — the big part — multi-agent topology + agent handoffs across every Python transport (an in-process contextvars carrier, o
How it works
- What the SDK is
- Step-by-step integration
- Routing fields
- sessionid and userid — set them every time
- In-cluster vs public ingest
- Span types — pick the right one
- Guarding tool calls — @guard (enforcement, not just tracing)
- Data masking
- Multi-agent topology & handoffs
- Verification
SKILL.md preview
---
name: darkhunt-telemetry-python-integration
description: |
Use this skill when integrating `darkhunt-telemetry` (the Darkhunt trace-hub
PYTHON SDK, published on PyPI) into a
Python service. Covers: install (published on PyPI), singleton client setup,
trace + generation + span emission via the `with`-based active-context helpers,
backdated `start_time`, graceful shutdown, routing-field discipline (tenant_id /
workspace_id / application_id), creating an OBSERVABILITY application via the
Darkhunt MCP, in-cluster vs public ingest, and — the big part
— multi-agent topology + agent handoffs across every Python transport (an
in-process contextvars carrier, orchestrator-passed traces, a LangGraph state
field, the HTTP `traceparent` header, a queue metadata field, and Temporal
Headers via `HandoffInterceptor` + `child_args`). Auto-invoke when the user asks
to add LLM tracing/observability to a Python app, send spans to trace-hub, wire
`DarkhuntTelemetry` / `client.trace()` / `trace.generation()`, or build a
multi-agent Python system where agents hand off to each other (agent topology).
Also covers guarding tool calls with `@guard` (Darkhunt rules block a call before it
runs or withhold its output) and plugging Darkhunt in as the policy behind
Microsoft's Agent Governance Toolkit (AGT). Auto-invoke when the user asks to
guard, block or enforce policy on an agent's tool calls, or to integrate AGT.
---
# Darkhunt telemetry Python SDK — integration guide
This skill walks through wiring `darkhunt-telemetry` (the **Python** SDK) into a
Python service. It is the Python analog of the TypeScript
`darkhunt-telemetry-integration` skill — same wire contract and routing
semantics, adapted to Python idioms (keyword arguments,
`with` context managers, `contextvars`).
The patterns below are extracted from `temporal-demo-python`, an internal
six-domain multi-agent reference integration where each domain uses a *different*
orchestration style, so it demonstrates every handoff transport at once. File-level
pointers into it are listed at the end of this skill; if you don't have that repo
checked out, the inline snippets here are self-contained.
For anything the patterns below don't cover, read the SDK's own `README.md` — it
carries the full API reference. It ships inside the installed
package and is published at
<https://github.com/darkhunt-security/darkhunt-telemetry-python>.
## What the SDK is
A Darkhunt-specific span exporter built on OpenTelemetry primitives
(`TracerProvider`, `BatchSpanProcessor`, OTLP/protobuf) that ships spans — traces,
LLM generations, tool calls, retrievals, guardrails — to Darkhunt trace-hub.
Routing semantics (`tenant_id` / `workspace_id` / `application_id`) and the
attribute schema are Darkhunt-specific; trace-hub is the only intended receiver.
The SDK sends values verbatim — PII masking happens server-side in the Darkhunt
platform on ingest. Requires Python **3.9+**.
Key shapes:
- **`DarkhuntTelemetry`** — the client. One per process, lifetime-of-the-process.
- **`Trace`** — a single user-facing interaction. Carries routing fields.
- **`Generation`** — one LLM round-trip under a trace (`model`, messages, `usage`, `cost`).
- **`Span`** — anything else (tool calls, retrievals, guardrails, sub-agents). Use
`observation_type` to categorize.
## Step-by-step integration
### 1. Install — from PyPI
The SDK is published on public PyPI as **`darkhunt-telemetry`**. Install it directly:
```bash
pip install "darkhunt-telemetry[temporal]"
# or, with uv:
uv add "darkhunt-telemetry[temporal]"
```
or declare it as a project dependency:
```toml
# pyproject.toml
dependencies = ["darkhunt-telemetry[temporal]"]
```
then `uv sync`. Releases are **continuous** — every merge to the SDK's `main`
publishes `0.5.<build>` to PyPI, so the newest release is always the latest.
Leave the spec unpinned to track latest, or set a floor (`>=0.5.13`) / pin an
exact version for reproducibility.
**Extras:** `[temporal]` pulls in `temporalio` (only needed for the Temporal
handoff interceptors). The core package does not import it, so it loads with zero
Temporal packages installed.
Because it installs from PyPI, **Docker builds just work** — `pip install` /
`uv sync` pulls the wheel inside the image, with no build-context tricks.
**Alternatives (rarely needed):**
- **Local path** — only when hacking on the SDK itself beside your app.
With uv: `[tool.uv.sources] darkhunt-telemetry = { path = "../darkhunt-telemetry-python", editable = true }`;
plain pip: `pip install -e "../darkhunt-telemetry-python[temporal]"`. (This is
the one case that reintroduces the Docker sibling-path trap — a `context: .`
build can't `COPY` a dep outside the build context; use a named
`additional_contexts: { dhsdk: ../darkhunt-telemetry-python }` and `COPY --from=dhsdk`.)
- **Git dependency** — to pin an unreleased commit:
`darkhunt-telemetry @ git+https://github.com/darkhunt-security/darkhunt-telemetry-python@<ref>`.
### 2. Get an API key
A `dh-...` API key is required for public/external ingest (`internal=False` —
app servers / CLIs / workers calling the public endpoint). In-cluster
service-to-service callers using `internal=True` don't need one.
Create one in the dashboard (**app.darkhunt.ai → Settings → Security → API Keys →
+ Create API key**), copy it immediately (shown once), and put it in the
**`DARKHUNT_API_KEY`** env var — the name the SDK reads by default
(`options api_key ?? DARKHUNT_API_KEY`). If it's missing on the public endpoint the
constructor raises `ValueError: api_key is required for the public endpoint`.
> **The key, the base URL, and the tenant must all be the same environment.** A
> `dh-` key is scoped to one environment's tenant. Source `DARKHUNT_API_KEY`,
> `DARKHUNT_BASE_URL`, and `DARKHUNT_TENANT_ID` from the **same** place — enrolling
> the `darkhunt-cli` writes a matched trio to `~/.darkhunt/credentials.json`
> (`{ apiKey, apiBaseUrl, tenantId }`); prefer that set together.
> **Gotcha: `credentials.json apiBaseUrl` has NO `/trace-hub` suffix** (it stores
> the bare host, e.g. `https://api.darkhunt.ai`). The SDK default `base_url` DOES
> include it (`https://api.darkhunt.ai/trace-hub`). If you source `DARKHUNT_BASE_URL`
> from that field, **append `/trace-hub` yourself** — the exporter posts to
> `{base_url}/otlp/t/{tenant_id}/v1/traces`, so dropping it yields 404.
### 2b. Create an OBSERVABILITY application (get the `application_id`)
Every trace needs an `application_id` (a workspace-scoped UUID). **Create a new,
dedicated OBSERVABILITY app** for the integration — do not reuse a random existing
app (its traces would pollute that scope). In a multi-agent system, **one app per
domain / service group** is typical (agents within a domain are told apart by
`service.name`).
**Reach for the Darkhunt MCP first** (it reuses enrolled credentials). Check the
`darkhunt_*` tools are connected, then:
```text
darkhunt_status # confirm auth + tenant + reachable API
darkhunt_list_workspaces # → pick your workspaceId (UUID)
darkhunt_create_application { workspace, name, type: 'OBSERVABILITY', description? }
# → returns the NEW application's UUID (use this)
```
- Pass **`type: 'OBSERVABILITY'`** (default is `RED_TEAM`) so the app's **Tracing**
view is enabled.
- Put each UUID in `DARKHUNT_APP_<DOMAIN>` (or `DARKHUNT_APPLICATION_ID` for a
single-app service). If MCP + CLI are both unavailable, use the dashboard's
new-application flow; don't silently curl the REST API.
### 3. Singleton client (process-wide)
**Don't construct `DarkhuntTelemetry` per request.** Each construction spins up a
`TracerProvider` + `BatchSpanProcessor` and registers an `atexit` handler, so a
per-call client leaks resources and prevents batching.
**Several logical agents in ONE process → ONE client, and `agent` per trace.**
`service.name` is a per-client OTel Resource, so it cannot vary within a process —
but `dh.trace(..., agent="research")` emits `service.name` as a *span* attribute,
which outranks the Resource at ingest, so each agent gets its own topology node from
a single client. Prefer this over a client registry: it is one object instead of N
providers/exporters/timers, and it does not tie your deployment shape to the graph
you want. Two rules come with it — the trace becomes a **new root** (`handoff_from[0]`
is recorded as a link, not a parent, so the edge survives but the cross-agent parent
chain does not), and every agent in one run must share a `session_id`, because links
resolve within a session.
A memoized **client per distinct `(application_id, service_name)`** is still right
when the names really are separate *services* — different processes, or one process
fronting genuinely distinct deployments:
```python
from darkhunt_telemetry import DarkhuntTelemetry
_clients: dict[str, DarkhuntTelemetry] = {}
def client_for(service_name: str, application_id: str) -> DarkhuntTelemetry:
key = f"{application_id}::{service_name}"
c = _clients.get(key)
if c is None:
c = DarkhuntTelemetry(
# api_key / base_url / tenant / workspace read from env by default.
application_id=application_id,
service_name=service_name, # the topology node identity
internal=False, # public ingest with the dh- bearer key
)
_clients[key] = c
return c
```
### 4. Graceful degradation — never crash a host app without creds
The client **raises** if `api_key` is missing on the public endpoint, and
`client.trace()` **raises** if a routing field is missing. So gate on config
presence and return `None` when unconfigured; call sites use `if trace:` /
`trace.end() if trace else None`:
```python
import os
_REQUIRED = ("DARKHUNT_API_KEY", "DARKHUNT_TENANT_ID", "DARKHUNT_WORKSPACE_ID")
_ENABLED = os.environ.get("DARKHUNT_ENABLED") != "false" and all(os.environ.get(k) for k in _REQUIRED)
def open_agent_trace(domain, agent, *, session_id, user_id, handoff_from=None, input=None):
if not _ENABLED:
return None
app_id = os.environ.get(f"DARKHUNT_APP_{domain.upper()}")
if not app_id:
return None
client = client_for(f"{domain}.{agent}", app_id)
tokens = [t for t in (handoff_from or []) if t]
return client.trace(name=f"{domain}.{agent}", session_id=session_id, user_id=user_id,
handoff_from=tokens or None, input=input)
```
`temporal-demo-python/src/demo/telemetry.py` is this exact pattern, production-ready
(`open_agent_trace` / `open_gateway_trace` / `trace_chat` / `trace_gen` /
`trace_tool` / `flush_telemetry` / `shutdown_telemetry`). **Copy it as your
starting point.**
### 5. Wire shutdown on signals
Spans batch in the background. The SDK flushes at process exit via `atexit`, but
**signal-driven shutdown (SIGTERM/SIGINT — `docker stop`, `kill`) bypasses
`atexit`**, losing the in-memory batch. Wire it:
```python
# Long-running server:
try:
uvicorn.run(app, ...) # blocks; returns on SIGINT/SIGTERM
finally:
for c in _clients.values():
c.shutdown()
# One-shot script / per-task on a server:
client.flush() # before returning
```
The Temporal `Worker.run()` resolves on shutdown — call `shutdown()` in its
`finally`. For a per-request worker (HTTP/queue consumer), `client.flush()` after
each handled task is cheap insurance since `atexit` won't fire until the process
ends.
### 6. Wrap each LLM call — prefer the `with`-based active form
The Python SDK's active-context helpers are **synchronous `with` blocks**, which
fit a blocking `chat()` perfectly (and still work called from `async` code):
```python
def trace_chat(trace, name, model, input_messages, call):
if trace is None:
return call() # no-op passthrough
with trace.start_active_generation(name, model=model) as gen:
gen.update(input_messages=input_messages) # known at start
res = call() # span is ACTIVE here → real timing
gen.end(model=model, output_messages=[res.message],
usage={"input_tokens": res.usage["input_tokens"],
"output_tokens": res.usage["output_tokens"]})
return res
def trace_tool(trace, tool_name, input, run):
if trace is None:
return run()
with trace.start_active_span(tool_name, observation_type="tool",
tool_name=tool_name, input=input) as span:
out = run()
span.end(output=out)
return out
```
- `start_active_generation` / `start_active_span` **time the span automatically**,
make it the active OTel span (so provider auto-instrumentation nests under it),
end it on block exit, and mark ERROR on a raised exception. The inner `gen.end()`
/ `span.end()` is idempotent, so calling it yourself to attach the payload is
fine.
- `update()` is for fields known at start (`input_messages`, `system_instructions`);
`end()` for fields known at finish (`output_messages`, `usage`, `cost`).
- **Manual form** (streaming, or holding a span open across calls): open it *after*
work started and **backdate** with `start_time` (epoch **seconds**, captured
before the call): `gen = trace.generation(name, model=..., start_time=t0)`. Epoch
seconds, not ms — the Python convention (`time.time()`).
- **You usually don't need `cost`.** trace-hub auto-prices from `model` + `usage`
for known models. Only set `cost` for custom/unpriced models.
## Routing fields
Every span carries `tenant_id` / `workspace_id` / `application_id`. Set them once
on the client if constant for the process; pass per-trace if multi-tenant
(`client.trace(tenant_id=..., ...)`). The constructor merges
`constructor arg > env var > default`; `client.trace()` raises `ValueError` if any
is still missing. `assessment_run_id` is optional (Darkhunt-internal grouping);
omit for general production tracing.
Routing and `agent` are **independent**, so a multi-tenant host that also runs
several logical agents in the process passes both per trace — routing from the
request context, `agent` from whichever agent is running, and only `service_name`
on the client:
```python
dh = DarkhuntTelemetry(service_name="alludium-web") # one client, forever
dh.trace(
"research.run",
agent="research", # the topology node
session_id=req.session_id, # MUST be shared across the run's agents
tenant_id=req.tenant_id, workspace_id=req.ws_id, application_id=req.app_id,
)
```
## `session_id` and `user_id` — set them every time
Technically optional, but **every integration should set them**. Traces sharing a
`session_id` group into one conversation timeline; the policy engine keys per-user
signals off `user_id`. If not known at open, `trace.update(user_id=..., session_id=...)`
once they are — spans created after inherit the values. Routing identifiers
round-trip verbatim for exact-match grouping.
## In-cluster vs public ingest
| Caller | `internal` | Auth | URL |
| ----------------------------- | ----------------- | --------------------------------- | ------------------------------------------------ |
| In-cluster service-to-service | `True` | none (cluster policy gates it) | `POST {base_url}/internal/t/{tenant}/v1/traces` |
| External CLI / browser / app | `False` (default) | `Authorization: Bearer <api_key>` | `POST {base_url}/otlp/t/{tenant}/v1/traces` |
## Span types — pick the right one
| Work | Python API | `observation_type` |
| ----------------------- | -------------------------------------------------------------- | --------------------- |
| LLM round-trip | `trace.generation(name, model=...)` | (auto: `generation`) |
| External tool call | `trace.span(name, observation_type="tool", tool_name=...)` | `"tool"` |
| Vector search/retrieval | `trace.span(name, observation_type="retriever")` | `"retriever"` |
| Sub-agent step | `trace.span(name, observation_type="agent")` | `"agent"` |
| Input/output guardrail | `trace.span(name, observation_type="guardrail")` | `"guardrail"` |
| Embedding | `trace.span(name, observation_type="embedding")` | `"embedding"` |
| Generic work | `trace.span(name)` | `"span"` (default) |
| Fire-and-forget marker | `trace.event(name)` | `"event"` |
Every factory has a `start_active_*` variant. Spans nest naturally —
`parent.span(...)` / `parent.generation(...)`. For `tool` spans set `tool_name`
(and optionally `tool_call_id` / `tool_arguments`) so the dashboard shows the real
tool, not the generic type.
## Guarding tool calls — `@guard` (enforcement, not just tracing)
Tracing records what a tool did. `@guard` lets the **rules in the Darkhunt
dashboard stop it**. It calls the guardrail manager's `/verify` twice per call:
* **`TOOL_CALL`**, before the function runs: a block means it never runs;
* **`TOOL_RESULT`**, before the caller sees the output: a block means it is withheld.
Only add it when the user asks to guard, block or enforce. Plain observability
doesn't need it.
### Where it goes
```python
from darkhunt_telemetry.guard import configure_guard, guard
configure_guard(mode="shadow") # start in shadow; see "Rollout"
@function_tool # the framework's decorator on TOP...
@guard(name="send_referral") # ...guard UNDER it (functools.wraps keeps the signature)
def send_referral(to: str, subject: str, message: str) -> dict: ...
```
* **Wrap the function, not the framework.** Put `@guard` on each tool function,
under the framework's own decorator (`@function_tool`, `@tool`, `@mcp.tool()`).
A loop that dispatches by name can wrap its executor function instead.
* **`name=` must match what the rules match on** (`toolName==…`). Default: the
function's name.
* **Call tools inside the run.** Wrap the agent run in `with trace.activate():`
so each check carries the run's `session_id`, `user_id` and routing.
`activate()` makes the trace current without ending it. Outside a trace the
checks still run, but with no session, and the SDK warns once.
* **Reuse an existing tool span.** If the code already opens a tool span for the
call (`start_active_span(name, observation_type="tool", tool_name=name)`), the
guard puts its `guardrail` spans under it instead of opening a second one.
### Pick `on_deny` per tool, by who reads the result
| Who consumes the result | `on_deny` | Why |
| ----------------------- | --------- | --- |
| **A model** (the LLM picked the tool) | `"return"` (default): a refusal string | the model reads "Blocked by Darkhunt: …" and can explain it |
| **Code** that can do without the data | a callable, e.g. `lambda v: []` | the pipeline continues; the block is still recorded |
| **Code** that can't continue | `"raise"`: `DarkhuntBlocked` | fail loudly rather than run on a refusal string |
Never let a refusal *string* flow into code that expects a list or dict.
### Rollout and verification
1. **Start in `shadow`** (`DARKHUNT_GUARD_MODE=shadow`, the default). Checks run
and are recorded as "Would block (shadow)", but nothing is stopped.
2. **Then `enforce` with an explicit fail mode.** `enforce` *requires*
`DARKHUNT_GUARD_FAIL=open|closed`, so nobody fails open or closed by accident.
Use `closed` for tools that send or write sensitive data, and `open` where
availability matters more.
3. **Make blocks visible in the app's own result.** Pass
`configure_guard(on_verdict=…)` to add blocked or denied verdicts to the
run's output (e.g. a "blocked by policy" list).
4. **Verify.** Create a rule that blocks one tool by name
(`toolName==<tool>`, ENFORCE, `TOOL_CALL`) and run the agent. The tool must
not run, the result must show the block, and the dashboard's enforcement log
must list it under the run's session.
Defaults: TOOL_CALL timeout 1.5 s, TOOL_RESULT 5 s, results over 64 KB sent
truncated. Generators (streaming tools) are refused.
### If the app uses Microsoft AGT (Agent Governance Toolkit)
Keep AGT, and make Darkhunt its policy. Give the AGT manifest a
`type: custom, adapter: darkhunt` policy on `pre_tool_call` / `post_tool_call`,
and pass `policy_dispatcher=DarkhuntPolicy()` (`darkhunt_telemetry.agt`). The
README's AGT section has the manifest. Then:
* **AGT's framework adapters don't cover tools.** `guard_openai_agents_runner`
and similar check only a run's input and output. Wire the tools where they are
dispatched:
* `@agt_tool(control)` under the framework's decorator, for a tool function
(a sync function stays sync);
* `await run_governed(control, name, args, execute)` (or `run_governed_sync`),
for a loop that dispatches by name. `execute()` must return something JSON:
AGT puts the result in its snapshot.
* `await check_tool_point(control, "pre_tool_call" | "post_tool_call", name, args, result=…)`,
for allow/deny hooks only, e.g. Claude Agent SDK `PreToolUse` / `PostToolUse`.
A hook can't withhold a built-in tool's output; it can only tell the model
not to use it.
* **Pass the run explicitly** (`host=trace`) when the call isn't inside
`trace.activate()`. ACS evaluates the policy in a thread pool that drops
context variables.
* **Requirements:** pin `agent-control-specification==0.3.1b1`, the version
shipped with AGT's latest official release (v4.1.0). Manifests must declare
`agent_control_specification_version: 0.3.1-beta`. It's a pre-release, needs
Python 3.11+, and is a Rust extension (PyPI wheels are Linux x86-64 only).
Keep it optional in the host app. Only `allow` / `deny` / `warn` are mapped.
## Data masking
There is no client-side masking and no `mask=` option: inputs/outputs/messages/system
prompts/metadata/tool args are sent verbatim. PII masking happens server-side in the
Darkhunt platform on ingest, but it does **not** cover metadata values, tags or routing
IDs (`session_id`, `user_id`, `user_email`) — never put secrets or unhashed PII there.
## Multi-agent topology & handoffs
When the service is one agent in a multi-agent system, Darkhunt reconstructs the
**agent topology** (who handed off to whom). **Identity: one `service.name` per
agent** — that string is the topology node. The graph is drawn from the
cross-service `parentSpanId` chain, so the whole job is to make each agent's root
trace **nest under its caller**.
> **Several logical agents in ONE process?** `service_name` is the OTel Resource, fixed per
> `TracerProvider` (i.e. per client), so a shared client renders ONE node named after the
> process. Don't reach for a client-per-agent registry — pass **`agent`** per trace instead:
> `dh.trace("research.run", agent="research", session_id=sid)`. It emits `service.name` as a
> *span* attribute on the root and every child (span attrs outrank the Resource at ingest), so
> each agent gets its own node from one client. **Two rules come with it:** (1) an agent-scoped
> trace is deliberately a **new root** — it ignores `handoff_from[0]` and any ambient span when
> parenting, because node identity resolves once per trace id and two agents in one trace would
> silently collapse; upstreams stay `agent_handoff` links, so the edge survives but the
> cross-agent parent chain does not. (2) Since edges then come from links alone, and links
> resolve **within a session**, every agent in one run MUST share a `session_id` or no edge is
> drawn. Use a small stable set of agent names — each distinct value is a permanent node.
### The one thing to get right: nest via `handoff_from`
`client.trace(handoff_from=[caller_token])` makes the agent's root span a **child**
of `handoff_from[0]` (shared `trace_id`, `parentSpanId` set) **and** records an
`agent_handoff` link. Further entries are supplementary links (fan-in). The
caller's token is `trace.handoff_token()` (an opaque W3C `traceparent` string). The
SDK auto-registers the global OTel context manager + W3C propagator on construction,
so there is nothing else to wire — just pass `handoff_from`.
> **Why nesting matters:** ingestion **drops a contentless span** (an agent's root
> often is — its generations/tools are children) **unless** it is a cross-service
> entry (has a `parentSpanId` into another service). Nest → the root is a
> cross-service entry → kept → connected. Don't nest and don't link → the roots are
> dropped → **disconnected islands** (correct only when the agents truly never hand
> off — see the datastore/queue note below).
### Keep the token OUT of business signatures
The token is observability plumbing — never a typed `handoff` parameter on your
domain functions, entry inputs, message/return types, or graph state visible to
business code. Carry it out-of-band, exactly like OTel carries trace context. Which
carrier depends on the topology + transport:
| Topology / transport | Carrier | Reference file (temporal-demo-python) |
| ----------------------------------- | ------------------------------------------------------------------------------------------- | --------------------------------------------- |
| **Linear in-process chain** | a `contextvars.ContextVar` (the `AsyncLocalStorage` analog) — read `current_handoff()`, publish your own | `src/demo/handoff_context.py`, `domains/weather/` |
| **Branching / fan-in in-process** | **orchestrator-driven**: the coordinator opens each agent trace with an explicit `handoff_from` and passes the `Trace` into the agent | `domains/finance/run.py` |
| **In-process graph (LangGraph)** | a dedicated field in the graph **state** (node writes its token, next node reads it as `handoff_from`) | `domains/banking/graph.py` |
| **HTTP** | the W3C `traceparent` **header** (out of the body) | `domains/healthcare/http.py` |
| **Queue (Redis/Kafka/SQS/…)** | a dedicated **message metadata field**, kept out of `data` | `domains/devops/bus.py` |
| **Temporal** | a **Temporal Header** via `HandoffInterceptor` + per-edge `child_args` | `domains/security/` |
**Linear in-process — the contextvars carrier** (a single "current" slot; perfect
for `a → b → c`, wrong for branches/fan-in):
```python
from contextlib import contextmanager
from contextvars import ContextVar
_h: ContextVar = ContextVar("handoff", default=None)
@contextmanager
def with_handoff(token):
reset = _h.set(token)
try: yield
finally: _h.reset(reset)
def current_handoff(): return _h.get()
def publish_handoff(token): _h.set(token)
# gateway seeds the scope with its root token; each agent reads+publishes:
with with_handoff(root.handoff_token() if root else None):
plan = plan_task(inp) # opens its trace w/ handoff_from=[current_handoff()], then publish_handoff(its token)
weather = geodata(plan) # nests under coordinator; publishes its own → advisor nests under geodata
advisor(weather)
```
**Branching / fan-in in-process — orchestrator-driven** (the single ambient slot
can't express a DAG). The coordinator opens every trace with an explicit
`handoff_from` it computes per edge, and passes a `trace` handle into each agent as
a trailing, defaults-`None` telemetry param (the bounded exception to "no tokens in
signatures"):
```python
tok = lambda t: t.handoff_token() if t else None
root = open_gateway_trace("finance", ...); root_t = tok(root)
coord = atrace("coordinator", [root_t]); plan_task(inp, rt, coord); coord_t = tok(coord); coord.end()
research = atrace("research", [coord_t]); ...
# parallel-with-different-parents + fan-in are now trivial — each trace is explicit:
bull_th = atrace("bull", [research_t]); bear_th = atrace("bear", [research_t]) # both ← research
pm = atrace("pm", [quant_t, bull_re_t, bear_re_t]) # fan-in of 3 (handoff_from[0] = parent edge)
```
### Carrying the token across a transport — use the SDK helpers
```python
from darkhunt_telemetry.transports import (
handoff_to_http_headers, handoff_from_http_headers, # HTTP: traceparent header
handoff_to_message_meta, handoff_from_message_meta, # queue: single-parent
handoffs_from_messages, # queue: fan-in (ordered, de-duped)
TRACEPARENT_HEADER, HANDOFF_MESSAGE_META_KEY,
)
# HTTP producer / consumer:
requests.post(url, headers=handoff_to_http_headers(trace.handoff_token(), base_headers))
token = handoff_from_http_headers(request.headers) # → client.trace(handoff_from=[token])
# Queue producer / consumer (token in a DEDICATED field, never inside `data`):
xadd(stream, handoff_to_message_meta(trace.handoff_token(), fields))
token = handoff_from_message_meta(parsed_fields) # single parent
tokens = handoffs_from_messages([m1, m2, m3]) # fan-in → client.trace(handoff_from=tokens)
```
### Temporal — the hardest, and the biggest trap
```python
from temporalio.worker import Worker
from darkhunt_telemetry.temporal import HandoffInterceptor, current_handoff, child_args
```
- **Worker**: register ONE `HandoffInterceptor()` in `interceptors=[...]` — it wires
BOTH the activity side (exposes the inbound token to `current_handoff()`) and the
workflow side (propagates / relocates per-edge overrides into the Temporal Header).
- **Activities** hold ALL the telemetry: open `dh.trace(handoff_from=current_handoff()
or [])`, wrap chat/tools, `trace.end()` in `finally`. `current_handoff()` **works
inside SYNC activities** run in a `ThreadPoolExecutor` — temporalio copies the
contextvar context into the worker thread (verified against
`temporalio.worker._activity`). No async requirement.
- **NEVER put telemetry in workflow code** — it's a deterministic sandbox (no SDK,
no `dh.trace`). Its only Darkhunt reference is `child_args`, imported from the
**sandbox-safe** `darkhunt_telemetry.temporal.handoff_header` subpath (pure dict
helper, no `temporalio` import) inside `with workflow.unsafe.imports_passed_through():`.
- **⚠️ RETURN the token to build the DAG.** The interceptor defaults nest every
child under the coordinator's inbound token → a **star** (the call graph), not the
causal DAG. A workflow can neither mint nor read a token (sandboxed); only an
**activity** produces one. A Temporal Header flows parent→child only — so a child's
token comes home exactly one way: the **activity's return value**. Add an optional
`handoff` field to each activity result; the coordinator threads it into the next
`execute_child_workflow` via `child_args`:
```python
# activity returns its token:
return {**result, "handoff": trace.handoff_token() if trace else None}
# coordinator threads it per edge (child_args attaches a hidden override the
# interceptor relocates to the header and strips before the child sees it):
def edge(x, upstream): return child_args(x, [t for t in upstream if t])
plan = await execute_child_workflow(ReconWf.run, edge(recon_in, [coord_tok]))
spine = plan.get("handoff")
batch = await asyncio.gather(*[execute_child_workflow(AnalyzerWf.run, edge(a_in, [spine])) for ...])
a_toks = [b.get("handoff") for b in batch] # this round's analyzer tokens
taint = await execute_child_workflow(TaintWf.run, edge(taint_in, a_toks)) # fan-in = array
decision = await execute_child_workflow(PlannerWf.run, edge(plan_in, a_toks))
spine = decision.get("handoff") or spine # loop advances the spine
```
Rules of thumb: **fan-in** = pass the array of upstream tokens; a **loop** = later
rounds thread the downstream stage's token (advance a `spine` var); a **hub** (a
guardrail gate fired N times) = keep it on the coordinator's token.
A collection-returning activity (`list[Finding]`) can't carry a `handoff` field —
**wrap it** in `{"findings": [...], "handoff": ...}`.
- **The gateway→coordinator edge needs a hand-rolled client interceptor** — the SDK
ships workflow + activity interceptors but NO client interceptor, so
`client.start_workflow(...)` carries no header on its own. Inject the gateway
trace's token (from an ambient store the request handler sets):
```python
import temporalio.converter
from temporalio.client import Client, Interceptor, OutboundInterceptor, StartWorkflowInput
from darkhunt_telemetry.temporal import HANDOFF_HEADER
class _GwOut(OutboundInterceptor):
async def start_workflow(self, input: StartWorkflowInput):
token = _gateway_handoff.get() # a contextvars.ContextVar the handler set
if token:
input.headers = {**(input.headers or {}),
HANDOFF_HEADER: temporalio.converter.default().payload_converter.to_payload([token])}
return await self.next.start_workflow(input)
class _GwInt(Interceptor):
def intercept_client(self, next): return _GwOut(next)
client = await Client.connect(addr, namespace=ns, interceptors=[_GwInt()])
```
### The orchestrator / gateway node
Ingestion retains a trace ROOT even when contentless (it anchors the topology), so a
gateway that only fans work out survives on its own — open the root, put the task on
its `input`, and hand off from `root.handoff_token()` into the first agent:
```python
root = dh.trace(name=f"{domain}.gateway", session_id=task_id, user_id=user_id, input={"task": task})
first_token = root.handoff_token() # → first agent's handoff_from
```
### Edge rules (each was a real bug)
- **Link to the REAL producing agent, not the orchestrator.** Thread the token
wherever one agent's output becomes the next agent's input. Linking a downstream
agent to the *orchestrator* (because it spawned it) draws a plausible-but-WRONG
graph (e.g. `advisor` linked to `coordinator` instead of downstream of `geodata`).
- **Agent vs Worker**: a node with ≥1 `generation` renders as an **Agent** (model +
cost); a tools-only node is a **Worker** (no cost). Emit `trace.generation(...)`
for EVERY real LLM call, even "boilerplate" ones, or their cost never surfaces.
- **Deep repeat loops → self-loops (`↻ ×N`), not per-round back-edges.** For an
N-round loop over M agents, link every round to the SAME stable upstream so they
render as clean self-loops; linking each round to the prior round's output emits a
tangle of back-edges.
- **Small 2-agent cycle (`↺`)** — here a back-edge IS right (a retried step links the
step it retries; a debate rebuttal links the opposing thesis).
- **Fan-in is `handoff_from=[a, b, c]`** — `[0]` is the parent edge, the rest are
links.
- **Logical coupling through a datastore/queue ≠ a drawn edge.** Services related
only through a shared DB / bucket / a queue the token does NOT ride on have no
`parentSpanId` chain, so they render as **disjoint islands** — frequently correct
(classic RAG `ingest`→`answer`). **Do NOT synthesize an edge, and tell the user
explicitly** that connecting them is an *architecture change* (carry a
`handoff_token()` across the medium), not a telemetry setting.
## Verification
After wiring:
1. **Types/lint**: `uv run python -m compileall src && uv run ruff check .` (or the
project's checks), and import every entrypoint.
2. **Graceful no-op**: run with `DARKHUNT_ENABLED=false` and confirm the app still
runs and every helper no-ops.
3. **Server-side auth/routing probe** — the ONLY reliable programmatic check (a clean
`flush()`/`shutdown()` is NOT proof of ingestion; the BatchSpanProcessor swallows
export errors). A **400** on an empty body = auth + routing OK; **401** = wrong/
absent key; **404** = missing `/trace-hub`:
```bash
curl -s -o /dev/null -w '%{http_code}\n' -X POST \
-H "Authorization: Bearer $DARKHUNT_API_KEY" \
-H 'Content-Type: application/x-protobuf' \
-H "X-Workspace-Id: $DARKHUNT_WORKSPACE_ID" \
-H "X-Application-Id: $DARKHUNT_APPLICATION_ID" \
--data-binary '' \
"$DARKHUNT_BASE_URL/otlp/t/$DARKHUNT_TENANT_ID/v1/traces"
```
4. **Run the REAL instrumented path** and read its node in the dashboard. Verify with
the same `service.name` as the integration — a probe under a *different*
`serviceName` mints a **permanent phantom node** (there's no delete-trace API).
5. **The Darkhunt MCP cannot read traces back** — there is no tracing-query tool. So
the two checks are (1) the curl probe (auth+routing only) and (2) the human
opening the dashboard. Don't claim you verified ingestion programmatically.
The dashboard should show: one session-grouped trace per interaction; generations
rendered as chat bubbles with `input_messages`/`output_messages`; routing attributes
on the span detail; and token usage / model / computed cost on generations.
## On completion, report the topology shape to the user
Finishing the wiring is not the last step — tell the user what the Topology tab will
(and won't) show, proactively, before they open it. A **connected** graph (real
handoffs wired) → name the edges to expect (`coordinator → geodata → advisor`,
self-loops, fan-in). **Disconnected** nodes (independent processes with no live
handoff — standalone scripts, or a producer/consumer pair coupled only through a
datastore) are a **correct** result — state why (no `parentSpanId` chain) and that
connecting them is an architecture change. For OSS/example repos, persist that note
in the README too.
## Reference files in temporal-demo-python
These paths are inside the internal `temporal-demo-python` reference repo. Skip
this section if you don't have it — every pattern it indexes is already shown
inline above.
- `src/demo/telemetry.py` — the singleton/memoized clients + `open_agent_trace` /
`open_gateway_trace` / `trace_chat` / `trace_gen` / `trace_tool` / lifecycle.
- `src/demo/handoff_context.py` — the `contextvars` ambient carrier.
- `domains/weather/` — linear in-process (ambient carrier).
- `domains/finance/run.py` — orchestrator-driven DAG (fan-in, `trace_gen` for a
non-`chat()` LLM path).
- `domains/banking/graph.py` — LangGraph state-threaded handoff.
- `domains/healthcare/{http,handlers,orchestration}.py` — HTTP `traceparent` header +
consensus fan-in.
- `domains/devops/{bus,handlers,orchestration}.py` — Redis-stream field + rca fan-in.
- `domains/security/{activities,workflows,temporal_worker}.py` + `server.py` —
Temporal: `HandoffInterceptor`, activities return tokens, coordinator threads
`child_args` per edge, hand-rolled gateway client interceptor.
## Common pitfalls
1. **Constructing the client per call** → leaks. Memoize per `(application_id,
service_name)`.
2. **Pinning to an unpublished local checkout when PyPI is fine** → drift from the
released SDK. Prefer `pip install "darkhunt-telemetry[temporal]"` from PyPI (§1);
reserve the local-path/git deps for hacking on the SDK or pinning an unreleased ref.
3. **No signal-driven shutdown** → SIGTERM loses the in-memory batch. Wire
`shutdown()` in a `finally`.
4. **Missing `start_time` on the MANUAL form** → ~0ms duration. Prefer the
`start_active_*` form (auto-timed); if manual, backdate with `start_time` in epoch
**seconds**.
5. **`base_url` without `/trace-hub`** → 404 (esp. when sourced from
`credentials.json apiBaseUrl`, which omits it).
6. **Reusing an existing app / dropping to raw REST.** Create a NEW OBSERVABILITY app
via the MCP.
7. **Telemetry in Temporal workflow code** → sandbox violation / non-determinism.
Telemetry lives in activities + the gateway; workflows only reference the
sandbox-safe `child_args`.
8. **Temporal star instead of the DAG** → you didn't RETURN activity tokens +
`child_args` them per edge. See the Temporal section.
9. **Disconnected nodes handed over without explanation** → reads as broken. Report
the topology shape (correct-when-independent) proactively.
10. **Verifying with a throwaway probe under a different `serviceName`** → permanent
phantom node. Use the curl probe + the real path.
11. **`@guard` above the framework's tool decorator** → it wraps a tool *object*,
not a function, and raises `TypeError`. Put it underneath.
12. **`enforce` without `DARKHUNT_GUARD_FAIL`** → `ValueError` at configuration.
Choose `open` or `closed` deliberately.
13. **Guarded tools called outside the run** → checks carry no session, so
session rules can't apply and the enforcement log can't group them. Wrap the
run in `trace.activate()` (or pass `host=` on the AGT helpers).
Versions
| Repository | Score | Licence |
|---|---|---|
| This page · darkhunt-security/darkhunt-telemetry-python | 82 | Apache-2.0 |
| darkhunt-security/darkhunt-telemetry-python | 82 | Apache-2.0 |
Database snapshot: 13.11.2026 · package version 8eabdcf3876e (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 82darkhunt-telemetry-integration
Use this skill when integrating `@darkhunt-security/telemetry` (the Darkhunt trace-hub TypeScript SDK) into a Node.js / TypeScript service. Covers: install, creating an API key, singleton client setup, trace + generation + span emission, backdated `startTime`, graceful shutdown,… - 87python
The Vovk.ts Python client — generating a typed Python package from a Vovk API with `vovk-python`, the `py` / `pySrc` templates, `TypedDict` shapes, JSON + JSON Lines consumption, client-side validation, and PyPI publishing. Use whenever the user wants a typed Python SDK, ongoing… - 95python-packaging
Create distributable Python packages with proper project structure, setup.py/pyproject.toml, and publishing to PyPI. Use when packaging Python libraries, creating CLI tools, or distributing Python code. - 100python-packaging
Create distributable Python packages with proper project structure, setup.py/pyproject.toml, and publishing to PyPI. Use when packaging Python libraries, creating CLI tools, or distributing Python ... - 100python-packaging
Create distributable Python packages with proper project structure, setup.py/pyproject.toml, and publishing to PyPI. Use when packaging Python libraries, creating CLI tools, or distributing Python code. - 100python-packaging
Python package creation and PyPI distribution via pyproject.toml and entry points. Use when publishing a package or setting up build configuration.