Skip to content
Skills Library
Programming · Building & coding

darkhunt-telemetry-python-integration

/darkhunt-telemetry-python-integration

Use this skill when integrating `darkhunt-telemetry` (the Darkhunt trace-hub PYTHON SDK, published on PyPI) into a Python service. Covers: install (published on PyPI), singleton client setup, trace + generation + span emission via the `with`-based active-context helpers, backdated `start_time`, graceful shutdown, routing-field discipline (tenant_id / workspace_id / application_id), creating an OBSERVABILITY application via the Darkhunt MCP, in-cluster vs public ingest, and — the big part — multi-agent topology + agent handoffs across every Python transport (an in-process contextvars carrier, o

Licence
Apache-2.0
Last repository change
06.10.2026

Assessment and security

82/ 100Class: good
Format20 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability17 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content30 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
4
Deprecation

No signals detected. Repository changed on 06.10.2026, without references to retired models or APIs.

Licence

Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.

What it does and when to use it

Use this skill when integrating `darkhunt-telemetry` (the Darkhunt trace-hub PYTHON SDK, published on PyPI) into a Python service. Covers: install (published on PyPI), singleton client setup, trace + generation + span emission via the `with`-based active-context helpers, backdated `start_time`, graceful shutdown, routing-field discipline (tenant_id / workspace_id / application_id), creating an OBSERVABILITY application via the Darkhunt MCP, in-cluster vs public ingest, and — the big part — multi-agent topology + agent handoffs across every Python transport (an in-process contextvars carrier, o

Original author description

Use this skill when integrating `darkhunt-telemetry` (the Darkhunt trace-hub PYTHON SDK, published on PyPI) into a Python service. Covers: install (published on PyPI), singleton client setup, trace + generation + span emission via the `with`-based active-context helpers, backdated `start_time`, graceful shutdown, routing-field discipline (tenant_id / workspace_id / application_id), creating an OBSERVABILITY application via the Darkhunt MCP, in-cluster vs public ingest, and — the big part — multi-agent topology + agent handoffs across every Python transport (an in-process contextvars carrier, o

How it works

  1. What the SDK is
  2. Step-by-step integration
  3. Routing fields
  4. sessionid and userid — set them every time
  5. In-cluster vs public ingest
  6. Span types — pick the right one
  7. Guarding tool calls — @guard (enforcement, not just tracing)
  8. Data masking
  9. Multi-agent topology & handoffs
  10. Verification
Original SKILL.md headings · 735 lines, 5,108 words
SKILL.md preview
---
name: darkhunt-telemetry-python-integration
description: |
  Use this skill when integrating `darkhunt-telemetry` (the Darkhunt trace-hub
  PYTHON SDK, published on PyPI) into a
  Python service. Covers: install (published on PyPI), singleton client setup,
  trace + generation + span emission via the `with`-based active-context helpers,
  backdated `start_time`, graceful shutdown, routing-field discipline (tenant_id /
  workspace_id / application_id), creating an OBSERVABILITY application via the
  Darkhunt MCP, in-cluster vs public ingest, and — the big part
  — multi-agent topology + agent handoffs across every Python transport (an
  in-process contextvars carrier, orchestrator-passed traces, a LangGraph state
  field, the HTTP `traceparent` header, a queue metadata field, and Temporal
  Headers via `HandoffInterceptor` + `child_args`). Auto-invoke when the user asks
  to add LLM tracing/observability to a Python app, send spans to trace-hub, wire
  `DarkhuntTelemetry` / `client.trace()` / `trace.generation()`, or build a
  multi-agent Python system where agents hand off to each other (agent topology).
  Also covers guarding tool calls with `@guard` (Darkhunt rules block a call before it
  runs or withhold its output) and plugging Darkhunt in as the policy behind
  Microsoft's Agent Governance Toolkit (AGT). Auto-invoke when the user asks to
  guard, block or enforce policy on an agent's tool calls, or to integrate AGT.
---

# Darkhunt telemetry Python SDK — integration guide

This skill walks through wiring `darkhunt-telemetry` (the **Python** SDK) into a
Python service. It is the Python analog of the TypeScript
`darkhunt-telemetry-integration` skill — same wire contract and routing
semantics, adapted to Python idioms (keyword arguments,
`with` context managers, `contextvars`).

The patterns below are extracted from `temporal-demo-python`, an internal
six-domain multi-agent reference integration where each domain uses a *different*
orchestration style, so it demonstrates every handoff transport at once. File-level
pointers into it are listed at the end of this skill; if you don't have that repo
checked out, the inline snippets here are self-contained.

For anything the patterns below don't cover, read the SDK's own `README.md` — it
carries the full API reference. It ships inside the installed
package and is published at
<https://github.com/darkhunt-security/darkhunt-telemetry-python>.

## What the SDK is

A Darkhunt-specific span exporter built on OpenTelemetry primitives
(`TracerProvider`, `BatchSpanProcessor`, OTLP/protobuf) that ships spans — traces,
LLM generations, tool calls, retrievals, guardrails — to Darkhunt trace-hub.
Routing semantics (`tenant_id` / `workspace_id` / `application_id`) and the
attribute schema are Darkhunt-specific; trace-hub is the only intended receiver.
The SDK sends values verbatim — PII masking happens server-side in the Darkhunt
platform on ingest. Requires Python **3.9+**.

Key shapes:

- **`DarkhuntTelemetry`** — the client. One per process, lifetime-of-the-process.
- **`Trace`** — a single user-facing interaction. Carries routing fields.
- **`Generation`** — one LLM round-trip under a trace (`model`, messages, `usage`, `cost`).
- **`Span`** — anything else (tool calls, retrievals, guardrails, sub-agents). Use
  `observation_type` to categorize.

## Step-by-step integration

### 1. Install — from PyPI

The SDK is published on public PyPI as **`darkhunt-telemetry`**. Install it directly:

```bash
pip install "darkhunt-telemetry[temporal]"
# or, with uv:
uv add "darkhunt-telemetry[temporal]"
```

or declare it as a project dependency:

```toml
# pyproject.toml
dependencies = ["darkhunt-telemetry[temporal]"]
```

then `uv sync`. Releases are **continuous** — every merge to the SDK's `main`
publishes `0.5.<build>` to PyPI, so the newest release is always the latest.
Leave the spec unpinned to track latest, or set a floor (`>=0.5.13`) / pin an
exact version for reproducibility.

**Extras:** `[temporal]` pulls in `temporalio` (only needed for the Temporal
handoff interceptors). The core package does not import it, so it loads with zero
Temporal packages installed.

Because it installs from PyPI, **Docker builds just work** — `pip install` /
`uv sync` pulls the wheel inside the image, with no build-context tricks.

**Alternatives (rarely needed):**

- **Local path** — only when hacking on the SDK itself beside your app.
  With uv: `[tool.uv.sources] darkhunt-telemetry = { path = "../darkhunt-telemetry-python", editable = true }`;
  plain pip: `pip install -e "../darkhunt-telemetry-python[temporal]"`. (This is
  the one case that reintroduces the Docker sibling-path trap — a `context: .`
  build can't `COPY` a dep outside the build context; use a named
  `additional_contexts: { dhsdk: ../darkhunt-telemetry-python }` and `COPY --from=dhsdk`.)
- **Git dependency** — to pin an unreleased commit:
  `darkhunt-telemetry @ git+https://github.com/darkhunt-security/darkhunt-telemetry-python@<ref>`.

### 2. Get an API key

A `dh-...` API key is required for public/external ingest (`internal=False` —
app servers / CLIs / workers calling the public endpoint). In-cluster
service-to-service callers using `internal=True` don't need one.

Create one in the dashboard (**app.darkhunt.ai → Settings → Security → API Keys →
+ Create API key**), copy it immediately (shown once), and put it in the
**`DARKHUNT_API_KEY`** env var — the name the SDK reads by default
(`options api_key ?? DARKHUNT_API_KEY`). If it's missing on the public endpoint the
constructor raises `ValueError: api_key is required for the public endpoint`.

> **The key, the base URL, and the tenant must all be the same environment.** A
> `dh-` key is scoped to one environment's tenant. Source `DARKHUNT_API_KEY`,
> `DARKHUNT_BASE_URL`, and `DARKHUNT_TENANT_ID` from the **same** place — enrolling
> the `darkhunt-cli` writes a matched trio to `~/.darkhunt/credentials.json`
> (`{ apiKey, apiBaseUrl, tenantId }`); prefer that set together.

> **Gotcha: `credentials.json apiBaseUrl` has NO `/trace-hub` suffix** (it stores
> the bare host, e.g. `https://api.darkhunt.ai`). The SDK default `base_url` DOES
> include it (`https://api.darkhunt.ai/trace-hub`). If you source `DARKHUNT_BASE_URL`
> from that field, **append `/trace-hub` yourself** — the exporter posts to
> `{base_url}/otlp/t/{tenant_id}/v1/traces`, so dropping it yields 404.

### 2b. Create an OBSERVABILITY application (get the `application_id`)

Every trace needs an `application_id` (a workspace-scoped UUID). **Create a new,
dedicated OBSERVABILITY app** for the integration — do not reuse a random existing
app (its traces would pollute that scope). In a multi-agent system, **one app per
domain / service group** is typical (agents within a domain are told apart by
`service.name`).

**Reach for the Darkhunt MCP first** (it reuses enrolled credentials). Check the
`darkhunt_*` tools are connected, then:

```text
darkhunt_status                # confirm auth + tenant + reachable API
darkhunt_list_workspaces       # → pick your workspaceId (UUID)
darkhunt_create_application    { workspace, name, type: 'OBSERVABILITY', description? }
                               # → returns the NEW application's UUID (use this)
```

- Pass **`type: 'OBSERVABILITY'`** (default is `RED_TEAM`) so the app's **Tracing**
  view is enabled.
- Put each UUID in `DARKHUNT_APP_<DOMAIN>` (or `DARKHUNT_APPLICATION_ID` for a
  single-app service). If MCP + CLI are both unavailable, use the dashboard's
  new-application flow; don't silently curl the REST API.

### 3. Singleton client (process-wide)

**Don't construct `DarkhuntTelemetry` per request.** Each construction spins up a
`TracerProvider` + `BatchSpanProcessor` and registers an `atexit` handler, so a
per-call client leaks resources and prevents batching.

**Several logical agents in ONE process → ONE client, and `agent` per trace.**
`service.name` is a per-client OTel Resource, so it cannot vary within a process —
but `dh.trace(..., agent="research")` emits `service.name` as a *span* attribute,
which outranks the Resource at ingest, so each agent gets its own topology node from
a single client. Prefer this over a client registry: it is one object instead of N
providers/exporters/timers, and it does not tie your deployment shape to the graph
you want. Two rules come with it — the trace becomes a **new root** (`handoff_from[0]`
is recorded as a link, not a parent, so the edge survives but the cross-agent parent
chain does not), and every agent in one run must share a `session_id`, because links
resolve within a session.

A memoized **client per distinct `(application_id, service_name)`** is still right
when the names really are separate *services* — different processes, or one process
fronting genuinely distinct deployments:

```python
from darkhunt_telemetry import DarkhuntTelemetry

_clients: dict[str, DarkhuntTelemetry] = {}

def client_for(service_name: str, application_id: str) -> DarkhuntTelemetry:
    key = f"{application_id}::{service_name}"
    c = _clients.get(key)
    if c is None:
        c = DarkhuntTelemetry(
            # api_key / base_url / tenant / workspace read from env by default.
            application_id=application_id,
            service_name=service_name,   # the topology node identity
            internal=False,              # public ingest with the dh- bearer key
        )
        _clients[key] = c
    return c
```

### 4. Graceful degradation — never crash a host app without creds

The client **raises** if `api_key` is missing on the public endpoint, and
`client.trace()` **raises** if a routing field is missing. So gate on config
presence and return `None` when unconfigured; call sites use `if trace:` /
`trace.end() if trace else None`:

```python
import os

_REQUIRED = ("DARKHUNT_API_KEY", "DARKHUNT_TENANT_ID", "DARKHUNT_WORKSPACE_ID")
_ENABLED = os.environ.get("DARKHUNT_ENABLED") != "false" and all(os.environ.get(k) for k in _REQUIRED)

def open_agent_trace(domain, agent, *, session_id, user_id, handoff_from=None, input=None):
    if not _ENABLED:
        return None
    app_id = os.environ.get(f"DARKHUNT_APP_{domain.upper()}")
    if not app_id:
        return None
    client = client_for(f"{domain}.{agent}", app_id)
    tokens = [t for t in (handoff_from or []) if t]
    return client.trace(name=f"{domain}.{agent}", session_id=session_id, user_id=user_id,
                        handoff_from=tokens or None, input=input)
```

`temporal-demo-python/src/demo/telemetry.py` is this exact pattern, production-ready
(`open_agent_trace` / `open_gateway_trace` / `trace_chat` / `trace_gen` /
`trace_tool` / `flush_telemetry` / `shutdown_telemetry`). **Copy it as your
starting point.**

### 5. Wire shutdown on signals

Spans batch in the background. The SDK flushes at process exit via `atexit`, but
**signal-driven shutdown (SIGTERM/SIGINT — `docker stop`, `kill`) bypasses
`atexit`**, losing the in-memory batch. Wire it:

```python
# Long-running server:
try:
    uvicorn.run(app, ...)          # blocks; returns on SIGINT/SIGTERM
finally:
    for c in _clients.values():
        c.shutdown()

# One-shot script / per-task on a server:
client.flush()                     # before returning
```

The Temporal `Worker.run()` resolves on shutdown — call `shutdown()` in its
`finally`. For a per-request worker (HTTP/queue consumer), `client.flush()` after
each handled task is cheap insurance since `atexit` won't fire until the process
ends.

### 6. Wrap each LLM call — prefer the `with`-based active form

The Python SDK's active-context helpers are **synchronous `with` blocks**, which
fit a blocking `chat()` perfectly (and still work called from `async` code):

```python
def trace_chat(trace, name, model, input_messages, call):
    if trace is None:
        return call()                                    # no-op passthrough
    with trace.start_active_generation(name, model=model) as gen:
        gen.update(input_messages=input_messages)        # known at start
        res = call()                                     # span is ACTIVE here → real timing
        gen.end(model=model, output_messages=[res.message],
                usage={"input_tokens": res.usage["input_tokens"],
                       "output_tokens": res.usage["output_tokens"]})
        return res

def trace_tool(trace, tool_name, input, run):
    if trace is None:
        return run()
    with trace.start_active_span(tool_name, observation_type="tool",
                                 tool_name=tool_name, input=input) as span:
        out = run()
        span.end(output=out)
        return out
```

- `start_active_generation` / `start_active_span` **time the span automatically**,
  make it the active OTel span (so provider auto-instrumentation nests under it),
  end it on block exit, and mark ERROR on a raised exception. The inner `gen.end()`
  / `span.end()` is idempotent, so calling it yourself to attach the payload is
  fine.
- `update()` is for fields known at start (`input_messages`, `system_instructions`);
  `end()` for fields known at finish (`output_messages`, `usage`, `cost`).
- **Manual form** (streaming, or holding a span open across calls): open it *after*
  work started and **backdate** with `start_time` (epoch **seconds**, captured
  before the call): `gen = trace.generation(name, model=..., start_time=t0)`. Epoch
  seconds, not ms — the Python convention (`time.time()`).
- **You usually don't need `cost`.** trace-hub auto-prices from `model` + `usage`
  for known models. Only set `cost` for custom/unpriced models.

## Routing fields

Every span carries `tenant_id` / `workspace_id` / `application_id`. Set them once
on the client if constant for the process; pass per-trace if multi-tenant
(`client.trace(tenant_id=..., ...)`). The constructor merges
`constructor arg > env var > default`; `client.trace()` raises `ValueError` if any
is still missing. `assessment_run_id` is optional (Darkhunt-internal grouping);
omit for general production tracing.

Routing and `agent` are **independent**, so a multi-tenant host that also runs
several logical agents in the process passes both per trace — routing from the
request context, `agent` from whichever agent is running, and only `service_name`
on the client:

```python
dh = DarkhuntTelemetry(service_name="alludium-web")   # one client, forever
dh.trace(
    "research.run",
    agent="research",                 # the topology node
    session_id=req.session_id,        # MUST be shared across the run's agents
    tenant_id=req.tenant_id, workspace_id=req.ws_id, application_id=req.app_id,
)
```

## `session_id` and `user_id` — set them every time

Technically optional, but **every integration should set them**. Traces sharing a
`session_id` group into one conversation timeline; the policy engine keys per-user
signals off `user_id`. If not known at open, `trace.update(user_id=..., session_id=...)`
once they are — spans created after inherit the values. Routing identifiers
round-trip verbatim for exact-match grouping.

## In-cluster vs public ingest

| Caller                        | `internal`        | Auth                              | URL                                              |
| ----------------------------- | ----------------- | --------------------------------- | ------------------------------------------------ |
| In-cluster service-to-service | `True`            | none (cluster policy gates it)    | `POST {base_url}/internal/t/{tenant}/v1/traces`  |
| External CLI / browser / app  | `False` (default) | `Authorization: Bearer <api_key>` | `POST {base_url}/otlp/t/{tenant}/v1/traces`      |

## Span types — pick the right one

| Work                    | Python API                                                     | `observation_type`    |
| ----------------------- | -------------------------------------------------------------- | --------------------- |
| LLM round-trip          | `trace.generation(name, model=...)`                            | (auto: `generation`)  |
| External tool call      | `trace.span(name, observation_type="tool", tool_name=...)`     | `"tool"`              |
| Vector search/retrieval | `trace.span(name, observation_type="retriever")`              | `"retriever"`         |
| Sub-agent step          | `trace.span(name, observation_type="agent")`                  | `"agent"`             |
| Input/output guardrail  | `trace.span(name, observation_type="guardrail")`              | `"guardrail"`         |
| Embedding               | `trace.span(name, observation_type="embedding")`              | `"embedding"`         |
| Generic work            | `trace.span(name)`                                             | `"span"` (default)    |
| Fire-and-forget marker  | `trace.event(name)`                                            | `"event"`             |

Every factory has a `start_active_*` variant. Spans nest naturally —
`parent.span(...)` / `parent.generation(...)`. For `tool` spans set `tool_name`
(and optionally `tool_call_id` / `tool_arguments`) so the dashboard shows the real
tool, not the generic type.

## Guarding tool calls — `@guard` (enforcement, not just tracing)

Tracing records what a tool did. `@guard` lets the **rules in the Darkhunt
dashboard stop it**. It calls the guardrail manager's `/verify` twice per call:

* **`TOOL_CALL`**, before the function runs: a block means it never runs;
* **`TOOL_RESULT`**, before the caller sees the output: a block means it is withheld.

Only add it when the user asks to guard, block or enforce. Plain observability
doesn't need it.

### Where it goes

```python
from darkhunt_telemetry.guard import configure_guard, guard

configure_guard(mode="shadow")            # start in shadow; see "Rollout"

@function_tool                            # the framework's decorator on TOP...
@guard(name="send_referral")              # ...guard UNDER it (functools.wraps keeps the signature)
def send_referral(to: str, subject: str, message: str) -> dict: ...
```

* **Wrap the function, not the framework.** Put `@guard` on each tool function,
  under the framework's own decorator (`@function_tool`, `@tool`, `@mcp.tool()`).
  A loop that dispatches by name can wrap its executor function instead.
* **`name=` must match what the rules match on** (`toolName==…`). Default: the
  function's name.
* **Call tools inside the run.** Wrap the agent run in `with trace.activate():`
  so each check carries the run's `session_id`, `user_id` and routing.
  `activate()` makes the trace current without ending it. Outside a trace the
  checks still run, but with no session, and the SDK warns once.
* **Reuse an existing tool span.** If the code already opens a tool span for the
  call (`start_active_span(name, observation_type="tool", tool_name=name)`), the
  guard puts its `guardrail` spans under it instead of opening a second one.

### Pick `on_deny` per tool, by who reads the result

| Who consumes the result | `on_deny` | Why |
| ----------------------- | --------- | --- |
| **A model** (the LLM picked the tool) | `"return"` (default): a refusal string | the model reads "Blocked by Darkhunt: …" and can explain it |
| **Code** that can do without the data | a callable, e.g. `lambda v: []` | the pipeline continues; the block is still recorded |
| **Code** that can't continue | `"raise"`: `DarkhuntBlocked` | fail loudly rather than run on a refusal string |

Never let a refusal *string* flow into code that expects a list or dict.

### Rollout and verification

1. **Start in `shadow`** (`DARKHUNT_GUARD_MODE=shadow`, the default). Checks run
   and are recorded as "Would block (shadow)", but nothing is stopped.
2. **Then `enforce` with an explicit fail mode.** `enforce` *requires*
   `DARKHUNT_GUARD_FAIL=open|closed`, so nobody fails open or closed by accident.
   Use `closed` for tools that send or write sensitive data, and `open` where
   availability matters more.
3. **Make blocks visible in the app's own result.** Pass
   `configure_guard(on_verdict=…)` to add blocked or denied verdicts to the
   run's output (e.g. a "blocked by policy" list).
4. **Verify.** Create a rule that blocks one tool by name
   (`toolName==<tool>`, ENFORCE, `TOOL_CALL`) and run the agent. The tool must
   not run, the result must show the block, and the dashboard's enforcement log
   must list it under the run's session.

Defaults: TOOL_CALL timeout 1.5 s, TOOL_RESULT 5 s, results over 64 KB sent
truncated. Generators (streaming tools) are refused.

### If the app uses Microsoft AGT (Agent Governance Toolkit)

Keep AGT, and make Darkhunt its policy. Give the AGT manifest a
`type: custom, adapter: darkhunt` policy on `pre_tool_call` / `post_tool_call`,
and pass `policy_dispatcher=DarkhuntPolicy()` (`darkhunt_telemetry.agt`). The
README's AGT section has the manifest. Then:

* **AGT's framework adapters don't cover tools.** `guard_openai_agents_runner`
  and similar check only a run's input and output. Wire the tools where they are
  dispatched:
  * `@agt_tool(control)` under the framework's decorator, for a tool function
    (a sync function stays sync);
  * `await run_governed(control, name, args, execute)` (or `run_governed_sync`),
    for a loop that dispatches by name. `execute()` must return something JSON:
    AGT puts the result in its snapshot.
  * `await check_tool_point(control, "pre_tool_call" | "post_tool_call", name, args, result=…)`,
    for allow/deny hooks only, e.g. Claude Agent SDK `PreToolUse` / `PostToolUse`.
    A hook can't withhold a built-in tool's output; it can only tell the model
    not to use it.
* **Pass the run explicitly** (`host=trace`) when the call isn't inside
  `trace.activate()`. ACS evaluates the policy in a thread pool that drops
  context variables.
* **Requirements:** pin `agent-control-specification==0.3.1b1`, the version
  shipped with AGT's latest official release (v4.1.0). Manifests must declare
  `agent_control_specification_version: 0.3.1-beta`. It's a pre-release, needs
  Python 3.11+, and is a Rust extension (PyPI wheels are Linux x86-64 only).
  Keep it optional in the host app. Only `allow` / `deny` / `warn` are mapped.

## Data masking

There is no client-side masking and no `mask=` option: inputs/outputs/messages/system
prompts/metadata/tool args are sent verbatim. PII masking happens server-side in the
Darkhunt platform on ingest, but it does **not** cover metadata values, tags or routing
IDs (`session_id`, `user_id`, `user_email`) — never put secrets or unhashed PII there.

## Multi-agent topology & handoffs

When the service is one agent in a multi-agent system, Darkhunt reconstructs the
**agent topology** (who handed off to whom). **Identity: one `service.name` per
agent** — that string is the topology node. The graph is drawn from the
cross-service `parentSpanId` chain, so the whole job is to make each agent's root
trace **nest under its caller**.

> **Several logical agents in ONE process?** `service_name` is the OTel Resource, fixed per
> `TracerProvider` (i.e. per client), so a shared client renders ONE node named after the
> process. Don't reach for a client-per-agent registry — pass **`agent`** per trace instead:
> `dh.trace("research.run", agent="research", session_id=sid)`. It emits `service.name` as a
> *span* attribute on the root and every child (span attrs outrank the Resource at ingest), so
> each agent gets its own node from one client. **Two rules come with it:** (1) an agent-scoped
> trace is deliberately a **new root** — it ignores `handoff_from[0]` and any ambient span when
> parenting, because node identity resolves once per trace id and two agents in one trace would
> silently collapse; upstreams stay `agent_handoff` links, so the edge survives but the
> cross-agent parent chain does not. (2) Since edges then come from links alone, and links
> resolve **within a session**, every agent in one run MUST share a `session_id` or no edge is
> drawn. Use a small stable set of agent names — each distinct value is a permanent node.

### The one thing to get right: nest via `handoff_from`

`client.trace(handoff_from=[caller_token])` makes the agent's root span a **child**
of `handoff_from[0]` (shared `trace_id`, `parentSpanId` set) **and** records an
`agent_handoff` link. Further entries are supplementary links (fan-in). The
caller's token is `trace.handoff_token()` (an opaque W3C `traceparent` string). The
SDK auto-registers the global OTel context manager + W3C propagator on construction,
so there is nothing else to wire — just pass `handoff_from`.

> **Why nesting matters:** ingestion **drops a contentless span** (an agent's root
> often is — its generations/tools are children) **unless** it is a cross-service
> entry (has a `parentSpanId` into another service). Nest → the root is a
> cross-service entry → kept → connected. Don't nest and don't link → the roots are
> dropped → **disconnected islands** (correct only when the agents truly never hand
> off — see the datastore/queue note below).

### Keep the token OUT of business signatures

The token is observability plumbing — never a typed `handoff` parameter on your
domain functions, entry inputs, message/return types, or graph state visible to
business code. Carry it out-of-band, exactly like OTel carries trace context. Which
carrier depends on the topology + transport:

| Topology / transport                | Carrier                                                                                     | Reference file (temporal-demo-python)         |
| ----------------------------------- | ------------------------------------------------------------------------------------------- | --------------------------------------------- |
| **Linear in-process chain**         | a `contextvars.ContextVar` (the `AsyncLocalStorage` analog) — read `current_handoff()`, publish your own | `src/demo/handoff_context.py`, `domains/weather/` |
| **Branching / fan-in in-process**   | **orchestrator-driven**: the coordinator opens each agent trace with an explicit `handoff_from` and passes the `Trace` into the agent | `domains/finance/run.py`                       |
| **In-process graph (LangGraph)**    | a dedicated field in the graph **state** (node writes its token, next node reads it as `handoff_from`) | `domains/banking/graph.py`                     |
| **HTTP**                            | the W3C `traceparent` **header** (out of the body)                                          | `domains/healthcare/http.py`                  |
| **Queue (Redis/Kafka/SQS/…)**       | a dedicated **message metadata field**, kept out of `data`                                  | `domains/devops/bus.py`                       |
| **Temporal**                        | a **Temporal Header** via `HandoffInterceptor` + per-edge `child_args`                       | `domains/security/`                           |

**Linear in-process — the contextvars carrier** (a single "current" slot; perfect
for `a → b → c`, wrong for branches/fan-in):

```python
from contextlib import contextmanager
from contextvars import ContextVar
_h: ContextVar = ContextVar("handoff", default=None)

@contextmanager
def with_handoff(token):
    reset = _h.set(token)
    try: yield
    finally: _h.reset(reset)

def current_handoff(): return _h.get()
def publish_handoff(token): _h.set(token)

# gateway seeds the scope with its root token; each agent reads+publishes:
with with_handoff(root.handoff_token() if root else None):
    plan = plan_task(inp)      # opens its trace w/ handoff_from=[current_handoff()], then publish_handoff(its token)
    weather = geodata(plan)    # nests under coordinator; publishes its own → advisor nests under geodata
    advisor(weather)
```

**Branching / fan-in in-process — orchestrator-driven** (the single ambient slot
can't express a DAG). The coordinator opens every trace with an explicit
`handoff_from` it computes per edge, and passes a `trace` handle into each agent as
a trailing, defaults-`None` telemetry param (the bounded exception to "no tokens in
signatures"):

```python
tok = lambda t: t.handoff_token() if t else None
root = open_gateway_trace("finance", ...);  root_t = tok(root)
coord = atrace("coordinator", [root_t]);     plan_task(inp, rt, coord);  coord_t = tok(coord); coord.end()
research = atrace("research", [coord_t]);     ...
# parallel-with-different-parents + fan-in are now trivial — each trace is explicit:
bull_th = atrace("bull", [research_t]);  bear_th = atrace("bear", [research_t])   # both ← research
pm = atrace("pm", [quant_t, bull_re_t, bear_re_t])   # fan-in of 3 (handoff_from[0] = parent edge)
```

### Carrying the token across a transport — use the SDK helpers

```python
from darkhunt_telemetry.transports import (
    handoff_to_http_headers, handoff_from_http_headers,   # HTTP: traceparent header
    handoff_to_message_meta, handoff_from_message_meta,   # queue: single-parent
    handoffs_from_messages,                               # queue: fan-in (ordered, de-duped)
    TRACEPARENT_HEADER, HANDOFF_MESSAGE_META_KEY,
)

# HTTP producer / consumer:
requests.post(url, headers=handoff_to_http_headers(trace.handoff_token(), base_headers))
token = handoff_from_http_headers(request.headers)        # → client.trace(handoff_from=[token])

# Queue producer / consumer (token in a DEDICATED field, never inside `data`):
xadd(stream, handoff_to_message_meta(trace.handoff_token(), fields))
token = handoff_from_message_meta(parsed_fields)          # single parent
tokens = handoffs_from_messages([m1, m2, m3])             # fan-in → client.trace(handoff_from=tokens)
```

### Temporal — the hardest, and the biggest trap

```python
from temporalio.worker import Worker
from darkhunt_telemetry.temporal import HandoffInterceptor, current_handoff, child_args
```

- **Worker**: register ONE `HandoffInterceptor()` in `interceptors=[...]` — it wires
  BOTH the activity side (exposes the inbound token to `current_handoff()`) and the
  workflow side (propagates / relocates per-edge overrides into the Temporal Header).
- **Activities** hold ALL the telemetry: open `dh.trace(handoff_from=current_handoff()
  or [])`, wrap chat/tools, `trace.end()` in `finally`. `current_handoff()` **works
  inside SYNC activities** run in a `ThreadPoolExecutor` — temporalio copies the
  contextvar context into the worker thread (verified against
  `temporalio.worker._activity`). No async requirement.
- **NEVER put telemetry in workflow code** — it's a deterministic sandbox (no SDK,
  no `dh.trace`). Its only Darkhunt reference is `child_args`, imported from the
  **sandbox-safe** `darkhunt_telemetry.temporal.handoff_header` subpath (pure dict
  helper, no `temporalio` import) inside `with workflow.unsafe.imports_passed_through():`.
- **⚠️ RETURN the token to build the DAG.** The interceptor defaults nest every
  child under the coordinator's inbound token → a **star** (the call graph), not the
  causal DAG. A workflow can neither mint nor read a token (sandboxed); only an
  **activity** produces one. A Temporal Header flows parent→child only — so a child's
  token comes home exactly one way: the **activity's return value**. Add an optional
  `handoff` field to each activity result; the coordinator threads it into the next
  `execute_child_workflow` via `child_args`:

  ```python
  # activity returns its token:
  return {**result, "handoff": trace.handoff_token() if trace else None}

  # coordinator threads it per edge (child_args attaches a hidden override the
  # interceptor relocates to the header and strips before the child sees it):
  def edge(x, upstream): return child_args(x, [t for t in upstream if t])
  plan  = await execute_child_workflow(ReconWf.run,  edge(recon_in,  [coord_tok]))
  spine = plan.get("handoff")
  batch = await asyncio.gather(*[execute_child_workflow(AnalyzerWf.run, edge(a_in, [spine])) for ...])
  a_toks = [b.get("handoff") for b in batch]                 # this round's analyzer tokens
  taint = await execute_child_workflow(TaintWf.run, edge(taint_in, a_toks))   # fan-in = array
  decision = await execute_child_workflow(PlannerWf.run, edge(plan_in, a_toks))
  spine = decision.get("handoff") or spine                   # loop advances the spine
  ```

  Rules of thumb: **fan-in** = pass the array of upstream tokens; a **loop** = later
  rounds thread the downstream stage's token (advance a `spine` var); a **hub** (a
  guardrail gate fired N times) = keep it on the coordinator's token.
  A collection-returning activity (`list[Finding]`) can't carry a `handoff` field —
  **wrap it** in `{"findings": [...], "handoff": ...}`.
- **The gateway→coordinator edge needs a hand-rolled client interceptor** — the SDK
  ships workflow + activity interceptors but NO client interceptor, so
  `client.start_workflow(...)` carries no header on its own. Inject the gateway
  trace's token (from an ambient store the request handler sets):

  ```python
  import temporalio.converter
  from temporalio.client import Client, Interceptor, OutboundInterceptor, StartWorkflowInput
  from darkhunt_telemetry.temporal import HANDOFF_HEADER

  class _GwOut(OutboundInterceptor):
      async def start_workflow(self, input: StartWorkflowInput):
          token = _gateway_handoff.get()   # a contextvars.ContextVar the handler set
          if token:
              input.headers = {**(input.headers or {}),
                               HANDOFF_HEADER: temporalio.converter.default().payload_converter.to_payload([token])}
          return await self.next.start_workflow(input)
  class _GwInt(Interceptor):
      def intercept_client(self, next): return _GwOut(next)
  client = await Client.connect(addr, namespace=ns, interceptors=[_GwInt()])
  ```

### The orchestrator / gateway node

Ingestion retains a trace ROOT even when contentless (it anchors the topology), so a
gateway that only fans work out survives on its own — open the root, put the task on
its `input`, and hand off from `root.handoff_token()` into the first agent:

```python
root = dh.trace(name=f"{domain}.gateway", session_id=task_id, user_id=user_id, input={"task": task})
first_token = root.handoff_token()   # → first agent's handoff_from
```

### Edge rules (each was a real bug)

- **Link to the REAL producing agent, not the orchestrator.** Thread the token
  wherever one agent's output becomes the next agent's input. Linking a downstream
  agent to the *orchestrator* (because it spawned it) draws a plausible-but-WRONG
  graph (e.g. `advisor` linked to `coordinator` instead of downstream of `geodata`).
- **Agent vs Worker**: a node with ≥1 `generation` renders as an **Agent** (model +
  cost); a tools-only node is a **Worker** (no cost). Emit `trace.generation(...)`
  for EVERY real LLM call, even "boilerplate" ones, or their cost never surfaces.
- **Deep repeat loops → self-loops (`↻ ×N`), not per-round back-edges.** For an
  N-round loop over M agents, link every round to the SAME stable upstream so they
  render as clean self-loops; linking each round to the prior round's output emits a
  tangle of back-edges.
- **Small 2-agent cycle (`↺`)** — here a back-edge IS right (a retried step links the
  step it retries; a debate rebuttal links the opposing thesis).
- **Fan-in is `handoff_from=[a, b, c]`** — `[0]` is the parent edge, the rest are
  links.
- **Logical coupling through a datastore/queue ≠ a drawn edge.** Services related
  only through a shared DB / bucket / a queue the token does NOT ride on have no
  `parentSpanId` chain, so they render as **disjoint islands** — frequently correct
  (classic RAG `ingest`→`answer`). **Do NOT synthesize an edge, and tell the user
  explicitly** that connecting them is an *architecture change* (carry a
  `handoff_token()` across the medium), not a telemetry setting.

## Verification

After wiring:

1. **Types/lint**: `uv run python -m compileall src && uv run ruff check .` (or the
   project's checks), and import every entrypoint.
2. **Graceful no-op**: run with `DARKHUNT_ENABLED=false` and confirm the app still
   runs and every helper no-ops.
3. **Server-side auth/routing probe** — the ONLY reliable programmatic check (a clean
   `flush()`/`shutdown()` is NOT proof of ingestion; the BatchSpanProcessor swallows
   export errors). A **400** on an empty body = auth + routing OK; **401** = wrong/
   absent key; **404** = missing `/trace-hub`:

   ```bash
   curl -s -o /dev/null -w '%{http_code}\n' -X POST \
     -H "Authorization: Bearer $DARKHUNT_API_KEY" \
     -H 'Content-Type: application/x-protobuf' \
     -H "X-Workspace-Id: $DARKHUNT_WORKSPACE_ID" \
     -H "X-Application-Id: $DARKHUNT_APPLICATION_ID" \
     --data-binary '' \
     "$DARKHUNT_BASE_URL/otlp/t/$DARKHUNT_TENANT_ID/v1/traces"
   ```

4. **Run the REAL instrumented path** and read its node in the dashboard. Verify with
   the same `service.name` as the integration — a probe under a *different*
   `serviceName` mints a **permanent phantom node** (there's no delete-trace API).
5. **The Darkhunt MCP cannot read traces back** — there is no tracing-query tool. So
   the two checks are (1) the curl probe (auth+routing only) and (2) the human
   opening the dashboard. Don't claim you verified ingestion programmatically.

The dashboard should show: one session-grouped trace per interaction; generations
rendered as chat bubbles with `input_messages`/`output_messages`; routing attributes
on the span detail; and token usage / model / computed cost on generations.

## On completion, report the topology shape to the user

Finishing the wiring is not the last step — tell the user what the Topology tab will
(and won't) show, proactively, before they open it. A **connected** graph (real
handoffs wired) → name the edges to expect (`coordinator → geodata → advisor`,
self-loops, fan-in). **Disconnected** nodes (independent processes with no live
handoff — standalone scripts, or a producer/consumer pair coupled only through a
datastore) are a **correct** result — state why (no `parentSpanId` chain) and that
connecting them is an architecture change. For OSS/example repos, persist that note
in the README too.

## Reference files in temporal-demo-python

These paths are inside the internal `temporal-demo-python` reference repo. Skip
this section if you don't have it — every pattern it indexes is already shown
inline above.

- `src/demo/telemetry.py` — the singleton/memoized clients + `open_agent_trace` /
  `open_gateway_trace` / `trace_chat` / `trace_gen` / `trace_tool` / lifecycle.
- `src/demo/handoff_context.py` — the `contextvars` ambient carrier.
- `domains/weather/` — linear in-process (ambient carrier).
- `domains/finance/run.py` — orchestrator-driven DAG (fan-in, `trace_gen` for a
  non-`chat()` LLM path).
- `domains/banking/graph.py` — LangGraph state-threaded handoff.
- `domains/healthcare/{http,handlers,orchestration}.py` — HTTP `traceparent` header +
  consensus fan-in.
- `domains/devops/{bus,handlers,orchestration}.py` — Redis-stream field + rca fan-in.
- `domains/security/{activities,workflows,temporal_worker}.py` + `server.py` —
  Temporal: `HandoffInterceptor`, activities return tokens, coordinator threads
  `child_args` per edge, hand-rolled gateway client interceptor.

## Common pitfalls

1. **Constructing the client per call** → leaks. Memoize per `(application_id,
   service_name)`.
2. **Pinning to an unpublished local checkout when PyPI is fine** → drift from the
   released SDK. Prefer `pip install "darkhunt-telemetry[temporal]"` from PyPI (§1);
   reserve the local-path/git deps for hacking on the SDK or pinning an unreleased ref.
3. **No signal-driven shutdown** → SIGTERM loses the in-memory batch. Wire
   `shutdown()` in a `finally`.
4. **Missing `start_time` on the MANUAL form** → ~0ms duration. Prefer the
   `start_active_*` form (auto-timed); if manual, backdate with `start_time` in epoch
   **seconds**.
5. **`base_url` without `/trace-hub`** → 404 (esp. when sourced from
   `credentials.json apiBaseUrl`, which omits it).
6. **Reusing an existing app / dropping to raw REST.** Create a NEW OBSERVABILITY app
   via the MCP.
7. **Telemetry in Temporal workflow code** → sandbox violation / non-determinism.
   Telemetry lives in activities + the gateway; workflows only reference the
   sandbox-safe `child_args`.
8. **Temporal star instead of the DAG** → you didn't RETURN activity tokens +
   `child_args` them per edge. See the Temporal section.
9. **Disconnected nodes handed over without explanation** → reads as broken. Report
   the topology shape (correct-when-independent) proactively.
10. **Verifying with a throwaway probe under a different `serviceName`** → permanent
    phantom node. Use the curl probe + the real path.
11. **`@guard` above the framework's tool decorator** → it wraps a tool *object*,
    not a function, and raises `TypeError`. Put it underneath.
12. **`enforce` without `DARKHUNT_GUARD_FAIL`** → `ValueError` at configuration.
    Choose `open` or `closed` deliberately.
13. **Guarded tools called outside the run** → checks carry no session, so
    session rules can't apply and the enforcement log can't group them. Wrap the
    run in `trace.activate()` (or pass `host=` on the AGT helpers).

Versions

RepositoryScoreLicence
This page · darkhunt-security/darkhunt-telemetry-python82Apache-2.0
darkhunt-security/darkhunt-telemetry-python82Apache-2.0

Database snapshot: 13.11.2026 · package version 8eabdcf3876e (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 82darkhunt-telemetry-integration
    Use this skill when integrating `@darkhunt-security/telemetry` (the Darkhunt trace-hub TypeScript SDK) into a Node.js / TypeScript service. Covers: install, creating an API key, singleton client setup, trace + generation + span emission, backdated `startTime`, graceful shutdown,…
  • 87python
    The Vovk.ts Python client — generating a typed Python package from a Vovk API with `vovk-python`, the `py` / `pySrc` templates, `TypedDict` shapes, JSON + JSON Lines consumption, client-side validation, and PyPI publishing. Use whenever the user wants a typed Python SDK, ongoing…
  • 95python-packaging
    Create distributable Python packages with proper project structure, setup.py/pyproject.toml, and publishing to PyPI. Use when packaging Python libraries, creating CLI tools, or distributing Python code.
  • 100python-packaging
    Create distributable Python packages with proper project structure, setup.py/pyproject.toml, and publishing to PyPI. Use when packaging Python libraries, creating CLI tools, or distributing Python ...
  • 100python-packaging
    Create distributable Python packages with proper project structure, setup.py/pyproject.toml, and publishing to PyPI. Use when packaging Python libraries, creating CLI tools, or distributing Python code.
  • 100python-packaging
    Python package creation and PyPI distribution via pyproject.toml and entry points. Use when publishing a package or setting up build configuration.

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.