Skip to content
Skills Library
DevOps & cloud · Analysis

trace-analytics

/trace-analytics

Investigate distributed traces and spans in OpenSearch. Use this skill when the user wants to analyze traces, investigate slow spans, find error spans, track agent invocations, measure token usage, reconstruct trace trees, query service maps, or debug distributed systems through trace data. Activate even if the user says traceId, spanId, OpenTelemetry, OTel, distributed tracing, latency, span duration, service map, or trace investigation without mentioning OpenSearch.

Licence
Apache-2.0
Last repository change
19.08.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
2
Scripts
0
HTML comments
0
URLs in SKILL.md
2
Deprecation

No signals detected. Repository changed on 19.08.2026, without references to retired models or APIs.

Licence

Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.

What it does and when to use it

Investigate distributed traces and spans in OpenSearch. Use this skill when the user wants to analyze traces, investigate slow spans, find error spans, track agent invocations, measure token usage, reconstruct trace trees, query service maps, or debug distributed systems through trace data. Activate even if the user says traceId, spanId, OpenTelemetry, OTel, distributed tracing, latency, span duration, service map, or trace investigation without mentioning OpenSearch.

Original author description

Investigate distributed traces and spans in OpenSearch. Use this skill when the user wants to analyze traces, investigate slow spans, find error spans, track agent invocations, measure token usage, reconstruct trace trees, query service maps, or debug distributed systems through trace data. Activate even if the user says traceId, spanId, OpenTelemetry, OTel, distributed tracing, latency, span duration, service map, or trace investigation without mentioning OpenSearch.

How it works

  1. Prerequisites
  2. Optional MCP Servers
  3. Critical Rules (MUST follow)
  4. Key Rules
  5. Workflow
  6. GenAI Operation Types
  7. Reference Files
Original SKILL.md headings · 145 lines, 756 words
SKILL.md preview
---
name: trace-analytics
description: >
  Investigate distributed traces and spans in OpenSearch. Use this skill when
  the user wants to analyze traces, investigate slow spans, find error spans,
  track agent invocations, measure token usage, reconstruct trace trees,
  query service maps, or debug distributed systems through trace data.
  Activate even if the user says traceId, spanId, OpenTelemetry, OTel,
  distributed tracing, latency, span duration, service map, or trace
  investigation without mentioning OpenSearch.
compatibility: Requires a running OpenSearch cluster with OTel trace data. PPL queries require the SQL plugin (built-in).
metadata:
  author: opensearch-project
  version: "2.0"
---

# OpenSearch Trace Analytics

You are an OpenSearch trace analytics specialist. You help users investigate distributed traces, analyze span performance, debug errors, and understand service dependencies.

## Prerequisites

- A running OpenSearch cluster with OTel trace data (typically `otel-v1-apm-span-*`)
- `uv` installed (for running helper scripts)

## Optional MCP Servers

```json
{
  "mcpServers": {
    "ddg-search": {
      "command": "uvx",
      "args": ["duckduckgo-mcp-server"]
    },
    "opensearch-mcp-server": {
      "command": "uvx",
      "args": ["opensearch-mcp-server-py@latest"],
      "env": { "FASTMCP_LOG_LEVEL": "ERROR" }
    }
  }
}
```

- **`opensearch-mcp-server`** — Direct OpenSearch API access including PPL via `GenericOpenSearchApiTool`. Handles SigV4 auth for AOS/AOSS.
- **`ddg-search`** — Search OpenSearch documentation for trace analytics features.

### opensearch-mcp-server Configuration Variants

For basic auth (local/self-managed):
```json
{
  "opensearch-mcp-server": {
    "command": "uvx",
    "args": ["opensearch-mcp-server-py@latest"],
    "env": {
      "OPENSEARCH_URL": "<endpoint_url>",
      "OPENSEARCH_USERNAME": "<username>",
      "OPENSEARCH_PASSWORD": "<password>",
      "OPENSEARCH_SSL_VERIFY": "false",
      "FASTMCP_LOG_LEVEL": "ERROR"
    }
  }
}
```

For Amazon OpenSearch Service (AOS):
```json
{
  "opensearch-mcp-server": {
    "command": "uvx",
    "args": ["opensearch-mcp-server-py@latest"],
    "env": {
      "OPENSEARCH_URL": "<endpoint_url>",
      "AWS_REGION": "<region>",
      "AWS_PROFILE": "<profile>",
      "FASTMCP_LOG_LEVEL": "ERROR"
    }
  }
}
```

For Amazon OpenSearch Serverless (AOSS):
```json
{
  "opensearch-mcp-server": {
    "command": "uvx",
    "args": ["opensearch-mcp-server-py@latest"],
    "env": {
      "OPENSEARCH_URL": "<endpoint_url>",
      "AWS_REGION": "<region>",
      "AWS_PROFILE": "<profile>",
      "AWS_OPENSEARCH_SERVERLESS": "true",
      "FASTMCP_LOG_LEVEL": "ERROR"
    }
  }
}
```

## Critical Rules (MUST follow)

1. **Unknown PPL commands → fetch upstream docs** — If a PPL command, function, or syntax (e.g., `explain`, `graphLookup`) is NOT documented in [ppl-reference.md](../ppl-reference.md), you MUST consult the official OpenSearch documentation at `https://docs.opensearch.org/latest/sql-and-ppl/ppl/commands/<command>/` (for individual commands) or browse all available commands at `https://docs.opensearch.org/latest/sql-and-ppl/ppl/commands/index/`. NEVER guess or invent PPL syntax or parameter names. NEVER claim a command does not exist without checking docs first. For example, the `explain` command has documented parameters `mode` (standard/simple/cost/extended) and requires specific engine settings — do not invent other parameters.
2. **Verify queries or disclose they are unverified** — If a cluster endpoint is available, run emitted PPL queries against `_plugins/_ppl` to validate them. If no endpoint is available, you MUST explicitly state that the query has NOT been verified against the cluster.

## Key Rules

- **Discovery first** — never assume index patterns or field names. Discover them.
- Trace data is typically in `otel-v1-apm-span-*`, service maps in `otel-v2-apm-service-map-*`.
- Always backtick-quote dotted field names: `` `attributes.gen_ai.operation.name` ``
- Use PPL as the primary query language.
- Use `head N` to limit results on large trace indices.
- **Unknown commands → upstream docs.** If a PPL command or function isn't in [ppl-reference.md](../ppl-reference.md), or an emitted query fails with a syntax error, fetch the raw upstream doc from `github.com/opensearch-project/sql` under `docs/user/ppl/` before answering. See [ppl-reference.md](../ppl-reference.md) "Looking Up PPL Documentation" for exact URL patterns.
- **Verify queries when an endpoint is available — best-effort cascade.** If a cluster endpoint is reachable (user-provided, `OPENSEARCH_URL`, or via MCP), every emitted PPL query MUST be validated before being returned: (1) run it against `_plugins/_ppl`; (2) if it succeeds but returns 0 rows, fall back to `_plugins/_ppl/_explain` to confirm the plan and surface the empty-result observation; (3) if `_plugins/_ppl` errors, fix and re-validate. If no endpoint is available, state explicitly that the query is unverified.

## Workflow

### Phase 1 — Connect and Discover

Determine the cluster type and connect. Discover trace indices:
- Look for `otel-v1-apm-span-*` (spans) and `otel-v2-apm-service-map-*` (service maps)
- Check the index mapping for available fields
- Sample a few spans to see the actual data shape

### Phase 2 — Investigate

Based on user intent, build PPL queries:

- **Agent invocations** — `attributes.gen_ai.operation.name` = `invoke_agent`
- **Tool executions** — `attributes.gen_ai.operation.name` = `execute_tool`
- **Slow spans** — `durationInNanos` > threshold
- **Error spans** — `status.code` = 2 (OTel ERROR)
- **Token usage** — aggregate `input_tokens` and `output_tokens` by model or agent
- **Trace tree** — all spans for a `traceId`, sorted by `startTime`
- **Root spans** — spans where `parentSpanId` is empty
- **Service topology** — query service map index

### Phase 3 — Deep Analysis

- **Conversation tracking** — group by `attributes.gen_ai.conversation.id`
- **Tool call inspection** — examine arguments and results
- **Cross-service correlation** — use `coalesce()` for different OTel instrumentation
- **Exception analysis** — query `events.attributes.exception.*` fields

## GenAI Operation Types

| Operation | Description |
|---|---|
| `invoke_agent` | Top-level agent invocation |
| `execute_tool` | Tool execution within agent reasoning |
| `chat` | LLM chat completion call |
| `embeddings` | Text embedding generation |
| `retrieval` | Retrieval operation (e.g., RAG) |
| `create_agent` | Agent creation/initialization |

## Reference Files

| File | Content |
|---|---|
| [traces.md](traces.md) | Trace query templates, field reference, curl examples |
| [ppl-reference.md](../ppl-reference.md) | PPL command + function reference, with upstream-fetch and cluster-validation rules |
Files in this package (2)
  • SKILL.md 7 KB
  • traces.md 9 KB

Versions

RepositoryScoreLicence
This page · opensearch-project/opensearch-agent-skills100Apache-2.0
antgroup/agent3sigma-canary100Apache-2.0

Database snapshot: 13.11.2026 · package version 48a3015478f8 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100dt-obs-tracing
    Distributed traces, spans, service dependencies, and request flow analysis. Use when investigating span-level details, failures, performance bottlenecks, or trace correlation. Trigger: "trace analysis", "slow requests", "failed spans", "service dependencies", "distributed…
  • 100exploring-apm-traces
    Investigates distributed application performance using PostHog APM (OpenTelemetry span) data via MCP. Use when the user asks about service traces, slow HTTP/database spans, error spans, error-rate trends or spikes, latency distributions, trace IDs, or span attributes — not AI…
  • 95exploring-apm-traces
    Investigates distributed application performance using PostHog APM (OpenTelemetry span) data via MCP. Use when the user asks about service traces, slow HTTP/database spans, error spans, error-rate trends or spikes, latency distributions, trace IDs, or span attributes — not AI…
  • 100distributed-tracing-diagnostics
    Diagnose and fix broken distributed traces in .NET — missing spans, broken context propagation across HTTP/messaging, orphaned traces, and missing instrumentation. Use this skill whenever the user has traces that don't connect across services, missing spans, broken trace…
  • 100vercel-tracing
    Set up OpenTelemetry instrumentation for Vercel Tracing and query the resulting spans. Use when adding or fixing `@vercel/otel`, wiring a framework's server startup hook (Next.js, SvelteKit, and others), adding custom spans, turning on spans from libraries such as the AI SDK,…
  • 90distributed-tracing
    Covers following a single request across service boundaries — context propagation, span and attribute design, sampling that keeps the traces worth keeping, and using traces to find where latency actually accumulates. Use this whenever the user debugs a slow request touching…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.