Skip to content
Skills Library
DevOps & cloud · Building & coding

observability-setup

/observability-setup

Set up end-to-end observability for microservices. Use when someone asks to "add tracing", "set up monitoring", "configure OpenTelemetry", "build Grafana dashboards", "distributed tracing", "structured logging", "metrics collection", or "debug production issues". Covers OpenTelemetry instrumentation, collector configuration, Grafana LGTM stack deployment, dashboard provisioning, and alert rules.

Repository
skills ↗
Licence
Apache-2.0
Last repository change
25.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
2
Scripts
0
HTML comments
0
URLs in SKILL.md
7
Deprecation

No signals detected. Repository changed on 25.10.2026, without references to retired models or APIs.

Licence

Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.

What it does and when to use it

Set up end-to-end observability for microservices. Use when someone asks to "add tracing", "set up monitoring", "configure OpenTelemetry", "build Grafana dashboards", "distributed tracing", "structured logging", "metrics collection", or "debug production issues". Covers OpenTelemetry instrumentation, collector configuration, Grafana LGTM stack deployment, dashboard provisioning, and alert rules.

Triggers from the description

add tracingset up monitoringconfigure OpenTelemetrybuild Grafana dashboardsdistributed tracingstructured loggingmetrics collectiondebug production issues
Original author description

Set up end-to-end observability for microservices. Use when someone asks to "add tracing", "set up monitoring", "configure OpenTelemetry", "build Grafana dashboards", "distributed tracing", "structured logging", "metrics collection", or "debug production issues". Covers OpenTelemetry instrumentation, collector configuration, Grafana LGTM stack deployment, dashboard provisioning, and alert rules.

How it works

  1. Overview
  2. Instructions
  3. Examples
  4. Guidelines
Original SKILL.md headings · 201 lines, 922 words
SKILL.md preview
---
name: observability-setup
description: >-
  Set up end-to-end observability for microservices. Use when someone asks to
  "add tracing", "set up monitoring", "configure OpenTelemetry", "build Grafana
  dashboards", "distributed tracing", "structured logging", "metrics collection",
  or "debug production issues". Covers OpenTelemetry instrumentation, collector
  configuration, Grafana LGTM stack deployment, dashboard provisioning, and
  alert rules.
license: Apache-2.0
compatibility: "Node.js 18.19+ or 20.6+, Python 3.10+. Grafana LGTM stack via Docker."
metadata:
  author: terminal-skills
  version: "1.1.0"
  repository: "https://github.com/open-telemetry/opentelemetry-js"
  category: devops
  tags: "observability, opentelemetry, grafana, tracing, monitoring"
---

# Observability Setup

## Overview

This skill helps AI agents implement the three pillars of observability — traces, metrics, and structured logs — across microservice architectures. It uses OpenTelemetry as the instrumentation standard and Grafana's LGTM stack (Loki, Grafana, Tempo, Mimir) as the backend, though patterns apply to any OTel-compatible backend.

## Instructions

### OpenTelemetry Instrumentation (Node.js)

1. Install packages:
   ```bash
   npm install @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node \
     @opentelemetry/exporter-trace-otlp-http @opentelemetry/exporter-metrics-otlp-http
   ```

2. Create `src/instrumentation.ts` — this file must be loaded BEFORE any other imports:
   ```ts
   import { NodeSDK } from '@opentelemetry/sdk-node';
   import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
   import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
   import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http';
   import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';

   // Endpoint and service name come from OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_SERVICE_NAME;
   // the exporters append /v1/traces and /v1/metrics themselves.
   const sdk = new NodeSDK({
     traceExporter: new OTLPTraceExporter(),
     metricReader: new PeriodicExportingMetricReader({
       exporter: new OTLPMetricExporter(),
       exportIntervalMillis: 15000,
     }),
     instrumentations: [getNodeAutoInstrumentations()],
   });
   sdk.start();
   process.on('SIGTERM', () => sdk.shutdown().finally(() => process.exit(0)));
   ```
   (`@opentelemetry/sdk-metrics` is installed as a dependency of the SDK; add it explicitly if your package manager is strict.)

3. Load it before the app. Plain Node cannot `--require` a `.ts` file, so use the compiled file or a TypeScript runner:
   `node --require ./dist/instrumentation.js dist/index.js` or `npx tsx --import ./src/instrumentation.ts src/index.ts`. ESM apps additionally need `--experimental-loader=@opentelemetry/instrumentation/hook.mjs`. Set `OTEL_SEMCONV_STABILITY_OPT_IN=http` to get the stable `http.server.request.duration` metric used in the queries below.

4. Add custom spans for business-critical operations:
   ```ts
   import { trace, SpanStatusCode } from '@opentelemetry/api';
   const tracer = trace.getTracer('payment-service');
   
   async function processPayment(order) {
     return tracer.startActiveSpan('process-payment', async (span) => {
       span.setAttribute('order.id', order.id);
       span.setAttribute('payment.amount_cents', order.totalCents);
       try {
         const result = await paymentGateway.charge(order);
         span.setAttribute('payment.status', result.status);
         return result;
       } catch (error) {
         span.recordException(error);
         span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
         throw error;
       } finally {
         span.end();
       }
     });
   }
   ```

### OpenTelemetry Instrumentation (Python)

1. Install: `pip install opentelemetry-distro opentelemetry-exporter-otlp`, then `opentelemetry-bootstrap -a install` to add instrumentation for the libraries you already use.
2. Run: `opentelemetry-instrument --service_name notification-service python app.py` (endpoint from `OTEL_EXPORTER_OTLP_ENDPOINT`; set `OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true` to export logs too).
3. For structured logs with trace context:
   ```python
   import structlog
   from opentelemetry import trace

   def add_trace_context(logger, method_name, event_dict):
       span = trace.get_current_span()
       ctx = span.get_span_context()
       if ctx.is_valid:
           event_dict['trace_id'] = format(ctx.trace_id, '032x')
           event_dict['span_id'] = format(ctx.span_id, '016x')
       return event_dict

   structlog.configure(processors=[add_trace_context, structlog.dev.ConsoleRenderer()])
   ```

### Structured Logging with Trace Correlation

Every log line must include `trace_id` and `span_id`. Use JSON format:
```json
{"level":"error","msg":"payment failed","trace_id":"abc123...","span_id":"def456...","service":"payment-service","error":"timeout","timestamp":"2026-10-13T10:30:00Z"}
```

For Node.js, use pino with a custom mixin:
```js
const pino = require('pino');
const { trace } = require('@opentelemetry/api');

const logger = pino({
  mixin() {
    const span = trace.getActiveSpan();
    if (span) {
      const ctx = span.spanContext();
      return { trace_id: ctx.traceId, span_id: ctx.spanId };
    }
    return {};
  }
});
```

### OTel Collector Configuration

```yaml
receivers:
  otlp:
    protocols:
      grpc: { endpoint: "0.0.0.0:4317" }
      http: { endpoint: "0.0.0.0:4318" }

processors:
  batch:
    timeout: 5s
    send_batch_size: 1000
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128

exporters:
  otlphttp/tempo:
    endpoint: http://tempo:4318
  prometheusremotewrite:
    endpoint: http://mimir:9009/api/v1/push
  otlphttp/loki:                       # the old `loki` exporter was removed from contrib
    endpoint: http://loki:3100/otlp

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/tempo]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [loki]
```

Put `memory_limiter` first in each pipeline, and use the `otel/opentelemetry-collector-contrib` image (the core image lacks `prometheusremotewrite`). Validate with `otelcol-contrib validate --config=otel-collector-config.yaml`.

### Grafana Dashboard Provisioning

Create dashboards as JSON files in `grafana/dashboards/`. Key panels:

**Service Overview Dashboard:**
- Request rate: `sum(rate(http_server_request_duration_seconds_count[5m])) by (service_name)`
- P95 latency: `histogram_quantile(0.95, sum(rate(http_server_request_duration_seconds_bucket[5m])) by (le, service_name))`
- Error rate: `sum(rate(http_server_request_duration_seconds_count{http_response_status_code=~"5.."}[5m])) / sum(rate(http_server_request_duration_seconds_count[5m]))`

**Alert Rules:**
- Error rate > 5 % for 5 min → P2
- P99 latency > 2s for 5 min → P3
- Service health check failing for 1 min → P1

### Signal Correlation in Grafana

1. Loki data source: add a derived field linking to Tempo, with matcher regex `"trace_id":"(\w+)"` for JSON logs (`trace_id=(\w+)` for logfmt). Logs ingested over OTLP carry the trace ID as structured metadata instead.
2. Tempo data source: enable "Trace to logs" linking to Loki with filter `{service_name="$service"} | trace_id="$traceId"`.
3. Metrics → Logs: use exemplars in Mimir to link metric data points to trace IDs.

## Examples

### Example 1 — Quick OTel setup for Express app

**Input:** "Add observability to my Express API."

**Output:** Create `src/instrumentation.ts` with the Node SDK config above, add the `--require` flag to the start script, set `OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318` and `OTEL_SERVICE_NAME=api-gateway` in the environment. A successful run shows `api-gateway` in Grafana Explore under the Tempo data source within seconds.

### Example 2 — Local backend for development

**Input:** "Give me an observability backend I can run on my laptop."

**Output:** Grafana's all-in-one development image bundles the collector, Prometheus, Tempo, Loki, Pyroscope and Grafana (not for production):
```bash
docker run --rm -p 3000:3000 -p 4317:4317 -p 4318:4318 grafana/otel-lgtm
```
Open http://localhost:3000 (login admin/admin), send telemetry to `http://localhost:4318`, and find traces in Explore with the Tempo data source. For production use the separate Tempo, Loki and Mimir (or Prometheus) deployments behind your own collector config above, with pinned image tags from each project's releases page.

## Guidelines

- **Instrument before you optimize.** You cannot improve what you cannot see.
- **Auto-instrumentation first, custom spans second.** Auto-instrumentation covers 80 % of what you need. Add custom spans only for business-critical paths.
- **Always correlate signals.** A trace without logs is incomplete. A metric spike without a trace is unactionable.
- **Set resource limits on the collector.** Without `memory_limiter`, a traffic spike can OOM the collector and create a cascading failure.
- **Use service.name consistently.** It is the primary grouping key across all three signal types. Mismatched names break correlation.
Files in this package (2)
  • SKILL.md 9 KB
  • _scores.json 3 KB

Versions

RepositoryScoreLicence
This page · TerminalSkills/skills100Apache-2.0
thisiskurubaran/DevSecOps100MIT
devpilotX/ai-skill-ai100MIT
JayRHa/AgentSkills100MIT

Database snapshot: 13.11.2026 · package version 3b9a7ee5bb68 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 95observability
    Use this skill when the user says 'observability', 'logging', 'metrics', 'tracing', 'OpenTelemetry', 'Grafana', 'Prometheus', 'structured logs', 'distributed tracing', 'monitoring', or when setting up observability for a service. Covers: three pillars (logs, metrics, traces),…
  • 90FastAPI Observability
    This skill should be used when the user asks to "add logging", "implement metrics", "add tracing", "configure Prometheus", "setup OpenTelemetry", "add health checks", "monitor API", or mentions observability, APM, monitoring, structured logging, distributed tracing, or Grafana.…
  • 95observability
    OpenTelemetry, distributed tracing, structured logging, metrics (Prometheus, Grafana, Datadog). Use when implementing monitoring, tracing, or debugging production issues.
  • 100agent-observability-engineer
    Specialist subagent: OpenTelemetry, Prometheus, Grafana, distributed tracing, SLO design, and alerting specialist. Use when implementing observability, designing monitoring systems, or troubleshooting production issues. Trigger phrases: observability, monitoring, tracing,…
  • 95observability
    Use when adding structured logging, instrumenting Prometheus metrics, wiring up distributed tracing with OpenTelemetry, writing alerting rules, defining SLOs and error budgets, or building a Grafana golden-signals dashboard.
  • 95observability
    Use when adding structured logging, instrumenting Prometheus metrics, wiring up distributed tracing with OpenTelemetry, writing alerting rules, defining SLOs and error budgets, or building a Grafana golden-signals dashboard.

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.