Przejdź do treści
Biblioteka skilli
DevOps i chmura · Budowanie i kod

monitoring-observability

/monitoring-observability

Dobiera wzorce monitoringu: metryki Prometheus, dashboardy Grafana, śledzenie wywołań LLM przez Langfuse i wykrywanie dryfu jakości.

Repozytorium
orcinus-skills ↗
Licencja
MIT
Ostatnia zmiana repozytorium
11.11.2026

Ocena i bezpieczeństwo

95/ 100Klasa: dobry
Format25 / 25
Frontmatter z nazwą i opisem, poprawna nazwa, opis do 1024 znaków, nazwa zgodna z folderem.
Odkrywalność25 / 25
Opis mówi, kiedy użyć skilla, ma sensowną długość i jest w 3. osobie.
Treść30 / 35
Co najmniej 150 słów i do 500 linii, nagłówki, listy kroków, przykłady albo kod.
Higiena15 / 15
Bez zaślepek typu TODO, bez ścieżek Windows, z licencją.
Ryzykobrak sygnałów w skanie
Pliki w paczce
29
Skrypty
6
Komentarze HTML
1
Adresy URL w SKILL.md
14
Przestarzałość

Brak sygnałów. Repozytorium zmienione 11.11.2026, bez odwołań do wycofanych modeli i API.

Licencja

MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.

Co sprawdzić przed instalacją
  • 6 skryptów w paczce. Przejrzyj je przed instalacją, bo Claude może je uruchomić.
  • SKILL.md zawiera komentarze HTML (1). Nie widać ich w podglądzie Markdown, a model je czyta.

Skrypty: scripts/callback-handler.py, scripts/health-checks.ts, scripts/observe-decorator.py, scripts/opentelemetry-tracing.ts, scripts/prometheus-metrics.ts, scripts/structured-logging.ts

Co robi i kiedy

Dobiera wzorce monitoringu: metryki Prometheus, dashboardy Grafana, śledzenie wywołań LLM przez Langfuse i wykrywanie dryfu jakości.

Oryginalny opis (EN)

Monitoring and observability patterns for Prometheus metrics, Grafana dashboards, Langfuse v4 LLM tracing (as_type, score_current_span, should_export_span, LangfuseMedia), and drift detection. Use when adding logging, metrics, distributed tracing, LLM cost tracking, or quality drift monitoring. Only invoke via the orcinus router.

Jak działa

  1. Upstream coverage (do not restate)
  2. Quick Reference
  3. Quick Start
  4. Infrastructure Monitoring
  5. LLM Observability
  6. Silent Failures
  7. Key Decisions
  8. Detailed Documentation
  9. Related Skills
Sekcje z nagłówków SKILL.md, w oryginale · 137 linii, 1 063 słów
Podgląd SKILL.md
---
name: monitoring-observability
description: Monitoring and observability patterns for Prometheus metrics, Grafana dashboards, Langfuse v4 LLM tracing (as_type, score_current_span, should_export_span, LangfuseMedia), and drift detection. Use when adding logging, metrics, distributed tracing, LLM cost tracking, or quality drift monitoring. Only invoke via the orcinus router.
license: MIT
user-invocable: false
compatibility: Requires Python 3.11+ and the langfuse package for the Langfuse scripts; the LangChain callback example also imports the host application's own config module and is illustrative, not runnable as shipped.
---

# Monitoring & Observability

A wrap around Prometheus, Grafana, OpenTelemetry and Langfuse, not a re-teaching of them. This
skill carries OrchestKit's delta (version floors, house decisions, scars) and points at the
vendor for everything else. Start at `references/ork-delta.md`.

## Upstream coverage (do not restate)

These topics are fully covered first-party. Read the source, do not add a local copy.

| Topic | First-party source |
|-------|--------------------|
| Prometheus metric types, RED method, cardinality, PromQL | <https://prometheus.io/docs/practices/> |
| Alertmanager grouping, inhibition, escalation, runbooks | <https://prometheus.io/docs/alerting/latest/configuration/> |
| Grafana dashboards, Loki and LogQL, Promtail | <https://grafana.com/docs/> |
| OpenTelemetry spans, sampling, context propagation | <https://opentelemetry.io/docs/> |
| Langfuse Python SDK (`@observe`, `as_type`, `score_current_span`, `should_export_span`, `LangfuseMedia`) | <https://langfuse.com/docs/sdk/python> |
| Langfuse v2 to v4 Python and v3 to v5 JS migration paths | <https://langfuse.com/docs/sdk/python/v4-migration> |
| Langfuse self-hosting (ClickHouse, Redis, S3, Helm) | <https://langfuse.com/docs/deployment/self-host> |
| Langfuse cost tracking, model pricing, Metrics API v2 | <https://langfuse.com/docs/model-usage-and-cost> |
| Langfuse scores, online evaluators, annotation queues, prompt management | <https://langfuse.com/docs/scores/overview> |
| Langfuse framework integrations (LangChain, LangGraph, CrewAI, Pydantic AI, Bedrock, LiveKit) | <https://langfuse.com/docs/integrations> |
| Agent Graphs, observation types, rendered tool calls | <https://langfuse.com/docs/tracing-features/agent-graphs> |
| PSI, KS test, KL and JS divergence, Wasserstein, embedding drift | <https://www.evidentlyai.com/blog/data-drift-detection-large-datasets> |
| EWMA control charts | <https://www.itl.nist.gov/div898/handbook/pmc/section3/pmc324.htm> |
| structlog, Winston, correlation IDs, log sampling | <https://www.structlog.org/en/stable/> |

## Quick Reference

| Category | Rules | Impact | When to Use |
|----------|-------|--------|-------------|
| [Infrastructure Monitoring](#infrastructure-monitoring) | 1 | CRITICAL | Grafana dashboards, Golden Signals, SLO/SLI |
| [LLM Observability](#llm-observability) | 1 | HIGH | Langfuse tracing, observation types, agent graphs |
| [Silent Failures](#silent-failures) | 3 | HIGH | Tool skipping, quality degradation, loop/token spike alerting |

**Total: 5 rules across 3 categories.** Drift detection, cost tracking, eval scoring, Prometheus
instrumentation and alert-rule authoring moved to the upstream sources listed above.

## Quick Start

```python
# Langfuse v4 LLM tracing: semantic as_type plus inline scoring
from langfuse import observe, get_client

@observe(as_type="generation", name="analyze_content")
async def analyze_content(content: str):
    get_client().update_current_trace(
        user_id="user_123", session_id="session_abc",
        tags=["production", "orchestkit"],
    )
    result = await llm.generate(content)
    get_client().score_current_span(name="response_quality", value=0.85)
    return result
```

```python
# Prometheus RED method, wired the way this repo expects (bounded labels only)
from prometheus_client import Counter, Histogram

http_requests = Counter('http_requests_total', 'Total requests', ['method', 'endpoint', 'status'])
http_duration = Histogram('http_request_duration_seconds', 'Request latency',
    buckets=[0.01, 0.05, 0.1, 0.5, 1, 2, 5])
```

## Infrastructure Monitoring

Dashboard and health-check patterns. Metric instrumentation and alert-rule syntax are upstream.

| Rule | File | Key Pattern |
|------|------|-------------|
| Grafana Dashboards | `rules/monitoring-grafana.md` | Golden Signals, SLO/SLI, health checks |

> **CC 2.1.161 — OTEL resource attributes as metric labels:** `OTEL_RESOURCE_ATTRIBUTES` values are now attached as labels on metric datapoints, so usage metrics can be sliced by custom dimensions (team, repo, environment). Add label selectors to dashboards for multi-tenant / per-team cost and usage tracking.

## LLM Observability

Langfuse-based tracing for LLM applications. Cost tracking, scoring and drift statistics are
upstream; what stays here is how this repo wires traces.

| Rule | File | Key Pattern |
|------|------|-------------|
| Langfuse Traces | `rules/llm-langfuse-traces.md` | @observe decorator, OTEL spans, agent graphs |

## Silent Failures

Detection and alerting for silent failures in LLM agents.

| Rule | File | Key Pattern |
|------|------|-------------|
| Tool Skipping | `rules/silent-tool-skipping.md` | Expected vs actual tool calls, Langfuse traces |
| Quality Degradation | `rules/silent-degraded-quality.md` | Heuristics + LLM-as-judge, z-score baselines |
| Silent Alerting | `rules/silent-alerting.md` | Loop detection, token spikes, escalation workflow |

> **CC 2.1.169 — OTEL client-cert paths require trust:** untrusted project settings can no longer set OTEL client-certificate paths without a trust confirmation. If your OTEL exporter uses client certs configured in project `.claude/settings.json`, expect a one-time trust prompt on first use in an untrusted project — telemetry silently not flowing after 2.1.169 is usually this gate, not the collector.

## Key Decisions

| Decision | Recommendation | Rationale |
|----------|----------------|-----------|
| Metric methodology | RED method (Rate, Errors, Duration) | Industry standard, covers essential service health |
| Log format | Structured JSON | Machine-parseable, supports log aggregation |
| Tracing | OpenTelemetry | Vendor-neutral, auto-instrumentation, broad ecosystem |
| LLM observability | Langfuse (not LangSmith) | Open-source, self-hosted, built-in prompt management |
| LLM tracing API | `@observe(as_type=...)` + `score_current_span()` | v4: semantic types, inline scoring, span filtering |
| Langfuse APIs | Observations API v2 + Metrics API v2 | v4 (Mar 2026): faster querying, aggregations at scale |
| Hook telemetry transport | JSONL under `~/.claude/analytics/`, never an SDK in-process | Hooks are per-event processes; SDK init would be paid on every spawn (`references/ork-delta.md`) |

## Detailed Documentation

| Resource | Description |
|----------|-------------|
| `references/ork-delta.md` | **Start here.** Floors, house decisions and scars that upstream docs do not carry |
| `references/langfuse-js-v5.md` | **JS/TS SDK v5** delta from Python 4.x: package map, phantom packages, SpanProcessor vs exporter. Read before writing any JS Langfuse code |
| `references/experiments-api.md` | Langfuse experiments and dataset runs as this repo uses them |
| `references/evaluation-scores.md` | Score shapes and scoring pipeline wiring |
| `references/session-tracking.md` | Session and user grouping across multi-step workflows |
| `references/metrics-collection.md` | Claude Code OTEL metric inventory and collector-side joins |
| `references/dashboards.md` | Dashboard layout conventions |
| `references/structured-logging.md` | Structured log field conventions |
| `references/dev-agent-lens.md` | LiteLLM proxy layer (or Bifrost) for API-boundary observability |
| `examples/orchestkit-monitoring-dashboard.md` | Worked monitoring dashboard example |
| `scripts/` | Templates: Prometheus, OpenTelemetry, health checks, Langfuse |

The Langfuse scripts in `scripts/` are host-app examples, not runnable as shipped: `observe-decorator.py` needs the `langfuse` package (`pip install langfuse`) and a configured Langfuse host, and `callback-handler.py` additionally needs `langchain-anthropic` and imports the host application's own `app.core.config` module — copy it into your app and point the import at your own config.

## Related Skills

<!-- ORCINUS change notice (2026-11-08, round 7, N5): upstream
listed five OrchestKit siblings here (defense-in-depth,
devops-deployment, resilience-patterns, llm-evaluation, caching);
none of them exists in ORCINUS, so the links pointed nowhere in
every install. Replaced with the in-repo sibling `monitoring`.
See ATTRIBUTION.md. -->

- `monitoring` - The outside view of a shipped service: uptime probes, SLO burn-rate alerts, on-call escalation. The complement of this skill — this skill instruments from the inside (what the app emits: logs, metrics, traces), `monitoring` detects from the outside (is it alive, is it fast). The router routes "Health / monitoring" to `monitoring` and "Telemetry / instrumentation" here.
Pliki w paczce (29)
  • ATTRIBUTION.md 5 KB
  • LICENSE.txt 1 KB
  • SKILL.md 9 KB
  • metadata.json 1 KB
  • test-cases.json 3 KB
  • examples/orchestkit-monitoring-dashboard.md 14 KB
  • references/dashboards.md 1 KB
  • references/dev-agent-lens.md 6 KB
  • references/evaluation-scores.md 7 KB
  • references/experiments-api.md 14 KB
  • references/langfuse-js-v5.md 5 KB
  • references/metrics-collection.md 7 KB
  • references/ork-delta.md 5 KB
  • references/session-tracking.md 7 KB
  • references/structured-logging.md 1 KB
  • rules/_sections.md 1 KB
  • rules/_template.md 1 KB
  • rules/llm-langfuse-traces.md 6 KB
  • rules/monitoring-grafana.md 3 KB
  • rules/silent-alerting.md 2 KB
  • rules/silent-degraded-quality.md 3 KB
  • rules/silent-tool-skipping.md 2 KB
  • scripts/alerting-rules.yml 3 KB
  • scripts/callback-handler.py 9 KB
  • scripts/health-checks.ts 3 KB
  • scripts/observe-decorator.py 6 KB
  • scripts/opentelemetry-tracing.ts 4 KB
  • scripts/prometheus-metrics.ts 3 KB
  • scripts/structured-logging.ts 3 KB

Wersje

RepozytoriumWynikLicencja
Ta strona · DnsQueen/orcinus-skills95MIT
travisjneuman/.claude100MIT
komluk/scaffolding100MIT
yonatangross/orchestkit100MIT
yonatangross/orchestkit100MIT
mawero83/google-sre-skills100MIT
autohandai/community-skills100Apache-2.0
akillness/jeo-skills95MIT

Migawka bazy z 13.11.2026 · wersja paczki 98fbb953d737 (skrót SHA-256 całej paczki)

Pobrania w ostatnich 30 dniach: 0 · ranking

Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.

Podobne skille

  • 90observability-engineer
    Konfiguruje monitoring, logowanie, metryki, śledzenie i alerty, w tym dashboardy Prometheus i Grafana, OpenTelemetry i testy obciążeniowe.
  • 95observability
    Wdraża OpenTelemetry, śledzenie rozproszone, logi strukturalne i metryki w Prometheus, Grafanie lub Datadog.
  • 90FastAPI Observability
    Podaje wzorce obserwowalności FastAPI: logowanie strukturalne, metryki, tracing rozproszony, Prometheus, OpenTelemetry, Grafana, health checki i monitorowanie API.
  • 100agent-observability-engineer
    Wspiera obserwowalność: OpenTelemetry, Prometheus, Grafana, śledzenie rozproszone, projekt SLO i alerty przy diagnozowaniu problemów produkcyjnych.
  • 100monitoring-expert
    Konfiguruje systemy monitoringu, potoki logów, dashboardy Prometheus i Grafana, reguły alertów oraz śledzenie rozproszone i testy obciążeniowe.
  • 91observability-engineering
    Konfiguruje monitoring i OpenTelemetry, projektuje SLO, alerty, dashboardy Grafana i Prometheus oraz logi.

Uwagi i zgłoszenia

Wiadomość trafi do Bartosza Sokolińskiego. Adres e-mail jest potrzebny tylko do odpowiedzi. Szczegóły w polityce prywatności.