qa

---
name: qa
description: "Smoke test a running Centaur deployment from a new Slack thread. Use when asked to QA the stack, run a smoke test, verify a deployment, check deploy readiness, or prove Slack tools, file upload/download, company context, logs, metrics, and tool loading work end to end."
---

# Centaur QA

Run this skill from a new Slack thread in a channel where the bot is present. The goal is to prove the user-facing agent path works, not just that APIs respond.

Default behavior: start immediately, run the in-thread smoke test, and return a concise pass/fail report. Do not ask clarifying questions unless the current Slack channel or thread cannot be inferred.

If the user asks for deploy readiness, promotion gating, concurrency checks, or broad tool coverage, run the in-thread smoke test first, then the relevant extended checks below.

## Success Criteria

The smoke test passes only when all core workflows work from the running agent session:

- A list of tools loads and includes expected tools.
- Slack file upload to the current thread works.
- Slack file download from the current thread works.
- Slack file download from the current channel, then re-upload to the current thread, works.
- Slack token search works after bounded retries.
- Slack overall message search works.
- Current thread history works.
- Company context connects to its database and returns indexed documents or a valid empty result.
- VictoriaLogs and VictoriaMetrics are reachable, and VictoriaMetrics proves metric existence without asserting exact metric values.

Treat auth, permission, DNS, schema, and timeout errors as failures. Treat empty search results as warnings only when the tool successfully queried the backing service.

For promotion or deploy-readiness requests, the final status is `PASS` only if the requested extended checks also pass or are explicitly accepted by the owner.

## In-Thread Smoke Test

Use direct tool CLIs when available. Use `centaur-tools call <tool> <method> '<json>'` when a tool has no standalone CLI command for the method you need.

Slack file transfers must use the proxied commands, which route through the Centaur API server Slack proxy: `slack upload` and `slack download <file_id> <channel_id>`. Do not use `upload-direct`, `download-direct`, `slack files --download`, `centaur-tools call slack upload_file`, or raw `url_private` fetches; those bypass the proxy and do not prove the user-facing path.

First capture session context:

```bash
echo "THREAD_KEY=${CENTAUR_THREAD_KEY:-}"
echo "SLACK_CHANNEL_ID=${SLACK_CHANNEL_ID:-}"
echo "SLACK_THREAD_TS=${SLACK_THREAD_TS:-}"
echo "SLACK_CHANNEL_NAME=${SLACK_CHANNEL_NAME:-}"
```

If `SLACK_CHANNEL_ID` or `SLACK_THREAD_TS` is missing, infer it from `CENTAUR_THREAD_KEY` when possible. Slack thread keys are usually `slack:<channel_id>:<thread_ts>`.

### 1. Tool Loading

```bash
centaur-tools list
```

Verify that the list is non-empty and includes at least `slack`, `company_context`, `vlogs`, and `vmetrics`. If a tool is absent, record the failure before continuing.

### 2. Upload File To Current Thread

Upload a tiny deterministic file to the current Slack thread through the proxy:

```bash
QA_TOKEN="centaur-qa-$(date +%s)"
QA_UPLOAD_DIR="/tmp/centaur-qa-upload-${QA_TOKEN}"
mkdir -p "$QA_UPLOAD_DIR"
printf '%s\n' "$QA_TOKEN" > "$QA_UPLOAD_DIR/centaur-qa-upload.txt"
slack upload "${SLACK_CHANNEL_ID}" "$QA_UPLOAD_DIR/centaur-qa-upload.txt" \
  --thread "${SLACK_THREAD_TS}" \
  --comment "QA upload smoke test: ${QA_TOKEN}"
```

Verify the command prints `Uploaded centaur-qa-upload.txt` and a Slack file ID (`F...`). Save the file ID for the next step. If no file ID is printed, use thread history in step 3 to find the uploaded file's `id`.

### 3. Download File From Current Thread

If step 2 did not print a file ID, fetch the current thread history and take the uploaded file's `id`:

```bash
slack thread "${SLACK_CHANNEL_ID}:${SLACK_THREAD_TS}" --json --limit 20
```

Download it through the proxy to a sandbox-local temp directory:

```bash
QA_DOWNLOAD_DIR="/tmp/centaur-qa-files-${QA_TOKEN}"
slack download "${FILE_ID_FROM_CURRENT_THREAD}" "${SLACK_CHANNEL_ID}" --output "$QA_DOWNLOAD_DIR"
```

Verify the JSON output includes `output_path` and a non-zero `size_bytes`, and that the downloaded file contains `$QA_TOKEN`.

### 4. Current Thread History

```bash
slack thread "${SLACK_CHANNEL_ID}:${SLACK_THREAD_TS}" --json --limit 20
```

Verify the returned messages include the QA request and the file upload message. Record the number of messages returned.

### 5. Search Uploaded Token

Search for the unique token uploaded in step 2:

```bash
for attempt in 1 2 3; do
  slack search "$QA_TOKEN" --limit 5 --full && break
  sleep 10
done
```

Verify at least one result points to the current channel or current thread. Retry up to three total attempts with 10 seconds between attempts to handle Slack indexing lag. If all three attempts complete successfully but return no matching result, record `WARN: token not indexed yet` and continue.

### 6. Search Overall Messages

Run a separate broader message search for a stable term from the current channel:

```bash
slack search "${SLACK_CHANNEL_NAME:-centaur}" --limit 5 --full
```

Pass when the command succeeds and returns valid search output. Empty results are a warning only when the query reached Slack successfully.

### 7. Download File From Current Channel And Re-Upload

Find another accessible Slack file from the current channel. Prefer a result outside the current thread, but do not use files from other channels. `search_files` filters by filename or title, so start broad with an empty query:

```bash
slack search-files "${SLACK_CHANNEL_ID}" "" --limit 5
```

Pick a result whose `channels` includes `${SLACK_CHANNEL_ID}`. If possible, avoid the file uploaded earlier in this QA run so the check proves download-and-reupload of an existing channel file. Download it through the proxy by file ID:

```bash
QA_REUPLOAD_DIR="/tmp/centaur-qa-reupload-${QA_TOKEN}"
slack download "${FILE_ID_FROM_CURRENT_CHANNEL}" "${SLACK_CHANNEL_ID}" --output "$QA_REUPLOAD_DIR"
```

Save the `output_path` from the JSON output, then re-upload it to the current thread through the proxy:

```bash
slack upload "${SLACK_CHANNEL_ID}" "${DOWNLOADED_FILE_FROM_CURRENT_CHANNEL}" \
  --thread "${SLACK_THREAD_TS}" \
  --comment "QA re-upload from current channel"
```

Verify the current thread now shows the re-uploaded file. If no accessible current-channel file exists, record `SKIP: no readable file in current channel` and include the `search-files` result.

### 8. Company Context

This verifies the database-backed context path and row-level permissions:

```bash
company_context list --limit 3 --json
company_context search "centaur" --limit 3 --json
```

Pass when the tool returns a valid JSON payload with `status: ok`, even if no documents match. Fail on database connection errors, permission errors, missing `COMPANY_CONTEXT_DSN`, or malformed results.

If company context returns `upstream connection failed`, use runtime evidence before suggesting a code fix:

```bash
centaur-tools call vlogs thread_logs '{"thread_key":"'"${CENTAUR_THREAD_KEY}"'","start":"2h"}'
vlogs query '_time:2h ("upstream connection failed" OR "postgres upstream connect failed" OR "failed to fetch secret")' --limit 20 --json
```

Classify whether the failure is a database proxy upstream issue, missing database selection, secret-resolution failure, or tool schema/client error. Include the failing tool command and thread key in the report.

### 9. Logs Via VictoriaLogs

```bash
vlogs health
vlogs query '*' --limit 3 --json
vlogs query "centaur.thread_key:\"${CENTAUR_THREAD_KEY}\"" --limit 10 --json
```

Pass when VictoriaLogs is reachable and returns JSON log entries or a valid empty list for the current-thread query. Fail on DNS, HTTP, or LogsQL errors. Keep this check scoped to the current deployment state visible from the current thread.

Do not run broad error searches such as `level:error`, `vlogs errors`, or `error OR ERROR` as part of the default QA pass/fail decision. The goal of this step is to prove the vlogs connection works, not to audit whether unrelated services currently have errors. Only query error logs when a preceding check failed and you need runtime evidence to classify that specific failure.

### 10. Metrics Via VictoriaMetrics

```bash
vmetrics health
vmetrics query 'count(up)' --json
vmetrics series 'up' --limit 5 --json
```

Pass when VictoriaMetrics is reachable and the responses prove that at least one `up` series exists. Do not use catch-all selectors such as `{__name__=~".+"}`; they scan every series and are too expensive on shared environments. Do not assert exact sample values, counter values, gauge values, label values, or a specific `up` value; those are environment-dependent. Fail on DNS, HTTP, malformed response errors, or a valid response that proves no metric names or series exist.

## Extended Checks

Run these after the core smoke test when the user asks for deploy readiness, concurrency, broad tool coverage, or promotion confidence.

### Tool Health

For broad tool coverage, follow the `tool-health-smoke` skill. It runs `<tool> health` for every live tool CLI through the normal brokered-credential path. Use `health` as the canonical tool smoke surface; only run other read-only commands after recording a failed health result, to classify that failure.

### Concurrent Agent Turns

When asked to check concurrency or deadlocks, start 3-5 QA prompts in separate Slack threads in the same channel. Use distinct `QA_TOKEN` values and ask each agent to do a different read-only task:

- Read thread history and summarize earlier messages.
- Call two tools and summarize grounded results.
- Query logs for its own thread key.
- Search messages for a synthetic token.
- Use company context for a small internal DB lookup.

Pass when every turn reaches a terminal response in Slack, no thread remains stuck busy, and vlogs show one coherent execution per prompt without duplicate final delivery.

### Promotion Gate

A deployment is ready for promotion only when:

- The core in-thread smoke test passes.
- Requested extended checks pass or failures are explicitly accepted by the owner.
- The same commit/build was tested and is the one being promoted.
- The report includes enough evidence for another engineer to verify: Slack permalinks, thread key, execution IDs, and log query windows.

## Report Format

Reply in the Slack thread with a digestible QA report, not a prose paragraph. The
first line must answer the outcome:

```text
Overall: PASS|FAIL|PARTIAL - <one short reason>
```

Use `PASS` only when every required smoke check passed. Use `PARTIAL` when the
user-facing path mostly worked but at least one required check warned, skipped,
or could not be verified. Use `FAIL` when any required check hit an auth,
permission, DNS, schema, timeout, malformed-response, upload/download, or
backing-service error.

Then include a Slack-friendly digest. Do not use Markdown tables in Slack
responses; Slack does not render them reliably. Use this exact shape:

```text
*Setup*
- *Thread context:* PASS - C123:1712345678.000000, key slack:C123:...
- *Tool loading:* PASS - 72 tools; expected slack/company_context/vlogs/vmetrics present

*Slack*
- *Upload current-thread file:* PASS - F123, centaur-qa-upload.txt
- *Download current-thread file:* PASS - centaur-qa-upload.txt, 22 bytes, token matched
- *Re-upload current-channel file:* SKIP - no readable prior file found in channel
- *Search uploaded token:* WARN - 3 attempts, 0 results; likely indexing lag
- *Search overall messages:* PASS - query "centaur", 5 results
- *Current thread history:* PASS - 4 messages; QA request and upload present

*Data + Observability*
- *Company context:* PASS - list 0, search 0, status ok
- *VictoriaLogs:* PASS - health ok, sample 3, thread query 0
- *VictoriaMetrics:* PASS - reachable, `up` series found

*Extended*
- *Tool health:* SKIP - not requested
- *Concurrency:* SKIP - not requested
```

Rules for the digest:

- Keep each evidence phrase to one short sentence fragment. Prefer concrete IDs,
  counts, filenames, byte counts, query names, and attempt counts.
- Do not write `mostly PASS`. Use `PARTIAL` and put each warning or skipped item
  on its own line.
- Do not paste raw JSON, stack traces, credentials, tokens, or long command
  output. Summarize the failure class and the command or endpoint that failed.
- If Slack file upload creates visible artifacts, mention the uploaded filenames
  and file IDs so the user can correlate them with thread attachments.
- If a check is skipped because no suitable input exists, mark only that row
  `SKIP`; do not hide it in prose.
- If using the Slack API directly, a Block Kit version is acceptable only when
  the message still contains the same information: headline, grouped sections,
  per-check result, and evidence. Do not require Block Kit for normal assistant
  replies.

After the grouped digest, add at most three short bullets. Use Slack mrkdwn list items with `- ` prefixes:

- `Failures:` highest-signal failed rows and likely owner.
- `Warnings:` non-blocking warnings such as Slack indexing lag or empty-but-valid
  search/log results.
- `Promotion:` `ready`, `not ready`, or `not evaluated`, with one reason.

Omit any bullet that has no content. The whole report should fit comfortably in
one Slack message.

## Known Gotchas

| Symptom | Likely Cause | Action |
|---------|--------------|--------|
| Missing Slack env vars | Invocation did not come through Slack, or runtime metadata was not injected | Derive from `CENTAUR_THREAD_KEY`; otherwise fail early |
| `slack download` rejects arguments | Passed a URL or permalink instead of a file ID, or the channel is not where the file is shared | Read thread history or `search-files` JSON and pass the file `id` (`F...`) plus the channel ID it is shared in |
| Search cannot find just-uploaded token | Slack search indexing lag | Retry up to three total attempts, then warn and run the separate overall message search |
| `company_context` permission denied | Principal lacks DB-backed reader grant | Report principal/channel and ask owner to grant company context access |
| `company_context` upstream connection failed | Database proxy upstream, database selection, or secret-resolution failure | Check thread logs and vlogs for upstream connect and secret fetch errors before proposing a code change |
| Tool CLI import error | Package entrypoint or relative import packaging regression | Run `<tool> health`, `<tool> --help`, package build checks, and report the broken console script |
| Tool crashes when an API key env var is absent | Client assumes local env auth despite brokered credentials | Re-run `<tool> health` with the env var unset and verify client construction does not raise |
| Tool returns 401/403 | Missing grant, wrong secret, or header not injected by iron-proxy | Check the principal's grants and the tool's `[tool.centaur]` hosts/match rules, then follow the `auth-failure-log-triage` skill |
| `vlogs` or `vmetrics` DNS failure | Observability service unavailable from sandbox | Check local stack or cluster service deployment |
| Expected tools missing | Tool catalog did not load or overlay masked base tools | Report the missing tool names and include `centaur-tools list` output |
| Concurrent runs hang | Runtime assignment, execution queue, or final delivery issue | Check `session_executions`, `session_events`, and the vlogs thread trace |

## Failure Triage

When a flow fails, inspect runtime evidence before redesigning:

- Stuck execution: check `session_executions`, `session_events`, and `centaur-tools call vlogs thread_trace '{"thread_key":"..."}'`.
- Missing Slack response: check Slackbot logs, final delivery state, and the Slack thread surface.
- File failure: check `slack file-info <file_id> <channel_id>`, downloaded byte size, and the proxied upload response.
- Tool failure: classify credential, DNS, upstream, schema, timeout, or runtime errors separately.
- Brokered-auth failure: verify whether the tool should work without local env vars and whether the proxy injected the expected headers.
- Context bug: inspect thread history, requester context, and message ordering.

## References

| Reference | When To Read |
|-----------|--------------|
| [../tool-health-smoke/SKILL.md](../tool-health-smoke/SKILL.md) | Broad all-tool health coverage |
| [../auth-failure-log-triage/SKILL.md](../auth-failure-log-triage/SKILL.md) | A tool fails with an auth or permission error |
| [references/test-inputs.md](references/test-inputs.md) | Only for deeper read-only triage after a tool health check fails |

• SKILL.md 16 KB
• references/test-inputs.md 4 KB