Skip to content
Skills Library
AI & agents · Analysis

gaia-debugging

/gaia-debugging

Diagnose why a GAIA question failed — extract trace, classify failure mode, and propose a fix. Use when a GAIA benchmark run reports a failed/incorrect task_id and you need to root-cause it before resubmitting.

Repository
ruflo ↗
Licence
MIT
Last repository change
25.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 25.10.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Diagnose why a GAIA question failed — extract trace, classify failure mode, and propose a fix. Use when a GAIA benchmark run reports a failed/incorrect task_id and you need to root-cause it before resubmitting.

Arguments

/gaia-debugging <task_id> [--results=<path>]
Original author description

Diagnose why a GAIA question failed — extract trace, classify failure mode, and propose a fix. Use when a GAIA benchmark run reports a failed/incorrect task_id and you need to root-cause it before resubmitting.

How it works

  1. When to use
  2. Failure mode taxonomy
  3. Diagnostic workflow
  4. Quick reference: tool catalogue check
  5. Pattern storage
Original SKILL.md headings · 112 lines, 634 words
SKILL.md preview
---
name: gaia-debugging
description: Diagnose why a GAIA question failed — extract trace, classify failure mode, and propose a fix. Use when a GAIA benchmark run reports a failed/incorrect task_id and you need to root-cause it before resubmitting.
argument-hint: "<task_id> [--results=<path>]"
allowed-tools: Bash Read mcp__plugin_ruflo-core_ruflo__memory_search mcp__plugin_ruflo-core_ruflo__memory_store mcp__plugin_ruflo-core_ruflo__agentdb_pattern-search mcp__plugin_ruflo-core_ruflo__agentdb_pattern-store
---

# GAIA Debugging Skill

When a GAIA question fails, systematically diagnose the root cause and propose
a targeted fix.

## When to use

- A specific `task_id` returns the wrong answer or times out
- Pass-rate dropped between two runs and you need to find the regression
- You want to understand why a particular question class is consistently failing

## Failure mode taxonomy

| Code | Mode | Symptom | Fix direction |
|------|------|---------|--------------|
| TG | Tool Gap | Agent lacks a required tool (no image OCR, no PDF reader) | Add tool to catalogue |
| RM | Reasoning Miss | Agent has the right data but draws wrong conclusion | Improve system prompt, add CoT instruction |
| EB | Extraction Bug | Answer is in the trace but `FINAL_ANSWER:` regex fails | Fix answer extraction pattern |
| LI | Loop Issue | Agent loops (re-asks same tool call) and hits turn limit | Increase max-turns or add loop-detection |
| DS | Dataset Shift | Ground truth differs from what web currently shows | Flag for HAL dataset audit |
| AT | API Timeout | Tool call times out; agent never gets the result | Increase per-turn timeout |

## Diagnostic workflow

### Step 1 — Load the question trace

```bash
# Find the result for the task_id in the latest run
RESULTS=~/.cache/ruflo/gaia/results-latest.json
node -e "
  const r = JSON.parse(require('fs').readFileSync('$RESULTS'));
  const q = r.results.find(x => x.task_id === '$TASK_ID');
  console.log(JSON.stringify(q, null, 2));
"
```

### Step 2 — Classify the failure

Look at the trace output:

1. **No tools called at all** → RM or configuration issue
2. **Tool called but returned error** → TG or AT
3. **Tool returned data, wrong answer** → RM or EB
4. **Correct answer in trace but marked wrong** → EB
5. **max-turns hit** → LI or question too hard for current model

### Step 3 — Re-run with extended logging

```bash
node v3/@claude-flow/cli/bin/cli.js gaia-bench run \
  --level 1 --limit N \
  --models claude-sonnet-4-6 \
  --max-turns 20 \
  --output json
```

`gaia-bench run` cannot select one task: there is no `--task-id` flag (unknown flags are ignored silently). Use the smallest `--limit N` that reaches the question's position in the level's dataset order, and read only the trace for `$TASK_ID` from the output; completed questions are skipped on resume via the checkpoint file.

### Step 4 — Apply targeted fix

| Failure | Action |
|---------|--------|
| TG — missing web_browse | Verify `gaia-tools/index.ts` exports `web_browse`; check tool registration |
| TG — missing image OCR | Add `image_describe` tool call; verify `GOOGLE_AI_API_KEY` |
| RM — reasoning | Add a system prompt instruction: "Before answering, list all facts you have gathered" |
| EB — extraction | Test the `FINAL_ANSWER_RE` regex against the trace manually |
| LI — loop | Add a tool-call deduplication guard in `gaia-agent.ts` |
| AT — timeout | Set `DEFAULT_PER_TURN_TIMEOUT_MS` higher or use `--max-turns` flag |

### Step 5 — Verify fix and store pattern

```bash
# Re-run (no --task-id exists; use the smallest --limit that includes the question)
node … gaia-bench run --level $LEVEL --limit N --models $MODEL --output json

# If now passing, store the pattern
npx @claude-flow/cli@latest memory store \
  --namespace gaia-debug-patterns \
  --key "fix-$FAILURE_CODE-$(date +%Y%m%d)" \
  --value "task_id=$TASK_ID, mode=$FAILURE_CODE, fix=$FIX_DESCRIPTION"
```

## Quick reference: tool catalogue check

```bash
node -e "
  const { createDefaultToolCatalogue } = require('./v3/@claude-flow/cli/src/benchmarks/gaia-tools/index.js');
  const cat = createDefaultToolCatalogue({});
  console.log('Tools registered:', cat.definitions.map(t => t.name));
"
```

Expected: `web_search`, `file_read`, `web_browse`, `image_describe`, `python_exec`

## Pattern storage

After resolving a debugging session, store the finding:
```bash
npx @claude-flow/cli@latest memory store \
  --namespace gaia-debug-patterns \
  --key "session-$(date +%Y%m%d-%H%M)" \
  --value '{"task_id":"$TASK_ID","failure_mode":"$CODE","fix":"$FIX","verified":true}'
```

Search for similar past failures:
```bash
npx @claude-flow/cli@latest memory search \
  --namespace gaia-debug-patterns \
  --query "extraction bug final answer regex"
```

Versions

The database has no other skills named gaia-debugging.

Database snapshot: 13.11.2026 · package version 99e87232ec61 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 95agent-benchmarking
    Map agent capabilities to standard benchmarks (SWE-bench, WebArena, OSWorld, GAIA, TauBench) plus custom regression packs. Use when you need defensible numbers on agent quality, not just vibes — before shipping, after every model change, and for vendor comparisons.
  • 100fa-foundation-optimize-directory-tree
    Provides the whole-tree bootstrap that puts the Gaia context layer onto an existing directory tree — an instruction file and MEMORY.md topic store at every scope that earns one, a root registry, and the retrospective design documentation its projects never had — and re-validates…
  • 100fa-engineering-architecture
    Provides Gaia architecture documentation guidance that keeps docs, README messaging, and design decisions aligned to the current operating model. Use it by updating the relevant files under the consuming repository's docs/architecture/ tree (system components, use cases, class…
  • 100checkpoint
    Expert coverage of Check Point Quantum / Gaia OS across all versions: SmartConsole, layered policy model, ClusterXL HA, Maestro hyperscale, Multi-Domain Management, ThreatCloud AI, NAT, VPN, SecureXL acceleration, Infinity architecture, and mgmt_cli / Web API automation. Use for…
  • 90agentic-reasoning-paper-specialist
    When the user is specifically looking for research papers related to agentic reasoning, autonomous agents, or generalist AI agents. This skill has domain knowledge about key papers, authors, and trends in agentic reasoning research, can identify relevant papers from search…
  • 90runs
    Drive Atoma through its MCP — draft a run goal and acceptance criteria, start or continue a project run, follow it, read its status, files, cost and trace, and explain why a run failed or stopped incomplete. Use whenever the person mentions Atoma, atoma.run, or an Atoma project,…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.