multi-model-validation

---
name: multi-model-validation
description: Runs the same task across multiple AI models in parallel and aggregates verdicts. Use when the user wants a second opinion, multi-expert validation, or consensus from Grok, Gemini, GPT, or Kimi.
user-invocable: false
---

# Multi-Model Validation

**Version:** 3.3.0
**Purpose:** Patterns for running multiple AI models in parallel via Claudish proxy with **context-aware preferences**, dynamic model discovery, session-based workspaces, and performance statistics
**Status:** Production Ready

## Overview

Multi-model validation is the practice of running multiple AI models (Grok, Gemini, GPT, DeepSeek, etc.) in parallel to validate code, designs, or implementations from different perspectives. This achieves:

- **3-5x speedup** via parallel execution (15 minutes → 5 minutes)
- **Consensus-based prioritization** (issues flagged by all models are CRITICAL)
- **Diverse perspectives** (different models catch different issues)
- **Cost transparency** (know before you spend)
- **Free model discovery** (NEW v3.0) - find high-quality free models from trusted providers
- **Performance tracking** - identify slow/failing models for future exclusion
- **Data-driven recommendations** - optimize model shortlist based on historical performance

**Key Innovations:**

1. **Context-Aware Preferences** (NEW v3.3.0) - Automatically use saved model preferences per task type (debug/research/coding/review) from `.claude/multimodel-team.json`
2. **Dynamic Model Discovery** (v3.0) - Read the live catalog (`list_models`) for current available models (live, 24h cache)
3. **Session-Based Workspaces** (v3.0) - Each validation session gets a unique directory to prevent conflicts
4. **4-Message Pattern** - Ensures true parallel execution by using only Agent tool calls in a single message
5. **Pattern 7-8** - Statistics collection and data-driven model recommendations

This skill is extracted from the `/review` command and generalized for use in any multi-model workflow.

---

## ⚠️ MANDATORY: Learn and Reuse User Preferences

> **Model preferences are learned per context and reused automatically.**
>
> - First time a context is used → ASK user → SAVE to that context
> - Next time same context → VALIDATE the saved IDs against `list_models`, then use
>   the survivors automatically (no asking). "No asking" applies to the *selection*,
>   never to the catalog check — saved IDs go stale and must be re-checked every run.
> - User explicitly says "change models" or "different models" → ASK and UPDATE

```bash
# FIRST STEP - Read preferences file
cat .claude/multimodel-team.json 2>/dev/null
```

**Flow:**

```
1. Detect context from task keywords
   - "debug", "error", "bug", "fix" → debug
   - "research", "analyze", "investigate" → research
   - "implement", "build", "create", "code" → coding
   - "review", "audit", "check" → review

2. Check if contextPreferences[context] exists and is non-empty

   IF EXISTS (has models saved):
   → Call: list_models (claudish MCP) and KEEP ONLY the saved IDs it still lists
     Saved preferences are user policy, not a catalog snapshot — they go stale
     silently. This applies to defaultModels and contextPreferences alike; see
     claudish:claudish-usage → "Every field of the preferences file is untrusted"
   → Name every dropped ID in your reply
   → DO NOT ask the user to re-pick while at least one saved ID survives
   → If NOTHING survives, say so and offer live alternatives

   IF EMPTY/MISSING (first time for this context):
   → Call: list_models (claudish MCP — current models, pricing, capabilities)
   → Ask user to select models (AskUserQuestion)
   → Save to contextPreferences[context]
   → Proceed with validation

3. User override triggers (explicit request to change):
   - "use different models"
   - "change models"
   - "update model preferences"
   → Ask user to select new models
   → Update contextPreferences[context]
```

**Example - Learning Flow:**

```
# First debug task ever:
Task: "Debug this authentication error"
→ Context: debug
→ contextPreferences.debug is empty
→ ASK: "Which models for debug tasks?"
→ User selects: grok, glm, minimax
→ SAVE to contextPreferences.debug
→ Run with those models

# Second debug task:
Task: "Debug the API timeout"
→ Context: debug
→ contextPreferences.debug = ["grok", "glm", "minimax"]
→ USE directly (no asking)
→ Run with saved models

# User wants to change:
Task: "Debug this error, use different models"
→ Detected: "different models" override trigger
→ ASK: "Which models for debug tasks?"
→ User selects: gemini, <model-c>
→ UPDATE contextPreferences.debug
→ Run with new models
```

---

## Related Skills

**Cross-References:**

- **multimodel:error-recovery** - Handling failures: diagnosis, reporting, user-chosen re-runs

---

## Core Patterns

### Pattern 0: Session Setup and Model Discovery (NEW v3.0)

**Purpose:** Create isolated session workspace and discover available models dynamically.

**Why Session-Based Workspaces:**

Using a fixed directory like `ai-docs/reviews/` causes problems:
- ❌ Multiple sessions overwrite each other's files
- ❌ Stale data from previous sessions pollutes results
- ❌ Hard to track which files belong to which session

Instead, create a **unique session directory** for each validation:

```bash
# Generate unique session ID
TARGET_SLUG=$(echo "${TASK_NAME:-review}" | tr '[:upper:] ' '[:lower:]-' | sed 's/[^a-z0-9-]//g' | head -c20)
SESSION_ID="review-${TARGET_SLUG}-$(date +%Y%m%d-%H%M%S)-$(head -c 4 /dev/urandom | xxd -p)"
SESSION_DIR="ai-docs/sessions/${SESSION_ID}"

# Create session workspace
mkdir -p "$SESSION_DIR"

echo "Session: $SESSION_ID"
echo "Directory: $SESSION_DIR"

# Example output:
# Session: review-auth-impl-20251212-143052-a3f2
# Directory: ai-docs/sessions/review-auth-impl-20251212-143052-a3f2
```

**Benefits:**
- ✅ Each session is isolated (no cross-contamination)
- ✅ Traceable - can associate files with a specific session
- ✅ Session ID can be used for tracking in statistics
- ✅ Parallel sessions don't conflict
- ✅ Aligned with the `dev:dev` session pattern
- ✅ Git-ignored session scratch: it dies with the session and is never committed

> Use `ai-docs/sessions/`, not `/tmp/`: a `/tmp/` path is not tied to the project, and
> the `/team` hook denies `/tmp/` paths in agent prompts.

---

**Dynamic Model Discovery:**

**NEVER hardcode model lists.** Models change frequently — new ones appear, old ones deprecate, pricing updates. Instead, read the live catalog (`list_models`) for current available models:

Call the `list_models` MCP tool (claudish). It returns the current recommended
set — model IDs, pricing, context window, capabilities, and the `provider@model`
access prefixes — served from claudish’s catalog with a 24-hour cache.

For every live variant in one family, call `search_models` with the family name.

**Free models:** filter the `list_models` result by pricing. Keep no list here — free tiers
churn faster than any other part of the catalog.

**Model Selection Flow (Learn and Reuse):**

```
1. Read Preferences File
   → cat .claude/multimodel-team.json
   → If file NOT exists → create empty one

2. Detect Task Context
   → Parse task for keywords (case-insensitive):
     - "debug", "error", "bug", "fix", "trace", "issue" → debug
     - "research", "investigate", "analyze", "explore", "find" → research
     - "implement", "build", "create", "code", "develop", "feature" → coding
     - "review", "audit", "check", "validate", "verify" → review
   → If no keywords match → context = "default"

3. Check for Override Triggers in User Message
   → "use different models", "change models", "update preferences"
   → If found → force_ask = true

4. Load or Learn Models
   → models = contextPreferences[context]

   IF models exist AND NOT force_ask:
     → USE models directly (no asking)
     → Go to step 6

   IF models empty OR force_ask:
     → Read: the live catalog (list_models)
     → AskUserQuestion with multiSelect
     → Save user selection to contextPreferences[context]
     → Go to step 6

5. Save Updated Preferences
   → Write .claude/multimodel-team.json
   → Update lastUpdated timestamp

6. Execute with Models
   → Launch parallel validation
   → No further confirmation needed
```

**Context Keywords:**

| Context | Keywords |
|---------|----------|
| debug | debug, error, bug, fix, trace, issue |
| research | research, investigate, analyze, explore, find |
| coding | implement, build, create, code, develop, feature |
| review | review, audit, check, validate, verify |

**Override Triggers (force re-selection):**
- "use different models"
- "change models"
- "update model preferences"
- "select new models"

### Routing is Claudish's

**Send the `id` from `list_models`. Never build an address.** Claudish owns backend
selection, credentials and fallback; this repo implements none of it.

A prefix/backend/key table used to sit here. It is deleted — it had drifted to the wrong
separator (`/` where claudish uses `@`), listed alias env vars as if canonical, covered a
third of the providers, and marked models "collision-free" that had since gained a direct
provider. A second copy in `claudish-usage` had drifted differently, which is the point:
restating claudish's routing anywhere in this repo guarantees two versions of the truth and
no way to tell which is stale.

If a model will not route, that is a claudish bug — report it with `report_error`. Do not
work around it by choosing a different prefix here.

**Interactive Model Selection (AskUserQuestion with multiSelect):**

**CRITICAL:** Use AskUserQuestion tool with `multiSelect: true` to let users choose models interactively. This provides a better UX than just showing recommendations.

```typescript
// Use AskUserQuestion to let user select models
AskUserQuestion({
  questions: [{
    question: "Which external models should validate your code? (Internal Claude reviewer always included)",
    header: "Models",
    multiSelect: true,
    options: [
      // One option per model, built from this run's list_models result plus
      // ai-docs/llm-performance.json history when it exists. Illustrative shape:
      {
        label: "<id from list_models>",
        description: "<price from list_models> | Quality: <history %, or 'new'> | Avg: <history s>"
      },
      {
        label: "<free id from list_models> 🆓",
        description: "FREE | <context from list_models> | Quality: <history %, or 'new'>"
      }
    ]
  }]
})
```

**Remember Selection for Session:**

Store the user's model selection in the session directory so it persists throughout the validation:

```bash
# After user selects models, save to session
save_session_models() {
  local session_dir="$1"
  shift
  local models=("$@")

  # Always include internal reviewer
  echo "claude-embedded" > "$session_dir/selected-models.txt"

  # Add user-selected models
  for model in "${models[@]}"; do
    echo "$model" >> "$session_dir/selected-models.txt"
  done

  echo "Session models saved to $session_dir/selected-models.txt"
}

# Load session models for subsequent operations
load_session_models() {
  local session_dir="$1"
  cat "$session_dir/selected-models.txt"
}

# Usage:
# After AskUserQuestion returns selected models
save_session_models "$SESSION_DIR" "grok" "<free-model-a>"

# Later in the session, retrieve the selection
MODELS=$(load_session_models "$SESSION_DIR")
```

**Session Model Memory Structure:**

```
$SESSION_DIR/
├── selected-models.txt    # User's model selection (persists for session)
├── claude-review.md       # Internal review
├── grok-review.md         # External review (if selected)
├── free-model-a-review.md   # External review (if selected)
└── consolidated-review.md # Final consolidated review
```

**Why Remember the Selection:**

1. **Re-runs**: If validation needs to be re-run, use same models
2. **Consistency**: All phases of validation use identical model set
3. **Audit trail**: Know which models produced which results
4. **Cost tracking**: Accurate cost attribution per session

**Always Include Internal Reviewer:**

```
BEST PRACTICE: Always run internal Claude reviewer alongside external models.

Why?
✓ FREE (embedded Claude, no API costs)
✓ Fast baseline (usually fastest)
✓ Provides comparison point
✓ Works even if ALL external models fail
✓ Consistent behavior (same model every time)

The internal reviewer should NEVER be optional - it's your safety net.

It runs in a session of its own, never as a subagent of the session that wrote the code —
except on the nested route below, `cli-sessions.md` §3, the one place a subagent is allowed:
seat it as the `internal` slot in the `models` list of the ONE `team` call, beside the
externals, so it gets the same `require_pattern` shape check as every one of them. After
the panel settles, copy `response-<slots.internal>.md` to `claude-review.md` — only when
that slot is COMPLETED and matched the pattern — and list `claude-review.md` on REVIEWS:,
never `response-<slots.internal>.md` beside it: that is the same review counted twice.
Without the `team` tool, pick the route by §1 of `skills/core/team-gate/cli-sessions.md`
inside the dev plugin, never by "is claudish installed": inside a claudish child `team` is
absent too, and a CLI launched from there rebuilds the nesting claudish refuses. When §1
picks the nested route, follow its §3 — the one foreground subagent, no externals. When it
picks cli-sessions, run the internal reviewer alone as a `claude` CLI session; this skill
launches no external CLIs, and that file's §4.1 says the same. dev's root is the `installPath` of its `dev@` entry in
`claude plugin list --json`; with no such entry dev is not installed, and the route is
unavailable. Pattern 3 states the rule.
```

---

### Pattern 1: The 4-Message Pattern (MANDATORY)

**Why This Pattern Exists:** independent calls issued in one message run together; calls
split across messages wait for each other. So preparation comes first, every reviewer
starts in one message, and consolidation waits until the panel settles. What decides
whether two calls may share a message is data dependency, not tool type — and every
reviewer, the internal one included, is a slot of the one `team` call in Message 2.

**The Pattern:**

```
Message 1: Preparation (Bash Only)
  - Create workspace directories
  - Validate inputs (check the claudish `team` tool is registered)
  - Write the brief (input.md) — never a pre-computed diff: every reviewer is
    handed `TARGET: BRANCH` and captures its own surfaces through dev's
    `capture-review-surfaces.ts`. No `OUTPUT:` line; it ends
    `Return the full report as your answer.`
  - NO Agent calls

Message 2: Parallel Execution (ONE team call)
  - `internal` and every external model in a single `team` MCP call; the tool
    parallelises them internally. `internal` is always present (Pattern 3)
  - Pass require_pattern whenever the prompt mandates an output shape

Message 3: Auto-Consolidation (one Agent call)
  - Automatically triggered when the panel settles with N ≥ 2 reviews. At N = 1 there
    is nothing to consolidate: use that review's own verdict directly (Pattern 5)
  - Launch `dev:aggregator` — the only consolidator; it reads reviews, never code
  - Pass every review file path on REVIEWS: — `claude-review.md`, then one
    `response-<slot>.md` per external slot, never `slots.internal` — the three lines under 'Apply verdict
    thresholds' in `dev:reviewer`'s agent file on THRESHOLDS: — read at dispatch
    time, never recalled — and the consolidated file on OUTPUT: (Pattern 5)
  - Consensus analysis is the aggregator's; never send the reviews to dev:reviewer

Message 4: Present Results
  - Show user prioritized issues
  - Include consensus levels (unanimous, strong, majority)
  - Link to detailed reports
  - Cost summary (if applicable)
```

**Example: 5-Model Parallel Code Review**

```
Message 1: Preparation (Session Setup + Model Discovery)
  # Create unique session workspace
  Bash: SESSION_ID="review-$(date +%Y%m%d-%H%M%S)-$(head -c 4 /dev/urandom | xxd -p)"
  Bash: SESSION_DIR="ai-docs/sessions/${SESSION_ID}" && mkdir -p "$SESSION_DIR"
  # No code capture here. Every reviewer is handed TARGET: BRANCH and runs dev's
  # capture-review-surfaces.ts itself, in BRANCH mode. A range computed here would
  # be one more hand-rolled diff, which is the one thing no dispatcher may do.

  # Discover available models
  MCP:  list_models   # current models, pricing, capabilities

  # User selects models via AskUserQuestion (see Pattern 0)

Message 2: Start the panel (ONE team call)
  Bash: write the brief to "$SESSION_DIR/input.md" — the contract lines every
        reviewer gets: TARGET: BRANCH / FOCUS: code / MODELS: none, no OUTPUT:
        line, ending "Return the full report as your answer."

  claudish team(mode="run", path=$SESSION_DIR,
    models=["internal", "grok", "<free-model-a>", "gpt", "<free-model-c>"],
    input_file="$SESSION_DIR/input.md",
    require_pattern="\*\*Verdict\*\*: (PASS|CONDITIONAL|FAIL)", agent="dev:reviewer")

  All 5 reviewers run at once as slots of one call: `internal` is the
  always-present internal reviewer, shape-checked like the four externals.

  The team call RETURNS IMMEDIATELY with a slots map. It does not carry the reviews.

Message 2b: Poll to completion
  claudish team(mode="status", path=$SESSION_DIR)
  # repeat until no slot in `models` has state === "RUNNING"
  # bound the loop; read idle_seconds_by_slot with activity_by_slot before
  # concluding a quiet slot is stuck

Message 3: Auto-Consolidation
  (Automatically triggered - don't wait for user to request)

  # The team run wrote one file per slot, $SESSION_DIR/response-NN.md, named by
  # ANONYMOUS slot id rather than by model, because the vote is blind. Do not try
  # to attribute a file to a model before the verdict is in. Copy
  # response-<slots.internal>.md to $SESSION_DIR/claude-review.md if that slot
  # COMPLETED and matched. The aggregator is the only consolidator: it is given the
  # reviews and never the code, and no reviewer ever sees another reviewer's output.

  Agent(
    subagent_type: "dev:aggregator",
    run_in_background: false,
    description: "Consolidate code reviews",
    prompt: "REVIEWS: $SESSION_DIR/claude-review.md
             $SESSION_DIR/response-<slot>.md   (one line per external slot from the slots map that completed — never slots.internal)
             THRESHOLDS: <the three lines under 'Apply verdict thresholds' in
                          dev:reviewer's agent file, read at dispatch time, never
                          recalled>
             OUTPUT: $SESSION_DIR/consolidated-review.md
             Consolidate with consensus levels (unanimous / strong / majority / divergent).
             Compute the verdict line from your counts against THRESHOLDS.
             You are given reviews, never code. Do not review."
  )

Message 4: Present Results + Update Statistics
  # Track performance for each model (see Pattern 7)
  track_model_performance "claude-embedded" "success" 32 8 95
  track_model_performance "grok" "success" 45 6 87
  track_model_performance "<free-model-a>" "success" 52 5 82
  track_model_performance "gpt" "success" 68 7 89
  track_model_performance "<free-model-b>" "success" 48 5 84

  # Record session summary
  record_session_stats 5 5 0 68 245 3.6

  "Multi-model code review complete! 5 AI models analyzed your code.
   Session: $SESSION_ID

   Top 5 Issues (Prioritized by Consensus):
   1. [UNANIMOUS] Missing input validation on POST /api/users
   2. [UNANIMOUS] SQL injection risk in search endpoint
   3. [STRONG] Weak password hashing (bcrypt rounds too low)
   4. [MAJORITY] Missing rate limiting on authentication endpoints
   5. [MAJORITY] Insufficient error handling in payment flow

   Model Performance (this session):
   | Model                          | Time | Issues | Quality | Cost   |
   |--------------------------------|------|--------|---------|--------|
   | claude-embedded                | 32s  | 8      | 95%     | FREE   |
   | grok          | 45s  | 6      | 87%     | $0.002 |
   | <free-model-a>          | 52s  | 5      | 82%     | FREE   |
   | gpt        | 68s  | 7      | 89%     | $0.015 |
   | <free-model-b>   | 48s  | 5      | 84%     | FREE   |

   Parallel Speedup: 3.6x (245s sequential → 68s parallel)

   See $SESSION_DIR/consolidated-review.md for complete analysis.
   Performance logged to ai-docs/llm-performance.json"
```

**Performance Impact:**

- Sequential execution: 5 models × 3 min = 15 minutes
- Parallel execution: max(model times) ≈ 5 minutes
- **Speedup: 3x with perfect parallelism**

---

### Pattern 2: Parallel Execution Architecture

**Single Message, Multiple Tasks:**

The key to parallel execution is putting ALL Agent calls in a **single message** with the `---` delimiter:

```
✅ CORRECT - Parallel Execution:

Agent: <agent-1>
  Prompt: "Task 1 instructions"
---
Agent: <agent-2>
  Prompt: "Task 2 instructions"
---
Agent: <agent-3>
  Prompt: "Task 3 instructions"

All 3 execute simultaneously.
```

**Anti-Pattern: Sequential Execution**

```
❌ WRONG - Sequential Execution:

Message 1:
  Agent: <agent-1>
Message 2:
  Agent: <agent-2>
Message 3:
  Agent: <agent-3>
Each task waits for previous to complete (3x slower).
```

**Independent Tasks Requirement:**

Each Task must be **independent** (no dependencies):

```
✅ CORRECT - Independent:
  Task: review code for security
  Task: review code for performance
  Task: review code for style

  All can run simultaneously (same input, different perspectives).

❌ WRONG - Dependent:
  Task: implement feature
  Task: write tests for feature (depends on implementation)
  Task: review implementation (depends on tests)

  Must run sequentially (each needs previous output).
```

**Unique Output Files:**

Each Task MUST write to a **unique output file** within the session directory:

```
✅ CORRECT - Unique Files in Session Directory:
  Task: reviewer1 → $SESSION_DIR/claude-review.md
  Task: reviewer2 → $SESSION_DIR/grok-review.md
  Task: reviewer3 → $SESSION_DIR/free-model-a-review.md

❌ WRONG - Shared File:
  Task: reviewer1 → $SESSION_DIR/review.md
  Task: reviewer2 → $SESSION_DIR/review.md (overwrites reviewer1!)
  Task: reviewer3 → $SESSION_DIR/review.md (overwrites reviewer2!)

❌ WRONG - Fixed Directory (not session-based):
  Task: reviewer1 → ai-docs/reviews/claude-review.md  # May conflict with other sessions!
```

**Wait for All Before Consolidation:**

Do NOT consolidate until ALL tasks complete:

```
✅ CORRECT - Wait for All:
  Launch: Task1, Task2, Task3, Task4 (parallel)
  Wait: All 4 complete
  Check: results.filter(r => r.status === 'fulfilled').length
  If >= 2: Dispatch dev:aggregator; if any failed, also offer to re-run them
  If 1:    Use that review's own verdict; if any failed, offer to re-run them
  If 0:    Offer a re-run or abort

❌ WRONG - Premature Consolidation:
  Launch: Task1, Task2, Task3, Task4
  After 30s: Task1, Task2 done
  Consolidate: Only Task1 + Task2 (Task3, Task4 still running!)
```

---

### Pattern 3: Model Invocation via claudish MCP

**How models are invoked:**

External models are invoked via claudish MCP tools. The orchestrator calls the MCP tools
directly; no Bash invocation is needed. This is 100% reliable.

**Where the native reviewer sits — one rule.** It is the claudish `internal` slot, in the
`models` list of the ONE `team` call, and gets the same `require_pattern` shape check as
every external. It is never a subagent of the session that wrote the code: that shares
its lineage and escapes the shape check.

- **`/team` itself** — that procedure lives in this plugin's own `commands/team.md`
  (Step 2 onward) and is not restated here.
- **The code-review panels `dev` dispatches** (`/dev:dev` Phase 5, `/dev:audit`,
  `/dev:fix` Phase B) — the same slot; with no externals the call is
  `models=["internal"]`. Without the `team` tool, dev picks the route by §1 of
  `skills/core/team-gate/cli-sessions.md` inside the dev plugin — CLI sessions in terminal
  slots, or the nested route inside a claudish child (its
  root: the `installPath` of the `dev@` entry in `claude plugin list --json`). Every code-review example in
  this skill is this case.

**For a code-review panel:** write the brief to `input.md`, start the panel, poll it to
completion, then read the reviews off disk.
````
team(mode="run", path=SESSION_DIR, models=["internal", "grok", "gemini"],
  input_file=`${SESSION_DIR}/input.md`,
  require_pattern="\*\*Verdict\*\*: (PASS|CONDITIONAL|FAIL)", agent="dev:reviewer")
  // → { started: true, slots: { "internal": "01", "grok": "02", "gemini": "03" }, ... }

team(mode="status", path=SESSION_DIR)   // until no slot has state === "RUNNING"
  // → copy `response-<slots.internal>.md` to `${SESSION_DIR}/claude-review.md`
  //   when that slot COMPLETED; read `response-<slot>.md` for each external slot
````

The `team` tool runs all models in parallel internally. **`run` does not wait and does not
return the answers** — it starts the slots and hands back the slot map. Per-model status
(`state`, `exitCode`, `outputSize`, `error.reason`) comes from a settled `status` response.
There is no `timeout` parameter any more, and passing one is silently ignored. Full procedure: `claudish:claudish-usage` → "The three-step lifecycle". Requires claudish >= 8.0.0.

**The internal reviewer is always a `models` entry.** Its slot runs with the dev plugin
loaded, so `TARGET: BRANCH` resolves through dev's own capture script. Its copy,
`claude-review.md`, goes on REVIEWS: in place of its slot file, never beside it; a slot
that failed leaves no copy, and dev's team-gate says what that means for the gate.

**`require_pattern` is what turns exit 0 into a real success check.** A slot that finished
without producing the required shape is reported FAILED (state EMPTY, reason
`shape_mismatch`) instead of counted as a success. Exit code 0 also occurs on API errors and
on a child that simply ignored the format, so without this the panel can report a verdict it
never actually received.

**For single-model delegation:**
```
create_session(model="grok", prompt=TASK_PROMPT, timeout_seconds=300)
→ poll list_sessions(include_completed=true) about every 10 s:
  STARTING → RUNNING → COMPLETED / FAILED / TIMEOUT
→ get_output(session_id) to retrieve result
```

**Verification:**
- `team` tool: check `status.models[<slot>].state` on a SETTLED `status` response — never
  the `run` response, which returns before any model has answered
- `create_session`: the row's `state` `COMPLETED` confirms success; on `FAILED` or `TIMEOUT`
  the row's `reason_detail` says why in one line, and `get_diagnostics` gives the full cause

### Correct Pattern Example

````
// ✅ CORRECT: the internal reviewer and every external in ONE team call
team(mode="run", path=SESSION_DIR,
  models=["internal", "grok", "gemini"],
  input_file=`${SESSION_DIR}/input.md`,
  require_pattern="\*\*Verdict\*\*: (PASS|CONDITIONAL|FAIL)", agent="dev:reviewer")

// …then poll, copy the internal slot to claude-review.md, and read each external
// slot's review from response-<slot>.md
team(mode="status", path=SESSION_DIR)
````

```
// ❌ WRONG: the internal review counted twice
REVIEWS: ${SESSION_DIR}/claude-review.md
         ${SESSION_DIR}/response-<slots.internal>.md   // the same review again
         ${SESSION_DIR}/response-<slot>.md             // each external slot
```

`claude-review.md` is the copy of `response-<slots.internal>.md`. Listing both hands the
aggregator one review as two, and the consensus it reports is fake. REVIEWS: names the
copy once, then each external slot that completed.

---

### Pattern 4: Cost Estimation and Transparency

**Input/Output Token Separation:**

Provide separate estimates for input and output tokens:

```
Cost Estimation for Multi-Model Review:

Input Tokens (per model):
  - Code context: 500 lines × 1.5 = 750 tokens
  - Review instructions: 200 tokens
  - Total input per model: ~1000 tokens
  - Total input (5 models): 5,000 tokens

Output Tokens (per model):
  - Expected output: 2,000 - 4,000 tokens
  - Total output (5 models): 10,000 - 20,000 tokens

Cost Calculation (example rates):
  - Input: 5,000 tokens × $0.0001/1k = $0.0005
  - Output: 15,000 tokens × $0.0005/1k = $0.0075 (3-5x more expensive)
  - Total: $0.0080 (range: $0.0055 - $0.0105)

User Approval Gate:
  "Multi-model review will cost approximately $0.008 ($0.005 - $0.010).
   Proceed? (Yes/No)"
```

**Input Token Estimation Formula:**

```
Input Tokens = (Code Lines × 1.5) + Instruction Tokens

Why 1.5x multiplier?
  - Code lines: ~1 token per line (average)
  - Context overhead: +50% (imports, comments, whitespace)

Example:
  500 lines of code → 500 × 1.5 = 750 tokens
  + 200 instruction tokens = 950 tokens total input
```

**Output Token Estimation Formula:**

```
Output Tokens = Base Estimate + Complexity Factor

Base Estimates by Task Type:
  - Code review: 2,000 - 4,000 tokens
  - Design validation: 1,000 - 2,000 tokens
  - Architecture planning: 3,000 - 6,000 tokens
  - Bug investigation: 2,000 - 5,000 tokens

Complexity Factors:
  - Simple (< 100 lines code): Use low end of range
  - Medium (100-500 lines): Use mid-range
  - Complex (> 500 lines): Use high end of range

Example:
  400 lines of complex code → 4,000 tokens (high complexity)
  50 lines of simple code → 2,000 tokens (low complexity)
```

**Range-Based Estimates:**

Always provide a **range** (min-max), not a single number:

```
✅ CORRECT - Range:
  "Estimated cost: $0.005 - $0.010 (depends on review depth)"

❌ WRONG - Single Number:
  "Estimated cost: $0.0075"
  (User surprised when actual is $0.0095)
```

**Why Output Costs More:**

Output tokens are typically **3-5x more expensive** than input tokens:

```
Pricing: read each model's input and output rates from this run's `list_models`
result, never from this file. Output is typically several times the input rate.

Impact:
  If input = 5,000 tokens, output = 15,000 tokens:
    Input cost: $0.0005
    Output cost: $0.0075 (15x higher despite only 3x more tokens)
    Total: $0.0080 (94% is output!)
```

**User Approval Before Execution:**

ALWAYS ask for user approval before expensive operations:

```
Present to user:
  "You selected 5 AI models for code review:
   - internal (host Claude session, no external cost)
   - <model-a> (external, <estimate from list_models pricing>)
   - <model-b> (external, <estimate>)
   - <model-c> (external, <estimate>)
   - <model-d> (external, <estimate>)

   Estimated total cost: $0.008 ($0.005 - $0.010)

   Proceed with multi-model review? (Yes/No)"

If user says NO:
  Offer alternatives:
    1. Use only free embedded Claude
    2. Select fewer models
    3. Cancel review

If user says YES:
  Proceed with Message 2 (parallel execution)
```

---

### Pattern 5: Auto-Consolidation Logic

**Automatic Trigger:**

Consolidation happens **automatically** as soon as the panel settles with N ≥ 2
reviews. `dev:aggregator` is the only writer of the consolidated report. At N = 1 there is
nothing to consolidate, so the dispatcher does not send it: the single review and its own
verdict are the result. This is dev's rule (`aggregate-reviews`: dispatchers avoid sending
N = 1 and use the single review's verdict directly):

```
✅ CORRECT - Auto-Trigger:

const results = await Promise.allSettled([task1, task2, task3, task4, task5]);
const successful = results.filter(r => r.status === 'fulfilled');
const failed = results.length - successful.length;

if (successful.length >= 2) {
  // Auto-trigger consolidation (DON'T wait for user to ask).
  const reviewPaths = successful.map((r) => r.value.reviewFile); // one review file per slot
  const consolidated = await Agent({
    subagent_type: "dev:aggregator",
    run_in_background: false,     // formatResults() consumes the return value
    description: "Consolidate code reviews",
    prompt: `REVIEWS: ${reviewPaths.join("\n")}
THRESHOLDS: <the three lines under 'Apply verdict thresholds' in dev:reviewer's agent file, read at dispatch time, never recalled>
OUTPUT: ${SESSION_DIR}/consolidated-review.md
Consolidate with consensus levels (unanimous / strong / majority / divergent).
Compute the verdict line from your counts against THRESHOLDS.
You are given reviews, never code. Do not review.`
  });

  if (failed > 0) {
    // An addition to the dispatch above, never a substitute for it
    notifyUser(`${failed} of ${results.length} models failed. Re-run them and re-consolidate?`);
  }
  return formatResults(consolidated);
} else if (successful.length === 1) {
  // One review: its own verdict is the result; no aggregator
  if (failed > 0) notifyUser(`${failed} of ${results.length} models failed. Re-run them?`);
  return formatResults(successful[0].value.reviewFile);
} else {
  // All failed
  notifyUser("All models failed. Check the diagnostics and re-run?");
}

❌ WRONG - Wait for User:

const results = await Promise.allSettled([...]);
const successful = results.filter(r => r.status === 'fulfilled');

// Present results to user
notifyUser("3 reviews complete. Would you like me to consolidate them?");
// Waits for user to request consolidation...

❌ WRONG - Send one review to the aggregator:

if (successful.length >= 1) {
  await consolidate();   // at N = 1 this costs an Agent call and can only subtract
}
```

**Why Auto-Trigger:**

- Better UX (no extra user prompt needed)
- Faster workflow (no wait for user response)
- Expected behavior (user assumes consolidation is part of workflow)

**N = 1 needs no aggregator:**

Consensus needs at least two reviews, and consolidating one review can only subtract from
it. At N = 1 the dispatcher uses that review and its own verdict directly, as `/dev:fix`,
`/dev:audit` and dev's review phases do. If a single review reaches `dev:aggregator`
anyway, it passes it through unchanged with a `VERDICT:` line, so nothing breaks; it is
just a wasted call.

**Pass All Review File Paths:**

`dev:aggregator` needs the path of EVERY review file, one per `REVIEWS:` line. It
reads the reviews and never the code, so the paths are all it gets:

```
Agent(
  subagent_type: "dev:aggregator",
  run_in_background: false,
  description: "Consolidate code reviews",
  prompt: "REVIEWS: $SESSION_DIR/claude-review.md
           $SESSION_DIR/response-<slot>.md   (one line per external slot from the slots map —
                                             never slots.internal, which claude-review.md copies)
           THRESHOLDS: <the three lines under 'Apply verdict thresholds' in
                        dev:reviewer's agent file, read at dispatch time, never
                        recalled>
           OUTPUT: $SESSION_DIR/consolidated-review.md
           Consolidate with consensus levels (unanimous / strong / majority / divergent).
           Compute the verdict line from your counts against THRESHOLDS.
           You are given reviews, never code. Do not review."
)
```

**Don't Inline Full Reviews:**

```
❌ WRONG - Inline Reviews (context pollution):
  Prompt: "Consolidate these reviews:

           Claude Review:
           [500 lines of review content]

           Grok Review:
           [500 lines of review content]

           Free model A review:
           [500 lines of review content]"

✅ CORRECT - File Paths in Session Directory:
  prompt: "REVIEWS: $SESSION_DIR/claude-review.md
           $SESSION_DIR/response-<slot>.md   (one per external slot)
           THRESHOLDS: ...
           OUTPUT: $SESSION_DIR/consolidated-review.md
           ..."
```

---

### Pattern 6: Consensus Analysis

**Consensus Levels:**

Classify issues by how many models flagged them:

```
Consensus Levels (for N models):

UNANIMOUS (100% agreement):
  - All N models flagged this issue
  - VERY HIGH confidence
  - MUST FIX priority

STRONG CONSENSUS (67-99% agreement):
  - Most models flagged this issue (⌈2N/3⌉ to N-1)
  - HIGH confidence
  - RECOMMENDED priority

MAJORITY (50-66% agreement):
  - Half or more models flagged this issue (⌈N/2⌉ to ⌈2N/3⌉-1)
  - MEDIUM confidence
  - CONSIDER priority

DIVERGENT (< 50% agreement):
  - Only 1-2 models flagged this issue
  - LOW confidence
  - OPTIONAL priority (may be model-specific perspective)
```

**Example: 5 Models**

```
Issue Flagged By:              Consensus Level:    Priority:
─────────────────────────────────────────────────────────────
All 5 models                   UNANIMOUS (100%)    MUST FIX
4 models                       STRONG (80%)        RECOMMENDED
3 models                       MAJORITY (60%)      CONSIDER
2 models                       DIVERGENT (40%)     OPTIONAL
1 model                        DIVERGENT (20%)     OPTIONAL
```

**Keyword-Based Matching (v1.0):**

Simple consensus analysis using keyword matching:

```
Algorithm:

1. Extract issues from each review
2. For each unique issue:
   a. Identify keywords (e.g., "SQL injection", "input validation")
   b. Check which other reviews mention same keywords
   c. Count models that flagged this issue
   d. Assign consensus level

Example:

Claude Review: "Missing input validation on POST /api/users"
Grok Review: "Input validation absent in user creation endpoint"
Gemini Review: "No validation for user POST endpoint"

Keywords: ["input validation", "POST", "/api/users", "user"]
Match: All 3 reviews mention these keywords
Consensus: UNANIMOUS (3/3 = 100%)
```

**Model Agreement Matrix:**

Show which models agree on which issues:

```
Issue Matrix:

Issue                             Claude  Grok  Gemini  GPT  DeepSeek  Consensus
──────────────────────────────────────────────────────────────────────────────────
SQL injection in search              ✓      ✓     ✓       ✓       ✓      UNANIMOUS
Missing input validation             ✓      ✓     ✓       ✓       ✗      STRONG
Weak password hashing                ✓      ✓     ✓       ✗       ✗      MAJORITY
Missing rate limiting                ✓      ✓     ✗       ✗       ✗      DIVERGENT
Insufficient error handling          ✓      ✗     ✗       ✗       ✗      DIVERGENT
```

**Prioritized Issue List:**

Sort issues by consensus level, then by severity:

```
Top 10 Issues (Prioritized):

1. [UNANIMOUS - CRITICAL] SQL injection in search endpoint
   Flagged by: Claude, Grok, Gemini, GPT, DeepSeek (5/5)

2. [UNANIMOUS - HIGH] Missing input validation on POST /api/users
   Flagged by: Claude, Grok, Gemini, GPT, DeepSeek (5/5)

3. [STRONG - HIGH] Weak password hashing (bcrypt rounds too low)
   Flagged by: Claude, Grok, Gemini, GPT (4/5)

4. [STRONG - MEDIUM] Missing rate limiting on auth endpoints
   Flagged by: Claude, Grok, Gemini, GPT (4/5)

5. [MAJORITY - MEDIUM] Insufficient error handling in payment flow
   Flagged by: Claude, Grok, Gemini (3/5)

... (remaining issues)
```

**Future Enhancement (v1.1+): Semantic Similarity**

```
Instead of keyword matching, use semantic similarity:
  - Embed issue descriptions with sentence-transformers
  - Calculate cosine similarity between embeddings
  - Issues with >0.8 similarity are "same issue"
  - More accurate consensus detection
```

---

### Pattern 7: Statistics Collection and Analysis

**Purpose**: Track model performance to help users identify slow or poorly-performing models for future exclusion.

**Storage Location**: `ai-docs/llm-performance.json` (persistent across all sessions)

**When to Collect Statistics:**
- After each model completes (success, failure, or timeout)
- During consolidation phase (quality scores)
- At session end (session summary)

**File Structure (ai-docs/llm-performance.json):**

```json
{
  "schemaVersion": "2.0.0",
  "lastUpdated": "2025-12-12T10:45:00Z",
  "models": {
    "claude-embedded": {
      "modelId": "claude-embedded",
      "provider": "Anthropic",
      "isFree": true,
      "pricing": "FREE (embedded)",
      "totalRuns": 12,
      "successfulRuns": 12,
      "failedRuns": 0,
      "totalExecutionTime": 420,
      "avgExecutionTime": 35,
      "minExecutionTime": 28,
      "maxExecutionTime": 52,
      "totalIssuesFound": 96,
      "avgQualityScore": 92,
      "totalCost": 0,
      "qualityScores": [95, 90, 88, 94, 91],
      "lastUsed": "2025-12-12T10:35:22Z",
      "trend": "stable",
      "history": [
        {
          "timestamp": "2025-12-12T10:35:22Z",
          "session": "review-20251212-103522-a3f2",
          "status": "success",
          "executionTime": 32,
          "issuesFound": 8,
          "qualityScore": 95,
          "cost": 0
        }
      ]
    },
    "x-ai-grok": {
      "modelId": "grok",
      "provider": "X-ai",
      "isFree": false,
      "pricing": "<from list_models>",
      "totalRuns": 10,
      "successfulRuns": 9,
      "failedRuns": 1,
      "totalCost": 0.12,
      "trend": "improving"
    },
    "<free-model-a>": {
      "modelId": "<free-model-a>",
      "provider": "<from list_models>",
      "isFree": true,
      "pricing": "FREE",
      "totalRuns": 5,
      "successfulRuns": 5,
      "failedRuns": 0,
      "totalCost": 0,
      "trend": "stable"
    }
  },
  "sessions": [
    {
      "sessionId": "review-20251212-103522-a3f2",
      "timestamp": "2025-12-12T10:35:22Z",
      "totalModels": 4,
      "successfulModels": 3,
      "failedModels": 1,
      "parallelTime": 120,
      "sequentialTime": 335,
      "speedup": 2.8,
      "totalCost": 0.018,
      "freeModelsUsed": 2
    }
  ],
  "recommendations": {
    "topPaid": ["grok", "gemini"],
    "topFree": ["<free-model-a>", "<free-model-b>"],
    "bestValue": ["grok"],
    "avoid": [],
    "lastGenerated": "2025-12-12T10:45:00Z"
  }
}
```

**Key Benefits of Persistent Storage:**
- Track model reliability over time (not just one session)
- Identify consistently slow models
- Calculate historical success rates
- Generate data-driven shortlist recommendations

**How to Calculate Quality Score:**

Quality = % of model's issues that appear in unanimous or strong consensus

```
quality_score = (issues_in_unanimous + issues_in_strong) / total_issues * 100

Example:
- Model found 10 issues
- 4 appear in unanimous consensus
- 3 appear in strong consensus
- Quality = (4 + 3) / 10 * 100 = 70%
```

Higher quality means the model finds issues other models agree with.

**How to Calculate Parallel Speedup:**

```
speedup = sum(all_execution_times) / max(execution_time)

Example:
- Claude: 32s
- Grok: 45s
- Gemini: 38s
- GPT: 120s

Sequential would take: 32 + 45 + 38 + 120 = 235s
Parallel took: max(32, 45, 38, 120) = 120s
Speedup: 235 / 120 = 1.96x
```

**Performance Statistics Display Format:**

```markdown
## Model Performance Statistics

| Model                     | Time   | Issues | Quality | Status    |
|---------------------------|--------|--------|---------|-----------|
| claude-embedded           | 32s    | 8      | 95%     | ✓         |
| grok     | 45s    | 6      | 85%     | ✓         |
| gemini   | 38s    | 5      | 90%     | ✓         |
| gpt   | 120s   | 9      | 88%     | ✓ (slow)  |
| <model-d>    | TIMEOUT| 0      | -       | ✗         |

**Session Summary:**
- Parallel Speedup: 1.96x (235s sequential → 120s parallel)
- Average Time: 59s
- Slowest: gpt (2.0x avg)

**Recommendations:**
⚠️ gpt runs 2x slower than average - consider removing
⚠️ <model-d> timed out - check API status or remove from shortlist
✓ Top performers: claude-embedded, gemini (fast + high quality)
```

**Recommendation Logic:**

```
1. Flag SLOW models:
   if (model.executionTime > 2 * avgExecutionTime) {
     flag: "⚠️ Runs 2x+ slower than average"
     suggestion: "Consider removing from shortlist"
   }

2. Flag FAILED/TIMEOUT models:
   if (model.status !== "success") {
     flag: "⚠️ Failed or timed out"
     suggestion: "Check API status or increase timeout"
   }

3. Identify TOP PERFORMERS:
   if (model.qualityScore > 85 && model.executionTime < avgExecutionTime) {
     highlight: "✓ Top performer (fast + high quality)"
   }

4. Suggest SHORTLIST:
   sortedModels = models.sort((a, b) => {
     // Quality/speed ratio: higher quality + lower time = better
     scoreA = a.qualityScore / (a.executionTime / avgExecutionTime)
     scoreB = b.qualityScore / (b.executionTime / avgExecutionTime)
     return scoreB - scoreA
   })
   shortlist = sortedModels.slice(0, 3)
```

**Implementation (writes to ai-docs/llm-performance.json):**

```bash
# Track model performance after each model completes
# Updates historical aggregates and adds to run history
# Parameters: model_id, status, duration, issues, quality_score, cost, is_free
track_model_performance() {
  local model_id="$1"
  local status="$2"
  local duration="$3"
  local issues="${4:-0}"
  local quality_score="${5:-}"
  local cost="${6:-0}"
  local is_free="${7:-false}"

  local perf_file="ai-docs/llm-performance.json"
  local model_key=$(echo "$model_id" | tr '/:' '-')  # Handle colons in free model names

  # Initialize file if doesn't exist
  [[ -f "$perf_file" ]] || echo '{"schemaVersion":"2.0.0","models":{},"sessions":[],"recommendations":{}}' > "$perf_file"

  jq --arg model "$model_key" \
     --arg model_full "$model_id" \
     --arg status "$status" \
     --argjson duration "$duration" \
     --argjson issues "$issues" \
     --arg quality "${quality_score:-null}" \
     --argjson cost "$cost" \
     --argjson is_free "$is_free" \
     --arg now "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
     --arg session "${SESSION_ID:-unknown}" \
     '
     # Initialize model if not exists
     .models[$model] //= {"modelId":$model_full,"provider":"unknown","isFree":$is_free,
       "totalRuns":0,"successfulRuns":0,"failedRuns":0,
       "totalExecutionTime":0,"avgExecutionTime":0,"minExecutionTime":null,"maxExecutionTime":null,
       "totalIssuesFound":0,"avgQualityScore":null,"qualityScores":[],"totalCost":0,
       "lastUsed":null,"trend":"new","history":[]} |

     # Update aggregates
     .models[$model].totalRuns += 1 |
     .models[$model].successfulRuns += (if $status == "success" then 1 else 0 end) |
     .models[$model].failedRuns += (if $status != "success" then 1 else 0 end) |
     .models[$model].totalExecutionTime += $duration |
     .models[$model].avgExecutionTime = ((.models[$model].totalExecutionTime / .models[$model].totalRuns) | floor) |
     .models[$model].totalIssuesFound += $issues |
     .models[$model].totalCost += $cost |
     .models[$model].isFree = $is_free |
     .models[$model].lastUsed = $now |

     # Update min/max
     .models[$model].minExecutionTime = ([.models[$model].minExecutionTime, $duration] | map(select(. != null)) | min) |
     .models[$model].maxExecutionTime = ([.models[$model].maxExecutionTime, $duration] | max) |

     # Update quality scores and trend (if provided)
     (if $quality != "null" then
       .models[$model].qualityScores += [($quality|tonumber)] |
       .models[$model].avgQualityScore = ((.models[$model].qualityScores|add) / (.models[$model].qualityScores|length) | floor) |
       # Calculate trend (last 3 vs previous 3)
       (if (.models[$model].qualityScores | length) >= 6 then
         ((.models[$model].qualityScores[-3:] | add) / 3) as $recent |
         ((.models[$model].qualityScores[-6:-3] | add) / 3) as $previous |
         .models[$model].trend = (if ($recent - $previous) > 5 then "improving"
           elif ($recent - $previous) < -5 then "degrading"
           else "stable" end)
       else . end)
     else . end) |

     # Add to history (keep last 20)
     .models[$model].history = ([{"timestamp":$now,"session":$session,"status":$status,
       "executionTime":$duration,"issuesFound":$issues,"cost":$cost,
       "qualityScore":(if $quality != "null" then ($quality|tonumber) else null end)}] + .models[$model].history)[:20] |

     .lastUpdated = $now
     ' "$perf_file" > "${perf_file}.tmp" && mv "${perf_file}.tmp" "$perf_file"
}

# Usage examples:
# Paid models
track_model_performance "grok" "success" 45 6 87 0.002 false
track_model_performance "gpt" "success" 68 7 89 0.015 false

# Free models (cost=0, is_free=true)
track_model_performance "<free-model-a>" "success" 52 5 82 0 true
track_model_performance "<free-model-b>" "success" 48 5 84 0 true

# Embedded Claude (always free)
track_model_performance "claude-embedded" "success" 32 8 95 0 true

# Failed/timeout models
track_model_performance "some-model" "timeout" 120 0 "" 0 false
```

**Record Session Summary:**

```bash
record_session_stats() {
  local total="$1" success="$2" failed="$3"
  local parallel_time="$4" sequential_time="$5" speedup="$6"
  local total_cost="${7:-0}" free_models_used="${8:-0}"

  local perf_file="ai-docs/llm-performance.json"
  [[ -f "$perf_file" ]] || echo '{"schemaVersion":"2.0.0","models":{},"sessions":[],"recommendations":{}}' > "$perf_file"

  jq --arg session "${SESSION_ID:-unknown}" \
     --arg now "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
     --argjson total "$total" --argjson success "$success" --argjson failed "$failed" \
     --argjson parallel "$parallel_time" --argjson sequential "$sequential_time" --argjson speedup "$speedup" \
     --argjson cost "$total_cost" --argjson free_count "$free_models_used" \
     '.sessions = ([{"sessionId":$session,"timestamp":$now,"totalModels":$total,
       "successfulModels":$success,"failedModels":$failed,"parallelTime":$parallel,
       "sequentialTime":$sequential,"speedup":$speedup,"totalCost":$cost,
       "freeModelsUsed":$free_count}] + .sessions)[:50] | .lastUpdated = $now' \
     "$perf_file" > "${perf_file}.tmp" && mv "${perf_file}.tmp" "$perf_file"
}

# Usage:
# record_session_stats total success failed parallel_time sequential_time speedup total_cost free_count
record_session_stats 5 5 0 68 245 3.6 0.017 2
```

**Get Recommendations from Historical Data:**

```bash
get_model_recommendations() {
  local perf_file="ai-docs/llm-performance.json"
  [[ -f "$perf_file" ]] || { echo "No performance data yet."; return; }

  jq -r '
    (.models | to_entries | map(select(.value.successfulRuns > 0) | .value.avgExecutionTime) | add / length) as $avg |
    {
      "overallAvgTime": ($avg | floor),
      "slowModels": [.models | to_entries[] | select(.value.avgExecutionTime > ($avg * 2)) | .key],
      "unreliableModels": [.models | to_entries[] | select(.value.totalRuns >= 3 and (.value.failedRuns / .value.totalRuns) > 0.3) | .key],
      "topPaidPerformers": [.models | to_entries | map(select(.value.avgQualityScore != null and .value.avgQualityScore > 80 and .value.isFree == false and .value.avgExecutionTime <= $avg)) | sort_by(-.value.avgQualityScore)[:3] | .[].key],
      "topFreePerformers": [.models | to_entries | map(select(.value.avgQualityScore != null and .value.avgQualityScore > 75 and .value.isFree == true)) | sort_by(-.value.avgQualityScore)[:3] | .[].key],
      "bestValue": [.models | to_entries | map(select(.value.avgQualityScore != null and .value.totalCost > 0)) | sort_by(-(.value.avgQualityScore / (.value.totalCost / .value.totalRuns)))[:2] | .[].key],
      "degradingModels": [.models | to_entries[] | select(.value.trend == "degrading") | .key]
    }
  ' "$perf_file"
}

# Display formatted recommendations
display_recommendations() {
  local perf_file="ai-docs/llm-performance.json"
  [[ -f "$perf_file" ]] || { echo "No performance data yet. Run some validations first!"; return; }

  echo "## Model Recommendations (based on historical data)"
  echo ""

  # Top paid performers
  echo "### 💰 Top Paid Models"
  jq -r '.models | to_entries | map(select(.value.isFree == false and .value.avgQualityScore != null)) | sort_by(-.value.avgQualityScore)[:3] | .[] | "- \(.value.modelId): Quality \(.value.avgQualityScore)%, Avg \(.value.avgExecutionTime)s, Cost $\(.value.totalCost | . * 100 | floor / 100)"' "$perf_file"
  echo ""

  # Top free performers
  echo "### 🆓 Top Free Models"
  jq -r '.models | to_entries | map(select(.value.isFree == true and .value.avgQualityScore != null and .key != "claude-embedded")) | sort_by(-.value.avgQualityScore)[:3] | .[] | "- \(.value.modelId): Quality \(.value.avgQualityScore)%, Avg \(.value.avgExecutionTime)s"' "$perf_file"
  echo ""

  # Models to avoid
  echo "### ⚠️ Consider Avoiding"
  jq -r '
    (.models | to_entries | map(select(.value.successfulRuns > 0) | .value.avgExecutionTime) | add / length) as $avg |
    .models | to_entries[] |
    select(
      (.value.avgExecutionTime > ($avg * 2)) or
      (.value.totalRuns >= 3 and (.value.failedRuns / .value.totalRuns) > 0.3) or
      (.value.trend == "degrading")
    ) |
    "- \(.key): " +
    (if .value.avgExecutionTime > ($avg * 2) then "⏱️ Slow (2x+ avg)" else "" end) +
    (if .value.totalRuns >= 3 and (.value.failedRuns / .value.totalRuns) > 0.3 then " ❌ Unreliable (>\(.value.failedRuns)/\(.value.totalRuns) failures)" else "" end) +
    (if .value.trend == "degrading" then " 📉 Quality degrading" else "" end)
  ' "$perf_file"
}
```

---

### Pattern 8: Data-Driven Model Selection (NEW v3.0)

**Purpose:** Use historical performance data to make intelligent model selection recommendations.

**The Problem:**

Users often select models arbitrarily or based on outdated information:
- "I'll use GPT because it's famous"
- "Let me try this new model I heard about"
- "I'll use the same 5 models every time"

**The Solution:**

Use accumulated performance data to recommend:
1. **Top performers** (highest quality scores)
2. **Best value** (quality/cost ratio)
3. **Top free models** (high quality, zero cost)
4. **Models to avoid** (slow, unreliable, or degrading)

**Model Selection Algorithm:**

```
1. Load historical data from ai-docs/llm-performance.json

2. Calculate metrics for each model:
   - Success Rate = successfulRuns / totalRuns × 100
   - Quality Score = avgQualityScore (from consensus analysis)
   - Speed Score = avgExecutionTime relative to overall average
   - Value Score = avgQualityScore / (totalCost / totalRuns)

3. Categorize models:
   TOP PAID: Quality > 80%, Success > 90%, Speed <= avg
   TOP FREE: Quality > 75%, Success > 90%, isFree = true
   BEST VALUE: Highest Quality/Cost ratio among paid models
   AVOID: Speed > 2x avg OR Success < 70% OR trend = "degrading"

4. Present recommendations with context:
   - Show historical metrics
   - Highlight trends (improving/stable/degrading)
   - Flag new models with insufficient data
```

**Interactive Model Selection with Recommendations:**

Instead of just displaying recommendations, use AskUserQuestion with multiSelect to let users interactively choose:

```typescript
// Build options from the live catalog (claudish list_models MCP tool) + history
const catalog    = await listModels();   // [{ id, pricing, context, capabilities }, ...]
const paidModels = catalog.filter((m) => m.pricing?.average > 0);
const freeModels = catalog.filter((m) => !m.pricing?.average);
const history = loadPerformanceHistory();        // ai-docs/llm-performance.json

// Merge and build AskUserQuestion options
AskUserQuestion({
  questions: [{
    question: "Select models for validation (Claude internal always included). Based on 25 sessions across 8 models.",
    header: "Models",
    multiSelect: true,
    options: [
      // Top paid with historical data
      {
        label: "grok ⚡ (Recommended)",
        description: "<price from list_models> | Quality: 87% | Avg: 42s"
      },
      {
        label: "gemini 🎯",
        description: "<price from list_models> | Quality: 91% | Avg: 55s"
      },
      // Top free models
      {
        label: "<free-model-a> 🆓",
        description: "FREE | Quality: 82% | 262K | Coding-specialized"
      },
      {
        label: "<free-model-b> 🆓",
        description: "FREE | Quality: 84% | 262K | Dev-focused"
      }
      // Note: Models to AVOID are simply not shown in options
      // Note: New models show "(new)" instead of quality score
    ]
  }]
})
```

**Key Principles for Model Selection UI:**

1. **Put recommended models first** with "(Recommended)" suffix
2. **Include historical metrics** in description (Quality %, Avg time)
3. **Mark free models** with 🆓 emoji
4. **Don't show models to avoid** - just exclude them from options
5. **Mark new models** with "(new)" when no historical data
6. **Remember selection** - save to `$SESSION_DIR/selected-models.txt`

**After Selection - Save to Session:**

```bash
# User selected: grok, <free-model-a>
# Save for session persistence
save_session_models "$SESSION_DIR" "${USER_SELECTED_MODELS[@]}"

# Now $SESSION_DIR/selected-models.txt contains:
# claude-embedded
# grok
# <free-model-a>
```

**Warning Display (separate from selection):**

If there are models to avoid, show a brief warning before the selection:

```
⚠️ Models excluded from selection (poor historical performance):
- gpt: Slow (2.1x avg)
- some-model: 60% success rate
```

**Automatic Shortlist Generation:**

```bash
# Generate optimal shortlist based on criteria
generate_shortlist() {
  local criteria="${1:-balanced}"  # balanced, quality, budget, free-only
  local perf_file="ai-docs/llm-performance.json"

  case "$criteria" in
    "balanced")
      # 1 internal + 1 fast paid + 1 free
      echo "claude-embedded"
      jq -r '.models | to_entries | map(select(.value.isFree == false and .value.avgQualityScore > 80)) | sort_by(.value.avgExecutionTime)[0].key' "$perf_file"
      jq -r '.models | to_entries | map(select(.value.isFree == true and .key != "claude-embedded" and .value.avgQualityScore > 75)) | sort_by(-.value.avgQualityScore)[0].key' "$perf_file"
      ;;
    "quality")
      # Top 3 by quality regardless of cost
      echo "claude-embedded"
      jq -r '.models | to_entries | map(select(.value.avgQualityScore != null and .key != "claude-embedded")) | sort_by(-.value.avgQualityScore)[:2] | .[].key' "$perf_file"
      ;;
    "budget")
      # Internal + 2 cheapest performers
      echo "claude-embedded"
      jq -r '.models | to_entries | map(select(.value.avgQualityScore > 75 and .value.isFree == true)) | sort_by(-.value.avgQualityScore)[:2] | .[].key' "$perf_file"
      ;;
    "free-only")
      # Only free models
      echo "claude-embedded"
      jq -r '.models | to_entries | map(select(.value.isFree == true and .key != "claude-embedded" and .value.avgQualityScore != null)) | sort_by(-.value.avgQualityScore)[:2] | .[].key' "$perf_file"
      ;;
  esac
}

# Usage:
generate_shortlist "balanced"   # For most use cases
generate_shortlist "quality"    # When accuracy is critical
generate_shortlist "budget"     # When cost matters
generate_shortlist "free-only"  # Zero-cost validation
```

**Integration with Model Discovery:**

```
Workflow:
1. Call `list_models` (claudish MCP) → current models, pricing, capabilities
2. Load ai-docs/llm-performance.json → Get historical performance
3. Merge data:
   - New models (no history): Mark as "🆕 New"
   - Known models: Show performance metrics
   - Deprecated models: Filter out (absent from the live catalog)
4. Generate recommendations
5. Present to user with AskUserQuestion
```

**Why This Matters:**

| Selection Method | Outcome |
|------------------|---------|
| Random/arbitrary | Hit-or-miss, may waste money on slow models |
| Always same models | Miss new better options, stuck with degrading ones |
| Data-driven | Optimal quality/cost/speed balance, continuous improvement |

Over time, the system learns which models work best for YOUR codebase and validation patterns.

---

## Integrating Statistics in Your Plugin

**To add LLM performance tracking to your plugin's commands:**

### Step 1: Reference This Skill
Add to your command's frontmatter:
```yaml
skills: multimodel:multi-model-validation
```

### Step 2: Track Each Model Execution
After each external model completes:
```bash
# Parameters: model_id, status, duration_seconds, issues_found, quality_score
track_model_performance "grok" "success" 45 6 85
```

### Step 3: Record Session Summary
At the end of multi-model execution:
```bash
# Parameters: total, successful, failed, parallel_t

[…] pełny plik w paczce .zip