Przejdź do treści
Biblioteka skilli
Programowanie · Testowanie

ai-regression-testing

/ai-regression-testing

Pokazuje strategie testów regresyjnych dla kodu pisanego z AI: testy API w trybie sandbox bez bazy i wzorce wyłapujące ślepe punkty tego samego modelu.

Repozytorium
Nemo-Cli ↗
Licencja
Apache-2.0
Ostatnia zmiana repozytorium
05.11.2026

Ocena i bezpieczeństwo

100/ 100Klasa: dobry
Format25 / 25
Frontmatter z nazwą i opisem, poprawna nazwa, opis do 1024 znaków, nazwa zgodna z folderem.
Odkrywalność25 / 25
Opis mówi, kiedy użyć skilla, ma sensowną długość i jest w 3. osobie.
Treść35 / 35
Co najmniej 150 słów i do 500 linii, nagłówki, listy kroków, przykłady albo kod.
Higiena15 / 15
Bez zaślepek typu TODO, bez ścieżek Windows, z licencją.
Ryzykobrak sygnałów w skanie
Pliki w paczce
1
Skrypty
0
Komentarze HTML
1
Adresy URL w SKILL.md
1
Przestarzałość

Brak sygnałów. Repozytorium zmienione 05.11.2026, bez odwołań do wycofanych modeli i API.

Licencja

Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.

Co sprawdzić przed instalacją
  • SKILL.md zawiera komentarze HTML (1). Nie widać ich w podglądzie Markdown, a model je czyta.

Co robi i kiedy

Pokazuje strategie testów regresyjnych dla kodu pisanego z AI: testy API w trybie sandbox bez bazy i wzorce wyłapujące ślepe punkty tego samego modelu.

Oryginalny opis (EN)

Regression testing strategies for AI-assisted development. Sandbox-mode API testing without database dependencies, automated bug-check workflows, and patterns to catch AI blind spots where the same model writes and reviews code. Use when adding regression coverage to AI-assisted code, or when the same model both wrote and reviewed a change.

Jak działa

  1. When to Activate
  2. The Core Problem
  3. Sandbox-Mode API Testing
  4. Integrating Tests into Bug-Check Workflow
  5. Common AI Regression Patterns
  6. Strategy: Test Where Bugs Were Found
  7. Quick Reference
  8. DO / DON'T
Sekcje z nagłówków SKILL.md, w oryginale · 381 linii, 1 629 słów
Podgląd SKILL.md
---
name: ai-regression-testing
description: Regression testing strategies for AI-assisted development. Sandbox-mode API testing without database dependencies, automated bug-check workflows, and patterns to catch AI blind spots where the same model writes and reviews code. Use when adding regression coverage to AI-assisted code, or when the same model both wrote and reviewed a change.
metadata:
  origin: ECC
---

# AI Regression Testing

Testing patterns specifically designed for AI-assisted development, where the same model writes code and reviews it — creating systematic blind spots that only automated tests can catch.

## When to Activate

- AI agent (Nemo Code, Cursor, Nemo Code) has modified API routes or backend logic
- A bug was found and fixed — need to prevent re-introduction
- Project has a sandbox/mock mode that can be leveraged for DB-free testing
- Running `/bug-check` or similar review commands after code changes
- Multiple code paths exist (sandbox vs production, feature flags, etc.)

## The Core Problem

When an AI writes code and then reviews its own work, it carries the same assumptions into both steps. This creates a predictable failure pattern:

```
AI writes fix → AI reviews fix → AI says "looks correct" → Bug still exists
```

**Real-world example** (observed in production):

```
Fix 1: Added notification_settings to API response
  → Forgot to add it to the SELECT query
  → AI reviewed and missed it (same blind spot)

Fix 2: Added it to SELECT query
  → TypeScript build error (column not in generated types)
  → AI reviewed Fix 1 but didn't catch the SELECT issue

Fix 3: Changed to SELECT *
  → Fixed production path, forgot sandbox path
  → AI reviewed and missed it AGAIN (4th occurrence)

Fix 4: Test caught it instantly on first run PASS:
```

The pattern: **sandbox/production path inconsistency** is the #1 AI-introduced regression.

## Sandbox-Mode API Testing

Most projects with AI-friendly architecture have a sandbox/mock mode. This is the key to fast, DB-free API testing.

### Setup (Vitest + Next.js App Router)

```typescript
// vitest.config.ts
import { defineConfig } from "vitest/config"
import path from "path"

export default defineConfig({
  test: {
    environment: "node",
    globals: true,
    include: ["__tests__/**/*.test.ts"],
    setupFiles: ["__tests__/setup.ts"],
  },
  resolve: {
    alias: {
      "@": path.resolve(__dirname, "."),
    },
  },
})
```

```typescript
// __tests__/setup.ts
// Force sandbox mode — no database needed
process.env.SANDBOX_MODE = "true"
process.env.NEXT_PUBLIC_SUPABASE_URL = ""
process.env.NEXT_PUBLIC_SUPABASE_ANON_KEY = ""
```

### Test Helper for Next.js API Routes

```typescript
// __tests__/helpers.ts
import { NextRequest } from "next/server"

export function createTestRequest(
  url: string,
  options?: {
    method?: string
    body?: Record<string, unknown>
    headers?: Record<string, string>
    sandboxUserId?: string
  },
): NextRequest {
  const { method = "GET", body, headers = {}, sandboxUserId } = options || {}
  const fullUrl = url.startsWith("http") ? url : `http://localhost:3000${url}`
  const reqHeaders: Record<string, string> = { ...headers }

  if (sandboxUserId) {
    reqHeaders["x-sandbox-user-id"] = sandboxUserId
  }

  const init: { method: string; headers: Record<string, string>; body?: string } = {
    method,
    headers: reqHeaders,
  }

  if (body) {
    init.body = JSON.stringify(body)
    reqHeaders["content-type"] = "application/json"
  }

  return new NextRequest(fullUrl, init)
}

export async function parseResponse(response: Response) {
  const json = await response.json()
  return { status: response.status, json }
}
```

### Writing Regression Tests

The key principle: **write tests for bugs that were found, not for code that works**.

```typescript
// __tests__/api/user/profile.test.ts
import { describe, it, expect } from "vitest"
import { createTestRequest, parseResponse } from "../../helpers"
import { GET, PATCH } from "@/app/api/user/profile/route"

// Define the contract — what fields MUST be in the response
const REQUIRED_FIELDS = [
  "id",
  "email",
  "full_name",
  "phone",
  "role",
  "created_at",
  "avatar_url",
  "notification_settings", // ← Added after bug found it missing
]

describe("GET /api/user/profile", () => {
  it("returns all required fields", async () => {
    const req = createTestRequest("/api/user/profile")
    const res = await GET(req)
    const { status, json } = await parseResponse(res)

    expect(status).toBe(200)
    for (const field of REQUIRED_FIELDS) {
      expect(json.data).toHaveProperty(field)
    }
  })

  // Regression test — this exact bug was introduced by AI 4 times
  it("notification_settings is not undefined (BUG-R1 regression)", async () => {
    const req = createTestRequest("/api/user/profile")
    const res = await GET(req)
    const { json } = await parseResponse(res)

    expect("notification_settings" in json.data).toBe(true)
    const ns = json.data.notification_settings
    expect(ns === null || typeof ns === "object").toBe(true)
  })
})
```

### Testing Sandbox/Production Parity

The most common AI regression: fixing production path but forgetting sandbox path (or vice versa).

```typescript
// Test that sandbox responses match the expected contract
describe("GET /api/user/messages (conversation list)", () => {
  it("includes partner_name in sandbox mode", async () => {
    const req = createTestRequest("/api/user/messages", {
      sandboxUserId: "user-001",
    })
    const res = await GET(req)
    const { json } = await parseResponse(res)

    // This caught a bug where partner_name was added
    // to production path but not sandbox path
    if (json.data.length > 0) {
      for (const conv of json.data) {
        expect("partner_name" in conv).toBe(true)
      }
    }
  })
})
```

## Integrating Tests into Bug-Check Workflow

### Custom Command Definition

```markdown
<!-- .nemo/commands/bug-check.md -->

# Bug Check

## Step 1: Automated Tests (mandatory, cannot skip)

Run these commands FIRST before any code review:

    npm run test       # Vitest test suite
    npm run build      # TypeScript type check + build

- If tests fail → report as highest priority bug
- If build fails → report type errors as highest priority
- Only proceed to Step 2 if both pass

## Step 2: Code Review (AI review)

1. Sandbox / production path consistency
2. API response shape matches frontend expectations
3. SELECT clause completeness
4. Error handling with rollback
5. Optimistic update race conditions

## Step 3: For each bug fixed, propose a regression test
```

### The Workflow

```
User: "バグチェックして" (or "/bug-check")
  │
  ├─ Step 1: npm run test
  │   ├─ FAIL → Bug found mechanically (no AI judgment needed)
  │   └─ PASS → Continue
  │
  ├─ Step 2: npm run build
  │   ├─ FAIL → Type error found mechanically
  │   └─ PASS → Continue
  │
  ├─ Step 3: AI code review (with known blind spots in mind)
  │   └─ Findings reported
  │
  └─ Step 4: For each fix, write a regression test
      └─ Next bug-check catches if fix breaks
```

## Common AI Regression Patterns

### Pattern 1: Sandbox/Production Path Mismatch

**Frequency**: Most common (observed in 3 out of 4 regressions)

```typescript
// FAIL: AI adds field to production path only
if (isSandboxMode()) {
  return { data: { id, email, name } } // Missing new field
}
// Production path
return { data: { id, email, name, notification_settings } }

// PASS: Both paths must return the same shape
if (isSandboxMode()) {
  return { data: { id, email, name, notification_settings: null } }
}
return { data: { id, email, name, notification_settings } }
```

**Test to catch it**:

```typescript
it("sandbox and production return same fields", async () => {
  // In test env, sandbox mode is forced ON
  const res = await GET(createTestRequest("/api/user/profile"))
  const { json } = await parseResponse(res)

  for (const field of REQUIRED_FIELDS) {
    expect(json.data).toHaveProperty(field)
  }
})
```

### Pattern 2: SELECT Clause Omission

**Frequency**: Common with Supabase/Prisma when adding new columns

```typescript
// FAIL: New column added to response but not to SELECT
const { data } = await supabase
  .from("users")
  .select("id, email, name") // notification_settings not here
  .single()

return { data: { ...data, notification_settings: data.notification_settings } }
// → notification_settings is always undefined

// PASS: Use SELECT * or explicitly include new columns
const { data } = await supabase.from("users").select("*").single()
```

### Pattern 3: Error State Leakage

**Frequency**: Moderate — when adding error handling to existing components

```typescript
// FAIL: Error state set but old data not cleared
catch (err) {
  setError("Failed to load");
  // reservations still shows data from previous tab!
}

// PASS: Clear related state on error
catch (err) {
  setReservations([]);  // Clear stale data
  setError("Failed to load");
}
```

### Pattern 4: Optimistic Update Without Proper Rollback

```typescript
// FAIL: No rollback on failure
const handleRemove = async (id: string) => {
  setItems((prev) => prev.filter((i) => i.id !== id))
  await fetch(`/api/items/${id}`, { method: "DELETE" })
  // If API fails, item is gone from UI but still in DB
}

// PASS: Capture previous state and rollback on failure
const handleRemove = async (id: string) => {
  const prevItems = [...items]
  setItems((prev) => prev.filter((i) => i.id !== id))
  try {
    const res = await fetch(`/api/items/${id}`, { method: "DELETE" })
    if (!res.ok) throw new Error("API error")
  } catch {
    setItems(prevItems) // Rollback
    alert("削除に失敗しました")
  }
}
```

## Strategy: Test Where Bugs Were Found

Don't aim for 100% coverage. Instead:

```
Bug found in /api/user/profile     → Write test for profile API
Bug found in /api/user/messages    → Write test for messages API
Bug found in /api/user/favorites   → Write test for favorites API
No bug in /api/user/notifications  → Don't write test (yet)
```

**Why this works with AI development:**

1. AI tends to make the **same category of mistake** repeatedly
2. Bugs cluster in complex areas (auth, multi-path logic, state management)
3. Once tested, that exact regression **cannot happen again**
4. Test count grows organically with bug fixes — no wasted effort

## Quick Reference

| AI Regression Pattern       | Test Strategy                              | Priority |
| --------------------------- | ------------------------------------------ | -------- |
| Sandbox/production mismatch | Assert same response shape in sandbox mode | High     |
| SELECT clause omission      | Assert all required fields in response     | High     |
| Error state leakage         | Assert state cleanup on error              | Medium   |
| Missing rollback            | Assert state restored on API failure       | Medium   |
| Type cast masking null      | Assert field is not undefined              | Medium   |

## DO / DON'T

**DO:**

- Write tests immediately after finding a bug (before fixing it if possible)
- Test the API response shape, not the implementation
- Run tests as the first step of every bug-check
- Keep tests fast (< 1 second total with sandbox mode)
- Name tests after the bug they prevent (e.g., "BUG-R1 regression")

**DON'T:**

- Write tests for code that has never had a bug
- Trust AI self-review as a substitute for automated tests
- Skip sandbox path testing because "it's just mock data"
- Write integration tests when unit tests suffice
- Aim for coverage percentage — aim for regression prevention

Wersje

RepozytoriumWynikLicencja
Ta strona · SHRIJANAGAIN/Nemo-Cli100Apache-2.0
ginaservice/ginaservice-skills100MIT
yeaight7/agent-powerups100Apache-2.0

Migawka bazy z 13.11.2026 · wersja paczki 2df41b0013d0 (skrót SHA-256 całej paczki)

Pobrania w ostatnich 30 dniach: 0 · ranking

Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.

Podobne skille

  • 100api-testing-patterns
    Podaje wzorce testowania API: testy kontraktowe, testy REST i GraphQL oraz testy integracyjne, dla strategii testów API.
  • 100qe-api-testing-patterns
    Stosuje wzorce testowania API: testy kontraktowe, REST i GraphQL oraz testy integracyjne, przy projektowaniu strategii testów API.
  • 100tracking-regression-tests
    Śledzi zestawy testów regresyjnych w kolejnych wydaniach i zarządza nimi, w tym walidacją względem wersji bazowej.
  • 100tracking-regression-tests
    Śledzi zestawy testów regresyjnych w kolejnych wydaniach i zarządza nimi, w tym walidacją względem wersji bazowej.
  • 100playwright-testing
    Wspiera testy E2E w Playwright: wiele przeglądarek, regresja wizualna, testy API i emulacja mobilna, przy pisaniu testów lub konfiguracji automatycznych testów UI.
  • 100tests-that-catch-bugs
    Pisze kilka testów regresyjnych, z których każdy nazywa błąd i sprawdza wynik tam, gdzie widzi go użytkownik.

Uwagi i zgłoszenia

Wiadomość trafi do Bartosza Sokolińskiego. Adres e-mail jest potrzebny tylko do odpowiedzi. Szczegóły w polityce prywatności.