Skip to content
Skills Library
AI & agents · Testing

evaluate-conversation-memory-layer

/evaluate-conversation-memory-layer

Evaluate Conversation Memory — measure recall accuracy, compression quality, PII scrubbing, memory relevance over time, storage efficiency. Use when: evaluate, test memory quality.

Repository
frootai ↗
Licence
MIT
Last repository change
26.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
3
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 26.10.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Evaluate Conversation Memory — measure recall accuracy, compression quality, PII scrubbing, memory relevance over time, storage efficiency. Use when: evaluate, test memory quality.

Original author description

Evaluate Conversation Memory — measure recall accuracy, compression quality, PII scrubbing, memory relevance over time, storage efficiency. Use when: evaluate, test memory quality.

How it works

  1. When to Use
  2. Memory Quality Metrics & Targets
  3. Step 1: Prepare Memory Test Scenarios
  4. Step 2: Evaluate Recall Accuracy
  5. Step 3: Evaluate Compression Quality
  6. Step 4: PII Scrubbing Validation
  7. Step 5: Cross-Session Continuity Test
  8. Step 6: Generate Report
  9. Evaluation Cadence
  10. Common Failure Patterns
Original SKILL.md headings · 100 lines, 665 words
SKILL.md preview
---
name: evaluate-conversation-memory-layer
description: "Evaluate Conversation Memory — measure recall accuracy, compression quality, PII scrubbing, memory relevance over time, storage efficiency. Use when: evaluate, test memory quality."
---

# Evaluate Conversation Memory Layer

## When to Use
- Evaluate recall accuracy (does the right memory surface for the query?)
- Measure compression quality (key facts preserved after compression?)
- Validate PII scrubbing completeness
- Assess memory relevance decay over time
- Gate deployments with memory quality thresholds

## Memory Quality Metrics & Targets

| Metric | Target | How to Measure |
|--------|--------|----------------|
| Recall precision | ≥ 85% | Relevant memories / returned memories |
| Recall speed | < 200ms | Vector search + Redis lookup time |
| Compression retention | ≥ 90% key facts | Compare compressed vs original |
| PII scrub completeness | 100% | Known PII in test data detected |
| Memory staleness | < 5% stale memories | User reports irrelevant old memories |
| Cross-session continuity | ≥ 80% | Agent references previous session correctly |
| Storage efficiency | < 1KB/session avg | Compressed memory per session |
| User deletion success | 100% | All user data removed on request |

## Step 1: Prepare Memory Test Scenarios
```json
{"user_id": "test-01", "sessions": [
  {"turns": ["My name is Alice", "I prefer dark mode"], "expected_memories": ["name: Alice", "preference: dark mode"]},
  {"turns": ["Remind me what I said last time"], "expected_recall": ["name: Alice", "preference: dark mode"]}
]}
```
Minimum: 20 multi-session test users with known memory expectations.

## Step 2: Evaluate Recall Accuracy
```bash
python evaluation/eval.py --test-set evaluation/test-set.jsonl --metrics recall_accuracy
```
- Query with new session prompt → check if relevant past memories surface
- Score: exact match, partial match, irrelevant, missing
- Track per-tier accuracy: short-term vs long-term vs episodic

## Step 3: Evaluate Compression Quality
- Compare original conversation (10 turns, ~4000 tokens) vs compressed (~200 tokens)
- Key facts preservation test: are names, preferences, decisions preserved?
- Lossy compression acceptable for filler; not for actionable information

## Step 4: PII Scrubbing Validation
```bash
python evaluation/eval.py --test-set evaluation/test-set.jsonl --metrics pii_scrub
```
- Inject known PII (SSN, email, phone, CC) into conversations
- Verify: PII removed from long-term and episodic tiers
- Short-term may retain PII (it expires in 15 min)

## Step 5: Cross-Session Continuity Test
- Session 1: User shares preferences and context
- Session 2 (next day): User asks "what did I say yesterday?"
- Agent should recall preferences from episodic memory
- Track: how many sessions before memory becomes stale?

## Step 6: Generate Report
```bash
python evaluation/eval.py --all --output evaluation/memory-report.json --ci-gate
```

### Quality Gate Decision
| Result | Action |
|--------|--------|
| All PASS | Deploy memory layer to production |
| Recall < 75% | Improve embedding model or lower similarity threshold |
| Compression loses facts | Refine compression prompt ("preserve all preferences") |
| PII found in long-term | Block — fix scrubber before deploying |
| Cross-session broken | Check Cosmos DB TTL and partition key |

## Evaluation Cadence
- **Pre-deployment**: Full multi-session memory evaluation
- **Weekly**: Recall accuracy spot-check on production users
- **Monthly**: PII scrub audit, storage growth review
- **On model change**: Re-evaluate compression quality

## Common Failure Patterns

| Symptom | Root Cause | Fix |
|---------|-----------|-----|
| Agent forgets previous session | Compression dropped key fact | Improve compression prompt |
| Wrong user's memory surfaces | Partition key not set | Add user_id filter to all queries |
| Memory bloated (>5KB/session) | No compression trigger | Trigger compression at 5 turns |
| Stale preferences recalled | No memory update mechanism | Add "update on contradiction" logic |
| GDPR delete incomplete | Vector embeddings not deleted | Delete from ALL tiers including vector index |
| Recall returns nothing | Embedding model mismatch | Ensure same model for storage and query |

## CI/CD Integration
```yaml
- name: Memory Recall Gate
  run: python evaluation/eval.py --metrics recall_accuracy --ci-gate --threshold 0.85
- name: PII Scrub Gate
  run: python evaluation/eval.py --metrics pii_scrub --ci-gate --threshold 1.0
- name: Cross-Session Gate
  run: python evaluation/eval.py --metrics cross_session --ci-gate --threshold 0.80
```
Files in this package (3)
  • SKILL.lean.md 5 KB
  • SKILL.md 5 KB
  • agents/openai.yaml 1 KB

Versions

The database has no other skills named evaluate-conversation-memory-layer.

Database snapshot: 13.11.2026 · package version bd7d48371dcc (SHA-256 digest of the entire package)

Downloads in the last 30 days: 1 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100tune-conversation-memory-layer
    Tune Conversation Memory — optimize compression ratio, recall thresholds, memory TTL per tier, embedding model for recall, storage cost per user. Use when: tune, optimize memory.
  • 100deploy-conversation-memory-layer
    Deploy Conversation Memory Layer — configure tiered memory (short-term Redis, long-term Cosmos DB, episodic vector store), compression pipeline, PII-aware storage, recall API. Use when: deploy, provision memory.
  • 88memory
    Persistent global memory across conversations. Read memory file at start of every conversation, write important context during conversation so it survives context compression. Use ALWAYS. Triggered automatically.
  • 90conversation-memory
    Persistent memory systems for LLM conversations including short-term, long-term, and entity-based memory Use when: conversation memory, remember, memory persistence, long-term memory, chat history.
  • 90conversation-memory
    Persistent memory systems for LLM conversations including short-term, long-term, and entity-based memory Use when: conversation memory, remember, memory persistence, long-term memory, chat history.
  • 90conversation-memory
    Persistent memory systems for LLM conversations including short-term, long-term, and entity-based memory Use when: conversation memory, remember, memory persistence, long-term memory, chat history.

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.