autocontext
/autocontext
Iterative strategy generation and evaluation system. Use when the user wants to evaluate agent output quality, run improvement loops, queue tasks for background evaluation, check run status, inspect runtime artifacts and session branch lineage, or discover available scenarios. Provides LLM-based judging with rubric-driven scoring.
- Author
- greyhaven-ai ↗
- Repository
- autocontext ↗
- Licence
- Apache-2.0
- Last repository change
- 26.10.2026
Assessment and security
- Files in this package
- 1
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 26.10.2026, without references to retired models or APIs.
Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.
What it does and when to use it
Iterative strategy generation and evaluation system. Use when the user wants to evaluate agent output quality, run improvement loops, queue tasks for background evaluation, check run status, inspect runtime artifacts and session branch lineage, or discover available scenarios. Provides LLM-based judging with rubric-driven scoring.
Original author description
Iterative strategy generation and evaluation system. Use when the user wants to evaluate agent output quality, run improvement loops, queue tasks for background evaluation, check run status, inspect runtime artifacts and session branch lineage, or discover available scenarios. Provides LLM-based judging with rubric-driven scoring.
How it works
- Available Tools
- Quick Start
- Configuration
- CLI Companion
SKILL.md preview
--- name: autocontext description: > Iterative strategy generation and evaluation system. Use when the user wants to evaluate agent output quality, run improvement loops, queue tasks for background evaluation, check run status, inspect runtime artifacts and session branch lineage, or discover available scenarios. Provides LLM-based judging with rubric-driven scoring. allowed-tools: autocontext_judge autocontext_improve autocontext_status autocontext_scenarios autocontext_queue autocontext_runtime_snapshot --- # autocontext autocontext is an iterative strategy generation and evaluation system that uses LLM-based judging to score and improve agent outputs. ## Available Tools - **autocontext_judge** — Evaluate agent output against a rubric. Returns a 0–1 score with reasoning and per-dimension breakdowns. - **autocontext_improve** — Run a multi-round improvement loop. The agent output is judged, revised based on feedback, and re-evaluated until the quality threshold is met or max rounds are exhausted. - **autocontext_queue** — Enqueue a task for background evaluation by the task runner daemon. - **autocontext_status** — Check the status of runs and queued tasks. - **autocontext_scenarios** — List available evaluation scenarios and their families. - **autocontext_runtime_snapshot** — Inspect run artifacts, package provenance, branchable session lineage, and recent event-stream entries. ## Quick Start ### 1. Evaluate output quality Use `autocontext_judge` with a task prompt, the agent's output, and a rubric: ``` autocontext_judge( task_prompt="Write a Python function to parse CSV files", agent_output="def parse_csv(path): ...", rubric="Correctness, error handling, edge cases, documentation" ) ``` ### 2. Improve output iteratively Use `autocontext_improve` to automatically revise output through judge-guided feedback loops: ``` autocontext_improve( task_prompt="Write a Python function to parse CSV files", initial_output="def parse_csv(path): ...", rubric="Correctness, error handling, edge cases, documentation", max_rounds=5, quality_threshold=0.85 ) ``` ### 3. Queue background tasks Use `autocontext_queue` with a scenario name to enqueue evaluation tasks for asynchronous processing: ``` autocontext_queue(spec_name="my_scenario") ``` Check results later with `autocontext_status`. For deeper context, use `autocontext_runtime_snapshot` with the run ID. Add `session_id` when you need the active branch path before continuing work: ``` autocontext_runtime_snapshot(run_id="run_123", session_id="sess_123") ``` ### 4. Discover scenarios Use `autocontext_scenarios` to see what evaluation scenarios are available: ``` autocontext_scenarios() autocontext_scenarios(family="agent_task") ``` ## Configuration The extension auto-detects configuration from these sources: 1. **Project config** — `.autoctx.json` in the working directory (created via `autoctx init`) 2. **Environment variables:** - `AUTOCONTEXT_AGENT_PROVIDER` or `AUTOCONTEXT_PROVIDER` — Provider type - `AUTOCONTEXT_AGENT_API_KEY` or `AUTOCONTEXT_API_KEY` — Provider API key - `AUTOCONTEXT_AGENT_DEFAULT_MODEL` or `AUTOCONTEXT_MODEL` — Model override - `AUTOCONTEXT_DB_PATH` — SQLite database path override 3. **Pi provider** — Falls back to Pi's configured LLM provider ## CLI Companion For standalone usage outside Pi, install the `autoctx` CLI: ```bash npm install -g autoctx autoctx init autoctx solve "your problem" --iterations 5 autoctx simulate --description "your simulation" --runs 3 ```
Versions
| Repository | Score | Licence |
|---|---|---|
| This page · greyhaven-ai/autocontext | 100 | Apache-2.0 |
| greyhaven-ai/autocontext | 100 | Apache-2.0 |
Database snapshot: 13.11.2026 · package version 49d43aa4d2b0 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100autocontext-creator
Use when an agent needs to CREATE knowledge with Autocontext - run a scenario or plain-language task through the improvement loop, judge or improve a single output, and inspect what the run produced. Host-agnostic; requires only the autoctx CLI. - 100agentic-evaluation-framework
This skill should be used when the user asks to "evaluate LLM output quality", "set up LLM-as-judge", "build an eval rubric", "compare model outputs pairwise", or "measure agent quality". - 98skill-101
Patterns and techniques for evaluating and improving AI agent outputs. Use this skill when implementing self-critique and reflection loops, building evaluator-optimizer pipelines for quality-critical generation, creating test-driven code refinement workflows, designing… - 100autocontext-consumer
Use when an agent needs to USE knowledge Autocontext already produced - find which scenarios have knowledge, read the playbook and lessons for one, understand the on-disk file and folder layout, and move knowledge between checkouts. Host-agnostic; requires only the autoctx CLI and the filesystem. - 100agent-evaluation
This skill should be used when the user asks to "evaluate agent performance", "build test framework", "measure agent quality", "create evaluation rubrics", "implement LLM-as-judge", "compare model outputs", "mitigate evaluation bias", or mentions multi-dimensional evaluation,… - 100eval
Evaluate everything the PA agent manages — tasks, skills, PA network health, billing, calendar connections, and memory quality. Use when: owner asks for an evaluation, wants to know what's working and what isn't, or requests a performance report. Combines supervisor status with quality scoring.