Skip to content
Skills Library
AI & agents · Review & audit

llm-api-optimization

/llm-api-optimization

Use when an LLM-backed application's API calls need to get cheaper, faster or more reliable — prompt caching and the prefix rules that decide whether it hits, batch processing, context engineering and what earns a place in the window, payload format economics, structured outputs and constrained decoding, model routing by task, streaming and latency, and measuring cost per resolved task rather than per call.

Licence
MIT
Last repository change
31.07.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
1
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 31.07.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What to check before installing
  • SKILL.md zawiera komentarze HTML (1). Nie widać ich w podglądzie Markdown, a model je czyta.

What it does and when to use it

Use when an LLM-backed application's API calls need to get cheaper, faster or more reliable — prompt caching and the prefix rules that decide whether it hits, batch processing, context engineering and what earns a place in the window, payload format economics, structured outputs and constrained decoding, model routing by task, streaming and latency, and measuring cost per resolved task rather than per call.

Original author description

Use when an LLM-backed application's API calls need to get cheaper, faster or more reliable — prompt caching and the prefix rules that decide whether it hits, batch processing, context engineering and what earns a place in the window, payload format economics, structured outputs and constrained decoding, model routing by task, streaming and latency, and measuring cost per resolved task rather than per call.

How it works

  1. 1. The cost stack, in order of leverage
  2. 2. Prompt caching — it is a prefix cache, and that governs everything
  3. 3. Batch processing
  4. 4. Context engineering — decide what earns a place in the window
  5. 5. Payload format economics
  6. 6. Structured outputs
  7. 7. Model routing
  8. 8. Latency
  9. 9. Measure cost per resolved task, not per call
  10. 10. Anti-patterns
Original SKILL.md headings · 180 lines, 1,475 words
SKILL.md preview
---
name: llm-api-optimization
description: Use when an LLM-backed application's API calls need to get cheaper, faster or more reliable — prompt caching and the prefix rules that decide whether it hits, batch processing, context engineering and what earns a place in the window, payload format economics, structured outputs and constrained decoding, model routing by task, streaming and latency, and measuring cost per resolved task rather than per call.
disambiguation: The API CALL — what you send, how it is cached and billed, and how the response is shaped. Designing what to RETRIEVE and from where is rag-architecture; deciding whether the system should be an agent or a workflow at all is agentic-architecture; routing spawned harness work to model tiers is smart-config; defending against injection in what you send is llm-security.
---

# LLM API optimisation

<!-- REVIEW-BY: 2027-01-31 -->
**Verified 2026-07-29.** Prices, multipliers and TTLs move constantly and differ per provider —
**every number below is a shape to reason with, and must be re-checked against current provider
documentation before it goes in a budget.**

## 1. The cost stack, in order of leverage

| Lever | Typical saving | Effort |
|---|---|---|
| **Prompt caching** | Up to ~90% on the repeated portion | Low — often a few lines |
| **Batch processing** | ~50% on both input and output | Low, if latency permits |
| **Context engineering** | 40–60% | Medium — ongoing discipline |
| **Payload format** | 20–50% on structured payloads | Low |
| **Model routing** | Large, task-dependent | Medium |

**Caching and batching stack multiplicatively** — combined they can take the repeated portion of a
workload to a small fraction of list price. Realistic end-to-end reduction on a well-engineered
application is around 70–80%.

**Do the cheap structural work before switching to a smaller model.** Model downgrades trade quality
for cost; caching does not.

## 2. Prompt caching — it is a prefix cache, and that governs everything

The cache keys on an **exact prefix** of the request. Everything before the change is reusable;
everything from the first differing byte onward is not.

**Therefore: static content first, dynamic content last.** System prompt, tool definitions and
documents at the top; the user's turn at the bottom. This one ordering rule decides whether caching
works at all.

Typical economics (verify against current docs):

| | Multiplier |
|---|---|
| Cache **write**, short TTL | ~1.25× base input |
| Cache **write**, extended TTL | ~2× base input |
| Cache **read** | ~0.1× base input |

**Break-even is one hit on a short TTL, roughly two on an extended one.** So the calculation is not
"is caching good" but "will this prefix be read again inside its TTL". A prefix read once costs you
money.

**Extended TTL wins where the natural gap between requests exceeds the short window** — a support
agent handling a case over half an hour, an interactive session with thinking time. Paying the write
premium once beats re-writing every few minutes.

**Do not cache what will not be re-read.** A one-shot call with a large unique document is the
canonical waste.

### The cache-miss traps

Every one of these silently drops the hit rate to zero while the code looks correct:

- **A timestamp, request id or "today's date" near the top of the system prompt.** The single most
  common cause. Move it to the end of the message, or omit it.
- **Tool definitions reordered** between calls — a dictionary iteration order change is enough.
- **Non-deterministic JSON serialisation** of tools or schemas: unsorted keys, varying whitespace.
- **Per-user or per-session content placed before shared content.** Shared prefix first, always.
- **Any prompt-version A/B test that edits the head** of the system prompt rather than appending.
- **Editing the system prompt on deploy**, which cold-starts every cache at once — expected, but plan
  for the cost spike.

**Instrument the hit rate.** Providers report cache-read and cache-write tokens per response. A
caching implementation nobody measured is usually a caching implementation that is not working.

## 3. Batch processing

Asynchronous batch submission typically returns within a day at around half price on input and
output, and **stacks with caching**.

Fits: evaluation runs, back-fills, bulk classification, summarising a document corpus, nightly
enrichment. Does not fit: anything a user is waiting for.

**Look for the interactive work that is not actually interactive.** Enrichment triggered by a user
action but consumed hours later is batch work wearing a synchronous costume.

## 4. Context engineering — decide what earns a place in the window

Prompt engineering is what you say; **context engineering is what you provide**. On production
systems the second dominates the bill and most of the quality.

- **Selective retrieval over stuffing.** More context is not more accuracy — see `rag-architecture` §5.
- **Summarise or compact conversation history** rather than replaying it turn by turn. Providers now
  offer first-class compaction and server-side memory for exactly this; a hand-rolled rolling summary
  works too.
- **Externalise durable facts.** Anything stable belongs in a cached prefix or a memory store, not
  re-sent every turn.
- **Trim tool definitions to what this call can plausibly use.** Tool schemas are input tokens on
  *every* request, and a large unused toolset also degrades selection.
- **Budget the window explicitly.** Overrun does not degrade gracefully — it truncates context, drops
  tool calls and malforms output.

## 5. Payload format economics

**JSON costs roughly twice the tokens of YAML or TSV for the same data** — quotes, braces and
repeated keys on every record.

- **Internal, machine-to-machine payloads**: use the compact format. Tabular data as TSV/CSV with one
  header row is dramatically cheaper than an array of objects repeating keys per row.
- **Model *output* that must be parsed**: keep JSON, and constrain it (§6). Output tokens are the
  expensive ones, but reliable parsing is worth more than the saving.
- **Strip what nobody reads** — nulls, empty arrays, internal audit fields, deep nesting that exists
  for a database's benefit rather than the model's.

**Measure before rewriting a serialiser.** This lever is large on high-volume structured payloads and
negligible on prose-shaped workloads.

## 6. Structured outputs

Two approaches, and the difference matters under load:

| Approach | How | Trade-off |
|---|---|---|
| **Constrained decoding** | The decoder can only emit tokens valid for the schema | Structurally guaranteed; needs provider or local support |
| **Schema-in-prompt + validate + retry** | Ask for a shape, validate, retry on failure | Portable; retries cost tokens and latency |

**Validate the output regardless of which you use.** Schema-valid is not the same as semantically
correct — a required field can be present and wrong. Types are not a business-rule check.

**Never parse structure out of prose.** Regexing an amount out of a sentence works in testing and
fails on the first unusual phrasing.

## 7. Model routing

**Route by task, not by habit.** Classification, extraction, routing and formatting rarely need the
largest model; multi-step reasoning, ambiguous judgement and code architecture do.

Build the routing decision as a policy in one place, not as a hard-coded model id at each call site —
that is what makes re-tiering a config change instead of a refactor. Inside this harness, that policy
is `smart-config`; in an application, a small resolver reading configuration does the same job.

**Escalate on evidence.** A cheap model plus a validity check, escalating on failure, is often
cheaper *and* better than the large model everywhere — provided the check is real.

## 8. Latency

- **Stream when a human is reading.** Time-to-first-token dominates perceived speed even when total
  time is unchanged.
- **Parallelise independent calls.** Sequential awaits on unrelated requests are the same waterfall
  mistake as in a front end.
- **Cache reads are faster as well as cheaper** — the prefix is already processed. Caching is a
  latency optimisation people forget to claim.
- **Speculative or eager calls burn real money.** Firing a request the user may not need is only
  worth it where the hit rate is measured and high.

## 9. Measure cost per resolved task, not per call

**Per-call cost is the wrong denominator.** A cheaper model that needs three attempts, a retry loop
that hides two extra calls, or an agent that retrieves the same document five times all look fine
per call and terrible per outcome.

Log per request: input tokens, **cache-read and cache-write tokens**, output tokens, model, latency,
and a task id that ties retries and sub-calls to one unit of work. Then track:

- **Cache hit rate** — the first number to check when a bill moves.
- **Tokens per resolved task** — the number that actually tracks the bill.
- **Retry and escalation rate** — where quality problems show up as cost.

**Set a per-task budget and alert on breach.** Agentic systems fail expensively in loops, and an
unbounded retry is a bill, not an error.

## 10. Anti-patterns

- **Dynamic content at the top of the prompt**, silently defeating the cache.
- **A timestamp in the system prompt.** The classic zero-hit-rate bug.
- **Caching a prefix that is read once.**
- **Never measuring the hit rate**, then assuming caching is working.
- **Synchronous calls for work nobody is waiting for.**
- **Replaying full conversation history** every turn instead of compacting.
- **Shipping every tool definition** on every call.
- **Verbose JSON for high-volume internal payloads.**
- **Parsing structure out of prose** instead of constraining the output.
- **Hard-coding a model id at every call site.**
- **Downgrading the model before doing the free structural work.**
- **Reporting cost per call** while the retries hide in another metric.

Versions

The database has no other skills named llm-api-optimization.

Database snapshot: 13.11.2026 · package version 5192c69a5cd2 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100llm-cost-optimization
    Use when an LLM feature costs too much. Covers prompt caching, context reduction, model routing, batching, output limits, and finding where the tokens actually go. Trigger on "our openai bill is too high", "llm costs are out of control", "where are all these tokens going", "set…
  • 100llm-cost-and-latency-optimization
    Guides reducing token cost and response latency of LLM-based agents without degrading quality. Use when a user asks to "reduce our LLM API bill," "make the agent respond faster," "our token usage is too high," "should we use a smaller/cheaper model here," decide where to apply…
  • 100cost-aware-llm-pipeline
    Cost optimization patterns for LLM API usage — model routing by task complexity, budget tracking, retry logic, and prompt caching. Use when LLM spend needs to come down, or when routing tasks across model tiers and budgets.
  • 100llm-cost-latency
    Cut LLM cost and latency with caching, model tiering, prompt diet, batching, and streaming UX. Use when the inference bill or response time needs engineering down.
  • 100cost-aware-llm-pipeline
    Cost optimization patterns for LLM API usage — model routing by task complexity, budget tracking, retry logic, and prompt caching. Use when LLM spend needs to come down, or when routing tasks across model tiers and budgets.
  • 100cost-latency-optimization-prompt-caching
    Use this skill when reducing LLM token costs and inference latency using provider prompt caching, model cascades, and batch APIs.

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.