Skip to content
Skills Library
AI & agents · Tool integration

server-inference

/server-inference

Use this skill when the user wants to run or debug MLX-VLM server inference, including uv run mlx_vlm.server, /v1/models, /v1/chat/completions, /v1/responses, streaming, OpenAI-compatible clients, health checks, metrics, model unload/reload, adapters, trust-remote-code, and server request/response failures.

Repository
mlx-vlm ↗
Licence
MIT
Last repository change
27.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
5
Deprecation

No signals detected. Repository changed on 27.10.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Use this skill when the user wants to run or debug MLX-VLM server inference, including uv run mlx_vlm.server, /v1/models, /v1/chat/completions, /v1/responses, streaming, OpenAI-compatible clients, health checks, metrics, model unload/reload, adapters, trust-remote-code, and server request/response failures.

Original author description

Use this skill when the user wants to run or debug MLX-VLM server inference, including uv run mlx_vlm.server, /v1/models, /v1/chat/completions, /v1/responses, streaming, OpenAI-compatible clients, health checks, metrics, model unload/reload, adapters, trust-remote-code, and server request/response failures.

How it works

  1. First Checks
  2. Startup
  3. Minimal Checks
  4. Endpoint Surface (beyond chat/responses)
  5. Debugging Rules
  6. Validation
Original SKILL.md headings · 83 lines, 423 words
SKILL.md preview
---
name: server-inference
description: Use this skill when the user wants to run or debug MLX-VLM server inference, including uv run mlx_vlm.server, /v1/models, /v1/chat/completions, /v1/responses, streaming, OpenAI-compatible clients, health checks, metrics, model unload/reload, adapters, trust-remote-code, and server request/response failures.
---

# Server Inference

Use this workflow for the FastAPI server and API-compatible inference.

## First Checks

1. Identify the server command, model, port, request endpoint, request body, and expected response.
2. Start with health/model-list checks before debugging generation.
3. Separate server startup failures from request-handling failures.
4. Keep streaming and non-streaming repros separate.

## Startup

```bash
uv run mlx_vlm.server \
  --model <model-or-path> \
  --port 8080
```

Useful startup flags include `--adapter-path`, `--trust-remote-code`, `--log-level`, `--enable-thinking`, `--thinking-budget`, `--draft-model`, `--draft-kind`, `--kv-bits`, `--kv-quant-scheme`, `--max-kv-size`, and `--vision-cache-size`.

## Minimal Checks

```bash
curl http://127.0.0.1:8080/health
curl http://127.0.0.1:8080/v1/models
curl http://127.0.0.1:8080/metrics
```

Minimal chat request:

```bash
curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "<model-or-path>",
    "messages": [{"role": "user", "content": "Say hello."}],
    "max_tokens": 32,
    "stream": false
  }'
```

Minimal Responses API request:

```bash
curl -s http://127.0.0.1:8080/v1/responses \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "<model-or-path>",
    "input": "Say hello.",
    "max_output_tokens": 32
  }'
```

## Endpoint Surface (beyond chat/responses)

The server exposes more than chat — cover or route to these as needed:

- **Anthropic Messages API:** `/v1/messages`, plus `/v1/messages/count` for token counting.
- **Audio:** `/v1/audio/speech` (TTS), `/v1/audio/transcriptions` and `/v1/audio/translations` (STT).
- **Images:** `/v1/images/generations` and `/v1/images/edits` (diffusion image models).
- **Cache & metrics:** `/v1/cache/stats`, `/v1/cache/reset`, `/v1/metrics`.
- **Models:** `/v1/models` lists loaded models and model candidates in the shared Hugging Face cache, with a `loaded` boolean indicating which are in memory — see `Skill("mlx-vlm-skills:hf-cache-models")`.

Match the model kind to the endpoint (an image endpoint needs a diffusion/image model, audio endpoints need an audio model); a mismatch returns a clear 4xx, not a crash.

## Debugging Rules

- Capture the server command, server logs, request body, response status, and response body.
- Test `/v1/models` after preload/load changes.
- For OpenAI client issues, reproduce with `curl` before blaming the client SDK.
- For streaming bugs, save raw event chunks and compare with non-streaming.
- For structured outputs, isolate the JSON schema and confirm whether the same request works without schema constraints.
- For tool calls, inspect native support and the parser inferred from the loaded processor's actual chat template. Follow [Tool Calling](../add-new-model/SKILL.md#tool-calling) for parser registration and validation.

## Validation

- For route/schema changes, run `uv run pytest mlx_vlm/tests/test_server.py -q`.
- For structured output changes, include `uv run pytest mlx_vlm/tests/test_generate.py -q -k 'thinking_aware or json_schema'`.
- For tool parser changes, add compact regressions to `TestProcessToolCalls` in `mlx_vlm/tests/test_server.py` and run the shared tool streaming tests in that module, plus any existing tests for affected parsers. Reuse consolidated tests instead of creating a new test file.
- If the result is a user-facing bug report, switch to `Skill("mlx-vlm-skills:reproducible-github-issues")`.

Versions

The database has no other skills named server-inference.

Database snapshot: 13.11.2026 · package version ad6bafac0b78 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100cli-inference
    Use this skill when the user wants to run or debug MLX-VLM inference from the command line, including uv run mlx_vlm.generate, image/audio/video inputs, local model paths, Hugging Face model IDs, deterministic repro commands, and CLI errors around processors, prompts, model…
  • 100mlx-vlm
    Runs and fine-tunes vision language models (image, audio and video plus text) locally on Apple Silicon Macs with MLX, from a CLI, a Python API or an OpenAI-compatible server. Use when installing mlx-vlm, describing or extracting text from images with a local model, batch…
  • 100iblai-api-inference
    Run inference against an ibl.ai deployment through its OpenAI-compatible API — same request/response shape as OpenAI's /v1/chat/completions, routed to any configured provider/model (e.g. openai/gpt-5). List models, then POST OpenAI-format messages for a single completion or a…
  • 100hf-cache-models
    Use this skill when the user wants to list, inspect, or report cached, custom, or loaded MLX-VLM models through the server's model discovery endpoint.
  • 100minicpm5-deploy-lmstudio
    Run MiniCPM5-1B or MiniCPM5-2B in LM Studio (desktop GUI) using either the GGUF runtime (cross-platform) or the MLX runtime (Apple Silicon, faster). Includes OpenAI-compatible local server. Use when the user mentions "LM Studio", desktop GUI inference, "lms" CLI, or wants a…
  • 100azure-ai-inference
    Use when calling Azure-hosted AI models via the Azure.AI.Inference SDK. Covers chat completions, embeddings, streaming, model selection, and Azure AI model catalog integration. USE FOR: calling Azure AI model catalog models, Azure OpenAI chat completions, generating embeddings…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.