molmo
/molmo
Visual pointing and Q&A via the Molmo VLM served from a self-hosted vLLM endpoint (OpenAI-compatible API). Use when a workflow needs a single pixel coordinate for a named object (point_prompt) or Molmo-grade visual question answering and a vLLM server is available; for an API-only zero-GPU alternative use the gemini-er bundle.
- Author
- graph-robots ↗
- Repository
- open-robot-skills ↗
- Licence
- Apache-2.0
- Last repository change
- 12.09.2026
Assessment and security
- Files in this package
- 5
- Scripts
- 1
- HTML comments
- 0
- URLs in SKILL.md
- 2
No signals detected. Repository changed on 12.09.2026, without references to retired models or APIs.
Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.
- 1 scripts in this package. Review them before installing, as Claude may execute them.
Scripts: tools.py
What it does and when to use it
Visual pointing and Q&A via the Molmo VLM served from a self-hosted vLLM endpoint (OpenAI-compatible API). Use when a workflow needs a single pixel coordinate for a named object (point_prompt) or Molmo-grade visual question answering and a vLLM server is available; for an API-only zero-GPU alternative use the gemini-er bundle.
Original author description
Visual pointing and Q&A via the Molmo VLM served from a self-hosted vLLM endpoint (OpenAI-compatible API). Use when a workflow needs a single pixel coordinate for a named object (point_prompt) or Molmo-grade visual question answering and a vLLM server is available; for an API-only zero-GPU alternative use the gemini-er bundle.
How it works
- Hosting recipe (vLLM)
- Config
- Notes
SKILL.md preview
---
name: molmo
description: Visual pointing and Q&A via the Molmo VLM served from a
self-hosted vLLM endpoint (OpenAI-compatible API). Use when a workflow needs
a single pixel coordinate for a named object (point_prompt) or Molmo-grade
visual question answering and a vLLM server is available; for an API-only
zero-GPU alternative use the gemini-er bundle.
license: MIT
compatibility: requires gap>=0.1
metadata: {category: perception, tags: [perception, pointing, molmo, vllm]}
gap:
requires: {env: [GAP_MOLMO_BASE_URL]}
serving:
command: ["python", "-m", "gap_core.rpc.server", "--bundle", "molmo"]
protocol: stdio-msgpack
tools:
- molmo.point_prompt: Point at a named object in an image; returns pixel coordinates.
- molmo.query: Free-form visual question answering via Molmo.
- molmo.query_yes_no: Yes/no visual question answering via Molmo; coerces the reply to a bool.
---
# molmo
Molmo visual pointing + Q&A. The bundle itself is zero-GPU (httpx client
only), but Molmo has **no hosted API** — you must serve it yourself with
vLLM and point `GAP_MOLMO_BASE_URL` at it. If you can't self-host, use
`gemini-er.detect` (hosted Gemini Robotics-ER) instead.
## Hosting recipe (vLLM)
Lifted from the dev tree's run book (`training/README.md`):
```bash
# Serve Molmo2-8B on an OpenAI-compatible endpoint
CUDA_VISIBLE_DEVICES=0 PYTHONNOUSERSITE=1 \
python -m vllm.entrypoints.openai.api_server \
--model allenai/Molmo2-8B \
--trust-remote-code \
--dtype bfloat16 \
--port 8122 \
--gpu-memory-utilization 0.5 \
--max-model-len 4096 \
--max-num-batched-tokens 4096
# Smoke-test
curl -s http://127.0.0.1:8122/v1/models | jq '.data[0].id'
# Point the bundle at it
export GAP_MOLMO_BASE_URL=http://127.0.0.1:8122/v1
```
Operational notes from the dev tree: pin Molmo to its own GPU when running
alongside other perception services — under heavy parallel evaluation it
becomes the throughput bottleneck if co-located; on a dedicated GPU you can
push `--gpu-memory-utilization 0.85 --max-num-batched-tokens 8192` for ~2×
perception throughput. The server can also run on a remote machine and be
port-forwarded in (the 4090 real-robot profile did exactly this).
## Config
| Env | Meaning | Default |
|----------------------|--------------------------------------|----------------------|
| `GAP_MOLMO_BASE_URL` | vLLM OpenAI-compatible base URL | — (required) |
| `GAP_MOLMO_MODEL` | Model name served by vLLM | `allenai/Molmo2-8B` |
## Notes
- `molmo.point_prompt` sends the canonical `"Point at <query>"` prompt and
parses all four Molmo point output formats (Molmo2 `<points coords=...>`,
Molmo1 `<point x= y=>`, legacy `<points x1= y1= ...>`, plain `x, y`
fallback), converting normalized coordinates to pixels. `found=False`
means the model emitted no parseable point.
- `molmo.query_yes_no` coerces with the source-verbatim rule: answer is true
iff `"yes"` appears in the lowercased reply.
- Backend unreachable after 3 retries raises `ToolError` (route `on_error`).
Files in this package (5)
- .python-version 1 KB
- SKILL.md 3 KB
- pyproject.toml 1 KB
- tools.py 11 KB
- uv.lock 131 KB
Versions
The database has no other skills named molmo.
Database snapshot: 13.11.2026 · package version 27eab239bce7 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 100minicpm5-deploy-vllm
Serve MiniCPM5-1B or MiniCPM5-2B via vLLM as an OpenAI-compatible HTTP server. Use when the user wants high-throughput production serving on NVIDIA GPU, asks for "vLLM", "OpenAI server", "REST API for MiniCPM5", or "production deployment". - 100vllm-setup
Deploy a vLLM inference server on an NVIDIA DGX Station GB300 with validated container, GPU targeting, and tuning parameters. Use when the user asks to serve a model with vLLM, start a vLLM endpoint, or set up OpenAI-compatible inference on DGX Station. - 100vllm
Complete vLLM v0.19.0 Python API reference for high-throughput LLM inference: offline batch generation, chat, embeddings, classification, structured outputs, LoRA adapters, multimodal inputs, and OpenAI-compatible server. Triggers on: vLLM questions, Python LLM serving, GPU… - 90vlm
Free-form and yes/no visual question answering against a hosted vision-language model (OpenRouter API by default; Vertex AI Gemini selectable by config). Use when a workflow needs scene descriptions, semantic checks ("is the drawer open?"), or LLM-judged verification of a camera… - 100nvidia-nemoclaw-user-configure-inference
Connects NemoClaw to a local inference server. Use when setting up Ollama, vLLM, TensorRT-LLM, NIM, or any OpenAI-compatible local model server with NemoClaw. Trigger keywords - nemoclaw local inference, ollama nemoclaw, vllm nemoclaw, local model server, openai compatible… - 100vllm
Operate, configure, benchmark, and troubleshoot vLLM inference servers: Docker and Kubernetes deployment, quantization-aware model configuration (tensor parallelism, KV cache), OpenAI-compatible API serving, throughput and latency benchmarking, continuous batching tuning, GPU…