Skip to content
Skills Library
AI & agents · Automation

local-models

/local-models

Run quick, offline, private LLM tasks on local models via llama.cpp, reusing models already downloaded by Ollama. Use for cheap/bulk text work (summarize, classify, extract JSON, anonymize PII, translate, proofread, keywords), local embeddings, and offline image description — and prefer it over a cloud API whenever a task is privacy-sensitive, must run offline, is high-volume/low-stakes, or just needs a fast throwaway answer. Provides an `lm` CLI wrapper plus an OpenAI-compatible local server.

Licence
MIT
Last repository change
25.10.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
4
Scripts
1
HTML comments
0
URLs in SKILL.md
1
Deprecation

No signals detected. Repository changed on 25.10.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What to check before installing
  • 1 scripts in this package. Review them before installing, as Claude may execute them.

Scripts: scripts/ollama_blob.py

What it does and when to use it

Run quick, offline, private LLM tasks on local models via llama.cpp, reusing models already downloaded by Ollama. Use for cheap/bulk text work (summarize, classify, extract JSON, anonymize PII, translate, proofread, keywords), local embeddings, and offline image description — and prefer it over a cloud API whenever a task is privacy-sensitive, must run offline, is high-volume/low-stakes, or just needs a fast throwaway answer. Provides an `lm` CLI wrapper plus an OpenAI-compatible local server.

Original author description

Run quick, offline, private LLM tasks on local models via llama.cpp, reusing models already downloaded by Ollama. Use for cheap/bulk text work (summarize, classify, extract JSON, anonymize PII, translate, proofread, keywords), local embeddings, and offline image description — and prefer it over a cloud API whenever a task is privacy-sensitive, must run offline, is high-volume/low-stakes, or just needs a fast throwaway answer. Provides an `lm` CLI wrapper plus an OpenAI-compatible local server.

How it works

  1. When to use this skill
  2. The core trick: reuse Ollama's models
  3. Usage
  4. Critical gotchas
  5. Reference material
  6. Requirements
Original SKILL.md headings · 104 lines, 678 words
SKILL.md preview
---
name: local-models
description: Run quick, offline, private LLM tasks on local models via llama.cpp, reusing models already downloaded by Ollama. Use for cheap/bulk text work (summarize, classify, extract JSON, anonymize PII, translate, proofread, keywords), local embeddings, and offline image description — and prefer it over a cloud API whenever a task is privacy-sensitive, must run offline, is high-volume/low-stakes, or just needs a fast throwaway answer. Provides an `lm` CLI wrapper plus an OpenAI-compatible local server.
---

# local-models

Quick access to local LLMs through **llama.cpp**, reusing the GGUF models already
pulled by Ollama (no re-download for text and embeddings). Everything runs on the
machine — no API key, no network, no per-token cost.

## When to use this skill

Reach for local models instead of a cloud API when the task is:
- **Privacy-sensitive** — redacting PII, processing personal notes, health data, secrets-adjacent text. The data never leaves the machine.
- **Offline** — no network available, or the user explicitly wants local-only.
- **High-volume / low-stakes** — classifying or tagging hundreds of items, where a small model is good enough and cloud cost/latency would add up.
- **A fast throwaway** — a quick summary, translation, or "what is this" where round-tripping to a frontier model is overkill.

Prefer a frontier (Claude) model when the task needs strong reasoning, long
context, careful code, or high accuracy — these local models are small (0.6–4B).

## The core trick: reuse Ollama's models

Ollama stores model weights as extension-less GGUF blobs under
`~/.ollama/models/blobs/`. **These are ordinary GGUF files** — llama.cpp loads
them directly. `scripts/ollama_blob.py` reads Ollama's manifests and resolves a
friendly name (e.g. `qwen2.5:3b`) to its weights blob path. No conversion, no
duplicate downloads.

## Usage

The entry point is `scripts/lm`. Run `scripts/lm help` for the full list. Invoke
it with an absolute path, e.g. `~/ai_projects/claude-skills/local-models/scripts/lm`.

```bash
lm models                       # list local models (text / vision / embed)
lm ask [MODEL] "PROMPT"         # one-shot prompt (default qwen2.5:3b)
lm chat [MODEL]                 # interactive REPL

# Text presets — accept a file path, inline text, OR stdin:
lm summarize  report.md
cat notes.txt | lm tldr
lm keywords   article.txt
lm anonymize  transcript.txt         # → [NAME] [EMAIL] [PHONE] [ADDRESS] ...
lm proofread  draft.md
lm translate  German "Good morning"
lm classify   "praise,complaint,question"  feedback.txt   # → one label
lm extract    "invoice_number, total, due_date"  invoice.txt   # → JSON

# Vision (downloads model+projector once via HuggingFace — see note below):
lm describe-image photo.jpg
lm tag-image      screenshot.png
lm vision photo.jpg "What brand is the shoe?"

# Embeddings & serving:
lm embed "text to embed"             # → OpenAI-style JSON vector
lm serve qwen2.5:3b 8080             # OpenAI-compatible server on :8080
```

Output is clean (just the answer) — the wrapper drives `llama-completion` in
single-turn mode and strips the chat-template scaffolding and llama.cpp logs.

### Choosing a model

Defaults are tuned for clean, fast output and can be overridden per call:
- General text presets → `qwen2.5:3b` (`LM_TEXT_MODEL`)
- Classify / extract → `qwen2.5:3b` (`LM_REASON_MODEL`), run at temperature 0
- Embeddings → `jeffh/intfloat-multilingual-e5-large:f16` (`LM_EMBED_MODEL`)
- Other envs: `LM_NTOK` (max tokens), `LM_VISION_HF` (vision repo), `LM_DEBUG=1` (show llama.cpp logs)

Pass an explicit model as the first argument to `ask`/`chat`/`embed`/`serve`
(e.g. `lm ask qwen3:4b "..."`).

## Critical gotchas

- **Ollama's `gemma3` GGUF does NOT load in stock llama.cpp.** It fails with
  `key not found in model: gemma3.attention.layer_norm_rms_epsilon` because
  Ollama writes custom metadata keys mainline llama.cpp doesn't read. Use a
  `qwen*` model instead, or pull a community gemma3 GGUF via `-hf`. This is why
  the defaults are qwen, not gemma3.
- **`qwen3:4b` emits `<think>…</think>` reasoning blocks** before its answer.
  Fine for `ask`/`chat`, but it pollutes preset output (JSON, labels) — the
  presets default to `qwen2.5:3b` to avoid this.
- **Vision has no Ollama blob to reuse.** Ollama did not store an `mmproj`
  (vision projector) for `qwen2.5vl`, and llama.cpp needs one. So the vision
  commands use `llama-mtmd-cli -hf ggml-org/Qwen2.5-VL-3B-Instruct-GGUF`, which
  downloads model+projector (~2–3 GB) into `~/.cache/llama.cpp` on first use,
  then runs offline. Warn the user before the first vision call.
- **Each one-shot call reloads the model** (a few seconds for these small
  models). For many sequential calls, start a server once with `lm serve` and
  hit `http://localhost:8080/v1/chat/completions` — see
  [references/serving-and-embeddings.md](references/serving-and-embeddings.md).

## Reference material

- [references/serving-and-embeddings.md](references/serving-and-embeddings.md) —
  running `llama-server` as an OpenAI-compatible endpoint (and pointing the `llm`
  CLI or any OpenAI client at it), plus local embeddings / RAG patterns with
  `llama-embedding`.

## Requirements

- `llama.cpp` installed (`brew install llama.cpp`) — provides `llama-completion`,
  `llama-mtmd-cli`, `llama-embedding`, `llama-server`.
- Ollama with at least one pulled model (for the blob-reuse path). `python3` for
  the resolver. No API keys.
Files in this package (4)
  • SKILL.md 5 KB
  • references/serving-and-embeddings.md 3 KB
  • scripts/lm 7 KB
  • scripts/ollama_blob.py 4 KB

Versions

RepositoryScoreLicence
This page · glebis/claude-skills100MIT
haedolabs/local-models-hub96MIT

Database snapshot: 13.11.2026 · package version ce639cd627e4 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100ollama
    Run and manage local LLMs via Ollama REST API — text generation, chat completions, embeddings, tool calling, structured output, and model management. Use when code imports ollama, references localhost:11434, or user asks about local LLM inference.
  • 100gemma4-local-deploy
    在本机 Mac 或 Apple Silicon 上部署 Gemma 4 12B。本地安装/升级 llama.cpp,下载 GGUF 量化模型,用 llama-server 暴露 OpenAI-compatible API,或用 Ollama 暴露本地模型服务;按用户需求在默认 Q4_K_M、64K/128K 长上下文、QAT Q4_0 @ 256K、左右对比演示之间选择,配置 tmux 后台运行,验证健康检查、问答接口、资源占用和常见故障。当用户说部署 Gemma 4、Gemma 4…
  • 100open-webui
    Open WebUI is a self-hosted web chat interface for local and cloud language models that connects to Ollama and any OpenAI-compatible API. Use when a user asks to install or deploy Open WebUI with Docker or pip, connect Open WebUI to Ollama, llama.cpp, vLLM or OpenAI, configure…
  • 100llama-cpp
    Guide for llama.cpp, the C/C++ LLM inference framework by ggml-org. Covers the C API (llama.h), GGUF format, quantization (Q4_K_M, Q8_0, IQ4_XS), CMake builds, GPU backends (CUDA, Vulkan, Metal, ROCm), HTTP server with OpenAI-compatible API, embeddings, grammar constraints,…
  • 100minicpm5-deploy-llama-cpp
    Run MiniCPM5-1B or MiniCPM5-2B with llama.cpp using the released GGUF artifacts (F16 / Q8_0 / Q4_K_M). Use when the user wants CPU-only / consumer-GPU / cross-platform native deployment, asks for "llama.cpp", "llama-cli", "llama-server", "GGUF", or has no Python available.
  • 100llamacpp
    Complete llama.cpp C/C++ API reference covering model loading, inference, text generation, embeddings, chat, tokenization, sampling, batching, KV cache, LoRA adapters, and state management. Triggers on: llama.cpp questions, LLM inference code, GGUF models, local AI/ML inference,…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.