create-adapter
/create-adapter
Guide for building a new LimaCharlie USP adapter in this repo — researching the source, implementing it by copying the closest existing adapter, writing thorough tests with a mock/fake of the source, optionally live-testing against real credentials, validating the mock's fidelity, and shipping a documented PR. Use when the user wants to add a new adapter / data source / integration (e.g. "add an adapter for <vendor>", "ingest <product> logs into LimaCharlie").
- Author
- refractionPOINT ↗
- Repository
- usp-adapters ↗
- Licence
- Apache-2.0
- Last repository change
- 27.10.2026
Assessment and security
- Files in this package
- 3
- Scripts
- 0
- HTML comments
- 0
- URLs in SKILL.md
- 0
No signals detected. Repository changed on 27.10.2026, without references to retired models or APIs.
Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.
What it does and when to use it
Guide for building a new LimaCharlie USP adapter in this repo — researching the source, implementing it by copying the closest existing adapter, writing thorough tests with a mock/fake of the source, optionally live-testing against real credentials, validating the mock's fidelity, and shipping a documented PR. Use when the user wants to add a new adapter / data source / integration (e.g. "add an adapter for <vendor>", "ingest <product> logs into LimaCharlie").
Triggers from the description
Original author description
Guide for building a new LimaCharlie USP adapter in this repo — researching the source, implementing it by copying the closest existing adapter, writing thorough tests with a mock/fake of the source, optionally live-testing against real credentials, validating the mock's fidelity, and shipping a documented PR. Use when the user wants to add a new adapter / data source / integration (e.g. "add an adapter for <vendor>", "ingest <product> logs into LimaCharlie").
How it works
- Phase 0 — Scope it with the user
- Phase 1 — Research the source
- Phase 2 — Implement by copying the closest adapter
- Phase 3 — Write a mock/fake of the source
- Phase 4 — Run the tests
- Phase 5 — Live-test with real credentials (if available)
- Phase 6 — Document and open a draft PR
- Checklist
SKILL.md preview
---
name: create-adapter
description: Guide for building a new LimaCharlie USP adapter in this repo — researching the source, implementing it by copying the closest existing adapter, writing thorough tests with a mock/fake of the source, optionally live-testing against real credentials, validating the mock's fidelity, and shipping a documented PR. Use when the user wants to add a new adapter / data source / integration (e.g. "add an adapter for <vendor>", "ingest <product> logs into LimaCharlie").
---
# Building a new USP adapter
A USP adapter reads events from some external source and ships them to
LimaCharlie as USP `DataMessage`s. Sources vary widely — REST APIs you poll,
vendor SDKs (AWS/GCP/Azure), streaming sockets, mailboxes, local files, stdin —
so **there is no single template**. What every adapter shares is one contract
(see `references/patterns.md`); everything else you get by **copying the existing
adapter closest to your source**.
The strongest, best-tested examples to imitate are `threatlocker/` and `gmail/`
(both have full mock-backed test suites). But pick by source type:
- HTTP API you poll → `threatlocker/`, `1password/`, `gmail/`
- a vendor SDK / cloud client → `s3/`, `sqs/` (AWS), `pubsub/`, `gcs/`,
`bigquery/` (GCP), `azure_event_hub/` (Azure)
- streaming socket / push → `syslog/`, `azure_event_hub/`
- mailbox → `imap/`, `gmail/`
- local files / OS sources → `file/`, `evtx/`, `wel/`, `stdin/`
`references/patterns.md` maps source types to examples and lists the shared
contract and the patterns worth copying regardless of source. Read it plus your
chosen example(s) before writing code.
Work through the phases in order. Don't skip research — getting the source's real
shape right up front is what makes the implementation and the mock correct.
## Phase 0 — Scope it with the user
Before writing anything, confirm:
- **What source / vendor** and **which events** they want ingested.
- **How you reach the source**: a documented HTTP API, an official SDK, a
socket/stream, a file? This decides which existing adapter you copy.
- **Auth method**: API key/token, OAuth refresh token, basic auth, cloud
credentials/service account, etc. This drives the config struct and the mock's
auth check.
- **Whether they have working credentials** you can use to live-test (ask now;
you'll need it in Phase 5). If they do, agree how they'll provide them — env
vars are simplest. **Never commit credentials.** This repo already uses files
like `*-do-not-commit.txt` for local secrets; follow that and check
`.gitignore`.
## Phase 1 — Research the source
Read the real vendor documentation / SDK reference — do not rely on memory. Use
WebFetch / WebSearch. Capture, before coding:
- **Authentication** — exact mechanism (header + token format, OAuth refresh
flow, SDK credential object), token lifetime/refresh.
- **How you list/receive events** — for an HTTP API: the endpoint(s), method,
request params, and **how pagination works** (page number, cursor/`next`,
`has_more`, offset/limit, time window) — pagination is the #1 source of adapter
bugs. For an SDK: which client method streams/lists the events and how it
signals "no more". For a stream/queue: how messages are received and
acknowledged.
- **The event shape** — a real sample payload, the **stable per-record id field**
(for dedup) and the **timestamp field** (for event time), with exact
paths/formats. For HTTP, the response envelope (bare array vs. `{"data":[...]}`).
- **Rate limits / error model** — status codes, retry-after, throttling.
Record the doc URLs and a couple of **real sample payloads** — sanitized
versions go into the mock fixtures and the README. Grab an OpenAPI/Swagger spec,
Postman collection, or SDK example if one exists; it's the most reliable source
of shapes.
## Phase 2 — Implement by copying the closest adapter
Create package directory `myadapter/` (lowercase; the Go package is
`usp_myadapter`). Most adapters are a single `client.go`; the well-tested ones
add `*_test.go` and sometimes split transport into `api.go` (as `threatlocker`
does for HTTP). **Mirror the file layout of whichever adapter you're copying.**
Every adapter, whatever the source, must honor the universal contract (detailed
in `references/patterns.md`):
- Config struct embedding `uspclient.ClientOptions` as `client_options`
(json+yaml tags on every field) with a `Validate()` that fills defaults.
- `func NewMyAdapter(ctx, conf) (*MyAdapter, chan struct{}, error)` and an
idempotent `Close() error`; the channel closes when the adapter stops itself.
- Read from the source however it works, and **ship each event verbatim** as a
`protocol.DataMessage{JsonPayload, EventType, TimestampMs}` through the usp
client. Don't reshape payloads — LimaCharlie parses/maps in the cloud.
Patterns worth copying from the well-tested adapters (all in
`references/patterns.md`): a **test seam** that puts the external dependency
behind a small interface so a fake/mock can stand in (`threatlocker`'s `uspSink`
+ unexported `newThreatLockerAdapter(ctx,conf,sink)`); lifecycle via
`utils.NewEvent()` + `WaitGroup` + `sync.Once` `Close`; **exactly-once shipping**
via `utils.NewLocalDeduper` keyed on a stable id when the source can re-deliver;
robust transport (`HTTPError` + `isTransientError`, backoff, **stop on 401/403**)
for HTTP sources; JSON fidelity via `utils.UnmarshalCleanJSON`/`utils.Dict`; and
backpressure handling on `uspclient.ErrorBufferFull`.
Then **register** the adapter (see `references/registration.md` — easiest is to
grep an existing adapter like `threatlocker` across these files and replicate):
- `containers/conf/all.go`: import the package, add a field to `GeneralConfigs`.
- `containers/general/tool.go`: import it, add the `else if method == "myadapter"`
dispatch branch.
- root `README.md`: add a short section with a run example.
Build with `go build ./containers/general` and `go fmt ./myadapter/...`.
## Phase 3 — Write a mock/fake of the source
The whole point is end-to-end tests that run **without live credentials**, by
substituting a stand-in for the external source through the test seam. Model on
`threatlocker/mock_test.go` and `gmail/mock_test.go`:
- For an **HTTP** source: an `httptest.NewServer` whose handler **reproduces the
real API** — checks auth the way the real API does, paginates over an in-memory
dataset with the *same* mechanism, returns the *same* envelope shape.
- For an **SDK / non-HTTP** source: a fake implementing the small interface your
adapter calls, returning realistic results and exercising the same "more
data / no more" and error behaviors.
- **Realistic fixtures** matching the Phase-1 sample payloads — nested objects,
arrays, mixed scalar types, the real id and timestamp field names/formats.
End-to-end tests to write:
- All events ship; event type, parsed timestamp, and **verbatim payload** match.
- Re-reading/re-polling does not re-ship (dedup works, when applicable).
- An event appearing mid-run ships exactly once.
- A multi-page / multi-batch dataset is fully consumed, each event once.
- Multiple event types (if applicable) are tagged correctly.
- Bad auth → adapter stops, ships nothing.
## Phase 4 — Run the tests
Standard `go test`:
```
go test ./myadapter/...
```
All tests must pass and be **fully enabled** — never skip a test or weaken an
assertion to get green. If a `run_local_test.sh` ever exists at the repo root,
prefer it (it handles local system dependencies). There isn't one today, so
`go test` is correct for a self-contained adapter.
## Phase 5 — Live-test with real credentials (if available)
If the user provided credentials in Phase 0, run the real adapter against the
live source to prove it works end-to-end. Easiest is via the built binary
(matches production), pointed at a test OID:
```
go build ./containers/general
./general myadapter \
client_options.identity.oid=$OID \
client_options.identity.installation_key=$INSTALLATION_KEY \
client_options.platform=json \
client_options.sensor_seed_key=myadapter \
<your-auth-config>=$CREDENTIAL ...
```
Confirm a new sensor appears in the user's LimaCharlie org and events flow.
**Capture the real responses/messages** while it runs (log the raw payloads, or
use a proxy) — these are gold for Phase 5b. If you write a throwaway integration
test to do the capture, **remove it before the PR**; do not leave a skipped test
behind.
### Phase 5b — Validate the mock against reality
With real data in hand, **diff it against your mock**:
- Does the real **pagination / batching** match what the mock assumes (sizes,
the end-of-data signal, cursor/`has_more`)?
- Is the **envelope / result shape** exactly what the mock returns?
- Do the **id** and **timestamp** field paths/formats match? Are timestamps in a
format your parser handles?
- Do the **fixtures** carry the field types the real data has (big integers,
nullable fields, nested arrays)?
- Does **auth** rejection behave as the mock simulates?
Update the mock and fixtures to mirror captured reality, so the tests would have
caught any shape mistake and future maintainers can evolve the adapter
confidently **without** live credentials. Re-run Phase 4.
If the user has **no** credentials, say so plainly, make the mock as faithful as
possible from the documented shapes (Phase 1), and note in the PR that it hasn't
been live-verified.
## Phase 6 — Document and open a draft PR
Write `myadapter/README.md` (model on `threatlocker/README.md`):
- One-line description + links to the vendor and **the API/SDK docs** (Phase 1).
- **Authentication**: how to obtain the credential and how it's supplied.
- **Configuration table**: every config key, required?, description, defaults.
- **What the data looks like**: which events are collected, the event types, a
sample payload.
- **How it works** (polling/streaming, pagination/batching, dedup) and
operational caveats (rate limits, required filters, region/instance, etc.).
- CLI and YAML run examples.
Add the short root-`README.md` entry too and make sure the example matches.
Then open the PR **as a draft**:
```
git checkout -b add-myadapter-adapter
git add myadapter/ containers/conf/all.go containers/general/tool.go README.md
git commit -m "Add MyAdapter adapter (<one-line of what it ingests>)" # honor the repo commit-trailer rule
git push -u origin add-myadapter-adapter
gh pr create --draft --title "Add MyAdapter adapter" --body "<summary>"
```
PR body: what it ingests, API/SDK docs link, auth method, event types/sample
data, how it was tested (mock coverage + whether live-verified), follow-ups. Be
careful to stage **only** your adapter's files — don't sweep in unrelated
working-tree changes.
## Checklist
- [ ] Source researched from real docs; auth, how-events-arrive, pagination/
batching, id+timestamp fields, sample payload recorded.
- [ ] Closest existing adapter identified and copied; universal contract honored
(embeds `ClientOptions`, `New…/Close`, ships verbatim `DataMessage`s).
- [ ] Patterns copied where applicable: test seam, lifecycle, dedup, transient/
auth handling, JSON fidelity, backpressure.
- [ ] Registered in `containers/conf/all.go` and `containers/general/tool.go`;
`go build ./containers/general` passes.
- [ ] Mock/fake reproduces the real source (auth, pagination/batching, shape);
realistic fixtures.
- [ ] Tests pass, fully enabled (`go test ./myadapter/...`).
- [ ] Live-tested if creds available; mock validated against captured data.
- [ ] `myadapter/README.md` + root README entry written with doc links and
sample data.
- [ ] Draft PR opened with only the adapter's files staged.
Files in this package (3)
- SKILL.md 12 KB
- references/patterns.md 6 KB
- references/registration.md 2 KB
Versions
| Repository | Score | Licence |
|---|---|---|
| This page · refractionPOINT/usp-adapters | 100 | Apache-2.0 |
| routecraftjs/routecraft | 100 | Apache-2.0 |
| marcus/sidecar | 100 | MIT |
| gtme-run/gtme | 100 | Apache-2.0 |
Database snapshot: 13.11.2026 · package version 526c3e397920 (SHA-256 digest of the entire package)
Downloads in the last 30 days: 0 · ranking
Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.
Similar skills
- 95adapter-assistant
Complete adapter lifecycle assistant for LimaCharlie. Supports External Adapters (cloud-managed), Cloud Sensors (SaaS/cloud integrations), and On-prem USP adapters. Dynamically researches adapter types from local docs and GitHub usp-adapters repo. Creates, validates, deploys,… - 100test-process-adapter
Writes the unit test for one process adapter method and asserts the engine command: process, message or signal name, correlation and variables. Use when testing a process adapter or after create-process-adapter. - 90review-adapter
Review a benchmark adapter's fidelity, grading, and parity evidence against Harbor's adapter specification and pinned upstream behavior. Use when reviewing an adapter PR or investigating a parity discrepancy. - 100apps
Author and deploy LimaCharlie "apps" — self-contained AI-generated HTML mini-apps that render in a sandboxed iframe in the LimaCharlie web UI and call LimaCharlie APIs through a brokered, permission-scoped runtime (window.lc). Use when a user wants to create, edit, or understand… - 100add-adapter
Playbook for adding a new BarefootJS template adapter package (IR → target template language + native runtime). Use when: adding an adapter for a new template engine or backend language, porting the bf helpers / ParsedExpr evaluator to a new language, or wiring an adapter into… - 100add-adapter
Add a vendor-official or community adapter to the Chat SDK catalog and docs — the @chat-adapter/catalog entry (listing metadata and capabilities), the docs MDX page, meta.json, and a changeset. Use when a developer wants to add, list, register, or submit a third-party…