Skip to content
Skills Library
AI & agents · Analysis

perceiving-objects-oneshot

/perceiving-objects-oneshot

Lightweight one-shot 3D object perception. Runs Grounding-DINO broad detection, a SINGLE VLM set-of-marks letter pick over the labeled boxes, SAM3 box segmentation, depth back-projection, and geometry.filter_and_compute_obb. No pairwise tournament, no multi-view safe-gate. Returns a clean not_found output (no exception) when the VLM answers "none" or DINO emits no detections — making this the right skill for clean-all-items loops whose natural termination signal is "no more matching objects in view". Use when a multi-item loop needs a clean no-match exit, or for generic target descriptions on

Licence
Apache-2.0
Last repository change
12.09.2026

Assessment and security

100/ 100Class: good
Format25 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
6
Scripts
1
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 12.09.2026, without references to retired models or APIs.

Licence

Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.

What to check before installing
  • 1 scripts in this package. Review them before installing, as Claude may execute them.

Scripts: scripts/perceive_simple.py

What it does and when to use it

Lightweight one-shot 3D object perception. Runs Grounding-DINO broad detection, a SINGLE VLM set-of-marks letter pick over the labeled boxes, SAM3 box segmentation, depth back-projection, and geometry.filter_and_compute_obb. No pairwise tournament, no multi-view safe-gate. Returns a clean not_found output (no exception) when the VLM answers "none" or DINO emits no detections — making this the right skill for clean-all-items loops whose natural termination signal is "no more matching objects in view". Use when a multi-item loop needs a clean no-match exit, or for generic target descriptions on

Triggers from the description

noneno more matching objects in view
Original author description

Lightweight one-shot 3D object perception. Runs Grounding-DINO broad detection, a SINGLE VLM set-of-marks letter pick over the labeled boxes, SAM3 box segmentation, depth back-projection, and geometry.filter_and_compute_obb. No pairwise tournament, no multi-view safe-gate. Returns a clean not_found output (no exception) when the VLM answers "none" or DINO emits no detections — making this the right skill for clean-all-items loops whose natural termination signal is "no more matching objects in view". Use when a multi-item loop needs a clean no-match exit, or for generic target descriptions on

How it works

  1. When to use
  2. When NOT to use
  3. Recommended subgraph state flow
Original SKILL.md headings · 98 lines, 455 words
SKILL.md preview
---
name: perceiving-objects-oneshot
description: >
  Lightweight one-shot 3D object perception. Runs Grounding-DINO broad
  detection, a SINGLE VLM set-of-marks letter pick over the labeled
  boxes, SAM3 box segmentation, depth back-projection, and
  geometry.filter_and_compute_obb. No pairwise tournament, no multi-view
  safe-gate. Returns a clean not_found output (no exception) when the
  VLM answers "none" or DINO emits no detections — making this the right
  skill for clean-all-items loops whose natural termination signal is
  "no more matching objects in view". Use when a multi-item loop needs a
  clean no-match exit, or for generic target descriptions on uncluttered
  scenes with reasonably sized targets.
license: MIT
compatibility: requires gap>=0.1
metadata:
  category: perception
  tags: [perception, dino, vlm, one-shot, set-of-marks]
gap:
  allowed_tools:
    - robot.get_observation
    - grounding-dino.detect
    - vlm.query
    - sam3.segment_box
    - geometry.mask_to_world_points
    - geometry.filter_and_compute_obb
  exit_conditions:
    found: Target detected; OBB and mask bound in subgraph outputs.
    not_found: VLM replied "none" or DINO emitted no detections. In clean-all-items loops route to done; in normal pick-and-place route to abort.
  produces_outputs:
    "<name>_obb": OrientedBoundingBox
    "<name>_mask": Mask
    "<name>_cloud": PointCloud
  errors:
    - "NOT_FOUND: No detection matched the target description."
  hard_rules:
    - perception_pipeline_invariants.md#emit-both-obb-and-mask
    - geometry_calling_conventions.md#obb-field-binding
  canonical_scripts:
    - perceive_simple: scripts/perceive_simple.py
  prompts:
    vlm_one_shot: prompts/vlm_one_shot.md
  references:
    - title: Perception pipeline invariants (emit obb + mask + cloud)
      path: references/perception_pipeline_invariants.md
    - title: Geometry tool calling conventions (output field binding)
      path: references/geometry_calling_conventions.md
  examples:
    - title: Canonical perception subgraph (observe → perceive → filter_obb)
      path: examples/canonical_subgraph.json
  streaming: false
---

# perceiving-objects-oneshot

Single VLM call over a set-of-marks overlay. Pipeline:

```text
observe → perceive → filter_obb
```

`perceive` runs:

1. ``grounding-dino.detect`` with a broad ``object.`` text prompt
2. One ``vlm.query`` showing the image with letter-labeled boxes:
   "Which letter is the *<target>*? Reply with one letter or 'none'."
3. On ``none``: emit ``found: False`` so the subgraph exits
   ``not_found``. On a letter: ``sam3.segment_box`` on the chosen box,
   ``geometry.mask_to_world_points`` for the cloud.

## When to use

- Clean-all-items / multi-item loops where the cycle needs a clean
  "no match" signal to terminate via ``target.not_found → done``.
- Tasks where the target description is generic ("any item on the
  floor", "the next remaining grocery item") rather than a specific
  scene-spec id.
- Uncluttered scenes with distinct, reasonably sized targets where the
  set-of-marks letter pick is reliable.

## When NOT to use

- Small / cluttered targets (< 40 px wide) — prefer ``perceiving-objects``
  whose pairwise crop tournament is far more reliable in that regime.

## Recommended subgraph state flow

3 states: ``observe → perceive → filter_obb`` (mirrors ``perceiving-objects``).

State details:

> **About `object_name` below:** it is a literal Python string — the
> natural noun phrase for the object you are perceiving, drawn from this
> subgraph's description (e.g. `"alphabet soup"`, `"basket"`,
> `"any grocery item on the floor"`). It is a constant per subgraph
> instance, NOT a binding. **DO NOT** write `Ref("in.object_name")` or
> any other `Ref(...)`; the coordinator does not declare `object_name`
> as a subgraph input. Write the string directly,
> e.g. `"object_name": "any grocery item on the floor"`.
> The same rule applies to `object_description` if you set it.

1. **`observe`** — `type: tool`, `tool: "robot.get_observation"`,
   `inputs: {}`. Connector tool; flat name only.
2. **`perceive`** — `type: script`, file `scripts/<sg>/perceive_simple.py`
   from this bundle. Inputs:
   `cameras=Ref("observe.cameras")`,
   `object_name="<noun phrase from the subgraph description>"`,
   plus any optional literals (`object_description`, `dino_prompt`).
   Returns `{found, cloud, mask, score}`. When the VLM picks "none" or
   DINO emits no detections, ``found`` is `False` and the downstream
   `filter_obb` step then raises (empty cloud) — caught by the
   subgraph's `on_error: "not_found"` exit.
3. **`filter_obb`** — `type: tool`,
   `tool: "geometry.filter_and_compute_obb"`,
   `inputs={"points": Ref("perceive.cloud")}`. Returns
   `{"obb": <OrientedBoundingBox>}`.

### Wiring the exit (HARD)

Linear `perceive → filter_obb → found → END`. The `filter_obb` tool
raises on empty clouds (the not-found path), and the subgraph's
`on_error: "not_found"` catches that. Do NOT add conditional edges on
`perceive` — the linear path plus `set_on_error` is sufficient.

```python
sg.add_node("filter_obb", type="tool",
            tool="geometry.filter_and_compute_obb",
            inputs={"points": Ref("perceive.cloud")})
sg.add_exit("found")
sg.add_edge("perceive", "filter_obb")
sg.add_edge("filter_obb", "found")
sg.add_edge("found", END)
sg.set_on_error("not_found")
```

Bind the subgraph outputs (ALL THREE — required, no exceptions):

```python
sg.set_outputs(
    target_obb=Ref("filter_obb.obb"),
    target_mask=Ref("perceive.mask"),
    target_cloud=Ref("perceive.cloud"),
)
```

Note that `geometry.filter_and_compute_obb` returns `{"obb": ...}`, so
the OBB binding walks into the `obb` field (`Ref("filter_obb.obb")`,
NOT a bare `Ref("filter_obb")`). See
`references/geometry_calling_conventions.md`.
Files in this package (6)
  • SKILL.md 6 KB
  • examples/canonical_subgraph.json 1 KB
  • prompts/vlm_one_shot.md 2 KB
  • references/geometry_calling_conventions.md 2 KB
  • references/perception_pipeline_invariants.md 1 KB
  • scripts/perceive_simple.py 8 KB

Versions

The database has no other skills named perceiving-objects-oneshot.

Database snapshot: 13.11.2026 · package version 1165e7ac5051 (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100perceiving-objects
    Fast single-path 3D object perception. Runs Grounding-DINO broad detection, a pairwise VLM crop tournament to identify the target box, SAM3 box segmentation, and depth back-projection to a world-frame point cloud, finished by geometry.filter_and_compute_obb for a clean oriented…
  • 100grounding-dino
    Grounding DINO zero-shot object detection — natural-language queries to labeled 2D bounding boxes with confidence scores. Use when a workflow needs to locate named objects in an RGB image before segmenting or grasping them.
  • 100tao-train-grounding-dino
    Grounding DINO for open-set object detection. Combines DINO-style detection with a BERT text encoder for language-guided detection — detects objects described by text prompts without a fixed class vocabulary. Use when training, evaluating, exporting, quantizing, or running…
  • 100tao-train-mask-grounding-dino
    Mask Grounding DINO for grounded instance segmentation. Extends Grounding DINO with a mask-prediction head for open-set segmentation guided by text prompts. Use when training, evaluating, exporting, quantizing, or running inference for a TAO Mask-Grounding-DINO model. Trigger…
  • 100vlm-segmentation
    Choose and evaluate VLM or segmentation pipelines, including text-conditioned detection, masks, part labels, model-license constraints, and measured GPU deployment choices. Use when a task has a VLM or segmentation component; route pure diffusion prompting, training, or serving…
  • 100tao-train-dino
    DINO (DETR with Improved DeNoising Anchor Boxes) for 2D object detection. Transformer-based detector with denoising training, multi-scale features, and optional distillation support. Use when training, evaluating, exporting, distilling, quantizing, or running inference for a TAO…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.