Przejdź do treści
Biblioteka skilli
AI i agenci · Analiza

perceiving-objects

/perceiving-objects

Szybko lokalizuje jeden nazwany obiekt w 3D: detekcja Grounding-DINO, turniej par w VLM, segmentacja SAM3 i rzutowanie głębi na chmurę punktów z OBB.

Licencja
Apache-2.0
Ostatnia zmiana repozytorium
12.09.2026

Ocena i bezpieczeństwo

100/ 100Klasa: dobry
Format25 / 25
Frontmatter z nazwą i opisem, poprawna nazwa, opis do 1024 znaków, nazwa zgodna z folderem.
Odkrywalność25 / 25
Opis mówi, kiedy użyć skilla, ma sensowną długość i jest w 3. osobie.
Treść35 / 35
Co najmniej 150 słów i do 500 linii, nagłówki, listy kroków, przykłady albo kod.
Higiena15 / 15
Bez zaślepek typu TODO, bez ścieżek Windows, z licencją.
Ryzykobrak sygnałów w skanie
Pliki w paczce
6
Skrypty
1
Komentarze HTML
0
Adresy URL w SKILL.md
0
Przestarzałość

Brak sygnałów. Repozytorium zmienione 12.09.2026, bez odwołań do wycofanych modeli i API.

Licencja

Apache-2.0, otwarta: można używać i zmieniać, z zachowaniem licencji i informacji o zmianach.

Co sprawdzić przed instalacją
  • 1 skrypt w paczce. Przejrzyj je przed instalacją, bo Claude może je uruchomić.

Skrypty: scripts/perceive_dino_vlm.py

Co robi i kiedy

Szybko lokalizuje jeden nazwany obiekt w 3D: detekcja Grounding-DINO, turniej par w VLM, segmentacja SAM3 i rzutowanie głębi na chmurę punktów z OBB.

Oryginalny opis (EN)

Fast single-path 3D object perception. Runs Grounding-DINO broad detection, a pairwise VLM crop tournament to identify the target box, SAM3 box segmentation, and depth back-projection to a world-frame point cloud, finished by geometry.filter_and_compute_obb for a clean oriented bounding box, mask, and cloud. Multi-camera rigs get a verified wrist-camera fallback gate. Use when a manipulation workflow needs to localize one named object quickly — uncluttered scenes with visually distinct targets, latency-bound loops, or platforms deploying only DINO + VLM + SAM3 + geometry.

Jak działa

  1. When to use
  2. When NOT to use
  3. Recommended subgraph state flow
  4. Hard rules
  5. Required end states
  6. See also
Sekcje z nagłówków SKILL.md, w oryginale · 172 linii, 974 słów
Podgląd SKILL.md
---
name: perceiving-objects
description: >
  Fast single-path 3D object perception. Runs Grounding-DINO broad
  detection, a pairwise VLM crop tournament to identify the target box,
  SAM3 box segmentation, and depth back-projection to a world-frame point
  cloud, finished by geometry.filter_and_compute_obb for a clean oriented
  bounding box, mask, and cloud. Multi-camera rigs get a verified
  wrist-camera fallback gate. Use when a manipulation workflow needs to
  localize one named object quickly — uncluttered scenes with visually
  distinct targets, latency-bound loops, or platforms deploying only
  DINO + VLM + SAM3 + geometry.
license: MIT
compatibility: requires gap>=0.1
metadata:
  category: perception
  tags: [perception, dino, vlm, sam3, single-method]
gap:
  allowed_tools:
    - robot.get_observation
    - grounding-dino.detect
    - vlm.query
    - vlm.query_yes_no
    - sam3.segment_box
    - sam3.segment_text
    - geometry.mask_to_world_points
    - geometry.exclude_robot_points
    - geometry.filter_and_compute_obb
  exit_conditions:
    found: Target detected; OBB and mask bound in subgraph outputs.
    not_found: Target not visible in any view. In clean-all-items loops route to done; in normal pick-and-place, route to abort.
  produces_outputs:
    "<name>_obb": OrientedBoundingBox
    "<name>_mask": Mask
    "<name>_cloud": PointCloud
  errors:
    - "NOT_FOUND: Object not detected in any camera view."
  hard_rules:
    - perception_pipeline_invariants.md#emit-both-obb-and-mask
    - geometry_calling_conventions.md#obb-field-binding
  canonical_scripts:
    - perceive_dino_vlm: scripts/perceive_dino_vlm.py
  prompts:
    vlm_pairwise: prompts/vlm_pairwise.md
  references:
    - title: Perception pipeline invariants (emit obb + mask + cloud)
      path: references/perception_pipeline_invariants.md
    - title: Geometry tool calling conventions (output field binding)
      path: references/geometry_calling_conventions.md
  examples:
    - title: Canonical perception subgraph (observe → perceive → filter_obb)
      path: examples/canonical_subgraph.json
  streaming: false
---

# perceiving-objects

Single-path perception: detect → disambiguate (pairwise crop tournament)
→ segment → fuse to 3D → extract OBB. Each DINO detection is cropped and
upscaled, and the target is found via binary "A or B?" comparisons of
crop pairs — far more reliable on small targets than a one-shot
Set-of-Marks letter pick (~30% → 97% on the LIBERO-PosVar object-ID
study).

Multi-camera handling uses a **`safe` wrist-fallback gate** (not blind
KD-tree fusion): identification defaults to the exterior view; the
wrist (eye-in-hand) view is consulted ONLY when the exterior pick fails
its own close-up verify AND the wrist pick passes its own. On the
4-suite / 200-frame regression study this was the only zero-regression
policy (+2.5% net, 0/189 frames regressed; blind fuse/verify→wrist/
wrist-only all regressed). See `perceive_dino_vlm.run` docstring.

On the verified-exterior path the wrist views still contribute **cloud
geometry** (never identity): wrist clouds of the same object — gated by
the multiview intersection check, with a geometry-seeded SAM fallback
(exterior cloud projected into the wrist frame) — are fused into the
output cloud so the OBB recovers the top face / far side a single front
view misses. A lone front view yields a sliver OBB biased toward the
camera by half the object depth, and that off-centre pinch is the
measured slip-during-transport failure mode on tall bottles/cartons.

## When to use

- Uncluttered scenes with visually distinct targets.
- Platforms where only DINO + VLM + SAM3 + geometry are deployed.
- The default single-target 3D perception skill.

## When NOT to use

- Cluttered scenes with many similar nearby distractors: strengthen the
  pairwise tournament by passing `object_description` shape/appearance hints
  (see the note above) rather than relying on the bare label alone.
- Clean-all-items / multi-item loops that need a clean "no match" loop
  terminator. Prefer `perceiving-objects-oneshot`.

## Recommended subgraph state flow

3 states:

```text
observe → perceive → filter_obb
```

State details:

> **About `object_name` below:** it is a literal Python string — the natural
> noun phrase for the object you are perceiving, drawn from this subgraph's
> description (e.g. `"alphabet soup can"`, `"basket"`, `"red bowl"`). It is
> a constant per subgraph instance, NOT a binding. **DO NOT** write
> `Ref("in.object_name")` or any other `Ref(...)`; the coordinator does
> not declare `object_name` as a subgraph input. Write the string directly,
> e.g. `"object_name": "basket"`.
>
> **About `object_description` (wire it whenever the task gives hints):**
> also a literal Python string. When the task/workflow description carries
> shape or appearance hints for the target (e.g. an "Object context" block
> with `shape_hint` / `expected_label`, or adjectives in the instruction),
> pass them through verbatim, e.g.
> `"object_description": "small rectangular box, blue and white packaging,
> ~5 cm wide"`. The description is injected into BOTH the pairwise
> tournament prompt ("It looks like: …") and the close-up verification
> question ("It should look like: …"). This is what disambiguates
> look-alike packaging (several LIBERO grocery items are small blue/white
> boxes) and keeps the verify gate from rejecting a correct pick whose
> rendered asset reads as a generic box — a rejection forces the
> wrist-camera fallback, whose single top-down view degrades the OBB
> height and downstream grasps. Omit it (default `""`) only when the task
> provides no hints.

1. **`observe`** — `type: tool`, `tool: "robot.get_observation"`,
   `inputs: {}`. Connector tool; flat name only.
2. **`perceive`** — `type: script`, file `scripts/<sg>/perceive_dino_vlm.py`
   from this bundle. Inputs:
   `cameras=Ref("observe.cameras")`,
   `object_name="basket"` (replace with the actual target noun phrase
   from this subgraph's description),
   `object_description="..."` (the task's shape/appearance hints — see the
   note above; strongly recommended whenever hints exist),
   plus any optional fields (`dino_prompt`, etc.).
   Returns `{found, cloud, mask, score}`.
3. **`filter_obb`** — `type: tool`,
   `tool: "geometry.filter_and_compute_obb"`,
   `inputs={"points": Ref("perceive.cloud")}`. Returns `{"obb": <OrientedBoundingBox>}`.

### Wiring the exit (HARD)

Use the linear edge `filter_obb → found → END`. When the target isn't
found, `perceive` returns an empty cloud and `filter_obb` raises on it,
so the subgraph's `on_error: "not_found"` catches that path
automatically. Do **NOT** add any conditional edges on `perceive` — the
linear path plus `set_on_error` is sufficient.

✅ Correct (the literal `gap.builder` calls you should emit):

```python
sg.add_node("perceive", type="script",
            script="scripts/<sg>/perceive_dino_vlm.py",
            inputs={"cameras": Ref("observe.cameras"),
                    "object_name": "small blue and white cream cheese",
                    # from the task's Object context / shape_hint block:
                    "object_description": ("small rectangular box, blue "
                                           "and white packaging, ~5 cm wide")})

sg.add_node("filter_obb", type="tool",
            tool="geometry.filter_and_compute_obb",
            inputs={"points": Ref("perceive.cloud")})

# add_exit() creates the success-marker noop node AND registers the
# exit value. Do NOT also call sg.add_node("found", type="noop") — that
# would conflict with the node add_exit created.
sg.add_exit("found")

sg.add_edge("perceive", "filter_obb")
sg.add_edge("filter_obb", "found")
sg.add_edge("found", END)

sg.set_on_error("not_found")
```

Bind the subgraph outputs (ALL THREE — required, no exceptions):

```python
sg.set_outputs(
    target_obb=Ref("filter_obb.obb"),
    target_mask=Ref("perceive.mask"),
    target_cloud=Ref("perceive.cloud"),
)
```

(Replace `target_*` with this subgraph's actual name prefix — e.g.
`container_obb`, `container_mask`, `container_cloud` when authoring
the container subgraph.)

All three bindings walk into a field of the producing node's output
dict: `geometry.filter_and_compute_obb` returns `{"obb": ...}` (bind
`Ref("filter_obb.obb")`, NOT a bare `Ref("filter_obb")`), while
`<name>_mask` and `<name>_cloud` walk into fields of `perceive`'s
output dict. `perceive_dino_vlm.py` already produces all three;
emitting them unconditionally lets downstream subgraphs that need any
of them (e.g. learned-grasp skills require `<name>_cloud`) wire up
without you having to anticipate which skill they'll use.

## Hard rules

1. Subgraph-level outputs MUST emit ALL THREE: `<name>_obb`,
   `<name>_mask`, AND `<name>_cloud`. The cloud is the fused world-frame
   point cloud needed by learned-grasp skills; emit it unconditionally
   so the downstream agent can wire it without round-tripping. See
   `references/perception_pipeline_invariants.md`.
2. `geometry.filter_and_compute_obb` returns `{"obb": OrientedBoundingBox}`;
   bind via `Ref("filter_obb.obb")` (walk into the `obb` field).
   See `references/geometry_calling_conventions.md`.

## Required end states

| End state | Meaning |
|---|---|
| `found` | OBB + mask bound; route to next subgraph (typically a grasp skill). |
| `not_found` | Route to `abort` (or to `done` in clean-all-items loops). |


## See also

- `prompts/vlm_pairwise.md` — the pairwise-tournament VLM prompt template.
- `scripts/perceive_dino_vlm.py` — the canonical perception script.
Pliki w paczce (6)
  • SKILL.md 9 KB
  • examples/canonical_subgraph.json 1 KB
  • prompts/vlm_pairwise.md 1 KB
  • references/geometry_calling_conventions.md 2 KB
  • references/perception_pipeline_invariants.md 1 KB
  • scripts/perceive_dino_vlm.py 51 KB

Wersje

W bazie nie ma innych skilli o nazwie perceiving-objects.

Migawka bazy z 13.11.2026 · wersja paczki 5d82217a2341 (skrót SHA-256 całej paczki)

Pobrania w ostatnich 30 dniach: 0 · ranking

Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.

Podobne skille

  • 100perceiving-objects-oneshot
    Lekko lokalizuje obiekt w 3D jednym wyborem VLM po detekcji Grounding-DINO i segmentacji SAM3, a gdy nic nie znajdzie, zwraca czysty wynik not_found bez wyjątku.
  • 100perceiving-object-parts
    Wykrywa obiekt nadrzędny, kadruje obraz i segmentuje w nim nazwaną część (DINO i SAM3), zwracając OBB, maskę i chmurę punktów w układzie świata.
  • 100tao-train-mask-grounding-dino
    Trenuje Mask Grounding DINO TAO do segmentacji instancji sterowanej tekstem, bez stałego słownika klas.
  • 100tao-train-grounding-dino
    Trenuje model Grounding DINO TAO do detekcji obiektów opisanych tekstem, bez stałego słownika klas.
  • 100tao-train-dino
    Trenuje, ocenia i eksportuje detektor obiektów DINO w TAO, w tym destylację, kwantyzację i uruchamianie wnioskowania.
  • 100vlm-segmentation
    Dobiera i ocenia potoki VLM oraz segmentacji: detekcję sterowaną tekstem, maski, licencje modeli i wdrożenie na GPU.

Uwagi i zgłoszenia

Wiadomość trafi do Bartosza Sokolińskiego. Adres e-mail jest potrzebny tylko do odpowiedzi. Szczegóły w polityce prywatności.