Przejdź do treści
Biblioteka skilli
Produktywność osobista · Automatyzacja

nested-display

/nested-display

Steruje aplikacją GUI, instalatorem, grą lub emulatorem w zagnieżdżonym ekranie Wayland: zrzuty, kliknięcia i skróty klawiszowe.

Licencja
MIT
Ostatnia zmiana repozytorium
29.10.2026

Ocena i bezpieczeństwo

93/ 100Klasa: dobry
Format20 / 25
Frontmatter z nazwą i opisem, poprawna nazwa, opis do 1024 znaków, nazwa zgodna z folderem.
Odkrywalność25 / 25
Opis mówi, kiedy użyć skilla, ma sensowną długość i jest w 3. osobie.
Treść35 / 35
Co najmniej 150 słów i do 500 linii, nagłówki, listy kroków, przykłady albo kod.
Higiena13 / 15
Bez zaślepek typu TODO, bez ścieżek Windows, z licencją.
Ryzykobrak sygnałów w skanie
Pliki w paczce
9
Skrypty
0
Komentarze HTML
0
Adresy URL w SKILL.md
0
Przestarzałość

Brak sygnałów. Repozytorium zmienione 29.10.2026, bez odwołań do wycofanych modeli i API.

Licencja

MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.

Co robi i kiedy

Steruje aplikacją GUI, instalatorem, grą lub emulatorem w zagnieżdżonym ekranie Wayland: zrzuty, kliknięcia i skróty klawiszowe.

Oryginalny opis (EN)

Drive a GUI app, installer, game or emulator inside a controlled nested Wayland display - screenshots, absolute clicks, typing and key chords - using the nested-display MCP. Use this instead of injecting input into the real desktop.

Jak działa

  1. Golden rules
  2. Standard workflow
  3. Recipe: relative-only apps (emulators, captured-pointer games)
  4. Recipe: Wine app or game at a fixed resolution
  5. Recipe: an installer
  6. Host window handling
  7. Troubleshooting
  8. What not to use
Sekcje z nagłówków SKILL.md, w oryginale · 118 linii, 973 słów
Podgląd SKILL.md
---
name: nested-display
description: Drive a GUI app, installer, game or emulator inside a controlled nested Wayland display - screenshots, absolute clicks, typing and key chords - using the nested-display MCP. Use this instead of injecting input into the real desktop.
---

# Nested display automation

Run the target inside its own nested sway compositor (the `nested-display` MCP). The display is a
normal window on the desktop, but input and capture happen inside it: no host focus games, no
pointer acceleration, no black screenshots, and the rest of the desktop stays usable while the
task runs.

Requires the `nested-display` MCP server to be registered with your agent (see its README).

## Golden rules

1. **One display per app.** `display_open` with `app` + `args`; omit `app` for an empty desktop.
   Displays are independent: own `WAYLAND_DISPLAY`, own Xwayland `DISPLAY`.
2. **Screenshot coordinates are the truth.** `screenshot` returns a PNG plus `{width, height}`.
   Click/pointer coordinates are in that image's pixel space.
3. **Never guess.** `wait_for_window` before interacting; `screenshot` after every state-changing
   action. Do not trust "typed N characters" style success messages.
4. **Do not mix input paths.** For `shell: "xwayland"` windows the server uses XTest; for
   `xdg_shell` it uses wtype/wlrctl. Typing with both in one session garbles X11 text because the
   virtual-keyboard keymap lands in Xwayland.
5. **Clicks and keys are held briefly** (100 ms / 80 ms). Polling UIs like raylib and SDL sample
   input once per frame; an instant synthetic press+release is never seen.
6. **Close displays.** `display_close { all: true }` at the end; each display keeps a compositor
   and an Xwayland alive.
7. **Never `pkill -f <pattern>`** where the pattern appears in your own command line - it kills
   your shell. Use the MCP's `display_close` or exact pids.

## Standard workflow

```
display_open { title: "installer", width: 1024, height: 768, app: "wine", args: ["setup.exe"] }
wait_for_window { pattern: "Setup" }
screenshot {}
# read the image, decide coordinates
click { x: 412, y: 288 }
type { text: "C:\\GAMES" }
key { combo: "Return" }
screenshot {}
display_close { all: true }
```

* `windows` returns `[{id, name, app_id, class, shell, rect, focused, pid}]`. Use `window_id` with
  `click`, `screenshot_window`, `focus`, `type`, `key` when there are several windows.
* `click { x, y, window_id, relative: true }` interprets coordinates from the window's top-left.
* `resize { width, height }` rescales the display and its host window; screenshots follow.
* `fullscreen { on: true }` makes the focused window fill the display.
* `wait { ms }` only when waiting for an animation with no window event to hook.

## Recipe: relative-only apps (emulators, captured-pointer games)

These consume *relative* motion only - absolute moves are ignored once they grab the pointer.

1. `click {}` once on the window to let the app capture the mouse (emulators usually show this in a
   status line, e.g. "click to capture" changing to "press Ctrl+End to release").
2. Clamp to the origin: `pointer_move_relative { dx: -2000, dy: -2000 }`.
3. Calibrate once: move a known amount (e.g. `dx: 200, dy: 100`), screenshot, and compare where the
   app's cursor ended up. A typical emulator maps one host unit to about 2 guest pixels.
4. To hit a point, clamp again, then send `(target - origin) / scale` as a relative move, verify
   with a screenshot, and `click`.
5. Desktop shortcuts need `click { count: 2 }`; single clicks only select.

Shutting a Windows 9x guest down: exit the app, then `ctrl+esc`, `u`, `Enter`, `Enter` in the guest.
The emulator exits by itself when the guest powers off; wait for the window to disappear before
`display_close`, otherwise you get a hard power-off and a disk check on the next boot.

## Recipe: Wine app or game at a fixed resolution

```
display_open {
  width: 640, height: 480,
  app: "wine",
  args: ["C:\\Games\\App\\app.exe"],
  env: { WINEDLLOVERRIDES: "ddraw=n", WINEDEBUG: "-all" }
}
wait_for_window { pattern: "App" }
screenshot {}
```

Capture with `screenshot` (the composited frame), not `xwd`: 8-bit palettized apps go through the
compositor's conversion, so the image matches what is on screen. Use `screenshot_window` when you
need only the app's rectangle.

## Recipe: an installer

1. `display_open` with the installer executable (or a shell wrapping it).
2. `wait_for_window` for the first dialog, then `screenshot`.
3. Prefer keyboard: `key { combo: "alt+n" }` activates Win9x/Wine buttons by underlined letter.
4. `click` only when the target is unambiguous in the screenshot.
5. Verify each page with `screenshot` before advancing.

## Host window handling

* Default backend is `x11`: the display appears as a normal decorated window titled
  `nested display: <title>`, movable by dragging its title bar.
* `position: { x, y }` places it at a chosen screen position at startup.
* To find it from a shell: `xdotool search --name "^nested display"` then
  `xdotool getwindowgeometry --shell <id>` (host `DISPLAY` required).

## Troubleshooting

| Symptom | Cause / fix |
| --- | --- |
| `display_open` times out | host session lost `WAYLAND_DISPLAY`/`DISPLAY`; restart your agent from the desktop session |
| `xwayland_display` is null | the compositor could not start Xwayland (stale `/tmp/.X11-unix/X*` socket); close displays, check the sockets, retry |
| Typed text arrives garbled in a Wine/X11 app | something used the Wayland input path in that display; close and reopen, then use only `type`/`key` |
| Click lands but nothing happens | focus the window first (`focus { window_id }`); for slow-starting apps retry once |
| Screenshot is black or empty | the app is still starting; `wait_for_window`, then capture again |
| Drag does nothing on a Wayland app | `drag` is X11-only (the virtual pointer has no button-hold); use an X11 target or keyboard |
| Window list is empty | wrong display id, or the app has not mapped yet |

## What not to use

* Host-desktop screenshot tools - many desktops deny or prompt for screen capture; use
  `screenshot` inside the display.
* `wlrctl keyboard type` for named keys - it only types text; use `key` (wtype `-k`).
* Host-desktop input injection to drive a nested display - it has its own input path.
Pliki w paczce (9)
  • .gitignore 1 KB
  • AGENTS.md 6 KB
  • CHANGELOG.md 3 KB
  • LICENSE 1 KB
  • README.md 9 KB
  • SKILL.md 6 KB
  • package-lock.json 58 KB
  • package.json 1 KB
  • tsconfig.json 1 KB

Wersje

W bazie nie ma innych skilli o nazwie nested-display.

Migawka bazy z 13.11.2026 · wersja paczki c69f7b024b0e (skrót SHA-256 całej paczki)

Pobrania w ostatnich 30 dniach: 0 · ranking

Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.

Podobne skille

  • 100terminal-mac
    Steruje odizolowanym wirtualnym macOS przez zrzuty, kliknięcia i skróty, by testować aplikacje bez zmian na pulpicie.
  • 100paw
    Steruje myszą i klawiaturą tego komputera z wiersza poleceń: zrzuty ekranu, kliknięcia, pisanie i przełączanie okien.
  • 95pc-control
    Ogląda i steruje lokalnym pulpitem Linux lub natywną aplikacją: zrzuty ekranu, myszą, klawiaturą i przełączaniem okien.
  • 95pywayne-gui
    Automatyzuje pulpit Windows: rejestruje globalne skróty klawiszowe, znajduje i steruje oknami oraz automatyzuje operacje w interfejsie z pywin32, pyuserinput i pyautogui.
  • 100kwin-desktop-automation
    Steruje aplikacjami na pulpicie KDE Plasma/Wayland przez kwin-mcp: uruchamia, klika, pisze i robi zrzuty do testów GUI.
  • 100lan-computer-use
    Steruje Makiem i komputerami Windows w sieci lokalnej przez SSH: drzewo dostępności, kliknięcia, pisanie, zrzuty ekranu.

Uwagi i zgłoszenia

Wiadomość trafi do Bartosza Sokolińskiego. Adres e-mail jest potrzebny tylko do odpowiedzi. Szczegóły w polityce prywatności.