nested-display
/nested-display
Steruje aplikacją GUI, instalatorem, grą lub emulatorem w zagnieżdżonym ekranie Wayland: zrzuty, kliknięcia i skróty klawiszowe.
- Autor
- gashcrumb ↗
- Repozytorium
- nested-display-mcp ↗
- Licencja
- MIT
- Ostatnia zmiana repozytorium
- 29.10.2026
Ocena i bezpieczeństwo
- Pliki w paczce
- 9
- Skrypty
- 0
- Komentarze HTML
- 0
- Adresy URL w SKILL.md
- 0
Brak sygnałów. Repozytorium zmienione 29.10.2026, bez odwołań do wycofanych modeli i API.
MIT, otwarta: można używać i zmieniać, z zachowaniem informacji o autorze.
Co robi i kiedy
Steruje aplikacją GUI, instalatorem, grą lub emulatorem w zagnieżdżonym ekranie Wayland: zrzuty, kliknięcia i skróty klawiszowe.
Oryginalny opis (EN)
Drive a GUI app, installer, game or emulator inside a controlled nested Wayland display - screenshots, absolute clicks, typing and key chords - using the nested-display MCP. Use this instead of injecting input into the real desktop.
Jak działa
- Golden rules
- Standard workflow
- Recipe: relative-only apps (emulators, captured-pointer games)
- Recipe: Wine app or game at a fixed resolution
- Recipe: an installer
- Host window handling
- Troubleshooting
- What not to use
Podgląd SKILL.md
---
name: nested-display
description: Drive a GUI app, installer, game or emulator inside a controlled nested Wayland display - screenshots, absolute clicks, typing and key chords - using the nested-display MCP. Use this instead of injecting input into the real desktop.
---
# Nested display automation
Run the target inside its own nested sway compositor (the `nested-display` MCP). The display is a
normal window on the desktop, but input and capture happen inside it: no host focus games, no
pointer acceleration, no black screenshots, and the rest of the desktop stays usable while the
task runs.
Requires the `nested-display` MCP server to be registered with your agent (see its README).
## Golden rules
1. **One display per app.** `display_open` with `app` + `args`; omit `app` for an empty desktop.
Displays are independent: own `WAYLAND_DISPLAY`, own Xwayland `DISPLAY`.
2. **Screenshot coordinates are the truth.** `screenshot` returns a PNG plus `{width, height}`.
Click/pointer coordinates are in that image's pixel space.
3. **Never guess.** `wait_for_window` before interacting; `screenshot` after every state-changing
action. Do not trust "typed N characters" style success messages.
4. **Do not mix input paths.** For `shell: "xwayland"` windows the server uses XTest; for
`xdg_shell` it uses wtype/wlrctl. Typing with both in one session garbles X11 text because the
virtual-keyboard keymap lands in Xwayland.
5. **Clicks and keys are held briefly** (100 ms / 80 ms). Polling UIs like raylib and SDL sample
input once per frame; an instant synthetic press+release is never seen.
6. **Close displays.** `display_close { all: true }` at the end; each display keeps a compositor
and an Xwayland alive.
7. **Never `pkill -f <pattern>`** where the pattern appears in your own command line - it kills
your shell. Use the MCP's `display_close` or exact pids.
## Standard workflow
```
display_open { title: "installer", width: 1024, height: 768, app: "wine", args: ["setup.exe"] }
wait_for_window { pattern: "Setup" }
screenshot {}
# read the image, decide coordinates
click { x: 412, y: 288 }
type { text: "C:\\GAMES" }
key { combo: "Return" }
screenshot {}
display_close { all: true }
```
* `windows` returns `[{id, name, app_id, class, shell, rect, focused, pid}]`. Use `window_id` with
`click`, `screenshot_window`, `focus`, `type`, `key` when there are several windows.
* `click { x, y, window_id, relative: true }` interprets coordinates from the window's top-left.
* `resize { width, height }` rescales the display and its host window; screenshots follow.
* `fullscreen { on: true }` makes the focused window fill the display.
* `wait { ms }` only when waiting for an animation with no window event to hook.
## Recipe: relative-only apps (emulators, captured-pointer games)
These consume *relative* motion only - absolute moves are ignored once they grab the pointer.
1. `click {}` once on the window to let the app capture the mouse (emulators usually show this in a
status line, e.g. "click to capture" changing to "press Ctrl+End to release").
2. Clamp to the origin: `pointer_move_relative { dx: -2000, dy: -2000 }`.
3. Calibrate once: move a known amount (e.g. `dx: 200, dy: 100`), screenshot, and compare where the
app's cursor ended up. A typical emulator maps one host unit to about 2 guest pixels.
4. To hit a point, clamp again, then send `(target - origin) / scale` as a relative move, verify
with a screenshot, and `click`.
5. Desktop shortcuts need `click { count: 2 }`; single clicks only select.
Shutting a Windows 9x guest down: exit the app, then `ctrl+esc`, `u`, `Enter`, `Enter` in the guest.
The emulator exits by itself when the guest powers off; wait for the window to disappear before
`display_close`, otherwise you get a hard power-off and a disk check on the next boot.
## Recipe: Wine app or game at a fixed resolution
```
display_open {
width: 640, height: 480,
app: "wine",
args: ["C:\\Games\\App\\app.exe"],
env: { WINEDLLOVERRIDES: "ddraw=n", WINEDEBUG: "-all" }
}
wait_for_window { pattern: "App" }
screenshot {}
```
Capture with `screenshot` (the composited frame), not `xwd`: 8-bit palettized apps go through the
compositor's conversion, so the image matches what is on screen. Use `screenshot_window` when you
need only the app's rectangle.
## Recipe: an installer
1. `display_open` with the installer executable (or a shell wrapping it).
2. `wait_for_window` for the first dialog, then `screenshot`.
3. Prefer keyboard: `key { combo: "alt+n" }` activates Win9x/Wine buttons by underlined letter.
4. `click` only when the target is unambiguous in the screenshot.
5. Verify each page with `screenshot` before advancing.
## Host window handling
* Default backend is `x11`: the display appears as a normal decorated window titled
`nested display: <title>`, movable by dragging its title bar.
* `position: { x, y }` places it at a chosen screen position at startup.
* To find it from a shell: `xdotool search --name "^nested display"` then
`xdotool getwindowgeometry --shell <id>` (host `DISPLAY` required).
## Troubleshooting
| Symptom | Cause / fix |
| --- | --- |
| `display_open` times out | host session lost `WAYLAND_DISPLAY`/`DISPLAY`; restart your agent from the desktop session |
| `xwayland_display` is null | the compositor could not start Xwayland (stale `/tmp/.X11-unix/X*` socket); close displays, check the sockets, retry |
| Typed text arrives garbled in a Wine/X11 app | something used the Wayland input path in that display; close and reopen, then use only `type`/`key` |
| Click lands but nothing happens | focus the window first (`focus { window_id }`); for slow-starting apps retry once |
| Screenshot is black or empty | the app is still starting; `wait_for_window`, then capture again |
| Drag does nothing on a Wayland app | `drag` is X11-only (the virtual pointer has no button-hold); use an X11 target or keyboard |
| Window list is empty | wrong display id, or the app has not mapped yet |
## What not to use
* Host-desktop screenshot tools - many desktops deny or prompt for screen capture; use
`screenshot` inside the display.
* `wlrctl keyboard type` for named keys - it only types text; use `key` (wtype `-k`).
* Host-desktop input injection to drive a nested display - it has its own input path.
Pliki w paczce (9)
- .gitignore 1 KB
- AGENTS.md 6 KB
- CHANGELOG.md 3 KB
- LICENSE 1 KB
- README.md 9 KB
- SKILL.md 6 KB
- package-lock.json 58 KB
- package.json 1 KB
- tsconfig.json 1 KB
Wersje
W bazie nie ma innych skilli o nazwie nested-display.
Migawka bazy z 13.11.2026 · wersja paczki c69f7b024b0e (skrót SHA-256 całej paczki)
Pobrania w ostatnich 30 dniach: 0 · ranking
Zgłoś problem z tym skillem: szkodliwa treść, błędny opis albo prośba autora o usunięcie.
Podobne skille
- 100terminal-mac
Steruje odizolowanym wirtualnym macOS przez zrzuty, kliknięcia i skróty, by testować aplikacje bez zmian na pulpicie. - 100paw
Steruje myszą i klawiaturą tego komputera z wiersza poleceń: zrzuty ekranu, kliknięcia, pisanie i przełączanie okien. - 95pc-control
Ogląda i steruje lokalnym pulpitem Linux lub natywną aplikacją: zrzuty ekranu, myszą, klawiaturą i przełączaniem okien. - 95pywayne-gui
Automatyzuje pulpit Windows: rejestruje globalne skróty klawiszowe, znajduje i steruje oknami oraz automatyzuje operacje w interfejsie z pywin32, pyuserinput i pyautogui. - 100kwin-desktop-automation
Steruje aplikacjami na pulpicie KDE Plasma/Wayland przez kwin-mcp: uruchamia, klika, pisze i robi zrzuty do testów GUI. - 100lan-computer-use
Steruje Makiem i komputerami Windows w sieci lokalnej przez SSH: drzewo dostępności, kliknięcia, pisanie, zrzuty ekranu.