Skip to main content

Desktop Sandbox (Linux + noVNC)

The desktop-sandbox Docker Compose profile starts a reproducible Linux desktop (Xfce) with noVNC for operator takeover, plus a paired desktop-app node for Tyrum.

Quickstart​

export GATEWAY_TOKEN="$(openssl rand -hex 32)"
docker compose --profile desktop-sandbox up -d --build
  • noVNC takeover: http://localhost:6080/vnc.html?autoconnect=true
  • Gateway UI: http://localhost:8788/ui

Security note: the sandbox desktop is intentionally unauthenticated in v1 (VNC/noVNC). The desktop-sandbox profile binds ports to 127.0.0.1 by default; do not expose it to untrusted networks. Gateway takeover links still go through an admin-authenticated gateway route before redirecting to the local noVNC endpoint.

Pairing and takeover​

  1. Open http://localhost:8788/ui, paste $GATEWAY_TOKEN on the Connect page, and connect.
  2. Open the Pairing page, approve the pending pairing request, and use the Open takeover link to open the noVNC conversation.

Manual QA checklist (operator UI)​

  • Trigger a Desktop act approval (any flow that calls tool.desktop.act).
  • In Approvals, the card shows a Desktop summary (op + action + target) and an Open takeover link.
  • Approve and deny both work; the approval disappears after resolution.
  • In Runs, open the run → attempt → Artifacts and confirm:
    • screenshot artifacts render inline
    • a11y tree artifacts render as an expandable JSON viewer
    • artifacts are clearly marked Sensitive

Manual verification (desktop snapshot)​

After approving pairing, run a Desktop snapshot:

TOKEN="${GATEWAY_TOKEN}"
curl -sS -H "authorization: Bearer ${TOKEN}" -H "content-type: application/json" \
  -d '{"conversation_key":"agent:default:manual:default:channel:desktop-sandbox","steps":[{"type":"Desktop","args":{"op":"snapshot","include_tree":false}}]}' \
  "http://localhost:8788/workflow/start"

Manual verification (AT-SPI a11y)​

  1. Open the noVNC takeover and launch a GUI app (for example Xfce Terminal). You can also launch one from the host:
docker compose exec -T desktop-sandbox bash -lc 'DISPLAY=:0 xfce4-terminal --title "Tyrum A11y Smoke" >/tmp/tyrum-a11y-smoke.log 2>&1 &'
  1. Run snapshot → query → act with include_tree: true:
TOKEN="${GATEWAY_TOKEN}"
curl -sS -H "authorization: Bearer ${TOKEN}" -H "content-type: application/json" \
  -d '{"conversation_key":"agent:default:manual:default:channel:a11y-desktop-sandbox","steps":[{"type":"Desktop","args":{"op":"snapshot","include_tree":true,"max_nodes":512,"max_text_chars":8192}},{"type":"Desktop","args":{"op":"query","selector":{"kind":"a11y","name":"Tyrum A11y Smoke"},"limit":1}},{"type":"Desktop","args":{"op":"act","target":{"kind":"a11y","name":"Tyrum A11y Smoke"},"action":{"kind":"focus"}}}]}' \
  "http://localhost:8788/workflow/start"

If the run pauses for policy approvals, approve in the Gateway UI (or use the existing smoke script as a reference for automating approvals).

Smoke test​

bash scripts/smoke-desktop-sandbox.sh

Set TYRUM_SMOKE_KEEP_RUNNING=1 to leave containers running after the script finishes.

Running the desktop automation tests​

The gateway test suite includes an end-to-end smoke test that boots an in-process gateway, starts the desktop-sandbox container, approves pairing, then dispatches a Desktop snapshot and a safe mouse action.

This test currently requires Linux + a working Docker daemon, and may take a few minutes on first run while the sandbox image builds.

pnpm exec vitest run packages/gateway/tests/integration/desktop-sandbox-e2e.test.ts

Useful environment variables:

  • TYRUM_DESKTOP_SANDBOX_REBUILD=1 forces a Docker rebuild of the sandbox image.
  • TYRUM_DESKTOP_SANDBOX_IMAGE=<tag> uses an existing prebuilt image tag.

Notes​

  • The published ghcr.io/tyrumai/tyrum-desktop-sandbox:main image is currently linux/amd64 only. @nut-tree-fork/libnut-linux still bundles an x86_64 libnut.node, so linux/arm64 desktop-sandbox images fail native startup.
  • Workflow publication is not enough by itself; the GHCR package also has to be anonymously pullable. If the gateway reports an unauthorized pull for ghcr.io/tyrumai/tyrum-desktop-sandbox:*, either make the package public or build and tag a local fallback image.
  • Local fallback: run bash scripts/build-desktop-sandbox-image.sh [tag], then update default_image_ref via PUT /config/desktop-environments/defaults to the local tag the script prints.
  • On darwin/arm64, gateway-managed desktop environments using the published Tyrum sandbox image are launched with Docker --platform linux/amd64. This relies on Docker Desktop's amd64 emulation support.
  • The sandbox image includes DBus and AT-SPI packages (dbus, dbus-x11, at-spi2-core) to maximize a11y availability for Linux backends.
  • Override the URL shown in the pairing UI via TYRUM_DESKTOP_SANDBOX_TAKEOVER_URL (useful when the host is remote).