agent-desktop

agent-desktop functions by reading the accessibility tree of a desktop application and returning its structure as JSON. Each interactive element within the application is assigned a reference (ref), allowing an AI agent to perform actions on these specific elements. The agent can then re-read the UI to verify the outcome of its action, facilitating an observe, act, and verify loop.

The tool is designed to be called by agents or scripts, rather than being an agent itself. It does not embed an LLM, feature a GUI, or provide an interactive prompt; instead, every command outputs a single JSON envelope and then exits. While it focuses on desktop applications, for web content within Chromium or Electron apps, agent-desktop can open a DevTools port for integration with agent-browser.

agent-desktop operates by leveraging platform-specific accessibility APIs: AX API on macOS, UI Automation on Windows, and AT-SPI on Linux. The core logic is platform-agnostic, with platform-specific crates handling the interaction with the operating system. This architecture ensures that a binary is compiled to support the specific OS it's intended for.

The workflow typically involves four commands: mapping an app with a shallow skeleton snapshot, drilling down into a specific region, acting on a ref (e.g., clicking a button), and then verifying the result by checking the UI again. This structured approach ensures reliable desktop automation for AI agents.