How AI agents see your screen: screenshots vs the accessibility tree

To use a computer, an agent has to answer one question over and over: what’s on the screen right now? There are two ways to answer it.

Option 1: screenshots

Take a picture, send it to a vision model, ask "where is the Send button?", get back coordinates, click there.

It works on anything that’s visible, which is its strength. The costs:

• Guessing. The model infers what a blue rectangle is. Similar buttons get mixed up.
• Coordinates. A moved window, a different screen size or a scrolled page sends the click somewhere else.
• Weight. Images are large. Each step is slower and more expensive.
• Privacy. A screenshot contains everything on the screen, not only the part the task needs.

Option 2: the accessibility tree

Every major operating system describes its windows to screen readers, so blind users can work. That description is a tree: a window contains a toolbar, the toolbar contains a button named "Send", which is enabled.

An agent that reads the tree knows that something is the Send button, not that it looks like one. In ands, each element gets a short reference, and actions target that reference:

ands_look  → button "Send" @s7k2:e14
ands_act   → click @s7k2:e14

Why this is better for agents:

• Exact. It clicks the element, not a point.
• Light. Text, not images, so steps are fast.
• Checkable. The safety gate knows exactly what is about to be clicked, by name, so it can tell "Send" from "Save draft" in English or Hebrew. That’s what makes the approval tiers reliable.

When the tree isn’t enough

Some apps draw their own controls (games, some old programs, canvases). There the tree is thin, and ands can fall back to a screenshot. Because a screenshot can capture anything on the screen, taking one is a sensitive action and needs approval.

For developers

If you’re building an agent, you get both through MCP: ands_look for the tree, ands_act and ands_press to act on it, and ands_screenshot for the rare case. Setup is one line: see Give Claude Code control of your desktop apps.