We gave our agent GUI access. It immediately enabled sticky keys three times.

Our coding agent was working perfectly until we gave it GUI access. Then it started clicking random buttons, dragging windows around, and somehow managed to enable sticky keys three times in one session.

The problem wasn't the agent — it was our permission model. We'd given it binary access: either full desktop control or nothing. No middle ground. No guardrails. No scope.

Here's what we learned building a permission system that actually works:

Start with application-level permissions, not system-level

Don't give your agent access to "the desktop." Give it access to specific applications. Our agent needs to interact with VS Code, Terminal, and Chrome. It doesn't need to touch System Preferences or drag files around the desktop.

ALLOWED_APPS = [
  "com.microsoft.VSCode",
  "com.apple.Terminal", 
  "com.google.Chrome"
]

# Block everything else
if app_bundle_id not in ALLOWED_APPS:
  return "Application access denied"

Zone your screen real estate

Even within allowed apps, your agent doesn't need full window access. We created "safe zones" — specific screen regions where the agent can click and type. The code editor pane is fair game. The file deletion dialog is not.

SAFE_ZONES = {
  "vscode": {
    "editor_pane": (200, 100, 1400, 800),
    "terminal_pane": (200, 800, 1400, 1000)
  },
  "chrome": {
    "content_area": (0, 100, 1920, 1000)
  }
}

Implement action approval for destructive operations

Some GUI actions are irreversible. File deletions, system preference changes, anything involving sudo. These need human approval, even if they're in a safe zone.

Critical: Build a "destructive action" classifier. Train it on your specific use cases. The agent will try to convince you that restarting Docker is "just a quick refresh."

Use capability tokens, not session permissions

Instead of granting blanket GUI access for the entire session, issue time-limited capability tokens for specific tasks:

# 10-minute token for VS Code debugging
token = issue_capability_token(
  apps=["com.microsoft.VSCode"],
  actions=["click", "type", "scroll"],
  duration=600,
  task_context="debug React component"
)

When the token expires, the agent has to request a new one with justification. This prevents the "I'll just quickly check something" spiral that burns through your afternoon.

Build an audit trail that actually helps

Every GUI action gets logged with context:

What the agent was trying to accomplish
Which permission zone it used
Screenshot before and after the action
Success/failure outcome

This isn't just for security — it's for debugging. When your agent says "I clicked the submit button" but nothing happened, you need to see what it actually clicked.

The result? Our agent now handles complex GUI workflows without chaos. It can debug React components in the browser dev tools, manage terminal sessions, and edit code — all without accidentally enabling voice control or deleting system files.

Most importantly, we sleep better knowing it can't accidentally nuke the production database because it got confused about which terminal tab was active.