docs / agents

Hands

An agent acts in four ways. It runs an app's actions, it works in web pages in your own browser, it writes records the way your clicks do, and it runs commands in its own checkout. Each act is a tool call on the session's log, so you can see what it did and take over at any point.

The rule under all of them is parity: anything a person can do in the window, an agent can do through a tool, and a tool can do nothing a person could not. An agent changes a document with a targeted edit that names the text it expects to find, and is refused if that text has moved, rather than writing over someone's typing. It drives an app through the same actions a person's clicks run. Every change is on the session's log with who made it, and can be undone.

How it works

An app's actions. An app declares one table of actions, such as selecting a cell, sending a reply or posting to Slack. Its view's clicks run them, and an agent runs the same ones with run_action, so what the agent does and what you see cannot differ, and the app can draw the agent's selection as its own. An action can ask first: when the agent runs it, it does not take effect until you approve, and the agent is told it waits. An action can need a connection: run without it, it asks you to connect instead of failing, and runs once you do. Mod records an app's actions the first time its pane mounts, so an agent can list them with no pane open; running one needs the pane.

The agent can drive the pane in the space you are looking at, or name a document to drive the pane showing it in any other space, without moving your window. The Review app uses this to walk you through a branch file by file. The text editor uses it to bring a heading or a passage into view and select the words the agent means.

What it can do here. Its tools are the same every turn and are sent once. What changes is what it can do with what is in front of it, so each turn that it changed, the agent is told that too: each pane on screen with what works on it (an app's actions, the browser tools for a page, the edit tools for a document, asking first when writes are gated), a branch or a page in hand, and the documents in hand. The Hands tab in the bar above the composer shows the same list first, then the tools folded to one line (thread).

What it may do. The agent is told its limits before it hits one: whether this session may write or its writes ask first, what is connected and what is not, and what each app on screen may read and write. A change to any of them is sent on the next turn.

Tool groups. Every tool belongs to a group: the core tools (documents, tasks, sessions, the view, search, comments, run_action), then browser, host, apps, packages, schedules, code and each connector. A harness that defers tools itself, such as Claude Code, is handed all of them. One that lists every tool up front is handed the core tools and load_tools, which names a group's tools and lists them from the next turn, and use_tool, which runs one now.

Web pages in your browser. An agent can open a web page as a pane. It is your own browser, with your logins, and you can watch the page and take it back at any moment, which stops the agent acting on it. The agent reads a page as an outline of its headings, text, links, buttons and fields, each one it can act on tagged with a ref. It clicks and types by those refs, and reads again after the page changes, because old refs are refused. It takes a screenshot only for what the outline cannot say (vision). It can also film a page while it drives it, and the video is saved into the project for a task, a comment or a message to attach.

Some things on a page are left to you. A button that pays, books, sends or deletes is refused, and so is typing a password, a card number or a code. Pressing Enter is allowed only in a search box; any other form is submitted by clicking its button.

Writing records. An agent changes an app's records by writing the record, the same way your clicks change it, so there is no agent-only path into an app. An app can also offer package tools, which change its records with no window at all (tools). Documents are changed with a targeted edit that names the text it expects to find (documents).

Running commands. An agent runs commands in its own checkout through its harness, such as Claude Code's Bash tool (harnesses). In a VM, the checkout is a Linux machine with the usual tools, and you can add tools of your own once for every VM (vms). It holds no credential.

Work that needs your Mac or your identity is a host action: a named tool the Mac runs on the agent's behalf. Publishing a branch, merging a pull request and reading the Mac's log are host actions. So are recipes, the commands a project names for the Mac, such as building the iPhone app or deploying a worker. A recipe is not a shell: the agent passes named arguments that must each match a pattern, so approving one recipe approves nothing else. It runs on the Mac with the Mac's sign-ins, and the session is woken when it ends, with its exit code and the end of its output.

Approvals. With writes turned off in a project, a session asks before it runs a command, writes a file or makes any of Mod's own writes. A recipe you have not approved asks too. The question goes on the session's log with "This time", "Always in this project" and "No", and you can answer it from any surface, your phone included. Your standing yeses are kept on your Mac, where no agent can write them (authorisation).

Using it as you build

Run an app's actions. Call with no action first to list them:

run_action  { "documentId": "<reviewId>" }
run_action  { "documentId": "<reviewId>", "action": "focus", "input": { … } }

control_app is the same tool under its old name. To give your own app actions, declare them with mod.app:

const app = mod.app({
  actions: {
    select: { description: 'Select a cell', run: (input, ctx) => ctx.set({ selected: input.cell }) },
    send: { description: 'Send the reply', ask: 'always', review: (i) => `Send reply to ${i.to}`, run: send },
    postToSlack: { description: 'Post the summary', needs: ['slack'], run: post },
  },
})
td.onclick = () => app.run('select', { cell: 'B5' })   // the view's click runs the same action

ctx.by is 'person' for a click and 'agent' for run_action. ask is 'always' or 'first' (once per pane). An app that registers commands with mod.surface.control.register still works, and its commands are listed the same way. To check what is allowed before offering something, and hear when it changes:

const caps = await mod.capabilities.get()
mod.capabilities.onChange((c) => (slackButton.hidden = !c.connected.includes('Slack')))

Work in a web page:

browser_open        { "url": "https://example.com" }
browser_read        { "page": "p1" }
browser_click       { "page": "p1", "ref": "e9" }
browser_type        { "page": "p1", "ref": "e12", "text": "reset expiry" }
browser_scroll      { "page": "p1", "dy": 600 }
browser_navigate    { "page": "p1", "back": true }
browser_screenshot  { "page": "p1" }
browser_record      { "page": "p1", "action": "start", "seconds": 60 }
browser_record      { "page": "p1", "action": "stop", "path": "recordings/evidence.mp4" }

Write an app's record, or run a package tool:

update_document  { "documentId": "<instanceId>", "data": { … } }
run_app_tool     { "app": "<app>", "tool": "<tool>", "input": { … } }

From a VM, act on the Mac:

publish_branch
merge_pr
list_recipes
run_recipe      { "name": "ios-testflight", "args": { … } }
recipe_status   { "jobId": "<jobId>" }
read_host_log

Name a project's recipes in .mod/recipes.json, each with a name, a one-line description to approve against, its command, the folder it runs in and the arguments an agent may pass:

{
  "recipes": [
    {
      "name": "ios-testflight",
      "description": "Build the iPhone app and upload it to TestFlight",
      "command": ["./scripts/testflight.sh", "{lane}"],
      "args": { "lane": { "pattern": "beta|release" } }
    }
  ]
}

Turn writes off or on for a project with project_writes.

Limits

run_action and the browser tools need the Mac app's window. A worker with no window does not get them.
An app's actions are listed with no pane open, but running one needs its pane mounted.
The browser opens http and https pages only.
At most two pages are held off screen for agents at once.
browser_screenshot takes the visible part of the page only.
A recording runs for 60 seconds by default and 180 at most.
A recipe runs for 30 minutes by default and three hours at most.
A question waiting for your approval is refused for now after five minutes.
An approval holds for a recipe's name and what it runs. Change its command and it needs a new approval.

backVoice nextApps