Vision

An agent can work with pictures the way it works with prose. It can save a screenshot into a project, hang it on the task it is evidence for, embed it in a document, and read it back and see what is in it. It can also take a picture of a web page it has open.

How it works

An image is a document. Saving an image into a project makes a file document: it has a path, a title, a pane and comments like any page, and it syncs to everyone in the project. Its bytes live in that one document. A task or a document that shows the image holds only its id (documents).

Pictures in messages. You can attach a file to a message with the + button, by dropping it on the composer or by pasting it. Each attachment is a document of its own, and the message holds a pointer to it, so the session's log stays small. An image is shrunk before it is stored if its longest edge is over 2048 pixels, and the agent receives it as a real picture, not a filename. Any other file is stored and shown as a chip, and the agent is told its name and type. An attachment belongs to its message and is not listed in the project. Saving it is how it becomes a project document.

Bytes from anywhere. An agent can save an image from a message attachment, a path on its machine, bytes it is already holding, or a URL. A URL that needs a sign-in, such as a screenshot posted in Slack, is fetched through the connector that has that sign-in. The token stays in the connector and never reaches the agent or the tool.

On a task and in a document. A task names a picture of itself with a document reference whose role is image, and a label that captions it. Wherever the task is drawn, its expanded row shows the image as a thumbnail that opens it: in the session header, in a task list, and on a card in an app. A document embeds an image with markdown's own image syntax, and draws the picture inline, both when you read and when you edit.

Reading it back. When an agent reads an image document, it gets the picture. When it reads a document that embeds images, it gets each one in reading order, captioned with its alt text. When it reads a task, it gets the task's images, captioned with their labels. So a fresh session that picks up a task tomorrow can see the screenshots on it without being told what they show. The text that names each image always comes too, so a surface that cannot show pictures still knows they are there. An SVG is stored and drawn but never sent to a model, because no provider takes one.

Structure before pixels. An agent reads a document's text, an app's records or a web page's outline before it reaches for a picture. A screenshot is for what only shows on screen, such as a chart, a map or a layout. See hands.

A shared screen in a call. While an agent is listening in a call, it sees a screen someone shares as sampled frames, and never anyone's camera. Its tile in the call says whether it is listening or watching. See voice.

A picture of a web page. An agent with a web page open in your browser pane can take a screenshot of what the page is showing. The picture goes to the agent, and it is kept as an attachment document, so the thread shows what the agent looked at.

Using it as you build

Save an image into the project, from one source:

save_file  { "file": "/Users/you/Desktop/export.png", "path": "bugs/export-blank.png" }
save_file  { "bytes": "<base64 or data: URL>", "path": "bugs/export-blank.png" }
save_file  { "url": "https://example.com/chart.png" }
save_file  { "url": "<Slack url_private>", "via": "slack", "path": "bugs/export-blank.png" }

It returns the new document's id. Attach it to a task with a caption:

update_task  { "taskId": "<id>", "refs": [{ "type": "document", "target": "<imageId>", "role": "image", "label": "Export dialog renders blank" }] }

Embed it in a document, on its own line, by id or by path:

![Export dialog renders blank](<imageId>)

An image written with a real web address, such as ![alt](https://…), is drawn as an ordinary image.

Read it back. Each of these returns the pictures as well as the text:

read_document  { "documentId": "<imageId>" }
read_document  { "documentId": "<docThatEmbedsImages>" }
read_task      { "taskId": "<id>" }

Look at a web page:

browser_open        { "url": "https://example.com" }
browser_read        { "page": "p1" }
browser_screenshot  { "page": "p1" }

An app in a sandboxed frame cannot read a document's bytes. It asks for a downscaled picture over the app bridge with docs.thumbnail, scoped to the project the way docs.read is (apps).

Limits

• One read carries at most four pictures, each at most 4 MB. Past either, the agent gets a sentence naming the image, its size and its id instead.
• save_file stores files up to 20 MB.
• A message attachment is refused if it is still over 5 MB after shrinking.
• An image saved with save_file is stored at the size it arrived. Only message attachments are shrunk.
• browser_screenshot takes the visible part of the page. Scroll and take another for the rest.
• Pictures are sent as PNG, JPEG, GIF or WebP.