Computer Use Turbo

Check whether the helper has the permissions it needs: accessibility (required for everything) and screen capture (needed for screenshots). Use it when a tool fails with accessMissing or screenshots are missing. request: true shows the permission prompts / settings to the user; only do that when the user wants to set up permissions. You cannot grant them yourself.

Click in an app. Give exactly one target: element (preferred; from the latest observe_app) or x and y (pixels of the latest screenshot as delivered, origin top-left). The app does not need to be in front. Apps that draw their own UI are brought forward and clicked with the real pointer, which is put back right after, only while the user is idle; otherwise the click fails with userActive (nothing sent, retry shortly). button: left (default) / right / middle; times: 1-3 (2 = double-click). Requires a prior observe_app for this app in this session. Call observe_app afterwards to verify the result.

Press the left mouse button at (start_x, start_y), move to (end_x, end_y) and release. Coordinates are pixels of the latest observe_app screenshot (origin top-left); there is no element form. For sliders, moving items (files too), resizing, drawing. File drags that need the real mouse briefly bring the app forward and use it while the user is idle (put back afterwards). Requires a prior observe_app for this app in this session. Call observe_app afterwards to verify the result.

Replace the value of an editable element (text field, text area, slider, …) directly through accessibility. element comes from the latest observe_app; the element should be marked {editable}. A text field that is not focused is focused first. Never presses Return or submits: to submit, send_keys "return" (or invoke_action "Confirm" when the line lists actions=Confirm). A rename field only commits on Return; the note says so. The value is read back afterwards; if the app reverted or ignored it, the result says "did not keep" and shows what the field holds; then click_at the field and use write_text instead. If the app only reformatted it (e.g. "1000" → "1,000"), the note says so: do not type it again. A checkbox-like true/false value takes true/false, 1/0, yes/no, on/off or checked/unchecked. For a browser address bar prefer send_keys "cmd+l" then write_text "<url>\n". Refused for password fields. Requires a prior observe_app for this app in this session. Call observe_app afterwards to verify the result.

List the apps you can target: running apps first, then recently used ones, most recent first. One line per app: "Name — id — running/not running — last used <date> (<n> uses)". Pass the name or id as `app` to observe_app and the action tools (if a name does not resolve, use the id). Last come background processes marked "system panel / status items": the system's menu bar extras (Control Center, the clock that opens Notification Center and its widgets, Focus, volume, a search icon) and the panels they open. Observe one to see its status items and click one to open its panel. Does not ask the user for approval.

Search an app's menu commands — every item of its menu bar, with its shortcut — by words ("export pdf", "bold") or by shortcut ("cmd+shift+s"). Lines look like `Format ▸ Font ▸ Bold [cmd+b]`; "(unavailable now)" marks a command the app greys out right now (it may need a selection or a document). Run one with run_command. The list is read once per app and saved, then only checked for changes, so apps used before answer instantly. Without query: the menus with their command counts (every command when the app has few). Opens no menu and does not need observe_app first.

Call once the task is finished (or abandoned). Ends this session in the helper: forgets element numbers, change baselines and screenshots, drops "For this task" approvals and hides the status overlay and live preview. If the user pressed Stop (haltedByUser) or declined access, end the session and ask the user how to proceed; do not call observe_app or any action again until the user explicitly asks you to continue.

Run a named accessibility action of an element: one listed after actions= on its line or under "More actions on #n" in observe_app, by that name (e.g. Increment, Decrement, "Show Alternate UI", "Scroll Down", "Show Menu", Raise, Cancel, Confirm; case-insensitive). Use only names that are shown, for what a plain click cannot do. Requires a prior observe_app for this app in this session. Call observe_app afterwards to verify the result.

Look at an app: its UI as numbered lines plus a screenshot of its window. Call it before acting on an app and again after every action (look → act → look). Launches the app if needed (a password manager asks the user first, once per task). Lines look like `#12 text field "Name" value="Ada" {focused, editable} actions=Increment`; #12 is the element number for the action tools and stays the same for that element for the whole session. Static text shows as `#5 text: Hello`, links as `#7 link [Title](example.com/page)`; long lists show only their visible rows plus a "(… more rows hidden; scroll to see them)" note. Covered: the active window with its sheets, popovers and open menus; other windows are one summary line each (invoke_action "Raise" switches to one); the menu bar on the first observation and while a menu is open. Later observations list only changes — M modified, A added, D deleted — when that is shorter; full_tree: true returns everything. A footer names the element with keyboard focus and any selected text. x/y for click_at / scroll_view / drag_item are pixels of this screenshot (origin top-left), read straight off the image; large windows are downscaled by the helper (see the Window line) and mapped back for you, so never rescale. Windows with web content show "Page loading: yes (NN%)" / "no"; while it says yes, observe again before using page content (a note says when a page makes no progress: then use it as it is). Apps that draw their own UI (games, 3D tools) expose almost nothing to accessibility: the text then says so and you judge results from the screenshot. Repeat observations also say whether the screenshot changed and by how much.

Paste content into the app at the caret / over the selection, through the clipboard: the helper puts the content on the clipboard, pastes it (the Paste menu item, or an accessibility insert of plain text, while the app is in the background), waits until the app has read it and then restores the user's previous clipboard (unless the user copied something meanwhile). format: "text" (default), "markdown" or "html" — the last two paste rich text (headings, bold, lists, links) where the app supports it. Good for long text and formatted text; focus the target field first (click_at it). Refused for password fields. Requires a prior observe_app for this app in this session. Call observe_app afterwards to verify the result.

Select a piece of text inside an element's value, or put the caret right before/after it. text is the exact substring; text_before / text_after (the text right next to it) tell repeated occurrences apart. mode: "select" (default), "caret_before", "caret_after". Useful before write_text to replace or insert at a precise spot. Requires a prior observe_app for this app in this session. Call observe_app afterwards to verify the result.

Run a command from the app's menus by its path, as find_command lists it ("File ▸ Export As PDF…", "Format > Font > Bold"). It is pressed through accessibility with the app in the background: no menu opens and the user is not disturbed. A command the app greys out only while it is inactive briefly brings the app forward while the user is idle (handed back right after); one that is greyed out anyway fails (it needs a selection, a document or another window). Requires a prior observe_app for this app in this session. Call observe_app afterwards to verify the result.

Run a sequence of actions on an app in one call — faster than one call per action. Each step is an action tool with its usual arguments plus "tool" (no "app" needed: steps use the batch's app unless they give their own), or {"tool": "pause", "ms": n}. Example: [{"tool": "click_at", "element": 12}, {"tool": "write_text", "text": "Ada"}, {"tool": "send_keys", "combo": "Return"}]. Every step goes through the same checks as the single tools (approval, Stop, password fields, take-back), in order, one at a time. All steps are validated before the first one runs. The batch stops at the first failure (stop_on_error: false reports it and goes on, except after Stop, a declined approval, the user taking over, a locked screen or a timeout). The result lists every step with its outcome and, unless observe_after is false, ends with an observation of the app (text + screenshot), so you do not need a separate observe_app. Element numbers must come from the latest observation; use run_steps for sequences whose effect you can predict (filling a form, typing and submitting, repeated clicks) and observe between steps when the UI may change unexpectedly. At most 50 steps. Requires a prior observe_app for this app in this session.

Scroll inside an app at an element (element) or at a screenshot point (x and y); exactly one target. direction: up/down/left/right; pages: 0.1-20 (default 1 ≈ one visible page). Requires a prior observe_app for this app in this session. Observe again with observe_app to see the newly revealed content.

Press a key or key combination in the app, "+"-separated and case-insensitive, e.g. "Return", "Tab", "Escape", "cmd+a", "ctrl+shift+Tab", "Down", "Page_Down" / "Next", "Prior", "Home", "F5", "KP_Enter", "BackSpace", "Delete" (forward delete; lower-case "delete" is Backspace). Modifiers: cmd/command/super/meta, ctrl/control, alt/option, shift (each also with _L/_R), fn; a modifier alone ("Shift_L") is pressed by itself. Keys: letters, digits, punctuation or its name (comma, period, slash, bracketleft, exclam, …), Return/Enter, Tab, space, Escape, BackSpace, Delete, arrows, Home/End, Page_Up/Page_Down, Insert/Help, F1-F20, KP_* keypad keys, Caps_Lock. The app stays in the background when possible (a shortcut presses the matching menu item; select-all / copy / cut / paste on a text field go through accessibility). A shortcut the app disables while inactive, or Return in a rename field, brings the app forward for this one key press while the user is idle (handed back right after); if the user is busy it fails with userActive (nothing sent, retry shortly). Requires a prior observe_app for this app in this session. Call observe_app afterwards to verify the result.

Show (show: true) or hide (show: false) the live preview: a small panel at the top right of the window the user talks to you in that shows a live image of the window you are working on with your pointer moving in it. It appears by itself while you work (unless the user turned it off) and stays up until finish (or the user presses Stop / Esc). Use this only when the user asks to see or hide it.

Wait until something happens in the app instead of observing again and again: text appears / is gone, an element changes, a window or dialog opens, the app settles, or anything changes (see until). The helper listens to the app's accessibility events and checks again after each burst of changes, so it returns as soon as the condition holds (or after timeout_s). The result says whether it matched, lists what changed while waiting and, unless observe_after is false, ends with an observation (text + screenshot). Good after starting a download, an export, a page load, a search or a long operation. Stop ends the wait. Requires a prior observe_app for this app in this session.

Type text into the app: inserted through accessibility at the caret / over the selection when the element allows it, otherwise as keystrokes. It adds to what a field already holds; to replace a field's contents (an address or location bar, a search box) use fill_value. With element that element is focused first; otherwise the text goes to the focused element. "\n" presses Return, which may submit forms. Works with the app in the background; when that cannot work (an app that ignores background keys, Return in a rename field) the app is brought forward for this one call while the user is idle, else it fails with userActive (nothing sent; retry shortly). Refused for password fields (passwordGuard), also when Return or Tab moves focus into one partway (typing then stops): never type passwords or other credentials. To replace a field's whole contents, or for long text, use fill_value. Requires a prior observe_app for this app in this session. Call observe_app afterwards to verify the result.