dsh-cua - Windows Computer Use

Capture a window screenshot and SAVE IT TO A FILE. Provide hwnd or title substring. Crops to the window's CLIENT area by default, so image pixel (0,0) is the same point click_at calls (0,0) — no frame offset to guess. Returns bounds = the screen rect (x,y,width,height) the image maps to, plus `scale` (multiply image pixels by it to get screen pixels; 1.0 unless you asked for downscaling). Also reports dpi_verified; when false, image pixels and input coordinates cannot be safely paired. Defaults to JPEG q80 — a 1080p shot is ~6-10x smaller than PNG, which is what keeps a long session's request body under the provider's 32 MiB cap. The bytes are NOT inlined as base64 by default (that costs ~12,500 tokens per shot as text); read the saved_path with read_image instead, which attaches the same picture for ~50 tokens. Pass include_data_uri=true only when a caller genuinely cannot read a file. Read-only — no input injection, no focus change.

Click at client-area coordinates within a window. Coordinates relative to window content area (0,0 = top-left). Tries the ELEMENT path first (resolve the point via UI Automation and invoke that element) because it needs no focus and does not care about z-order; falls back to a raw cursor click, which does. The result's `method` says which was used: 'ax_press' or 'raw_event'. MUTATING: serialized across agent sessions; the raw_event path ALSO YIELDS TO THE USER (waits for input-quiet, then refuses with user-active rather than fighting them for the cursor); the ax_press path injects no physical input. The raw path verifies the addressed window actually owns that screen point and returns success=false instead of clicking another window. Pass dry_run=true to check where the click would land without moving the cursor, clicking, or taking any gate.

Read the current clipboard text. Read-only. Non-text clipboard content reports empty text rather than failing.

Replace the clipboard contents with text. MUTATING (soft): serialized across agent sessions — and it DESTROYS whatever the user last copied, so do not use it casually; announce it when you do.

Report the coexistence policy in force: whether the admission gate is enabled, the quiet/max-wait/mutex-timeout tunables, the pre-injection final-check window, named-mutex health, and how long ago the last physical input happened. Read-only — call it when an action was refused with user-active or arbiter-busy and you want the numbers behind the refusal.

Current cursor position plus the screen size, both physical pixels. Read-only.

Perform an action on a UI element by the `ref` from element_at_point — this is the focus-free path: a UI Automation pattern is delivered to the element, so the window is never raised and z-order never matters. MUTATING (soft): serialized across agent sessions; NO physical input is injected, so it can run while the user types — just do not operate the very window the user is working in. `action` must be one of the actions listed for that element (press, set_value, select, toggle, expand, collapse, scroll_into_view, focus); `set_value` also needs `text`. Refuses with the supported list when the element does not offer the action.

Act on the element that line `index` of the last skyshot named. This is the focus-free path for elements element_at_point cannot reach: the tree names the element, so the action goes to it and the window is never raised. MUTATING (soft): serialized across agent sessions; no physical input injected. `action` is one of press, set_value, select, toggle, expand, collapse, scroll_into_view, focus; set_value also needs `text`. Take a fresh skyshot if the UI has changed — a stale index is refused rather than acted on.

Identify the UI element at a PHYSICAL screen point via Windows UI Automation. Returns its name, role, rect, and which actions it supports, plus an opaque `ref` for read_element. This is how you turn 'what I see in the screenshot' into something addressable without clicking: coordinates here are the same space as capture bounds (both physical pixels). Read-only — moves no cursor and changes no focus.

Find UI elements by role and/or name substring and return refs that can be acted on with element_action. This is how you reach an element that cannot be hit-tested — the point of naming it is that you no longer need a pixel. `role` is the lowercase role from skyshot (button, edit, document, list_item, ...). At least one filter is required: an unfiltered call would be a tree dump, so use skyshot for that. The result reports `minimized` — a minimized window does not always expose its full tree, so a zero here is not proof that nothing matches. Read-only.

Find a window by title substring and return its hwnd and geometry. Read-only.

Get detailed position and size info for a window: absolute position, dimensions, and client area size. Read-only.

List monitors: index (1-based), primary flag, device name, and the monitor plus work-area rectangles in physical pixels. Read-only.

List all visible top-level windows with title, hwnd, PID, position and size, sorted largest first. Read-only — safe while the user works.

Open a file, folder, or URI with the shell's default handler — like double-clicking it in Explorer: an .exe path, a document, or a URL. MUTATING (soft): serialized across agent sessions; the launched app may take focus.

Re-read a UI element by the `ref` returned from element_at_point. Use this to check whether an element is still there before acting on it; a ref goes stale when the UI is rebuilt, and that is reported rather than guessed. Read-only.

Send keyboard shortcut to a window. Example: keys=['ctrl','s']. For Alt+key menu navigation, include 'alt' as first key (or set use_alt_key). MUTATING: serialized across agent sessions via the admission gate, and global hotkeys additionally YIELD TO THE USER — recent mouse/keyboard activity makes them wait, then refuse with reason user-active instead of interrupting; switch to element actions or retry later. Also refuses when the window is not the foreground window, because injected keys would go to the wrong window.

Read a window's UI as a compact TEXT tree instead of a screenshot — far cheaper than an image, and it names elements a hit-test cannot reach (e.g. an input inside a Chromium page, which element_at_point reports only as the enclosing 'document'). Each line is '{index} {indent}{role} {name}{Value: ...}{(state)}'. Every shot after the first is a DIFF against the previous one: unchanged lines are omitted, '~' marks a changed line, '+' an added one, and removed indices are summarised as ranges. So a second call is usually a few lines, not the whole tree — an unchanged window costs about 100 characters. INDEXES ARE POSITIONAL: each shot numbers the tree it just walked, so a number is only valid against the shot that printed it. '~' and '+' lines carry THIS shot's index and may be acted on; unchanged lines carry no index, so reaching one needs disable_diff=true for a full render. An index read from an older shot is not a durable handle — re-check it with a fresh shot before acting if the window may have changed. The result includes shot_key (the window the diff baseline belongs to — the baseline lives server-side and outlives your session, so this is how you confirm it was taken against the window you meant). Pass include_offscreen=true to include elements that are scrolled out or hidden. The result reports `minimized`, because a minimized window does not always expose its full tree (measured on Edge and on Explorer; Notepad was unaffected) — so a small tree is not proof that the window is empty, and include_offscreen does not recover it. Restore the window once and read again if the result matters. NOTE: this reads element VALUES, so the text can contain whatever is on screen in that window. Read-only — no input injection, no focus change; safe to call while the user works.

Type a text string into a window character-by-character (window-targeted PostMessage, not global keystrokes). The text control inside the window is resolved automatically — a top-level window does not forward WM_CHAR to its child edit, so passing a window handle alone used to post into nothing — and the control's text is read back so `effect_verified` says whether the text actually landed. MUTATING (soft): serialized across agent sessions; no physical input is injected so it can run while the user types — but the window may be RAISED to foreground, so avoid typing into a window the user is currently working in.