Skip to main content
This is the entry point for all automation. Obtain a machine with [Machine.local] or [Machine.android], then get apps, windows, and accessibility nodes from it — every handle stays bound to the machine it came from, and the same code drives any backend. Mouse, keyboard, and clipboard access are flat methods on the machine itself (moveMouse, typeText, getClipboardString, …).

​
Canonical coordinate space

This is the reference description of the coordinate space used throughout the library. Window.boundingBox(), AccessibilityNode.boundingBox(), screenshots, and grounding-model output all live in this same space, so coordinates round-trip between them without conversion. Absolute coordinates live on the global desktop: top-left origin at (0, 0) on the primary monitor (on Android, the device screen), with the OS’s native units. The unit is not the same on every OS:
  • Windows, Linux, and Android use physical pixels (raw hardware pixels).
  • macOS uses logical points — on a 2× Retina display one point spans two hardware pixels, so coordinates are half the physical-pixel count.
Within a single OS every function speaks that OS’s unit, so the round-trip guarantee holds; only code that crosses into a different coordinate system (e.g. an Electron overlay measured in CSS pixels) needs to account for the per-OS unit. These are also the native units the OS input/accessibility APIs expect, so they are not the browser logical/CSS pixel. Monitors arranged to the left of or above the primary display contribute negative coordinates, so callers should not assume x, y >= 0. Use [Machine.screens] / [Window.screen] to discover where the addressable region actually is.

​
Constructor

​
Returns

Machine

​
Properties

​
string
required
A human-readable identifier for the machine: the local hostname, or the adb serial (host:port) for an Android device.
​
Os
required
The operating system this machine runs: 'macos' | 'windows' | 'linux' | 'android'.

​
Methods

​
android()

Connect to an Android device at an adb endpoint (host:port). The device must be reachable over adb and run the uiautomator2 (appium) server; appiumBase is its base URL and defaults to the standard local forward (http://localhost:6790). The appium URL is per device: when several devices are connected, forward a distinct local port for each. Unlike the local arm, each Android machine owns its emulated cursor and modifier state, so input state is per-handle.

​
Parameters

​
string
required
​
null | string

​
Returns

Machine

​
app()

An installed application by exact name (no fuzzy matching). Locally this is the launcher name or path; on Android it is the package name (e.g. com.android.chrome).

​
Parameters

​
string
required

​
Returns

App

​
apps()

All installed applications.

​
Returns

App[]

​
asAndroid()

Phone-only operations of the underlying Android device, or null when this machine is the local desktop. Everything cross-platform lives on [Machine] and the handles it returns; this escape hatch is only for operations that have no desktop counterpart.

​
Returns

null | AndroidExtras

​
clearClipboard()

Clears all content from the machine’s clipboard, regardless of type.

​
defaultBrowser()

The machine’s default web browser.

​
Returns

App

​
dir()

A directory on the machine’s filesystem, opened — or created along with its parents, with createMissing — at path. Path resolution as in [Machine.file].

​
Parameters

​
string
required
​
boolean
required

​
Returns

Directory

​
file()

A file on the machine’s filesystem, opened — or created along with its parent directories, with createMissing — at path. Local: an absolute path is used as-is. A relative path is joined to the SimularFiles root (.. is kept — this is a default base, not a sandbox). Android: path must be absolute (there is no working directory on the device to resolve against).

​
Parameters

​
string
required
​
boolean
required

​
Returns

File

​
focusedRoot()

Root accessibility node of the machine’s focused application. Android has no per-application tree; this is a snapshot of everything currently shown on screen, the same as [Machine.systemRoot].

​
Returns

AccessibilityNode

​
focusedWindow()

The window that currently has keyboard focus, if any. Returns null when nothing is focused (desktop: no focused window; Android: no visible task, e.g. the screen is off) rather than treating that as an error.

​
Returns

null | Window

​
foregroundApp()

The application currently in the foreground.

​
Returns

Instance

​
fuzzyApp()

Find an installed app by fuzzy matching query against the machine’s app list (launcher names locally, package names on Android).

​
Parameters

​
string
required

​
Returns

App

​
getClipboardImage()

Gets the image content from the machine’s clipboard. Returns the clipboard image if available, or null if the clipboard doesn’t contain image data.

​
Returns

null | Image

​
getClipboardString()

Gets the string content from the machine’s clipboard. Returns the clipboard string if available, or null if the clipboard doesn’t contain string data or is empty.

​
Returns

null | string

​
key()

Sends an individual key event. It will enter the keysym (virtual key). Have a look at the [Machine.keyRaw] method, if you want to enter a keycode. Some of the keys are specific to a platform.

​
Parameters

​
Key
required
​
Direction
required

​
keyOther()

Sends a key event for a raw, platform-specific key value.

​
Parameters

​
number
required
​
Direction
required

​
keyRaw()

Sends a raw keycode. The keycode may or may not be mapped on the current layout. You have to make sure of that yourself. This can be useful if you want to simulate a press regardless of the layout (WASD on video games). Have a look at the [Machine.key] method, if you just want to enter a specific key and don’t want to worry about the layout/keymap. Windows only: If you want to enter the keycode (scancode) of an extended key, you need to set the high byte for the extended key too. You can for example do: keyRaw(0xE01D, Direction.Click) to simulate RControl.

​
Parameters

​
number
required
​
Direction
required

​
keyUnicode()

Sends an individual Unicode key event. Provide a single character.

​
Parameters

​
string
required
​
Direction
required

​
local()

The local desktop this process runs on. Infallible and zero-cost. Local machine handles share one global input state: every local machine drives the same physical mouse, keyboard, and clipboard.

​
Returns

Machine

​
loopback()

Loopback capture of the machine’s audio output (what its speakers are playing), in the given format. Not supported on Android yet: capturing device audio needs a device-side streamer, so this throws at construction. The signature will not change when support lands.

​
Parameters

​
AudioFormat
required

​
Returns

Loopback

​
mainScreen()

The machine’s main display (the primary monitor). Android: the device screen.

​
Returns

Screen

​
microphone()

The machine’s default microphone, recording in the given format from the moment it is opened. Not supported on Android yet — throws at construction; see [Machine.loopback].

​
Parameters

​
AudioFormat
required

​
Returns

Microphone

​
mouseButton()

Sends an individual mouse button event, e.g. to simulate a click of the left mouse key. Some of the buttons are specific to a platform. On Android there is no real cursor: a left press+release becomes a tap or swipe at the location recorded by [Machine.moveMouse].

​
Parameters

​
Button
required
​
Direction
required

​
mouseLocation()

Get the location of the mouse in the canonical global-desktop space (OS-native units; see [Machine]). On Android this is the emulated cursor’s recorded location.

​
Returns

[number, number]

​
moveMouse()

Move the mouse cursor to the specified x and y coordinates. You can specify absolute coordinates or relative from the current position. With absolute coordinates, (x, y) is in the global desktop space described on [Machine]: top-left origin at (0, 0), OS-native units, and secondary monitors arranged above / to the left of the primary may have negative coordinates. With relative coordinates, a positive x moves the cursor x pixels to the right; a positive y moves it down.

​
Parameters

​
number
required
​
number
required
​
Coordinate
required

​
nodeAtPoint()

The accessibility element at global desktop coordinates (x, y) via the platform hit-test, or null when the point has no accessible element (empty desktop, gaps between controls). Throws on Android (no accessibility hit-test channel) and on genuine backend failures. (x, y) are in the canonical coordinate space (see [Machine]) — the same space .boundingBox() and mouseLocation use.

​
Parameters

​
number
required
​
number
required

​
Returns

null | AccessibilityNode

​
pasteText()

Types text by pasting it from the clipboard. Locally this saves the previous clipboard text and image (when present), sets the clipboard to the specified string, verifies it was set correctly, then simulates Command+V (or Ctrl+V) to paste it; after pasting, the saved snapshot is reapplied. On Android the text is committed through the device’s input channel without touching the host clipboard.

​
Parameters

​
string
required

​
player()

A player queueing sounds on the machine’s default audio output. Not supported on Android yet — throws at construction; see [Machine.loopback].

​
Returns

AudioPlayer

​
screenFromMouse()

The display the mouse cursor is currently on (the main screen when the cursor is on none). Android: the device screen — the emulated pointer is always on it. Capture the returned handle once and reuse it when screen identity matters: resolving per capture can hit a different display whenever the mouse moves between calls.

​
Returns

Screen

​
screens()

Every display connected to the machine. The order is platform-defined; do not rely on it. Android: the device’s single physical screen.

​
Returns

Screen[]

​
screenshotCropped()

Screenshot of a region of the machine’s screen, addressed in the canonical global desktop coordinates (see [Machine]). Local: a native cropped capture. Android: the full screen is captured and cropped to the region, which must lie entirely within the screen (hideCursor is ignored).

​
Parameters

​
number
required
​
number
required
​
number
required
​
number
required
​
boolean
required

​
Returns

Screenshot

​
scroll()

Send a mouse scroll event. A positive length will result in scrolling down/right and negative ones up/left.

​
Parameters

​
number
required
​
number
required

​
setClipboardImage()

Replaces the contents of the machine’s clipboard with the given image. For example used to copy a screenshot to the clipboard. Returns a [ClipboardContent] snapshot of what the clipboard held before this call (see [Machine.setClipboardString]).

​
Parameters

​
Image
required

​
Returns

ClipboardContent

​
setClipboardString()

Replaces the contents of the machine’s clipboard with the given string. Returns a [ClipboardContent] snapshot of what the clipboard held before this call. Nothing is restored automatically; see the snapshot’s docs for how to write it back (and the one-format-per-write caveat).

​
Parameters

​
string
required

​
Returns

ClipboardContent

​
snapshot()

Playwright-style aria snapshot of the machine’s focused application (Android: everything currently shown on screen).

​
Returns

string

​
systemRoot()

Root accessibility node spanning everything currently accessible: on desktops the system-wide root (its children are the running applications), on Android a snapshot of everything currently shown on screen (which can span the foreground app, the system bars, and a split-screen neighbor).

​
Returns

AccessibilityNode

​
tempDir()

A new uniquely named directory in the machine’s temp location. It is not removed automatically; call [Directory.delete] when done.

​
Returns

Directory

​
typeText()

Enter the text. You can use unicode here like: ❤吅. This works regardless of the current keyboard layout. You cannot use this function for entering shortcuts or something similar. For shortcuts, use the [Machine.key] method instead.

​
Parameters

​
string
required

​
windowAtPoint()

The top-level window at global desktop coordinates (x, y), or null when no window sits under the point. (x, y) are in the canonical coordinate space (see [Machine]) — the same space Window.boundingBox and mouseLocation use, so a cursor location can be passed straight in. Platform notes:
  • macOS: returns the containing AXWindow of the element under the cursor — the window itself, not the whole application.
  • Windows: returns the top-level window (GA_ROOT) of whatever control sits under the cursor.
  • Linux: always null (no per-point window hit-test).
  • Android: throws — tasks have no per-point hit-test.

​
Parameters

​
number
required
​
number
required

​
Returns

null | Window

​
windows()

All windows on the machine: visible top-level windows across every process locally, one window per live task (recents entry) on Android.

​
Returns

Window[]