Vue (Sight): seeing the screen

Vue is the screen perception organ. It captures, converts coordinates, locates visible text and remembers the topology of your displays. All its tools are read-only and are not subject to the Mains (Hands) ceiling. Vue does not read the screen for you: it returns images and positions, and it is up to the model to interpret them.

Tool	Role	Level
vue_displays	List the active displays	read
vue_snapshot	Capture a display (PNG image and coordinate contract)	read
vue_resolve	Convert between image pixel and global point, without capturing	read
vue_find	Locate a visible label by OCR	read
vue_calibrate	Save the display topology under a name	read
vue_profile	Read the profile matched to the current environment	read
vue_profile_brief	Bootstrap brief for a sub-agent	read
vue_pilot_brief	Per-task delegation package	read

Common errors

A Vue error never crashes the server: it comes back in the form {"error": "<code>", "message": "<detail>"}.

Code	Meaning
permission_denied	A system permission is missing (Screen Recording)
invalid_arguments	An argument is invalid
index_out_of_range	The display index does not exist
capture_failed	The capture failed
exclusions_unreadable	The exclusions file is broken; the message names the file and the line, never the faulty value

If in doubt about a permission, call daimon_doctor first.

vue_displays

Lists all active displays. This is the starting point: the index returned is the value to pass as display to the other tools.

Parameters: none.

Returns: a list of objects {index, width, height, is_main, origin: {x, y}, dpi}.

Example call: vue_displays with no arguments.

Pitfalls: the origin of a secondary display can be negative in the global desktop.

vue_snapshot

Captures a display and returns it as an image, after censoring and redaction.

Parameter	Type	Default	Required	Role
display	integer	0	no	Display index, as listed by vue_displays
max_width	integer	720	no	Downscaling cap for the served image, in image pixels; 0 disables it
region	object {x, y, width, height}	whole display	no	Sub-region to capture, in points, local to the display, clamped to the display

Level: read.

Example call

{ "display": 0, "max_width": 1440, "region": { "x": 0, "y": 0, "width": 800, "height": 600 } }

Returns: two content blocks, in this order: (1) a text block containing the coord_space JSON contract, (2) a base64 PNG image block (image/png). All of coord_space is expressed in points, the unit in which main_click acts; image_scale is the number of image pixels served per point. coord_space.region reports what was actually captured, and coord_space.secret_rects_available says whether masking by secret field could run.

Censoring. The windows of the applications you protect (apps, secret_apps of the exclusions) are never refused: they are censored, replaced by an opaque flat tint #3A2A5C, like the declared zones and the secret fields; the rest of the screen is served. When a rule exists, the text block also carries:

Field	Values	Meaning
censored	boolean	A censoring was applied
censored_reason	none, zones, unstable, unknown	Reason
censored_rects	list of {x, y, width, height}	Censored zones, in global points, clipped to the capture

Errors: see Common errors.

Pitfalls

• A #3A2A5C zone is masked content, not an empty screen.
• With unstable or unknown, the protected windows could not be placed, even after an automatic new capture: the whole image is the flat tint. Try again.
• On macOS, the menu bar is also censored as long as a protected application is in the foreground.
• The default value of max_width is 720: ask for more for fine detail.
• On Windows, a point is a physical pixel of the virtual screen and backing_scale is always 1.0.
• Daimon does not interpret the image: look at it yourself.

vue_resolve

Converts an image pixel (seen in a vue_snapshot of display) to a global point, and vice versa, without capturing. Use it rather than computing offset and scale yourself: a resolved pair is directly clickable.

Parameter	Type	Default	Required	Role
display	integer	0	no	Display index
max_width	integer	720	no	The same cap as the original snapshot; 0 disables it
region	object {x, y, width, height}	whole display	no	The same region as the snapshot, in points, local to the display
image_x, image_y	number	none	no	Image pixel to convert (both together)
global_x, global_y	number	none	no	Global point to convert to an image pixel (both together)
source	string	"probe"	no	"probe" reads the live display geometry; "profile" reads the matched calibration profile, falling back to probe if none matches

Level: read.

Example call

{ "display": 0, "max_width": 720, "image_x": 360, "image_y": 200 }

Returns: {global_x, global_y} if you pass image_x/image_y, {image_x, image_y} if you pass global_x/global_y.

Errors: see Common errors.

Pitfalls: max_width and region must be identical to those of the snapshot, otherwise the conversion is wrong.

vue_find

Vue's fallback: locates a visible label by on-device OCR and returns clickable global coordinates, in points. Useful when touche_tree and touche_probe stay silent (old Win32 applications, custom-drawn ones, Electron) and there is therefore no accessibility tree to click.

Parameter	Type	Default	Required	Role
text	string	none	yes	Text to locate on the screen
display	integer	0	no	Display index
max_width	integer	720	no	Downscaling cap for the internal capture; 0 disables it
region	object {x, y, width, height}	whole display	no	Search sub-region, in points, local to the display
min_score	number	0.6	no	Minimum score, in [0, 1], to count as a hit
source	string	"probe"	no	"probe" or "profile", as for vue_resolve

Level: read.

Example call

{ "text": "Enregistrer", "display": 0, "min_score": 0.6 }

Returns: {found, text, score, image_x, image_y, global_x, global_y, candidates}. On failure, candidates contains the 5 closest visible labels. Pass global_x and global_y as is to main_click (default global space).

Errors: see Common errors. On Windows, a missing OCR language is reported as an error, never as an empty failure.

Pitfalls

• The engine is the system's: Apple Vision on a macOS build compiled with --features apple-vision, Windows.Media.Ocr on Windows. Any other build embeds the dummy FakeOcr engine: no real recognition, always a failure.
• The OCR reads the censored image: no word touching a censored zone is returned. With an exclusion rule, the response adds censored and censored_reason.
• It is a locator, not an interpreter: it returns a position.

vue_calibrate

Freezes the current display topology under a name (idempotent write: same name, update) and returns the saved profile. A profile lets vue_resolve and vue_find use source="profile".

Parameter	Type	Default	Required	Role
name	string	none	yes	Profile name

Level: read (the tool saves a calibration profile, it does not act on the screen).

Example call: { "name": "bureau-deux-ecrans" }

Returns: the saved profile.

vue_profile

Indicates the calibration profile matched to the current display topology, or reports an unknown environment along with the known profile names.

Parameters: none. Level: read.

vue_profile_brief

Produces the bootstrap brief of a delegated sub-agent: profile matching gate and addressable display indices.

Parameter	Type	Default	Required	Role
expected	string	none	no	Profile name to check

Level: read. Example call: { "expected": "bureau-deux-ecrans" }

vue_pilot_brief

Produces the delegation package of a task: a go/no-go gate and a ready-to-paste prompt for a sub-agent, for the given objective. No AI call is made.

Parameter	Type	Default	Required	Role
objective	string	none	yes	Objective of the delegated task
expected	string	none	no	Expected profile name

Level: read.

Example call

{ "objective": "Relever le numéro de version affiché dans la fenêtre À propos", "expected": "bureau-deux-ecrans" }

Pitfalls: if the gate is not ready, calibrate first (vue_calibrate) rather than driving blind.