Text recognition

Recognise on-screen text with Apple's Vision framework, entirely on the device — no network and no screenshot upload. Prefer element lookups (clickElement) where they work: the element tree is structured, fast and exact. OCR is for text drawn into images or canvases that the element tree does not contain.

At a glance

·Set the language and speed once with ocrConfigure; the whole task uses them
·Chinese needs iOS 14 or later, Japanese / Korean / Russian / Thai / Vietnamese need iOS 16 or later, and Vision does not support Arabic
·Coordinates come back in points, ready for client.click

ocr#

Recognise the text on screen and return it line by line in reading order.

typescript

ocr(region?: [number, number, number, number] | null): Line[] & { text: string };

type Line = {
  text: string;
  confidence: number;
  x: number; y: number; width: number; height: number;
  centerX: number; centerY: number;
};

Parameters

Name	Type	Required	Default	Description
`region`	`[x, y, w, h] \| null`	No	—	Only read this area; omit it for the whole screen. A smaller area is several times faster. Written as in colour and image matching: 0–1 screen ratios when all four numbers are between 0 and 1, otherwise points.

Returns

Type	Description
`{ text, confidence, x, y, width, height, centerX, centerY }[]`	One item per line, coordinates in points. The array also carries .text: every line joined with newlines.

Throws

Error	Raised when
`WDAError`INVALID_ARGUMENT	An argument is wrong: an empty keyword, a wait that is not a number or exceeds 600, a region that is not [x, y, w, h], an old-style {}, and so on.
`WDAError`OCR_LANG_UNSUPPORTED	This device's Vision framework does not support the configured language.
`WDAError`FEATURE_UNAVAILABLE	The app on the device is too old to support text recognition.

Example

javascript

var lines = ocr([0.05, 0.12, 0.9, 0.25]);
logi(lines.text);
for (var i = 0; i < lines.length; i++) {
  logi(lines[i].text + ' @' + lines[i].centerX + ',' + lines[i].centerY);
}

findText#

Find a piece of text on screen and return the line that contains it.

typescript

findText(keyword: string | RegExp, timeoutSec?: number, region?: [number, number, number, number] | null, exact?: boolean): Line | null;

Parameters

Name	Type	Required	Default	Description
`keyword`	`string \| RegExp`	Yes	—	The text to find (a line containing it counts), or a regular expression. Must not be an empty string.
`timeoutSec`	`number`	No	`0`	The longest wait, in seconds. 0 means look once; above 0 it re-captures and re-reads every 0.5 seconds until found or out of time.
`region`	`[x, y, w, h] \| null`	No	—	Only read this area; omit it for the whole screen. A smaller area is several times faster. Written as in colour and image matching: 0–1 screen ratios when all four numbers are between 0 and 1, otherwise points.
`exact`	`boolean`	No	`false`	Pass true to require the whole line to match, not just contain the keyword.

Returns

Type	Description
`{ text, confidence, x, y, width, height, centerX, centerY } \| null`	The matching line, or null.

Throws

Error	Raised when
`WDAError`INVALID_ARGUMENT	An argument is wrong: an empty keyword, a wait that is not a number or exceeds 600, a region that is not [x, y, w, h], an old-style {}, and so on.
`WDAError`OCR_LANG_UNSUPPORTED	This device's Vision framework does not support the configured language.
`WDAError`FEATURE_UNAVAILABLE	The app on the device is too old to support text recognition.

Example

javascript

var hit = findText('Balance', 5);
if (hit) {
  logi(hit.text);
}
var price = findText(/\$\s*[\d.]+/);

clickText#

Wait for a piece of text to appear and tap it, at a slightly random spot within the line.

typescript

clickText(keyword: string | RegExp, timeoutSec?: number, region?: [number, number, number, number] | null, exact?: boolean): boolean;

Parameters

Name	Type	Required	Default	Description
`keyword`	`string \| RegExp`	Yes	—	The text to find (a line containing it counts), or a regular expression. Must not be an empty string.
`timeoutSec`	`number`	No	`10`	The longest wait, in seconds. It re-captures and re-reads every 0.5 seconds.
`region`	`[x, y, w, h] \| null`	No	—	Only read this area; omit it for the whole screen. A smaller area is several times faster. Written as in colour and image matching: 0–1 screen ratios when all four numbers are between 0 and 1, otherwise points.
`exact`	`boolean`	No	`false`	Pass true to require the whole line to match, not just contain the keyword.

Returns

Type	Description
`boolean`	True when tapped, false when the text never appeared.

Throws

Error	Raised when
`WDAError`INVALID_ARGUMENT	An argument is wrong: an empty keyword, a wait that is not a number or exceeds 600, a region that is not [x, y, w, h], an old-style {}, and so on.
`WDAError`OCR_LANG_UNSUPPORTED	This device's Vision framework does not support the configured language.
`WDAError`FEATURE_UNAVAILABLE	The app on the device is too old to support text recognition.

Notes

By default the task log records whether the text was tapped or not found; turn it off with setClickLog(false).

Example

javascript

clickText('Claim now', 8);

ocrConfigure#

Set this task's recognition languages and speed, validating the languages up front. Optional — the default is Simplified Chinese plus English.

typescript

ocrConfigure(languages?: string[] | string, level?: "accurate" | "fast", correction?: boolean): { ok: true; languages: string[]; level: string };

Parameters

Name	Type	Required	Default	Description
`languages`	`string[] \| string`	No	—	The recognition languages, for example ['zh-Hans', 'en-US'] or ['ja-JP', 'en-US']; a single language can be a plain string.
`level`	`"accurate" \| "fast"`	No	`"accurate"`	accurate is precise, fast is quick.
`correction`	`boolean`	No	`false`	Pass true to let the system correct results against its dictionary.

Returns

Type	Description
`{ ok, languages, level }`	The settings now in effect.

Throws

Error	Raised when
`WDAError`OCR_LANG_UNSUPPORTED	This device's Vision framework does not support the configured language.
`WDAError`INVALID_ARGUMENT	The languages are not an array of strings, level is not accurate / fast, or an old-style {} was passed.
`WDAError`FEATURE_UNAVAILABLE	The app on the device is too old to support text recognition.

Notes

Its value is failing early: without it, a device that cannot read Chinese only shows the problem deep inside a loop.

Example

javascript

ocrConfigure(['zh-Hans', 'en-US'], 'accurate');

ocrCapabilities#

Which languages this device can actually recognise.

typescript

ocrCapabilities(): { engine: string; revision: number; accurate: string[]; fast: string[]; supportsChinese: boolean };

Returns

Type	Description
`{ engine, revision, accurate, fast, supportsChinese }`	accurate / fast list the language codes each mode supports.

Throws

Error	Raised when
`WDAError`FEATURE_UNAVAILABLE	The app on the device is too old to support text recognition.

Example

javascript

logi(JSON.stringify(ocrCapabilities()));