Text recognition

Recognise on-screen text with Apple's Vision framework, entirely on the device — no network and no screenshot upload. Prefer element lookups (clickElement) where they work: the element tree is structured, fast and exact. OCR is for text drawn into images or canvases that the element tree does not contain.

نظرة عامة

·Set the language and speed once with ocrConfigure; the whole task uses them
·Chinese needs iOS 14 or later, Japanese / Korean / Russian / Thai / Vietnamese need iOS 16 or later, and Vision does not support Arabic
·Coordinates come back in points, ready for client.click

ocr#

Recognise the text on screen and return it line by line in reading order.

typescript

ocr(region?: [number, number, number, number] | null): Line[] & { text: string };

type Line = {
  text: string;
  confidence: number;
  x: number; y: number; width: number; height: number;
  centerX: number; centerY: number;
};

المعاملات

الاسم	النوع	مطلوب	القيمة الافتراضية	الوصف
`region`	`[x, y, w, h] \| null`	لا	—	Only read this area; omit it for the whole screen. A smaller area is several times faster. Written as in colour and image matching: 0–1 screen ratios when all four numbers are between 0 and 1, otherwise points.

القيمة المُعادة

النوع	الوصف
`{ text, confidence, x, y, width, height, centerX, centerY }[]`	One item per line, coordinates in points. The array also carries .text: every line joined with newlines.

الاستثناءات

الاستثناء	متى يحدث
`WDAError`INVALID_ARGUMENT	An argument is wrong: an empty keyword, a wait that is not a number or exceeds 600, a region that is not [x, y, w, h], an old-style {}, and so on.
`WDAError`OCR_LANG_UNSUPPORTED	This device's Vision framework does not support the configured language.
`WDAError`FEATURE_UNAVAILABLE	The app on the device is too old to support text recognition.

مثال

javascript

var lines = ocr([0.05, 0.12, 0.9, 0.25]);
logi(lines.text);
for (var i = 0; i < lines.length; i++) {
  logi(lines[i].text + ' @' + lines[i].centerX + ',' + lines[i].centerY);
}

انظر أيضًا findText、ocrConfigure

findText#

Find a piece of text on screen and return the line that contains it.

typescript

findText(keyword: string | RegExp, timeoutSec?: number, region?: [number, number, number, number] | null, exact?: boolean): Line | null;

المعاملات

الاسم	النوع	مطلوب	القيمة الافتراضية	الوصف
`keyword`	`string \| RegExp`	نعم	—	The text to find (a line containing it counts), or a regular expression. Must not be an empty string.
`timeoutSec`	`number`	لا	`0`	The longest wait, in seconds. 0 means look once; above 0 it re-captures and re-reads every 0.5 seconds until found or out of time.
`region`	`[x, y, w, h] \| null`	لا	—	Only read this area; omit it for the whole screen. A smaller area is several times faster. Written as in colour and image matching: 0–1 screen ratios when all four numbers are between 0 and 1, otherwise points.
`exact`	`boolean`	لا	`false`	Pass true to require the whole line to match, not just contain the keyword.

القيمة المُعادة

النوع	الوصف
`{ text, confidence, x, y, width, height, centerX, centerY } \| null`	The matching line, or null.

الاستثناءات

الاستثناء	متى يحدث
`WDAError`INVALID_ARGUMENT	An argument is wrong: an empty keyword, a wait that is not a number or exceeds 600, a region that is not [x, y, w, h], an old-style {}, and so on.
`WDAError`OCR_LANG_UNSUPPORTED	This device's Vision framework does not support the configured language.
`WDAError`FEATURE_UNAVAILABLE	The app on the device is too old to support text recognition.

مثال

javascript

var hit = findText('Balance', 5);
if (hit) {
  logi(hit.text);
}
var price = findText(/\$\s*[\d.]+/);

انظر أيضًا clickText、ocr

clickText#

Wait for a piece of text to appear and tap it, at a slightly random spot within the line.

typescript

clickText(keyword: string | RegExp, timeoutSec?: number, region?: [number, number, number, number] | null, exact?: boolean): boolean;

المعاملات

الاسم	النوع	مطلوب	القيمة الافتراضية	الوصف
`keyword`	`string \| RegExp`	نعم	—	The text to find (a line containing it counts), or a regular expression. Must not be an empty string.
`timeoutSec`	`number`	لا	`10`	The longest wait, in seconds. It re-captures and re-reads every 0.5 seconds.
`region`	`[x, y, w, h] \| null`	لا	—	Only read this area; omit it for the whole screen. A smaller area is several times faster. Written as in colour and image matching: 0–1 screen ratios when all four numbers are between 0 and 1, otherwise points.
`exact`	`boolean`	لا	`false`	Pass true to require the whole line to match, not just contain the keyword.

القيمة المُعادة

النوع	الوصف
`boolean`	True when tapped, false when the text never appeared.

الاستثناءات

الاستثناء	متى يحدث
`WDAError`INVALID_ARGUMENT	An argument is wrong: an empty keyword, a wait that is not a number or exceeds 600, a region that is not [x, y, w, h], an old-style {}, and so on.
`WDAError`OCR_LANG_UNSUPPORTED	This device's Vision framework does not support the configured language.
`WDAError`FEATURE_UNAVAILABLE	The app on the device is too old to support text recognition.

ملاحظات

By default the task log records whether the text was tapped or not found; turn it off with setClickLog(false).

مثال

javascript

clickText('Claim now', 8);

انظر أيضًا findText、clickElement

ocrConfigure#

Set this task's recognition languages and speed, validating the languages up front. Optional — the default is Simplified Chinese plus English.

typescript

ocrConfigure(languages?: string[] | string, level?: "accurate" | "fast", correction?: boolean): { ok: true; languages: string[]; level: string };

المعاملات

الاسم	النوع	مطلوب	القيمة الافتراضية	الوصف
`languages`	`string[] \| string`	لا	—	The recognition languages, for example ['zh-Hans', 'en-US'] or ['ja-JP', 'en-US']; a single language can be a plain string.
`level`	`"accurate" \| "fast"`	لا	`"accurate"`	accurate is precise, fast is quick.
`correction`	`boolean`	لا	`false`	Pass true to let the system correct results against its dictionary.

القيمة المُعادة

النوع	الوصف
`{ ok, languages, level }`	The settings now in effect.

الاستثناءات

الاستثناء	متى يحدث
`WDAError`OCR_LANG_UNSUPPORTED	This device's Vision framework does not support the configured language.
`WDAError`INVALID_ARGUMENT	The languages are not an array of strings, level is not accurate / fast, or an old-style {} was passed.
`WDAError`FEATURE_UNAVAILABLE	The app on the device is too old to support text recognition.

ملاحظات

Its value is failing early: without it, a device that cannot read Chinese only shows the problem deep inside a loop.

مثال

javascript

ocrConfigure(['zh-Hans', 'en-US'], 'accurate');

انظر أيضًا ocrCapabilities、ocr

ocrCapabilities#

Which languages this device can actually recognise.

typescript

ocrCapabilities(): { engine: string; revision: number; accurate: string[]; fast: string[]; supportsChinese: boolean };

القيمة المُعادة

النوع	الوصف
`{ engine, revision, accurate, fast, supportsChinese }`	accurate / fast list the language codes each mode supports.

الاستثناءات

الاستثناء	متى يحدث
`WDAError`FEATURE_UNAVAILABLE	The app on the device is too old to support text recognition.

مثال

javascript

logi(JSON.stringify(ocrCapabilities()));

انظر أيضًا ocrConfigure