Every agent framework breaks in the same place: the interface. APIs rate-limit, extensions get detected, drivers need install rights. ACID CAT goes underneath all of it — a small ESP32-S3 board that plugs into USB and enumerates as an ordinary HID keyboard and mouse. No driver, no extension, nothing to integrate. To the target machine it is a person typing, which is the one integration surface no vendor can withdraw.
The device does more on one cable than it looks like it should. It runs as a composite USB device — HID keyboard,
HID mouse and a CDC serial console simultaneously — while also serving a JSON HTTP API on port 8080 over WiFi, running a
six-characteristic BLE GATT service, and driving a 240x280 touchscreen with its own menu system. Six ways in: serial,
HTTP, BLE, the on-device touch UI, a Web Bluetooth page that provisions WiFi from a browser tab with nothing installed,
and a one-line shell wrapper. HID mice only speak relative deltas, so the firmware maintains a virtual cursor and
implements absolute moveto as an interpolated stream of int8-clamped reports. The JSON parsing is hand-rolled
indexOf scanning rather than a library — no allocator, no heap fragmentation on a device meant to stay up for days.
The half nobody expects is that it can see. A pure input device is blind, so we built the eyes out-of-band: the host
takes a screenshot and reconstructs a UI tree from raw pixels — tesseract lines regrouped into elements, then classified
into buttons, inputs, links and tabs by Canny edge density and interior variance, modals found by dual-threshold contour
geometry so it works on light and dark themes, and the mouse cursor located by template-matching a pointer shape
synthesized in code. That last piece closes the loop: the mouse calibrates by slamming into the screen corner, then
verifies every move against a fresh screenshot and issues corrective deltas when the OS's pointer acceleration eats the
difference. Clicks can demand proof — screenshot before, screenshot after, retry with jitter if the screen didn't
change. On top sits a declarative JSON macro runner with wait_for, verify, click-by-label and live TOTP generation;
the reference macro logs into a 2FA-protected site end to end with zero software on the target. And there is a bongo cat
on the front that drums the correct paw for whichever half of the keyboard your current character lives on, because we
built the hardware and could.
// Technical highlights
- Composite USB device: HID keyboard + HID mouse + CDC serial console on one port, plus a WiFi HTTP API, a BLE GATT service and a touchscreen UI — 1,508 lines of firmware on one ESP32-S3.
- Screenshots become structured UI JSON: OCR lines, then buttons/inputs/links classified by Canny edge density and interior variance; modals detected by dual-threshold contour geometry in both light and dark themes.
- Absolute mouse positioning on a relative-only protocol: corner-slam calibration, interpolated int8-clamped HID reports, screenshot verification with a 20-pixel tolerance and automatic corrective deltas.
- Clicks that prove they worked: before/after screen-state diff, then up to 3 retries with ±5px jitter.
- Bongo cat animation ships only the 145x40 rectangle in which the frames actually differ — 11.6 KB per frame instead of 134 KB, a ~23x cut that keeps the paws in sync with typing at 120 ms per swap.
- WiFi setup with nothing installed on either side: Web Bluetooth page → six custom GATT characteristics → RSSI-ranked scan results → credentials written to NVS.
- Declarative JSON macros with
wait_for,verify, OCR click-by-label and live TOTP — the sample macro completes a 2FA browser login using nothing but a physical keyboard and a screenshot loop.