otto monitor — the observation pipeline¶
The monitor answers “what are the hosts doing while my tests run?” — a pipeline that observes the lab rather than commanding it, careful about two hazards: polluting the logs it observes through, and writing a database onto a filesystem that can’t take it.
Frontend: React, not vanilla JS
The monitor backend was reworked behind a stable
MetricCollector facade: the collector
decomposes into store/db/broadcast/history modules, dashboard
metadata is typed TabSpec/ChartSpec models served at /api/meta, and a
project-level register_parsers() joins the per-host registry. The
dashboard itself was ported from a single vanilla-JS file to a React + Vite
TypeScript single-page app (
web/, built tostatic/dist/) behind the same observable surface. See otto monitor for the frontend dev workflow (make web-dev) —tests/e2e/monitor/dashboard/pins the exact ids/classes/behaviors that must survive any further change to either side.
Collection. build_monitor_collector (otto/monitor/factory.py) turns a
host list into MonitorTargets and picks each host’s collection mode:
Shell mode — the target pairs the host with a dict of
MetricParserobjects, each keyed by the command whose output it parses (/proc/stat,/proc/meminfo, …). Per-host parser sets are pluggable via theHOST_PARSERSregistry.SNMP mode — hosts with an
snmptable in lab data are polled by OID instead; SNMP metric descriptors have their own registry. This is how non-shell targets (and embedded targets that can’t afford console polling) stay observable.
Only Unix hosts and SNMP-enabled hosts are monitored — there is no generic way to poll an arbitrary console target without disturbing it.
The tick loop. MetricCollector polls
every target concurrently each tick (one asyncio.gather per tick, one
shared timestamp per tick so cross-host data lines up). The factory sets each
polled host’s log disposition to NEVER — a monitor sampling ten hosts every
two seconds would otherwise bury the transcript; because LogMode gates
command I/O only, real errors from those hosts still surface
(Logging).
Events. Suites stamp markers onto the same timeline
(start_monitor / add_monitor_event from OttoSuite),
so “CPU spiked” and “test_load started” correlate.
Serving and persistence. A live dashboard
(MonitorServer) binds an OS-assigned port and
serves the collector’s buffer to the built React frontend. With --db,
samples persist to SQLite — WAL journaling on local disks, DELETE on network
filesystems (Data at the boundary) — and --file replays a
saved run without touching any host.
Gating. otto monitor gates itself per branch rather than in the
preamble: live collection runs the reservation gate; --file replay reads a
local file and is gate-exempt by design (The command lifecycle).