otto monitor — the observation pipeline

The monitor answers “what are the hosts doing while my tests run?” — a pipeline that observes the lab rather than commanding it, careful about two hazards: polluting the logs it observes through, and writing a database onto a filesystem that can’t take it.

Frontend: React, not vanilla JS

The monitor backend was reworked behind a stable MetricCollector facade: the collector decomposes into store/db/broadcast modules — joined by session (a run’s identity and lab snapshot) and export (the format:1 producer) — dashboard metadata is typed TabSpec/ChartSpec models, reshaped into each session’s SessionMeta for the GET /api/monitor_sessions/GET /api/stream wire (spec 2026-07-12 monitor-live-streaming), and a project-level register_parsers() joins the per-host registry. The dashboard itself was ported from a single vanilla-JS file to a React + Vite

  • TypeScript single-page app (web/, built to static/dist/) behind the same observable surface, and later gained a real live producer: both live and review boot through the same /api/monitor_sessions endpoint, and a live tab keeps growing via /api/stream SSE fragments rather than reloading. See otto monitor for the frontend dev workflow (make web-dev) — tests/e2e/monitor/dashboard/ pins the exact ids/classes/behaviors that must survive any further change to either side.

monitor hosts lab hosts (Unix or SNMP-enabled) factory factory MonitorTarget per host shell parsers or SNMP source host.log → NEVER hosts->factory collector collector tick loop concurrent poll, one shared timestamp per tick factory->collector db SQLite session archive (--db) WAL local / DELETE on network FS collector->db dash web dashboard OS-assigned port collector->dash events suite events start_monitor / add_monitor_event events->collector replay otto monitor <source> review (no hosts touched) replay->dash

Collection. build_monitor_collector (otto/monitor/factory.py) turns a host list into MonitorTargets and picks each host’s collection mode:

  • Shell mode — the target pairs the host with a dict of MetricParser objects, each keyed by the command whose output it parses (/proc/stat, /proc/meminfo, …). Per-host parser sets are pluggable via the HOST_PARSERS registry.

  • SNMP mode — hosts with an snmp table in lab data are polled by OID instead; SNMP metric descriptors have their own registry. This is how non-shell targets (and embedded targets that can’t afford console polling) stay observable.

Only Unix hosts and SNMP-enabled hosts are monitored — there is no generic way to poll an arbitrary console target without disturbing it.

The tick loop. MetricCollector polls every target concurrently each tick (one asyncio.gather per tick, one shared timestamp per tick so cross-host data lines up). The factory sets each polled host’s log disposition to NEVER — a monitor sampling ten hosts every two seconds would otherwise bury the transcript; because LogMode gates command I/O only, real errors from those hosts still surface (Logging).

Events. Suites stamp markers onto the same timeline (start_monitor / add_monitor_event from OttoSuite), so “CPU spiked” and “test_load started” correlate.

Serving and persistence. A live dashboard (MonitorServer) binds an OS-assigned port and serves the collector’s buffer to the built React frontend: an initial GET /api/monitor_sessions snapshot (the same format:1 shape review mode loads) plus a live GET /api/stream SSE feed of format:1-shaped fragments that grows an already-open tab in real time. With --db, each run’s samples persist as one session in a SQLite archive — WAL journaling on local disks, DELETE on network filesystems (Data at the boundary); running against the same --db path again appends another session rather than overwriting the archive. The positional otto monitor <source> form instead replays a saved .json export or .db archive without touching any host.

Gating. otto monitor gates itself per branch rather than in the preamble: live collection (--live) runs the reservation gate; reviewing a saved <source> reads a local file and is gate-exempt by design (The command lifecycle).

otto monitor --help

otto monitor --help Usage: otto monitor [OPTIONS] [SOURCE] Launch an interactive performance monitoring dashboard, or review a saved export. Exactly one of ``--live`` or ``<source>`` must be given — never both, and never neither (bare ``otto monitor`` prints usage and exits 2). Output-dir creation moved to the shared leaf-invoke :func:`~otto.cli.invoke.command_preamble` (monitor's spec declares ``output_dir=True``), so a ``--help`` invocation can never create a spurious dir. Neither the reservation gate nor the lab requirement is uniform: monitor's spec declares ``gate=False`` AND ``lab_free=True``, so this body gates and lab-loads only the ``--live`` branch (via :func:`~otto.cli.invoke.ensure_lab_session`) — reviewing a saved ``<source>`` is both gate-exempt and lab-free (it reads a local file and never touches live hardware or a lab). ╭─ Arguments ──────────────────────────────────────────────────────────────────╮ │ [source] PATH Review a saved .json or .db monitor export instead of │ │ collecting live. │ ╰──────────────────────────────────────────────────────────────────────────────╯ ╭─ Options ────────────────────────────────────────────────────────────────────╮ │ --live Collect from lab hosts (explicit │ │ opt-in; never the default). │ │ --hosts REGEX Regex matched against host IDs (via │ │ re.search). Default: all hosts. │ │ --interval -i SECONDS [x>=1.0] Collection interval in seconds. │ │ [default: 5.0] │ │ --db PATH SQLite file to persist live metric │ │ data for later historical viewing. │ │ --label TEXT Human-readable label to store with │ │ this live session. │ │ --note TEXT Free-form note to store with this live │ │ session. │ │ --help -h Show this message and exit. │ ╰──────────────────────────────────────────────────────────────────────────────╯