otto test — the test pipeline¶

otto test is a thin, deliberate bridge: suites are ordinary classes, tests are ordinary async methods, and the runner underneath is stock pytest with an otto plugin — not a bespoke test framework.

From class to subcommand¶

A suite extends OttoSuite with a Test-prefixed class name (matching pytest’s own python_classes = Test* collection rule), which triggers __init_subclass__ to call register_suite_class(). Registration does three things at import time (which, for repo test files, means during bootstrap phase 2 — The command lifecycle):

1. Reads the suite’s Options class — any dataclass works; an @options pydantic dataclass adds validation (Options classes) — and synthesizes a Typer subcommand whose flags mirror its fields: the same options-to-parameters machinery instructions use, so suites and instructions share option classes.

2. Registers the suite in the SUITES registry under its class name.

3. Makes re-registration idempotent per source file: pytest re-importing the same file is expected and harmless, while a second suite of the same name from a different file is a loud collision.

otto test <SuiteName> therefore gets registry-backed completion and --list-suites for free, like every other registry (Registries and the pluggable CLI):

Handing off to pytest¶

The suite’s synthesized subcommand builds the options instance and calls run_suite (otto/cli/test.py), which invokes pytest.main() scoped to the suite’s source file, with otto’s plugin installed. Conftest loading is cut at the owning repo’s root (--confcutdir), so the user repo’s full conftest hierarchy applies while otto’s own never leaks in. pytest keeps what it is good at — collection, fixtures, parametrize, markers, reporting — and the plugin (OttoPlugin) layers on otto’s concerns:

• Artifacts — each test gets its own directory under the invocation’s output dir (Logging), exposed to the suite as testDir.

• Stability modes — --iterations N / --duration re-run tests via the runtest protocol and aggregate per-test pass rates, reporting Unstable rather than failing on the first flake.

• Retry — @pytest.mark.retry(n) re-runs a failing test in place.

• Monitoring and coverage — test start/end events are stamped onto the monitor timeline, and coverage runs fetch embedded counters after the session (otto monitor — the observation pipeline, otto cov — the coverage pipeline).

Selection runs: pytest-native, no suite required¶

--tests NAME[,NAME…] and -m/--markers EXPR also work without naming a suite. The selection path resolves test names to exact pytest nodeids (unknown names fail with a did-you-mean), then runs one pytest session per matching repo — a repo with no match is skipped rather than reported as “collected 0 items”, and per-repo --results files fan out automatically. Combining --tests with a suite subcommand is a loud usage error, not a silent intersection. Suite Options are default-constructed per class in selection runs; a suite whose options are required points you back at the single-suite form.

This is the deliberate second door into the same pipeline: plain pytest functions (no OttoSuite at all) are first-class here, which is what the init scaffold demonstrates.

--tests tab-completes test names — bare functions and suite methods alike, matched by base name (a bare test_x runs every test_x[...] parametrization):

Two layers feed it. The always-available floor is a static ast scan of def test_* / Test* methods — instant, never runs your test code. On top of it sits a pytest-collected set that also includes dynamically generated tests (pytest_generate_tests, conftest fixtures) that a source scan can’t see. That set is warmed by any real collection: an otto test --list-tests run fills it for free, and otherwise the first --tests TAB spawns a single bounded collection in the background (a one-time slower TAB — capped, and it falls back to the floor if it can’t finish in time) and caches the result, so every later TAB is fast and complete. A test-file edit invalidates the cache via the same fingerprint the rest of the cache uses, so the collected set never goes stale silently. The completer itself still never runs user code — the collection happens in a disposable subprocess, never in the process answering the keystroke.

Non-fatal assertions¶

self.expect(...) records a failed expectation — with the captured source line and locals — and keeps the test running; the accumulated failures raise one combined AssertionError at the end (ExpectCollector). This exists because hardware tests are expensive to reach: when a board takes minutes to provision, “check everything, then fail with the full list” beats fail-fast.

Suites vs instructions¶

Both are registered callables with option classes; the split is intent. Instructions (instruction()) are procedures — deploy, flash, collect — with one body and an exit code from their returned Result. Suites are verdicts: many independent test methods, pytest semantics, stability statistics, per-test artifacts. Shared repo-wide options classes keep the two consistent (Options classes).

otto test --help¶