The command lifecycle

Every otto invocation walks the same path before a first-party command takes over: compose the process, dispatch one command, prepare the invocation, run it, tear it down deterministically. This page covers that shared path; the pages below cover what each command does once it has control.

lifecycle entry entry() — console script completion completion fast path cache hit → zero user code entry->completion completion request discovery bootstrap phase 1: discovery OTTO_* env + settings.toml (no user code runs) entry->discovery registration bootstrap phase 2: registration init modules + test files (per-file failures contained) discovery->registration dispatch dispatch resolve only the target command; every other command stays a help stub registration->dispatch preamble invoke preamble load + merge labs → OttoContext → output dir + log sinks → reservation gate (lab_free commands skip lab and gate) dispatch->preamble body command body (command-specific — pages below) preamble->body teardown teardown HostScope closes remaining hosts; exit code derived from the Result body->teardown

The front door looks like this — and every terminal block in these pages is captured from the real CLI at build time (a scaffolded demo repo, real --help output, real completion candidates), so what you see here is what the current code does:

otto --help Usage: otto [OPTIONS] COMMAND [ARGS]... O.T.T.O. (Our Trusty Testing Orchestrator) If a development repo is under test, then OTTO_SUT_DIRS must be set in your environment. It is a list of paths to repo root directories, separated by ``,`` or the OS path separator (``:`` on Linux/macOS, ``;`` on Windows). ╭─ Options ────────────────────────────────────────────────────────────────────╮ │ --lab -l COMMA SEPARATED Name of lab(s) │ │ LIST to reserve and │ │ use. │ │ [env var: │ │ OTTO_LAB] │ │ --xdir -x PATH Directory in │ │ which to store │ │ logs and │ │ artifacts. │ │ [env var: │ │ OTTO_XDIR] │ │ [default: .] │ │ --field --debug Use field or │ │ debug products. │ │ [env var: │ │ OTTO_FIELD_PROD… │ │ [default: debug] │ │ --log-days INTEGER RANGE Number of days │ │ [x>=0] to retain logs. │ │ [env var: │ │ OTTO_LOG_DAYS] │ │ [default: 30] │ │ --log-level LOG LEVEL Level at which │ │ to log. │ │ [env var: │ │ OTTO_LOG_LEVEL] │ │ [default: INFO] │ │ --rich-log-file --no-rich-log-… Determines │ │ whether log │ │ files have rich │ │ formatting. │ │ [env var: │ │ OTTO_LOG_RICH] │ │ [default: │ │ no-rich-log-fil… │ │ --show-time Show per-line │ │ timestamps on │ │ the live console │ │ (log files are │ │ always │ │ timestamped). │ │ --lab-depth INTEGER RANGE Depth for │ │ [x>=0] --show-lab │ │ output (0 = │ │ unlimited). │ │ [default: 3] │ │ --list-labs List all │ │ available lab │ │ names. │ │ --show-lab Show specified │ │ lab details. │ │ --list-hosts Show all valid │ │ host IDs. │ │ --dry-run -n Preview what │ │ would be │ │ executed without │ │ running commands │ │ on hosts. │ │ --version Show program │ │ version and │ │ exit. │ │ --clear-autocomp… Delete the │ │ shell-completion │ │ cache file and │ │ exit. │ │ --as-user USERNAME Check │ │ reservations as │ │ USERNAME instead │ │ of the current │ │ user. Use when a │ │ teammate has the │ │ shared lab │ │ booked under │ │ their name. │ │ --skip-reservati… -R Bypass the │ │ reservation │ │ check entirely. │ │ Intended only │ │ for emergencies │ │ when the │ │ scheduler is │ │ wrong or │ │ unreachable. │ │ --install-comple… Install │ │ completion for │ │ the current │ │ shell. │ │ --show-completion Show completion │ │ for the current │ │ shell, to copy │ │ it or customize │ │ the │ │ installation. │ │ --help -h Show this │ │ message and │ │ exit. │ ╰──────────────────────────────────────────────────────────────────────────────╯ ╭─ Commands ───────────────────────────────────────────────────────────────────╮ │ run Run a registered instruction on the lab. │ │ test Run a registered OttoSuite test suite. │ │ monitor Launch an interactive performance dashboard. │ │ cov Generate coverage reports from otto test --cov output. │ │ host Run commands and transfer files on lab hosts. │ │ docker Build images and orchestrate compose stacks on docker-capable │ │ lab hosts. │ │ reservation Inspect and verify lab reservations. │ │ schema Export JSON Schema for hosts.json / settings.toml / │ │ reservations. │ │ init Scaffold a new otto repo or validate an existing one. │ ╰──────────────────────────────────────────────────────────────────────────────╯

Bootstrap: two phases, contained failures

otto.bootstrap.bootstrap() replaces what used to be import-time side effects with an explicit composition root:

  • Phase 1 — discovery. Parse the OTTO_* environment variables and every repo’s .otto/settings.toml into an OttoEnvSettings plus a list of Repo objects. No user code runs. Environment-level failures raise — nothing can degrade gracefully if OTTO_SUT_DIRS itself is broken — but a single repo’s malformed settings file is framed and skipped.

  • Phase 2 — registration. Add each repo’s libs to sys.path, import its init modules, and import its test files. Every user-module exec is wrapped: one broken file becomes a framed BootstrapError in the returned BootstrapResult instead of a traceback that bricks the process. The CLI prints one warning line per contained error; actually dispatching into broken code fails loud.

bootstrap() is idempotent: the CLI entry point calls it before argv parsing, open_context() calls it lazily for library users, and repeated calls return the same result.

Lab loading is deliberately not part of bootstrap. otto --help, --list-* flags, and shell completion never open hosts.json, and a missing or malformed lab file only matters once a command that needs the lab runs.

The preamble, and who opts out

For CLI commands, the invoke preamble (otto/cli/invoke.py) runs just before the leaf callback: load and merge labs (--lab may repeat), build and install the OttoContext, create the per-command output directory and wire the log sinks (Logging), and run the reservation gate. Each first-party command declares what it needs on its CommandSpec (Registries and the pluggable CLI):

Command

Needs a lab

Output dir

Reservation gate

run

yes

yes

yes

test

yes

yes

yes

host

yes

yes

yes

monitor

yes

yes

self-gated per branch: live collection gates, --file replay doesn’t

docker

yes

yes

no — containers ride the parent’s reservation

cov

yes

no — reads existing run dirs

no

reservation

no (lab_free) — check loads lab data itself

no

no — it is the gate, made inspectable

schema

no (lab_free)

no

no

init

no (lab_free)

no

no

--lab itself tab-completes — the lab names are tags on hosts in the hosts.json files, read data-only (no host construction, no user code), and the option is comma-separated so each segment completes in turn:

otto --lab <TAB><TAB> example_lab $ otto --lab

OttoContext: the per-invocation runtime

OttoContext is a plain dataclass holding exactly what one invocation needs: the active lab, the dry_run and log_command_output flags, the invocation’s output_dir, and a HostScope. Its methods are the canonical host accessors:

  • get_host() — look up one host by id, apply per-call option overrides, register it with the scope.

  • all_hosts() — iterate the fleet. The built-in local host and Docker container hosts are excluded unless opted in (include_local=True / include_containers=True): deploy, monitor, and coverage sweeps must never silently operate on the runner itself.

  • do_for_all_hosts() / run_on_all_hosts() — fan a call out across the fleet (concurrently by default) with per-host error isolation: the returned dict maps host id to either the result or the exception that host raised, so one dead host cannot abort the sweep.

The context is installed in a ContextVar via set_context() and read back with get_context() (raising) or try_get_context() (returning None). The context-variable is plumbing, not a global: explicit ctx passing is first-class, and the zero-argument convenience accessors (all_hosts(), get_host, run_on_all_hosts, …) simply delegate to the active context’s method of the same name. Anything that wants its dependency visible takes a ctx parameter — CLI commands can declare ctx: OttoContext and have it injected.

HostScope: deterministic teardown, no __del__

Hosts hold real resources — SSH connections, telnet consoles, docker exec channels. otto deliberately has no __del__-based cleanup: garbage collection is non-deterministic, and relying on it caused resource churn. Instead every host handed out by a context is registered (deduplicated by identity) with the context’s HostScope, and the scope closes anything still connected when the invocation ends.

That yields three equally valid usage modes, mirroring file descriptors:

async with ctx.get_host("router1") as h:   # 1. tight, early scoping
    await h.run("uptime")

h = ctx.get_host("router1")                # 2. no ceremony — the scope
await h.run("uptime")                      #    closes it at command end

await h.close()                            # 3. explicit manual control

close() is idempotent, so an early per-host close and the end-of-scope sweep never collide.

Library use: open_context()

Scripts and notebooks get the same lifecycle without the CLI:

import otto

async with otto.open_context(lab="my_lab") as ctx:
    result = await ctx.run_on_all_hosts("uname -a")

open_context() runs bootstrap() (lazily, idempotently), loads and merges the requested lab(s), installs the context, and tears everything down — scope included — on exit. It does not run the reservation gate; that is a CLI-preamble concern, and scripts that want it call check_reservations explicitly. See Using otto as a library for the user-facing walkthrough.

First Party Commands