cli.test¶

Run a registered OttoSuite test suite, or a suite-less test selection.

Each Test*-named OttoSuite subclass auto-registers as a subcommand of otto test. Suite-specific options (declared in the suite’s inner Options dataclass) are automatically registered as Typer parameters with full type enforcement and --help documentation.

When no suite subcommand is given, --tests and/or --markers run a suite-less selection instead: exact test names (optionally Class::name qualified) and/or a marker expression are resolved against every repo’s collected tests, and pytest runs once per repo whose selection matched. Plain pytest functions (not just OttoSuite classes) are runnable this way.

Markers

integration: Requires live Vagrant VMs. Skip with --markers "not integration".
timeout(seconds): Fail the test if it runs longer than seconds.
retry(n): Retry a failing test up to n times before reporting failure.

Listing tests

--list-suites List test suites with run syntax and exit.

--list-tests List the selected tests (optionally narrowed by a suite name: and/or --markers) and exit.

--list-markers List the markers available to --markers and exit.

Options on ``otto test`` (before the suite name)

--markers / -m EXPRESSION: pytest -m marker expression applied after collection. With no suite subcommand, runs the marker selection across every repo (one pytest session per repo) instead of requiring a suite name.
--tests NAME[,NAME...]: Run specific tests by exact name — no suite subcommand needed. Comma-separated; bare names match every collected test with that function name (all parametrizations, across suites and repos); TestClass::name disambiguates. Combine with --markers to narrow. Unknown names raise a loud error with did-you-mean suggestions.
--iterations / -i N: Repeat each test N times within a single setup/teardown cycle (0 = disabled).
--duration / -d SECONDS: Repeat tests for N seconds within a single setup/teardown cycle (0 = disabled).
--threshold FLOAT: Minimum per-test pass rate percentage required in stability mode (0-100, default: 100).
--results PATH: Write test results (JUnit XML) to PATH (default: auto-written to the log directory).

When both --iterations and --duration are specified, testing stops when either limit is reached first.

--cov: Fetch .gcda files from remote hosts after the suite finishes and place them in a cov/ directory in the suite’s output directory.
--cov-dir PATH: Write coverage data to PATH instead of the default <output_dir>/cov. Implies --cov. The directory is created if missing; if it already exists and is non-empty, the command aborts unless --overwrite-cov-dir is also given.
--overwrite-cov-dir: Clear the contents of the --cov-dir destination before the run so stale data from a previous invocation cannot be mixed with the new results.
--cov-clean / --no-cov-clean: Delete .gcda files on remote hosts before the test run. Enabled by default; use --no-cov-clean to keep stale data.
--cov-report / -r: After coverage collection, render an HTML report. Implies --cov. Default location: <output_dir>/cov_report.
--cov-report-dir PATH: Write the HTML report to PATH instead of the default. Implies --cov-report (and therefore --cov). Empty/overwrite rules match --cov-dir: created if missing, aborts if non-empty unless --overwrite-cov-report-dir is also given.
--overwrite-cov-report-dir: Clear the contents of --cov-report-dir before the report is rendered.
--project-name STR: Title shown in the HTML report header (only used with --cov-report).
--monitor: Enable host performance monitoring for the duration of the run. Samples every host (or those matched by --monitor-hosts) on a fixed interval and emits per-test start/end events automatically. At the end of the run a JSON snapshot of all metrics and events is written to <output_dir>/monitor.json.
--monitor-interval SECONDS: Sampling interval for --monitor (default: 5).
--monitor-output PATH: Override the destination for the captured monitor data. Format inferred from the suffix: .json (default) writes a self-contained snapshot, .db writes a SQLite database loadable via otto monitor --file.
--monitor-hosts REGEX: Restrict --monitor to host IDs matching this regex (re.search).

Examples:

otto test --list-suites
otto test --list-markers
otto test --list-tests
otto test --list-tests --markers slow
otto test --list-tests TestMyDevice
otto test --tests test_login
otto test --tests TestB::test_login,test_plain
otto test -m slow
otto test TestMyDevice --help
otto test TestMyDevice --device-type switch --firmware 2.1
otto test --iterations 50 --threshold 95 TestMyDevice
otto test --duration 300 --threshold 90 TestMyDevice
otto test --iterations 100 --duration 60 TestMyDevice
otto test --cov TestMyDevice
otto test --cov-dir /tmp/myrun TestMyDevice
otto test --cov-dir /tmp/myrun --overwrite-cov-dir TestMyDevice
otto test --cov --no-cov-clean TestMyDevice
otto test --cov --cov-report TestMyDevice
otto test -r --cov-report-dir /tmp/myreport TestMyDevice

class otto.cli.test.TestRunOptions(markers: str = '', tests: str = '', iterations: int = 0, duration: int = 0, threshold: float = 100.0, results: str = '', cov: bool = False, cov_dir: Path | None = None, cov_clean: bool = True, cov_report: bool = False, cov_report_dir: Path | None = None, overwrite_cov_report_dir: bool = False, project_name: str = 'Coverage Report', monitor: bool = False, monitor_interval: float = 5.0, monitor_output: Path | None = None, monitor_hosts: str | None = None)¶

Bases: object

Shared otto test run options, set by the suite_app callback and read by run_suite.

Stored in Typer ctx.meta (shared across the whole context chain by click’s design) rather than ctx.obj (whose parent->subcommand propagation broke under click 8.3).

markers : str = ''¶

tests : str = ''¶

iterations : int = 0¶

duration : int = 0¶

threshold : float = 100.0¶

results : str = ''¶

cov : bool = False¶

cov_dir : Path | None = None¶

cov_clean : bool = True¶

cov_report : bool = False¶

cov_report_dir : Path | None = None¶

overwrite_cov_report_dir : bool = False¶

project_name : str = 'Coverage Report'¶

monitor : bool = False¶

monitor_interval : float = 5.0¶

monitor_output : Path | None = None¶

monitor_hosts : str | None = None¶

otto.cli.test.resolve_suite(suite: str, repos: list[Repo]) → str¶: Expand a sut_dir-relative suite path to an absolute path for pytest.

otto.cli.test.run_suite(suite_class: type, suite_file: str, opts_instance: object | None, ctx: Context) → None¶

Execute a registered suite via pytest.main().

Runner options (--markers, --iterations, --duration, --threshold, --results) and coverage options (--cov / --cov-clean) are read from the TestRunOptions the otto test callback stored in ctx.meta[RUN_OPTIONS_KEY]. The context is passed in by the suite runner (Typer injects it), so this function never reaches into a global context stack.

otto.cli.test.run_selection(ctx: Context) → None¶: Run a suite-less selection (–tests and/or -m) — one session per repo.

otto.cli.test.list_suites_callback(value: bool) → None¶: Print all available test suites (one panel per repo) and exit when the flag is set.

otto.cli.test.list_markers_callback(value: bool) → None¶: Print the markers available to –markers (one panel per repo) and exit.

Collect otto test run options and store them in ctx.meta for suite runners.

Validates coverage and report directories and resolves implied flags (e.g. --cov-dir implies --cov). Output-directory creation and the reservation gate happen later, in the shared leaf-invoke command preamble.