cli.test¶

Run a registered OttoSuite test suite.

Each test suite decorated with @register_suite() appears as a subcommand of otto test. Suite-specific options (declared in the suite’s inner Options dataclass) are automatically registered as Typer parameters with full type enforcement and --help documentation.

Markers

integration: Requires live Vagrant VMs. Skip with --markers "not integration".
timeout(seconds): Fail the test if it runs longer than seconds.
retry(n): Retry a failing test up to n times before reporting failure.

Listing tests

--list-suites List test suites with run syntax and exit.

Options on ``otto test`` (before the suite name)

--markers / -m EXPRESSION: pytest -m marker expression applied after collection.
--iterations / -i N: Repeat each test N times within a single setup/teardown cycle (0 = disabled).
--duration / -d SECONDS: Repeat tests for N seconds within a single setup/teardown cycle (0 = disabled).
--threshold FLOAT: Minimum per-test pass rate percentage required in stability mode (0-100, default: 100).
--results PATH: Write test results (JUnit XML) to PATH (default: auto-written to the log directory).

When both --iterations and --duration are specified, testing stops when either limit is reached first.

--cov: Fetch .gcda files from remote hosts after the suite finishes and place them in a cov/ directory in the suite’s output directory.
--cov-dir PATH: Write coverage data to PATH instead of the default <output_dir>/cov. Implies --cov. The directory is created if missing; if it already exists and is non-empty, the command aborts unless --overwrite-cov-dir is also given.
--overwrite-cov-dir: Clear the contents of the --cov-dir destination before the run so stale data from a previous invocation cannot be mixed with the new results.
--cov-clean / --no-cov-clean: Delete .gcda files on remote hosts before the test run. Enabled by default; use --no-cov-clean to keep stale data.
--cov-report / -r: After coverage collection, render an HTML report. Implies --cov. Default location: <output_dir>/cov_report.
--cov-report-dir PATH: Write the HTML report to PATH instead of the default. Implies --cov-report (and therefore --cov). Empty/overwrite rules match --cov-dir: created if missing, aborts if non-empty unless --overwrite-cov-report-dir is also given.
--overwrite-cov-report-dir: Clear the contents of --cov-report-dir before the report is rendered.
--project-name STR: Title shown in the HTML report header (only used with --cov-report).
--monitor: Enable host performance monitoring for the duration of the run. Samples every host (or those matched by --monitor-hosts) on a fixed interval and emits per-test start/end events automatically. At the end of the run a JSON snapshot of all metrics and events is written to <output_dir>/monitor.json.
--monitor-interval SECONDS: Sampling interval for --monitor (default: 5).
--monitor-output PATH: Override the destination for the captured monitor data. Format inferred from the suffix: .json (default) writes a self-contained snapshot, .db writes a SQLite database loadable via otto monitor --file.
--monitor-hosts REGEX: Restrict --monitor to host IDs matching this regex (re.search).

Examples:

otto test --list-suites
otto test TestMyDevice --help
otto test TestMyDevice --device-type switch --firmware 2.1
otto test --iterations 50 --threshold 95 TestMyDevice
otto test --duration 300 --threshold 90 TestMyDevice
otto test --iterations 100 --duration 60 TestMyDevice
otto test --cov TestMyDevice
otto test --cov-dir /tmp/myrun TestMyDevice
otto test --cov-dir /tmp/myrun --overwrite-cov-dir TestMyDevice
otto test --cov --no-cov-clean TestMyDevice
otto test --cov --cov-report TestMyDevice
otto test -r --cov-report-dir /tmp/myreport TestMyDevice

class otto.cli.test.TestRunOptions(markers: str = '', iterations: int = 0, duration: int = 0, threshold: float = 100.0, results: str = '', cov: bool = False, cov_dir: Path | None = None, cov_clean: bool = True, cov_report: bool = False, cov_report_dir: Path | None = None, overwrite_cov_report_dir: bool = False, project_name: str = 'Coverage Report', monitor: bool = False, monitor_interval: float = 5.0, monitor_output: Path | None = None, monitor_hosts: str | None = None)¶

Bases: object

Shared otto test run options, set by the suite_app callback and read by run_suite. Stored in Typer ctx.meta (shared across the whole context chain by click’s design) rather than ctx.obj (whose parent->subcommand propagation broke under click 8.3).

markers : str = ''¶

iterations : int = 0¶

duration : int = 0¶

threshold : float = 100.0¶

results : str = ''¶

cov : bool = False¶

cov_dir : Path | None = None¶

cov_clean : bool = True¶

cov_report : bool = False¶

cov_report_dir : Path | None = None¶

overwrite_cov_report_dir : bool = False¶

project_name : str = 'Coverage Report'¶

monitor : bool = False¶

monitor_interval : float = 5.0¶

monitor_output : Path | None = None¶

monitor_hosts : str | None = None¶

otto.cli.test.resolve_suite(suite: str, repos: list[Repo]) → str¶: Expand a sut_dir-relative suite path to an absolute path for pytest.

otto.cli.test.run_suite(suite_class: type, suite_file: str, opts_instance: object | None, ctx: Context) → None¶

Execute a registered suite via pytest.main().

Runner options (--markers, --iterations, --duration, --threshold, --results) and coverage options (--cov / --cov-clean) are read from the TestRunOptions the otto test callback stored in ctx.meta[RUN_OPTIONS_KEY]. The context is passed in by the suite runner (Typer injects it), so this function never reaches into a global context stack.

otto.cli.test.list_suites_callback(value: bool) → None¶

otto.cli.test.main(ctx: ~typer.models.Context, list_suites: ~typing.Annotated[bool, <typer.models.OptionInfo object at 0x7fcc40a83d90>] = False, markers: ~typing.Annotated[str, <typer.models.OptionInfo object at 0x7fcc40a81540>] = '', iterations: ~typing.Annotated[int, <typer.models.OptionInfo object at 0x7fcc40a815d0>] = 0, duration: ~typing.Annotated[int, <typer.models.OptionInfo object at 0x7fcc40a80520>] = 0, threshold: ~typing.Annotated[float, <typer.models.OptionInfo object at 0x7fcc40a80190>] = 100.0, results: ~typing.Annotated[str, <typer.models.OptionInfo object at 0x7fcc40a83220>] = '', cov: ~typing.Annotated[bool, <typer.models.OptionInfo object at 0x7fcc40a81630>] = False, cov_dir: ~types.Annotated[~pathlib.Path | None, <typer.models.OptionInfo object at 0x7fcc40a83040>] | None = None, overwrite_cov_dir: ~typing.Annotated[bool, <typer.models.OptionInfo object at 0x7fcc40a821d0>] = False, cov_clean: ~typing.Annotated[bool, <typer.models.OptionInfo object at 0x7fcc40a83d30>] = True, cov_report: ~typing.Annotated[bool, <typer.models.OptionInfo object at 0x7fcc40a80160>] = False, cov_report_dir: ~types.Annotated[~pathlib.Path | None, <typer.models.OptionInfo object at 0x7fcc40a83370>] | None = None, overwrite_cov_report_dir: ~typing.Annotated[bool, <typer.models.OptionInfo object at 0x7fcc40a831c0>] = False, project_name: ~typing.Annotated[str, <typer.models.OptionInfo object at 0x7fcc40a82920>] = 'Coverage Report', monitor: ~typing.Annotated[bool, <typer.models.OptionInfo object at 0x7fcc40a82f20>] = False, monitor_interval: ~typing.Annotated[float, <typer.models.OptionInfo object at 0x7fcc40a82fb0>] = 5.0, monitor_output: ~types.Annotated[~pathlib.Path | None, <typer.models.OptionInfo object at 0x7fcc40a83460>] | None = None, monitor_hosts: ~types.Annotated[str | None, <typer.models.OptionInfo object at 0x7fcc40a83a00>] | None = None) → None¶