suite.run¶

Suite-run engine as a library call — run_suite without a Typer context.

This module holds the pytest-driving core that otto test used to carry inline in otto.cli.test: the run-options record, the inner pytest session, the stability report, and the coverage pre/post hooks. Extracting it here lets a suite run as a plain library call — run_suite(MySuite, output_dir=...) — returning a SuiteRunResult instead of raising typer.Exit.

The CLI (otto.cli.test) keeps its context-driven run_suite / run_selection wrappers and imports the moved helpers from here, so otto test behavior is unchanged.

Import-weight note: this module never imports typer (nor pytest) at module load — the typer/pytest-touching pieces (the inner session’s plugins, the coverage helpers) are imported lazily inside the functions that need them, so import otto.suite.run stays cheap for library callers.

exception otto.suite.run.NoTestsMatchedError¶

Bases: ValueError

A --tests / -m selection resolved to nothing to run.

Raised by run_selection() when the selection matched no repo — no repos configured, no repo carrying the marker, or (via otto.suite.selection.resolve_selection()) no test universe to search at all. It is distinct from otto.suite.selection.UnknownSelectionError (a genuinely unknown name against a real universe, carrying did-you-mean suggestions).

Subclasses ValueError so it stays catchable as one, but the CLI adapter catches this specifically — a broad except ValueError would misreport an unrelated pipeline ValueError as “No tests matched the selection.”

class otto.suite.run.RunOptions(markers: str = '', tests: str = '', iterations: int = 0, duration: int = 0, threshold: float = 100.0, results: str = '', cov: bool = False, cov_dir: Path | None = None, overwrite_cov_dir: bool = False, cov_clean: bool = True, cov_report: bool = False, cov_report_dir: Path | None = None, overwrite_cov_report_dir: bool = False, project_name: str = 'Coverage Report', monitor: bool = False, monitor_interval: float = 5.0, monitor_output: Path | None = None, monitor_hosts: str | None = None)¶

Bases: object

Shared suite-run options: markers/iterations/stability/coverage/monitor.

The otto test callback constructs one of these from its CLI flags and stores it in Typer ctx.meta[RUN_OPTIONS_KEY]; library callers pass one directly to run_suite(). Field defaults mirror the otto test CLI defaults exactly.

markers : str = ''¶

tests : str = ''¶

iterations : int = 0¶

duration : int = 0¶

threshold : float = 100.0¶

results : str = ''¶

cov : bool = False¶

cov_dir : Path | None = None¶

overwrite_cov_dir : bool = False¶

cov_clean : bool = True¶

cov_report : bool = False¶

cov_report_dir : Path | None = None¶

overwrite_cov_report_dir : bool = False¶

project_name : str = 'Coverage Report'¶

monitor : bool = False¶

monitor_interval : float = 5.0¶

monitor_output : Path | None = None¶

monitor_hosts : str | None = None¶

class otto.suite.run.SuiteRunResult(exit_code: int, junit_paths: list[Path], stability_report: Path | None, stability_unstable: bool, output_dir: Path)¶

Bases: object

Outcome of a run_suite() invocation.

exit_code is the ssh-like final code (pytest rc, with a stability threshold violation folded in). junit_paths are the JUnit XML files the session wrote. stability_report is the stability_report.txt path when a stability run produced one (else None); stability_unstable is True when any test fell below its pass-rate threshold.

exit_code : int¶

junit_paths : list[Path]¶

stability_report : Path | None¶

stability_unstable : bool¶

output_dir : Path¶

property passed : bool¶

True when the invocation succeeded (exit code 0).

otto.suite.run.resolve_output_dir(output_dir: Path | None) → Path¶

Explicit param → context output_dir → CWD (xdir-defaults-to-CWD philosophy).

otto.suite.run.find_suite(name: str) → type¶

Resolve a registered OttoSuite subclass by class name via SUITES.

otto.suite.run.run_suite(suite: type, *, options: object | None = None, run_options: RunOptions = RunOptions(markers='', tests='', iterations=0, duration=0, threshold=100.0, results='', cov=False, cov_dir=None, overwrite_cov_dir=False, cov_clean=True, cov_report=False, cov_report_dir=None, overwrite_cov_report_dir=False, project_name='Coverage Report', monitor=False, monitor_interval=5.0, monitor_output=None, monitor_hosts=None), output_dir: Path | None = None) → SuiteRunResult¶

Run a suite class via pytest.main and return its SuiteRunResult.

suite is an OttoSuite subclass (or any pytest-collectable class); options is its per-suite Options instance (or None); run_options carries the markers/stability/coverage/monitor settings; output_dir overrides where JUnit/coverage/stability artifacts land (defaults per resolve_output_dir()).

otto.suite.run.run_selection(*, run_options: RunOptions = RunOptions(markers='', tests='', iterations=0, duration=0, threshold=100.0, results='', cov=False, cov_dir=None, overwrite_cov_dir=False, cov_clean=True, cov_report=False, cov_report_dir=None, overwrite_cov_report_dir=False, project_name='Coverage Report', monitor=False, monitor_interval=5.0, monitor_output=None, monitor_hosts=None), output_dir: Path | None = None) → SuiteRunResult¶

Run a suite-less --tests / -m selection — one pytest session per matching repo.

Mirrors otto test --tests / otto test -m as a plain library call: exact test names (optionally Class::name qualified) and/or a marker expression are resolved against every configured repo’s collected tests (otto.suite.selection.resolve_selection() for --tests, otto.suite.selection.repos_with_marker_matches() for -m alone), and a pytest session runs once per repo whose selection matched. Sessions fold into a single SuiteRunResult: exit_code is the worst per-session exit code (via the same stability-aware rule run_suite uses internally), and junit_paths lists every session’s JUnit file, in run order.

Raises NoTestsMatchedError ("No tests matched the selection.") when the selection resolves to nothing to run — no repos, no matching marker, or (via resolve_selection()) no test universe to search at all. A genuinely unknown test name against a real, non-empty test universe instead raises UnknownSelectionError (a ValueError subclass carrying the did-you-mean message) from resolve_selection() itself. Both subclass ValueError; catch UnknownSelectionError before NoTestsMatchedError to distinguish the typo case.

Raises ValueError immediately if run_options carries neither tests nor markers: with both empty a selection would match every test in every repo, so — like the otto test callback, which only reaches this path when --tests/-m is set — the library refuses rather than silently running everything.