# `Mutare.Runner`
[🔗](https://github.com/foxbenjaminfox/mutare/blob/v0.2.1/lib/mutare/runner.ex#L1)

Compile once, then run the suite once per mutant in a fresh OS process.

The flow protects the one-compile invariant: we compile the sandbox a single time, run the tests as a baseline to ensure it passes, then launch one `mix test` process per mutant with `MUTARE_MUTANT_NAMESPACE` and `MUTARE_ACTIVE_MUTANT` set to its file and its id within that file. Sources never change between runs, so mix's incremental compiler finds nothing to rebuild — the per-mutant cost is process boot plus the suite (only up to the first failure for a kill), never recompilation.

The compile step distinguishes Mix dependency validation from actual compile-poisoning. A dependency failure returns `:dependency_failed` immediately: dropping mutant ids cannot repair copied dependency state, so it never enters poison recovery. The Mix task then points remediation at the original project rather than the disposable sandbox. The compile also carries a wall-clock cap (`:compile_timeout`, default 30 minutes, `nil` to disable): a config-hosted sibling of the per-mutant timeout watcher self-halts a pathological compile, surfaced as `:compile_timed_out` — likewise never fed to poison recovery, since there is no error to attribute and a rebuild cannot make an oversized compile faster.

`run/2` returns `{:ok, %Mutare.Run{}}`: the `Mutare.Schema` that was run, the list of per-mutant `Mutare.Result`s, the sandbox path, the baseline run's wall-clock in milliseconds, and whether an early-stop condition (`:max_survivors` or `:time_budget`) stopped the run early.

## Baseline + coverage probe

Before the per-mutant loop we run the test suite once and ensure it passes (`Mutare.Runner.Baseline`) — then build a per-mutant test selection (`Mutare.Runner.CoverageProbe`) which picks the test files each mutant needs (or marks it `:no_coverage`). The two are split on purpose: a failing baseline test aborts, while coverage is advisory and degrades to running everything. The baseline can be run more than once (`:baseline_runs`) to catch a flaky suite: runs that disagree abort with `:baseline_flaky` rather than let a flaky test manufacture false mutant kills. A consistently red baseline attempt can also be retried (`:baseline_retries`, default 0) to survive startup/load flakes without weakening `:baseline_runs`' mixed-outcome check. See those modules for the selection modes.

## Unanimous kill reruns

`:kill_runs` (`--kill-runs`, default 1) handles the narrower case where a residual flaky test appears only under one mutant's timing. Only kill outcomes are rerun, and every attempt must kill. If a later attempt passes, the mutant is recorded as `:survived`; if a later attempt persistently hits the harness, it remains `:harness_error`. Harness retries are the inner infrastructure layer; kill reruns combine settled test-suite verdicts.

## Parallel workers and timeouts

The per-mutant phase runs `:workers` mutants concurrently (default: half `System.schedulers_online/0`, capped at 4 — each worker is a full `mix test` BEAM that itself uses every scheduler, so a parallel suite already scales with the machine and extra workers only fill the serial/IO gaps one run leaves), each its own OS process in the shared sandbox. Each run has a wall-clock cap: an explicit `:timeout` in ms, or `baseline × :timeout_multiplier` (default 3.0) scaled by half the concurrent lanes — the baseline is measured *uncontended*, so wall time under contention legitimately inflates with the lane count — with a floor. A mutation can turn a terminating loop infinite, so the run is capped; a capped run counts as `:timeout` — a kill, since the hang is observable misbehavior.

Even a scaled cap can be overrun by a slow-but-finite run, and survivors are the most exposed (a kill exits at its first failing test; a survivor must run its entire selected set). A false `:timeout` is a false kill hiding a true survivor, so by default (`:confirm_timeouts`) a streamed `:timeout` is *provisional*: after the stream drains, each timed-out mutant is re-run sequentially — no contention — with the same cap, and that verdict is recorded instead. Only a repeat overrun records `:timeout`; a genuine hang pays one extra cap. `confirm_timeouts: false` (`--no-confirm-timeouts`) records the first overrun as-is. A wall-clock `:time_budget` covers this confirmation pass too: confirmations already launched are allowed to finish, but no new confirmation run starts after the deadline.

## Early stop: survivor cap (`:max_survivors`) or time budget (`:time_budget`)

Two conditions can stop the per-mutant loop before every mutant runs, whichever fires first. `:max_survivors` (`--max-survivors`) stops once that many survivors (`:survived` results) have surfaced — useful for finding and fixing a few concrete test gaps at a time. `:time_budget` (`--time-budget`, a duration string like `"10m"` parsed by `Mutare.Duration`) stops once that much wall-clock elapses in the per-mutant phase — a "see what I can get in ten minutes" run. The clock starts as the phase begins (compile/baseline/probe are not charged against it) and is checked just before a task announces and launches a real mutant run, so ordered result buffering cannot hide an expired budget and allow more mutants to start.

Unlike `:max_mutants` (a `Mutare.Schema` cap on candidate *sites*), both leave every mutant compiled in — only the *run* halts early. The per-mutant stream is consumed `ordered: true`, so a survivor stop is deterministic: the Nth survivor in source order, regardless of which worker finished first, and the reported survivors are exactly the first N. (A time-budget stop is not deterministic — it depends on how far the run got.) Runs already in flight when either condition trips are *drained* (not killed), so the sandbox teardown never races a live `mix` subprocess. If the budget elapses after every mutant has already launched, the result set is still complete unless the budget also prevents a provisional timeout from being confirmed. Otherwise the returned run carries `stopped_early`; on an early stop the harness-error abort guard is skipped (the score is already budget-limited or a partial prefix — the Mix task notes it and skips the `--min-score` gate too), since aborting would discard the very survivors the user asked to find.

## Per-worker partitioning (DB isolation)

Optionally (`:partition_env`, off by default), each concurrent run is handed a distinct partition id under a named env var (default `MIX_TEST_PARTITION`), so a stateful suite can point each worker at its own database — the `mix test --partitions` convention. The ids come from a bounded, recycled pool (`Mutare.Runner.Partitions`) sized to `:workers`, so two live runs never share a partition and only `:workers` databases are needed. The one compile, the baseline, and the coverage probe (all sequential, pre-pool) take a fixed partition — the compile too, since it evaluates the target's config, where a partitioned default-less `System.fetch_env!` would otherwise raise. Inert when unset.

## Harness errors are kept out of the score

A mutant run interrupted by a compile error, a missing dependency, or a filesystem race provides no verdict on the mutation. It is recorded as `:harness_error` and excluded from the score's denominator. Treating every non-zero exit as a kill would miscount these failures.

Two knobs harden this against flakiness and systemic breakage:

  * `:harness_retries` (default 2) re-runs a harness-errored mutant before recording it, so a *transient* failure (a filesystem/lock race) gets another chance; a real verdict is never retried.
  * `:max_harness_error_rate` (default 0.5, `nil` to disable) aborts the whole run — `{:error, :too_many_harness_errors, detail}` — when persistent harness errors exceed that fraction of the mutants that *ran*. Past that, the sandbox is broken, not the mutations tested, and a score over the surviving denominator would mislead; better to fail loudly.

## Boot-failure: a known-transient harness error retried harder

One harness-error *cause* is recognised by name (`Output.boot_failure?/1` → the `:boot_failure` outcome): the sandbox node dies during boot with its own diagnostic erased by a secondary `:standard_error` failure. It is almost always concurrent workers contending on shared singletons at startup (a test DB, a connection pool), so it clears on a retry that doesn't re-collide with the boot stampede. It gets its own retry budget (`@boot_failure_retries`), independent of `:harness_retries` and with a short jittered backoff, plus a *specific* warning that stops pointing at output that can't help (the real cause is unrecoverable) and names the actual contention levers — `--workers` and `--partition-db`/`--partition-env`. (Not `--harness-retries`: a `:boot_failure` draws only from its own dedicated budget, so raising that knob would not retry it more.) The verdict is unchanged (a harness error, out of the score); only the messaging and retry effort differ.

## SIGKILL (likely OOM): a harness error never retried, and the `:max_heap_mb` cap

The mirror-image refinement: a run the OS killed with SIGKILL (exit 137, the `:sigkilled` outcome) is recognised so it is **never** retried — the opposite of `:boot_failure`. Its signature cause is the kernel OOM killer reaping a mutant whose mutation made it allocate without bound (a dropped guard turning a function unconditionally self-recursive can exhaust tens of GB in under a second — faster than any wall-clock watcher can react), and that failure is *deterministic*: a back-to-back retry re-detonates the same blowup on the host. The verdict stays a harness error (out of the score), with a specific warning naming the likely cause and the mitigation.

The mitigation is `:max_heap_mb` (`--max-heap-mb`, off by default): a per-process BEAM heap cap injected into every *runtime* sandbox run — baseline, coverage probe, per-mutant — so a runaway-allocation mutant dies as an ordinary, fast test failure inside its own run instead of endangering the host. The baseline running under the same cap validates up front that the suite itself fits under it. The one metamutant compile is deliberately not capped. Mechanism and sizing guidance: `Mutare.Sandbox.Command.Invocation.heap_cap_env/1`.

# `error`

```elixir
@type error() ::
  {:error,
   :compile_failed
   | :compile_timed_out
   | :dependency_failed
   | :baseline_failed
   | :baseline_flaky
   | :nothing_to_mutate
   | :too_many_harness_errors, String.t()}
```

# `run`

```elixir
@type run() :: Mutare.Run.t()
```

# `check_with_schema`

```elixir
@spec check_with_schema(
  Mutare.Schema.t(),
  Path.t(),
  Mutare.Run.Context.t() | Mutare.Options.t() | keyword()
) ::
  {:ok, %{schema: Mutare.Schema.t(), recovery: Mutare.Run.recovery() | nil}}
  | error()
```

Compile-only preflight (`mix mutare --check`): materialise the sandbox and run the one
compile — recovering from compile-poisoning exactly like a full run — then stop before
the baseline and the per-mutant phase.

Returns `{:ok, %{schema: schema, recovery: recovery}}`, where `schema` is the (possibly
rebuilt) schema the compile succeeded against and `recovery` summarises any poison
recovery it took (`t:Mutare.Run.recovery/0`, or `nil` when the metamutant compiled
clean on the first attempt). Errors are the compile-stage subset of `t:error/0`. The
`:on_phase` hook receives the same `:compiling` / `{:poison_round, info}` /
`{:compiled, ms}` events as a full run.

# `run`

```elixir
@spec run(Path.t(), Mutare.Run.Context.t() | Mutare.Options.t() | keyword()) ::
  {:ok, run()} | error()
```

Run mutation testing against the project at `root`.

`opts` is a `Mutare.Run.Context` (or a `Mutare.Options` / keyword list resolved
into one). Returns `{:ok, %Mutare.Run{}}` or `{:error, reason, detail}`.

# `run_with_schema`

```elixir
@spec run_with_schema(
  Mutare.Schema.t(),
  Path.t(),
  Mutare.Run.Context.t() | Mutare.Options.t() | keyword()
) :: {:ok, run()} | error()
```

Run a pre-built schema (lets a caller report the mutant count before launching).

`opts` may be a `Mutare.Run.Context`, a `Mutare.Options` struct, or a keyword
list. The resolved context supplies the sandbox options, run options, and live
progress hooks.

Live progress hooks:

  * `:reporter` — called with each `Mutare.Result` the run keeps, in source order.
    Every reported result appears in the returned run's `:results` (an early stop
    discards the runs still in flight when it trips — those are never reported), and
    the calls are serialized, so the hook needs no synchronization of its own.
  * `:on_start` — called with each `Mutare.Site` just before its test run starts.
    Unlike `:reporter`, this fires concurrently from every worker, and a site whose
    run is later discarded by an early stop still announces its start.
  * `:on_phase` — called as the run enters `:compiling`, `:baseline`,
    `:coverage_probe`, and `{:running, total}`.

`:on_phase` may also receive detail events:

  * `{:seed_app_build, summary}` — emitted during `:compiling`, once the sandbox is
    materialised: what the app-build `_build` seed did
    (`Mutare.Sandbox.Seed.summary/0` — `:seeded` with reused/recompiled beam
    counts, a `:fallback` to a cold compile, or `:skipped`). `--verbose` renders
    the first two.
  * `{:inference_override_declined, %{file: file, reason: reason}}` — emitted during
    `:compiling`, once per `mix.exs` whose inference override did not land
    (`Mutare.Sandbox.CompilerOptions.project_source/1`). That project compiles
    with type-signature inference on, which can make the one compile far slower;
    `reason` describes the cause in a short phrase. `--verbose` renders it.
  * `{:poison_round, info}` — one compile-poison recovery round: the compile
    failed, the implicated mutants were dropped, and a rebuild + recompile is
    starting. `info` is `%{dropped: [%{id: id, file: file, line: line,
    mutator: family}], escalated: [t:Mutare.Run.escalation/0]}` — the mutants
    dropped individually this round, and any unknown block macro escalated
    wholesale. Fired on every round (not just verbose), since each one is a
    full recompile the user would otherwise read as a hang.
  * `{:compiled, ms}`
  * `{:baseline_done, ms}`
  * `{:coverage_done, summary}`
  * `{:run_config, cfg}`
  * `{:confirming_timeouts, count}` — the sequential re-run of provisional
    timeouts is starting (see the timeouts section above)

Custom hooks should ignore phase or detail events they do not recognise.

The run uses the resolved `:test_selection`, `:workers`, `:timeout`,
`:timeout_multiplier`, `:max_heap_mb`, `:baseline_runs`, `:baseline_retries`,
`:kill_runs`, `:confirm_timeouts`, `:harness_retries`, `:max_harness_error_rate`,
and `:max_survivors` options. When
`:max_survivors` stops the run early, the returned run has
`stopped_early: true`.

---

*Consult [api-reference.md](api-reference.md) for complete listing*