Evals and replay

EvalDataset

Bases: _Model

A suite of cases for one task (DESIGNS No. 23).

graders_for

graders_for(case: EvalCase) -> tuple[GraderSpec, ...]

The suite's graders followed by the case's own, labelled.

EvalCase

Bases: _Model

One case: the input the task runs on and what the graders compare against.

SuiteResult

Bases: _Model

A suite's cases, its pass rate and whether it meets its threshold.

CaseResult

Bases: _Model

Grade

Bases: _Model

What an Evaluator says about one sample.

of classmethod

of(passed: bool, reason: str | None = None) -> Self

A pass or a fail, scored 1 or 0.

GraderResult

Bases: Grade

Baseline

Bases: _Model

A suite's accepted results: .harness/baselines/<task>.<suite>.json.

OnlineSample

Bases: _Model

A production run sampled at run.end and graded (evals.online).

It names the run instead of copying its content: the run's record holds that.

ShadowGuardReport

Bases: _Model

How often a log_only guard would have fired, and how well (R8).

GuardFiring

Bases: _Model

A guard whose verdict wasn't allow, read from the run's trace.

Replay

ReplayReport

Bases: BaseModel

A recorded run, run again against the current code and spec.

differs property

differs: bool

Whether the status or the output changed.

TraceChange

Bases: BaseModel

A decision one run's trace has and the other's doesn't.