Testing

The test kit, harnessweaver.testing. Testing in the tutorial shows it in use.

harness_for_test

harness_for_test(
    spec: Mapping[str, Any]
    | str
    | Path
    | EffectiveSpec
    | None = None,
    *,
    model: ModelProvider | None = None,
    tools: Iterable[FakeTool] | ToolBackend = (),
    deciders: Mapping[str, FakeDecider] | None = None,
    clock: Clock | None = None,
    catalog: PluginCatalog | None = None,
    telemetry: Telemetry | None = None,
) -> Harness

A harness for a test: offline, on a fake clock, with fakes where you give them.

Parameters:

Name	Type	Description	Default
spec	Mapping[str, Any] | str | Path | EffectiveSpec | None	A harness.yaml path, a spec document as a dict (apiVersion, kind and metadata may be left out), or a loaded spec.	None
model	ModelProvider | None	Replaces models.provider, usually a scripted model.	None
tools	Iterable[FakeTool] | ToolBackend	Fake tools, or any tool backend.	()
deciders	Mapping[str, FakeDecider] | None	Fake routing answers by router name.	None
clock	Clock | None	Defaults to a FakeClock.	None
catalog	PluginCatalog | None	Where plugins are found; defaults to the installed ones.	None
telemetry	Telemetry | None	Spans and metrics go here (for example an in-memory SDK).	None

Raises:

Type	Description
SpecError	For an invalid spec, as Harness.load would.

scripted

scripted(
    *replies: ScriptedReply | Mapping[str, Any] | str,
    rules: Iterable[Mapping[str, Any]] = (),
    repeat_last: bool = True,
) -> ScriptedModel

A ScriptedModel from replies (strings are text replies) and optional rules.

text

text(value: str, **fields: Any) -> ScriptedReply

A reply that answers with value.

output

output(value: JsonValue, **fields: Any) -> ScriptedReply

A reply with structured output (also sent as JSON text).

tool_call

tool_call(
    name: str,
    /,
    *,
    id: str | None = None,
    **arguments: JsonValue,
) -> ScriptedReply

A reply that asks for one tool call.

reply

reply(**fields: Any) -> ScriptedReply

A scripted reply with any of text, output, tool_calls, usage, cost_usd, model, chunks, finish_reason or fail.

FakeTool dataclass

FakeTool(
    name: str,
    result: JsonValue | ToolHandler = None,
    description: str = "",
    input_schema: dict[str, JsonValue] = (
        lambda: dict[str, JsonValue]({"type": "object"})
    )(),
    side_effect: SideEffect = NONE,
    risk: Risk = LOW,
    data_access: tuple[str, ...] = (),
    egress: tuple[str, ...] = (),
    scopes: tuple[str, ...] = (),
    untrusted: bool = True,
    raises: Exception | None = None,
    governance: dict[str, JsonValue] = dict[
        str, JsonValue
    ](),
    calls: list[ToolCall] = list[ToolCall](),
)

A tool for tests. result is returned (or called with the arguments); every call is kept in calls. raises makes it fail instead.

governance class-attribute instance-attribute

governance: dict[str, JsonValue] = field(
    default_factory=dict[str, JsonValue]
)

More manifest spec keys, such as approval or rate_limit (DESIGNS No. 17).

manifest

manifest() -> ToolManifest

The tool's manifest, as an app would write it (S2).

FakeDecider

FakeDecider(
    choice: str | None = None,
    *,
    confidence: float | None = None,
    reason: str = "fake decider",
)

A routing stage that always gives the same answer; requests are kept in requests.

With no choice it abstains. With a confidence the decision is probabilistic.

registration

registration(router: str) -> Registration

Middleware that answers route.decide for router.

assert_trace

assert_trace(
    trace: RunResult | RunTrace | Iterable[TraceEvent],
    *expected: str | Step,
    exact: bool = False,
) -> None

Check that the trace has the expected events, in that order.

A string is a hook point that fired ("tool.before"); a step matches any event by its fields. Other events may come in between, unless exact is set: then the expected strings must be exactly the hooks that fired.

Raises:

Type	Description
AssertionError	With the trace rendered, when something is missing.

step

step(
    hook: str | None = None,
    *,
    kind: str | None = None,
    source: str | None = None,
    outcome: str | None = None,
    code: str | None = None,
    band: str | None = None,
    reason: str | None = None,
) -> Step

An expected trace event, for assert_trace.

hooks_fired

hooks_fired(
    trace: RunResult | RunTrace | Iterable[TraceEvent],
) -> list[str]

The hook points that fired, in order (one entry per firing).

render_trace

render_trace(
    trace: RunResult | RunTrace | Iterable[TraceEvent],
) -> str

The trace as text, one event per line, for assertion messages.

assert_no_content

assert_no_content(where: Any, *values: str) -> None

Check that none of values appears in where: a result, a trace, log lines, span attributes or any mix of them (R9).

Cassette

Cassette(
    path: str | Path,
    *,
    provider: ModelProvider | None = None,
    mode: Mode = "auto",
    scrub: Callable[[ModelResponse], ModelResponse]
    | None = None,
)

A model provider that replays recorded answers, or records them from provider.

Parameters:

Name	Type	Description	Default
path	str | Path	The cassette file (JSON).	required
provider	ModelProvider | None	The real provider to record from; not needed to replay.	None
mode	Mode	replay never calls out; record always calls out and overwrites; auto replays if the file exists, otherwise records.	'auto'
scrub	Callable[[ModelResponse], ModelResponse] | None	Applied to every recorded answer before it is saved (S7).	None

Raises:

Type	Description
SpecError	When recording is needed but no provider was given.

save

save() -> None

Write the cassette (recording mode saves after every answer).