Budgets and deadlines

A model that keeps asking for repairs, or an agent that keeps calling tools, costs money and time. A budget limits one run: what it may spend, how many calls it may make, and how long it may take. The harness checks it before every call, so a run stops at its limit instead of after it.

Set limits

harness.yaml

apiVersion: harnessweaver/v1
kind: Harness
metadata:
  name: support-desk
  owner: support-team

models:
  provider: openai.chat
  base_url: http://localhost:4000
  api_key_env: LITELLM_API_KEY
  default: fast
  aliases: { fast: fast-chat, strong: strong-chat }
  prices:
    fast-chat: { input_per_mtok: 0.25, output_per_mtok: 2.00 }
    strong-chat: { input_per_mtok: 3.00, output_per_mtok: 15.00 }

budgets:
  per_run: { max_usd: 0.50, deadline: 2m }

tasks:
  triage:
    runner: direct
    model: fast
    prompt: triage@2
    input: helpdesk.models:Ticket
    output: helpdesk.models:Triage
    budget:
      max_model_calls: 3
      max_usd: 0.01
      deadline: 30s
      overridable: [max_model_calls, max_usd]

validation:
  max_repairs: 1
  on_failure: escalate
  rules:
    - id: bugs-are-urgent
      when: "output.category == 'bug' && output.priority > 2"
      message: "a bug is priority 1 or 2"
    - id: short-summary
      when: "size(output.summary) > 120"
      message: "keep the summary under 120 characters"

Limit	Meaning
max_usd	Total cost in US dollars
max_tokens	Input plus output tokens
max_model_calls	Model calls, repairs included
max_tool_calls	Tool calls
deadline	Wall-clock time for the whole run, such as 30s, 5m or 1h30m

Limits come from up to three places, and the tightest value of each one applies:

1. budgets.per_run, for every run of every task. An org baseline usually caps it.
2. The task's budget.
3. The caller, for one run, but only for the fields the task lists in overridable, and only downward (RULES R10).

Hit a limit

The offline model from the last chapter needs a repair on every run, so each run makes two model calls. Run once normally, then once with a caller's limit of one call:

main.py

import asyncio

from helpdesk.models import Ticket

import harnessweaver as hw

TICKET = Ticket(text="The app crashes every time I log in since this morning's update.")


async def main() -> None:
    harness = hw.load("harness.yaml")

    result = await harness.run("triage", TICKET)
    print(f"{result.status.value}, cost ${result.cost_usd:.6f}")

    tight = await harness.run("triage", TICKET, budget={"max_model_calls": 1})
    if tight.error is not None:
        print(f"{tight.status.value} ({tight.error.details['code']}): {tight.error.message}")


if __name__ == "__main__":
    asyncio.run(main())

$ HARNESSWEAVER_ENV=offline uv run python main.py
completed, cost $0.000155
blocked (budget_exceeded): budget max_model_calls (1) reached

The second run was stopped before its repair call: blocked, with the code budget_exceeded. The repair never reached the model.

A caller can only lower a limit. Asking for more, or overriding a field the task doesn't list, fails before the run starts:

await harness.run("triage", ticket, budget={"max_model_calls": 5})
# BoundViolation: a run may only lower tasks.triage.budget.max_model_calls (R10)
await harness.run("triage", ticket, budget={"deadline": "10s"})
# BoundViolation: tasks.triage.budget.deadline can't be overridden per run; ...

Where budgets are checked

Check	When	What happens
The next model call	model.before	Blocked when the call count, tokens or cost has reached its limit, or when the request's estimated input won't fit in the tokens left
Actual usage and cost	model.after	Adds the response's usage and cost. Going over max_tokens or max_usd blocks the run
Tool calls	Before tool.before	Counts the call against max_tool_calls
The deadline	The whole run	When it passes, the run fails with DeadlineExceeded

Each check is a harness.budget note in the trace (RULES R13). While a run waits for a person to approve something (Approvals), its deadline is paused.

Info

A runner or adapter that can't reach model.before and model.after can't enforce a model budget. If one is set, the spec fails to load (UnreachableHook) rather than run unchecked (RULES R4).

Durations

Durations are strings such as 250ms, 20s, 5m, 1h30m or 1d2h, or a plain number of seconds. The units are d, h, m, s and ms, largest first, each used once, with an optional fraction (1.5s). A string needs a unit: "5" is an error, while the number 5 is five seconds.

Recap

• A budget limits a run's cost, tokens, model calls, tool calls and time. The tightest of the spec's, the task's and the caller's limits applies.
• Callers may only lower the limits a task marks overridable.
• Budgets are checked before each call, so a run stops at its limit with budget_exceeded.