Tool Contract Tests vs End-to-End Tests for AI Agents | Hokstad Consulting

Tool Contract Tests vs End-to-End Tests for AI Agents

Tool Contract Tests vs End-to-End Tests for AI Agents

Tool contract tests check whether tools work as agreed. End-to-end tests check whether the agent completes the task. Run contract checks first, then use sandboxed workflow tests to block releases that fail critical tasks.

For a £49.99 refund, a valid tool response is only part of the test. The agent must also refund the right order, avoid duplicate payments after a timeout and report the result correctly.

Quick Comparison

Criterion Tool contract tests End-to-end tests
Scope Requests, responses, permissions and agreed errors Decisions, tool calls, state, recovery and final results
Dependencies Isolated tools and controlled test data Agent, model and connected services within the test boundary
Speed and cost Usually faster and cheaper; often no model calls Usually slower, with model and setup costs
Repeatability More stable with fixed inputs Can vary with model output, timing and data
Failures caught Changed fields, types and interface rules Wrong actions, lost state, unsafe retries and incorrect answers
Release role Check interface changes early Gate critical workflows before release

Use both layers, with clear pass criteria. Keep test data isolated, record versions and traces, and investigate failures rather than rerunning tests until they pass.

::: @figure Tool Contract Tests vs End-to-End Tests for AI Agents{Tool Contract Tests vs End-to-End Tests for AI Agents} :::

Schema-Based Contract Testing: Preventing API Drift in Practice

Tool Contract Tests: Check Interface Rules

Contract tests check interface rules, not task success. Treat each tool contract as a versioned agreement between the agent client and the provider, based on the requests and responses the client actually uses.

Define Inputs, Outputs and Error Handling

Cover tool names, operation IDs, required and optional fields, JSON Schema types, enums, nullability, response formats and authentication. For HTTP tools, also check methods, paths, headers and status codes. Assert only the required response fields.

Test authentication failures, permission errors, rate limits, timeouts and retry rules. Where idempotency is required, repeat a request with the same key and check that it creates no duplicate resource. Schema validation alone cannot prove access control or side effects.

Verify Providers Against Consumer Contracts

With Pact, consumer tests run the consumer client against a mock provider and generate a versioned contract.[3] Provider verification then checks the recorded interactions against the running provider. It uses known provider states and, where appropriate, stubbed downstream dependencies.[1][2]

A mock alone does not prove that the real provider is compatible. Both stages matter, but they verify the interface - not whether the agent can finish the task.

Detect Interface Drift, Not Task Success

Contract tests catch renamed fields and changed error formats. They do not prove that the model chose the right tool or that the user’s task succeeded. End-to-end tests pick up where contract tests stop.

End-to-End Tests: Check Complete Agent Tasks

Once the interface contract passes, check whether the agent can finish the job. End-to-end tests take a realistic user request through the full agent workflow. Set a clear test boundary: use the real orchestration, prompts, routing, state and adapters under test. Sandbox any systems that send messages, change accounts or make payments. Use recorded responses or stubs only for dependencies outside the scenario’s scope.

Check Decisions, State and Recovery

Check that the agent follows instructions, selects suitable tools, keeps context and respects permissions across multiple calls. Test missing information, expired permissions, timeouts and partial failures.

Judge the outcome, not the exact tool order. Accept different tool sequences when they produce the correct result and authorisation comes first. Verify saved records, audit entries, authorised side effects and the final answer.

After a timeout, the agent should check status before retrying, avoid duplicates and escalate when recovery is unsafe. End-to-end tests catch wrong tool choices, lost state and unsafe retries that contract tests cannot detect. To diagnose failures, record model calls, tool arguments and results, guardrail events, hand-offs and state changes in traces.[4][5]

Test Outcomes in Safe Environments

A safe sandbox lets you test task outcomes and recovery without affecting live accounts or payments.

In a sandboxed £49.99 refund test, seed an eligible order, verify the agent issues the correct refund, then simulate a timeout and confirm it retains the refund status, avoids a duplicate refund and does not report completion while the outcome remains uncertain.[4][5]

Use isolated fixtures, synthetic records and resettable databases. Record fixture versions, prompts, model versions, tool schemas and permissions so tests can be reproduced. Keep broken fixtures and unavailable test services separate from agent failures.

Define outcome criteria before each run, then repeat critical scenarios and track pass rates. Allow valid alternative paths and equivalent wording, but require correct state, safe recovery and accurate reporting.[6]

Contract Tests vs End-to-End Tests: Scope, Cost and Failures

The practical difference is which layer catches a failure at the lowest cost.

Use the cheaper layer to catch interface drift, and the end-to-end layer to check failures involving state or decisions.

Dimension Tool contract tests End-to-end tests
Scope Consumer/provider requests, responses, data types and agreed error behaviour Complete agent tasks across decisions, tools, state and the final outcome
Dependencies Controlled fixtures and isolated provider verification; downstream services can be stubbed Agent runtime, model, tool services, identity, databases and other dependencies within the test boundary
Execution speed Usually faster because each check covers a narrow interaction Usually slower because model calls, network requests and retries add up
Cost Usually no live-model cost; limited infrastructure and setup Inference charges when using a real model, plus integrated environments, data stores and clean-up
Repeatability Usually high with controlled inputs and dependencies Usually more variable because model outputs, timing and data can change
Failure detection Missing fields, changed types, incompatible errors and authentication assumptions; usually pinpoints the broken interface expectation Wrong tool choice, broken state handling, unsafe recovery and outcome failures covered by explicit assertions; often needs traces to diagnose the cause

Speed and cost vary by model, caching and environment. Before setting a budget, measure median and 95th percentile run time, cost, setup time and reruns.

What Each Test Layer Can Miss

Coverage depends on the assertions, not the test label.

Failure scenario Contract-test coverage End-to-end coverage
A tool response field changes from refund_id to id Provider verification should fail if the consumer contract requires refund_id. Must reach that response and check parsing or its effect on the outcome.
The agent chooses the wrong tool Usually missed: the wrong tool can still honour its contract. Must establish the intended action and assert tool choice or the resulting business action.
An agent retry causes two refunds Idempotency guarantees can be contract-tested, but do not prove safe orchestration. Must trigger a retry and check that only one refund is recorded.
The agent’s final answer is incorrect Interface checks do not establish answer accuracy. Must check required facts or consistency with saved state, not merely a non-empty answer.

These trade-offs shape which checks run early and which block release. Neither suite proves overall correctness: contracts check compatibility risk, while workflow tests check decision or state risk. High-risk behaviour needs checks at both layers.

Use Both Layers in the Release Pipeline

Match Test Runs to Component Changes

Run contract checks when schemas, clients, auth, providers or responses change. Add smoke end-to-end tests to selected pull requests and release candidates. Run broader workflow suites after changes to models, prompts, tools or orchestration. Prioritise multi-tool tasks, personal or financial data, permission-sensitive operations, customer-facing decisions and irreversible actions.

Use unit tests for deterministic logic, model evaluations for output quality and production monitoring for live reliability. Monitoring does not replace pre-release checks. This order catches low-cost interface failures before they reach later workflow runs.

Set Up Fixtures, Traces and Release Gates

Give every versioned contract a named owner and a compatibility policy. Use stable fixtures, seeded data and isolated sandboxes for side effects, with explicit clean-up. Record the versioned inputs and traces needed to reproduce failures. Use those traces to separate interface, orchestration, model, data and provider issues. Restrict trace access, redact personal data and set justified retention periods that meet UK data protection obligations.[8][9]

Define release gates before running tests. Block incompatible interface changes unless a versioned transition has been approved. Also block critical workflows that fail their acceptance criteria. Classify each failure as an interface, orchestration, model, data or provider issue.

When results vary, compare failed traces with a known-good baseline and check whether you can reproduce the failure. Do not keep rerunning a flaky test until it passes. Instead, inspect the trace, reproduce the result and decide whether the test or implementation needs fixing.[7]

Once runs are reproducible, gate the release on the smallest test that proves the risk is controlled.

Build Contracts, Then Test Critical Workflows

The release flow should follow the risk split: compatibility first, task success second. Start with contract definitions, then add sandboxed workflow tests for high-risk paths. Cover success, invalid input, timeouts, partial failures, permission denial and provider unavailability.

Check traces against explicit criteria for decisions, tool arguments, state changes and final outcomes. Keep incompatible interfaces and failed critical workflows blocked. Expand coverage based on risk, near misses and incidents.

Conclusion: Combine Both Layers to Reduce Release Risk

Contract tests check tool compatibility; end-to-end tests check complete agent workflows.

Start with clearly defined tool contracts and a small set of critical workflows. Keep human review and operational controls in place for risks that tests alone cannot rule out.

FAQs

How should I split my AI testing budget?

Prioritise risk. Run automated contract tests frequently to check interface rules. Save more costly end-to-end workflow tests for nightly or pre-release runs to keep pipelines fast.

Use AI-driven test selection to run the highest-risk tests with each commit. Budget for model inference, tool usage and human review - not just model spend. Track cost per successful task so spending reflects accepted outcomes, while maintaining quality as you scale.

What pass rate should block an agent release?

There’s no universal pass rate for an agent release. Any failed mandatory security control or safety requirement must block release until the fault is fixed and retested.

For functional tasks, success means verifying that all acceptance criteria have been met. Agent activity or attempted runs alone don’t count as success. High-risk actions, such as those affecting payments or production states, always need human sign-off - regardless of automated pass rates.

How can I test agents with dynamic tool schemas?

Treat failed tool calls as contract failures. Track schema versions and use contract tests to pinpoint mismatches. Keep a reviewed tool allow-list that records every permitted operation and its input schema.

Validate arguments and targets outside the model. Treat unknown tools or altered schemas as failures.

Whenever schemas change, rerun test fixtures to check compatibility and catch schema drift before it breaks agent workflows.

Need help with your DevOps, cloud or AI plans?

Hokstad Consulting helps companies with DevOps transformation, cloud architecture and hands-on AI development — pragmatic consulting with measurable results.

Our services: DevOps on Retainer · Hosting & Cloud · AI Development & Strategy