Agent Handoffs: How Traces Reveal Context Loss | Hokstad Consulting

Agent Handoffs: How Traces Reveal Context Loss

Agent Handoffs: How Traces Reveal Context Loss

A delivered handoff can still lose the facts an agent needs. Traces should compare what the sender passed on, what the receiver could use and whether the later decision had the required support.

In the article’s hypothetical £180 refund, retaining four of five required facts means 80% context completeness. But if the missing fact is the approval limit, the agent must escalate - not approve.

Checking these gaps means:

  • Defining required facts, permissions and ownership in a versioned handoff contract.
  • Tracing field status and protected references without logging private content or internal reasoning.
  • Testing summaries, input limits, access failures, retries and conflicting branches.
  • Measuring fact retention, decision support and trace coverage, with unknown results kept separate from confirmed losses.
  • Assigning owners and turning each failure into a regression test.

The aim is simple: preserve required facts, check their use and stop decisions that lack support.

::: @figure Agent Handoffs: Trace Context Loss and Block Unsupported Decisions{Agent Handoffs: Trace Context Loss and Block Unsupported Decisions} :::

Find Context Loss Without Logging Sensitive Data

Record Each Handoff Checkpoint

Trace each request through sender, handoff, receiver, retrieval, tool and decision spans. Add checkpoints for creation, validation, serialisation, trace-context propagation, input reconstruction, evidence availability and the final outcome: accept, retry, escalate or reject.

At each checkpoint, record allow-listed field names, schema versions, expected and received counts, and keyed fingerprints. Different fingerprints show that the representation changed - not that facts were lost. Matching fingerprints do not prove correct use, either. Compare the required facts with the context the receiver can use, rather than checking delivery status alone.

Hypothetical UK e-commerce workflow: A support agent passes a £180 refund request to a refund agent. The contract requires five facts: order reference, purchase date, return status, refund amount and approval limit. Reconstruction drops the £100 approval limit. Record missing_fact=approval_limit, required_count=5, available_count=4 and decision_status=escalated - not customer details, payment details or message text. The receiver must escalate rather than assume it can authorise the refund.

Check Summaries, Input Limits, Permissions and Versions

A summary is acceptable only if the next agent retains every required fact or an authorised reference to it. Compare field presence before and after summarisation. Then check for renamed fields, changed types, truncated inputs and blocked evidence.

A minimum-context schema should label fields as required, optional, derived or prohibited, and specify source, freshness and authorisation requirements. Reject compaction if it removes required facts or the authorised shared-state reference.

Compare schema and prompt-template versions. Give each retry its own attempt number and state version, and record branch identifiers so conflicting results can be traced. Treat standard approval as an unverified assumption - not a replacement for the missing limit.

Set Privacy Rules for Trace Data

Use a controlled trace schema and filter attributes before storage. Keep the diagnostic information, not the payload content. Exclude raw messages, documents, direct identifiers, secrets, payment details and sensitive business or health data.

Prohibit credentials and personal data in OpenTelemetry baggage: propagated attributes can reach downstream services.[2] Store any necessary payloads separately in protected storage. Trace references should be opaque, short-lived and subject to authorisation, with retrieval audited.

Trace field Diagnostic purpose Safe representation Retention/access rule
Pseudonymous request, handoff and span IDs Connect steps and attempts Random or tokenised IDs Restricted access; documented deletion schedule
Agent, workflow, schema and prompt-template versions Locate implementation drift Release IDs or version strings Defined investigation period
Allow-listed field names and counts Detect missing facts Allow-listed names; expected/received counts No values; role-based access
Fact or representation fingerprints Detect changes Keyed fingerprints where guessing is possible Separate keys; short diagnostic retention
Token counts and truncation Locate input-limit loss Counts, limits and status codes Aggregate when individual traces are unnecessary
Opaque retrieval references Check evidence availability Short-lived reference IDs Protected target storage; audited retrieval
Policy codes Explain authorisation outcomes Policy version and allow/deny/escalate codes Restricted operational and compliance access
Tool outcomes Identify execution failures Success, denied or timeout codes Exclude arguments and results; purpose-limited retention
Latency Find slow steps and timeouts Duration in milliseconds Restrict detailed traces; share aggregates
Decision status and error categories Connect missing context to outcomes Enumerated statuses and error codes No free-text exceptions containing payloads

Pseudonyms and fingerprints may still be personal data under UK GDPR. Pseudonymisation does not make them anonymous.[1] Use keyed fingerprints for guessable values, and keep the keys outside the tracing system.

Document production sampling, access roles, audit trails, deletion and purpose-limited retention. Sample routine traces conservatively, with greater coverage for failures or high-risk decisions. Replace detailed records with aggregates when those are sufficient. Review attributes periodically and delete them on schedule.[3] Use the same allow-listed fields and fingerprints as fixtures for missing-fact regression tests.

OpenAI Agents SDK Handoffs | Deep Dive Tutorial

Build Regression Tests for Missing Facts

Base every regression case on a missing-fact pattern found during trace review. Use trace checkpoints to test where context gets lost. For each required fact, build a fact-transfer matrix covering its source, sender field, receiver field, allowed transformation, dependent decision and expected response when absent.

Test Required Fields and Fact Transfer

Use synthetic fixtures only. Redact values without changing the failure mode under test.[4]

Test valid, missing, null and malformed fields. Reject unsupported schema versions unless a compatibility adapter handles them. Where transfer must be lossless, check that non-sensitive markers, types, units and provenance are reconstructed exactly. For summaries, check only the required facts: a paraphrase must not count as confirmation.

Test payloads just below, at and above each transport and input limit. Put required markers at the beginning, middle and end. Then check that they remain available after serialisation, transport and receiver parsing. Include Unicode, escaped characters and nested objects. Assert complete reconstruction, rejection or escalation - not just successful delivery.

Check whether those same facts survive denied access, retries and branch joins.

Test Permissions, Retries and Parallel Branches

Verify that authorised retrieval works and that access to state without permission is denied. Trigger timeouts, partial acknowledgements and receiver restarts. Check that retries neither overwrite newer state nor repeat an external action. Use idempotency keys or side-effect-free replay.

Delay parallel branches and inject conflicting versions. The join must use only authorised, current evidence or remain incomplete. Unavailable evidence must stop authorisation, not prompt a guessed value.

Then replay the failing path to check that the same missing fact still blocks the decision.

Replay Decisions and Test Missing Facts

Replay from the first failing boundary. Fix the receiver snapshot, evidence references and permissions, and pin the policy, prompt and model versions. Across repeated trials, check policy compliance rather than identical wording.

Remove one required fact at a time, keeping everything else constant. Require detection, retrieval, escalation or safe refusal. If a required approval threshold is absent, assert refusal or escalation. Restoring it should allow the transfer path; removing it again should block approval.

Measure Fact Retention and Decision Support

Use the same trace checkpoints from the tests above to track repeated context loss in live traffic. Compare what the sender had, what the receiver could use and what the decision used.

Measure Fact Completeness and Supported Decisions

Define the expected facts in the versioned handoff contract, then calculate:

  • Context completeness = facts available to the receiver ÷ expected required facts.
  • Fact preservation = sender-present facts retained by the receiver ÷ sender-present facts.

Count only required facts in both measures, and record pre-send omissions separately. For either measure, report a zero denominator as n/a, not 100%.

Count a decision as supported only when its required facts were available, valid and authorised. Check this through controlled tests, policy checks or authorised review - not field names alone. Store only bounded field-status labels, validation codes and trace links.[5][7][8]

Track Truncation, Stale State, Data Exposure and Routing

Use the scorecard below to investigate signals from the same sender, receiver and decision spans. Every row should record workflow, agent, contract and schema versions, plus any other relevant version scope.

Set thresholds using workflow risk, contractual requirements and measured baselines - not a universal retention target. Some workflows may allow no loss of safety-critical facts. Latency and retry targets should reflect the route’s operating requirements.[6][10]

Metric Definition Trace fields Measurement point Denominator Workflow/version scope Severity threshold Alert condition Investigation action
Context completeness Facts available to the receiver ÷ expected required facts Required-field labels, presence or keyed status Receiver validation Expected required facts Workflow, agent and schema version Critical fields: zero loss tolerated Any critical field missing or rolling rate below target Compare sender-present facts, receiver input and contract revision
Fact preservation Sender-present facts retained by the receiver ÷ sender-present facts Sender and receiver fact-status results, trace_id, handoff_id Post-receive validation Sender-present required facts Handoff type, source and target agent versions and failure mode Any loss of a mandatory fact in a high-risk route Preservation below target or a new failure mode appears Check whether filtering, truncation, mapping or permissions caused the loss
Supported decisions Decisions meeting evidence, permission and policy-support criteria ÷ evaluated decisions Decision type, required-fact status, policy-check result Decision evaluation Evaluated decisions Decision policy and workflow version Set a minimum support rate by decision class Rate falls below target or differs materially by agent version Replay sampled cases and inspect decision prerequisites
Incomplete or invalid handoffs Handoffs failing required-field, type, permission or schema checks ÷ all handoffs Validation codes, contract/schema versions, source/target, retry flag Handoff validation All handoffs Failure mode and route Zero or near-zero for critical contracts Rate exceeds baseline or any critical failure Group by validation code; identify the common producer and release
Missing field occurrences Missing occurrences for a field ÷ expected occurrences Field label, expected-occurrence rule, presence result, coverage At each checkpoint where the field should exist Expected occurrences Field, route and version Field-specific Field rate exceeds limit Check extraction, mapping, redaction and conditional logic
Truncated handoffs Handoffs exceeding limits or losing required content at a limit ÷ inspected handoffs Size class, token estimate, truncation flag, dropped-field labels Before send and after receive Inspected handoffs Model, agent and schema version Critical fields: zero loss tolerated Any critical-field truncation or rate breach Compare payload size, field order, summariser output and limits
Stale-state decisions Decisions using state older than the workflow limit ÷ evaluated decisions State timestamp, decision timestamp, freshness limit Immediately before decision Evaluated decisions Workflow and data-source version Define maximum age per data source Any breach for time-sensitive decisions Check cache, retry, replication and clock behaviour
Sensitive-field exposure Detected unauthorised exposures ÷ inspected events Policy result, field class, permission outcome, inspection status Log, tool-call and output inspection Inspected events Permission and policy version Zero tolerance for confirmed exposure Any confirmed exposure or inspection gap Restrict access, preserve redacted evidence and assess scope
Handoff latency Time between sender dispatch and receiver acceptance Start/end timestamps, queue and retry durations Sender and receiver spans Completed handoffs Route and deployment version Use workflow-specific p95/p99 targets Tail latency exceeds target Separate queueing, processing, retry and downstream time
Retried handoffs Retried handoffs ÷ handoffs Attempt number, retry reason, idempotency key Each handoff completion All handoffs Transport and agent version Set a baseline by route Rate exceeds baseline or duplicates decisions Inspect timeouts, idempotency and receiver acknowledgements
Misrouting Invalid receiver selections ÷ evaluated routing decisions Capability, permission, schema checks Receiver selection Evaluated routing decisions Routing policy and version No prohibited selections Incompatible receiver selected Inspect selection rules

Define staleness by permitted age, revision or event boundary. Count retry attempts separately from logical handoffs. These signals guide investigation; they do not prove that facts were preserved or that no exposure occurred.[6][9][11]

Break down counts and rates by handoff type, source and target versions, schema version and failure mode. Compare releases over the same reporting window. Report checkpoint coverage against all handoffs, decision-support coverage against all decisions, and privacy inspection coverage against eligible events.

Keep confirmed_absent separate from redacted, not_inspected and telemetry_missing. A missing receiver span means unknown - not confirmed fact loss. Show unknown counts and small-sample warnings alongside results. If coverage falls below its agreed minimum, alert on the evidence gap rather than reporting a healthy retention rate.[6][7][10]

Conclusion: Make Context-Loss Reviews Repeatable

Start with one high-risk delegation route. Check sender payloads, receiver inputs and later decisions against its handoff contract. Use privacy-safe telemetry to create a versioned reliability baseline.

Track the scorecard’s existing metrics: completeness, supported decisions, truncation, staleness, permission failures, branch conflicts and exposure incidents. NIST recommends testing AI systems before deployment and regularly during operation, documenting measurements, benchmarks and uncertainty where relevant.[6] Use these same metrics to select the highest-risk failure as your first regression case.

Turn Trace Findings into Regression Tests

Build the smallest reproducible failure case from that trace. Pinpoint the handoff, contract version and receiver input. Identify exactly which fact is missing, then recreate the gap with a synthetic or safely redacted fixture.

Test that the required fact survives the handoff and that the receiver either makes a supported decision or declines safely. After the fix, update the contract, review telemetry and retention controls, deploy in stages and monitor the original metric. Rerun the regression whenever the prompt, model, knowledge base, tool or orchestration changes.[13][14]

Assign Owners and Map Common Failures

Give each recurring symptom one owner and one control.

Symptom Likely transfer failure Trace evidence Corrective control
Receiver makes a decision without a mandatory fact Sender generation or summarisation drops, renames or omits a required field Required-field bitmap, contract version and protected fact fingerprint Enforce schema validation and fail closed
Evidence exists but the receiver cannot use it Retrieval failure or insufficient permission Retrieval reference, result count, authorisation outcome and policy version Repair access scope, add retrieval tests and require an evidence check
A retry uses an earlier value Stale cache, replayed message or missing state version Attempt number, state version and event timestamp Add freshness limits, version checks and idempotent processing
Parallel branches produce conflicting conclusions Branch isolation or merge logic failure Parent trace ID, branch IDs, fact fingerprints and merge outcome Define precedence rules and block the decision until conflicts are resolved

Assign clear responsibilities:

  • Contract owner: schemas, required facts and compatibility.
  • Service owner: sender and receiver implementation.
  • Evaluation owner: regression tests and release gates.
  • Observability owner: trace propagation, redaction and retention.
  • Security or privacy owner: access and data handling.
  • Incident owner: post-failure review.

Require a review of any high-risk decision involving a missing required fact, an authorisation failure or an unexplained branch conflict.[12]

FAQs

How do I identify decision-critical facts?

Connect selection events to execution records with trace links, then compare the intended action with the tool actually used. Work backwards from the final response through its dependencies to locate the first visible contract breach. At each boundary, check the facts, payload structure and policy compliance.

Use sanitised test fixtures with fixed context and versions to check inputs, identities and permissions. Record any missing evidence so you can tell whether the failure occurred during dispatch, selection or downstream state handling.

How can I verify fact use without logging sensitive data?

Prioritise metadata over raw payloads. Use correlation IDs and span links to connect sender payloads, receiver inputs and later decisions without storing sensitive content.

Redact inputs, prompts and outputs before they leave the application. Replace personal data with stable identifiers or hashes. Keep only the metadata you need - such as agent IDs, timestamps, policy versions and task outcomes - in secure, access-controlled audit logs. These logs should support checks on which facts informed each decision.

How do I distinguish context loss from missing telemetry?

First, check your instrumentation: sampling, redaction, field naming, exporter errors, ingestion delays and trace-loss rates. Missing telemetry means there’s a gap in the evidence - not proof that everything worked.

If instrumentation is intact, use shared correlation IDs to compare sender payloads, receiver inputs and later decisions without logging sensitive context. If required information is missing, that points to context loss. Confirm it by replaying the first failed event with the missing value restored.

Need help with your DevOps, cloud or AI plans?

Hokstad Consulting helps companies with DevOps transformation, cloud architecture and hands-on AI development — pragmatic consulting with measurable results.

Our services: DevOps on Retainer · Hosting & Cloud · AI Development & Strategy