Memory Retrieval Tests Track Agent Answer Quality | Hokstad Consulting

Memory Retrieval Tests Track Agent Answer Quality

Memory Retrieval Tests Track Agent Answer Quality

A correct answer does not prove an agent found the right memory. Test retrieval and answers separately using fixed, labelled conversation histories, and track response times and cost in pounds.

A useful test checks:

  • What the agent finds: required facts, missed memories and irrelevant results.
  • What it answers: factual accuracy, use of current facts and whether it admits when information is missing.
  • What it costs: median and p95 response times, plus pounds per attempted case and per correct answer.
  • Whether it can be released: repeat runs, untouched test cases and preset quality, privacy and cost limits.

Include changed preferences, distant conversations, deleted records and questions with no known answer. Require zero cross-user memory access, and release only when every required check passes.

::: @figure Agent Memory Testing: From Conversations to Release{Agent Memory Testing: From Conversations to Release} :::

Using LongMemEval to Improve Agent Memory

2. Build Tests from Past Conversations

Turn past conversations into labelled test cases, using only records the organisation is authorised to evaluate. Preserve their original order and give each user or organisation a separate namespace. Replace names, contact details and unnecessary unstructured personal data with stable placeholders. Record the source conversations, relevant turns and expected facts, but keep the reference answer out of the agent input.[6][7]

Document how long records will be retained and how they will be deleted or anonymised. Apply role-based access to annotations, reports and raw conversations: all three may contain personal information.[10] With these controls in place, define each test’s expected facts and excluded memories.

Label Expected Facts and Excluded Memories

Hypothetical delivery case: On 4 March 2026, the customer says, “Saturday delivery would be convenient.” On 18 March 2026, they say, “Please update my preference: I need delivery on a weekday, not at the weekend.” On 25 March 2026, they ask, “Do you know what time the parcel will arrive?”

Test query: “What delivery preference do I currently have, and what time will the parcel arrive?” Expect the weekday request and latest parcel enquiry; exclude the superseded Saturday preference. The answer must state weekday delivery and acknowledge that no confirmed delivery time is available, without inventing an hour.

Use a compact, versioned schema such as:

{
  "case_id": "delivery-001",
  "conversation_ids": ["conv-104", "conv-119"],
  "relevant_turns": ["conv-104:t8", "conv-119:t3"],
  "query": "When should my parcel be delivered, and what time will it arrive?",
  "expected_memory_ids": ["mem-104-08", "mem-119-03"],
  "reference_facts": [
    "The customer requested weekday delivery.",
    "The customer later cancelled the earlier Saturday preference."
  ],
  "answer_criteria": [
    "States that delivery should occur on a weekday.",
    "Does not use the superseded Saturday preference.",
    "Says that no delivery time is available rather than inventing one."
  ],
  "excluded_memory_ids": ["mem-099-02"],
  "annotation_version": "1.0"
}

Use expected_memory_ids for the smallest supporting units and reference_facts for evaluator wording. Put required, prohibited and uncertain content in answer_criteria. Use excluded_memory_ids for obsolete, irrelevant, other-user or withheld memories. These labels set the baseline for retrieval precision, missed-memory and answer-quality checks.

Assign one shared fact_id to approved equivalents. Different wording does not automatically mean a different fact.

For example, “Please deliver on weekdays”, “Tuesday to Friday works” and “Avoid weekend delivery” may support the same canonical fact, delivery_days: weekdays, if the business rules confirm that interpretation.

Have at least two trained annotators assess difficult cases. Require adjudication when they disagree about chronology, equivalence, supersession, privacy or whether the question can be answered, and retain the agreed reasoning. Record borderline equivalence decisions and review them regularly so changes to business rules do not silently shift test outcomes.

Test Distant, Conflicting and Missing Facts

Include short-range, long-range, multi-session, conflicting and no-memory cases. Mix direct questions with questions that combine several requests. Add negative controls for unsupported details, expired instructions and cross-user records.[11]

Version Data and Reserve Held-Out Cases

Split development and held-out cases before tuning to prevent leakage and overfitting, leaving an independent check of release readiness. Keep related conversations from the same user or organisation together wherever overlap could leak answers.

Version source snapshots, redaction rules, annotations, equivalence mappings, exclusions and dataset splits. Record preference changes in a new dataset revision. Include the source snapshot timestamp and each question’s cut-off point so later messages cannot supply its answer.

Never use held-out results to choose retrieval settings, rewrite annotations or design special handling. If those results guide a change, use a new, untouched set for final evaluation.[8][9] Lock these versions before scoring retrieval and answer accuracy, with the fixed labels as the benchmark.

3. Measure Retrieval, Answers, Latency and Cost

Use the test set’s labelled expected facts to score retrieval, answer quality, latency and cost separately.

Measure Precision, Recall and Missed Memories

Score each question, turn or replay once. Define the denominators before testing, then check each labelled case against its expected facts and excluded memories.

For each case, let E be the expected set and R the retrieved set. Calculate precision as |R ∩ E| / |R| and recall as |R ∩ E| / |E|. Remove exact duplicates, group equivalent paraphrases under one canonical fact, and keep contradictory versions separate. For cases with expected memories, report which memories were missed, how many were missed and the missed-memory rate: 1 − recall. In no-memory cases, any retrieval is false; an empty result is correct. Record precision as not applicable and report the false-retrieval rate separately.[13]

Report micro scores using pooled counts. For macro scores, calculate scores within each conversation, then average across conversations. State how many cases each measure includes.

Report Recall@k as required-fact coverage, alongside the percentage of cases where retrieval found every required fact. Report MRR for the first relevant result, scoring zero when none appears. For graded relevance, use nDCG@k with scores of 2 for direct evidence, 1 for useful support and 0 for irrelevant content. Exclude no-memory cases from recall and ranking averages.[12][13]

Retrieval scores tell you whether the agent found the right facts. Answer scoring checks what it did with them.

Score Answers and Diagnose Failures

Use a separate rubric for accuracy, completeness, relevance, faithfulness to evidence and abstention. Score each dimension as 0 for wrong, 1 for partial and 2 for correct. Flag unsupported claims explicitly.

Define answer accuracy as the percentage of answers meeting all required factual criteria. Define faithfulness as the percentage whose factual claims are supported by the supplied evidence. An answer can be faithful yet still leave out required facts.

No-memory cases receive full credit only when the agent correctly says it cannot verify the answer or does not know. Confident invention is a failure. Validate automated judges against a reviewed sample, compare their scores with human labels, and send ambiguous cases for human review.[3]

Retrieval evidence Answer outcome Likely source Follow-up check
Required facts retrieved Wrong or incomplete Retrieved memories misused or ignored Inspect instructions, context ordering and response parsing
Required facts missing Wrong or incomplete Memories not stored or retrieved Check ingestion, indexing, metadata filters, embedding search and top-k limit
Irrelevant evidence retrieved Unsupported or confused Answer grounded in irrelevant memories Test stricter filtering, reranking and abstention
Correct answer without required memory Shortcut, leakage or incomplete labels Answer bypasses required memories Remove answer leakage, inspect system prompts and review expected-fact labels

Compare Quality, Latency and Cost in Pounds

After scoring quality, compare what each configuration costs to run.

Record end-to-end, search, reranking, prompt-building and generation timings. Report median and p95 latency. Log returned-memory counts, context length, input tokens, output tokens, total tokens, model and prompt versions, calls, retries, cache use and errors.[14]

Cost per case must include every billable attempt, retrieval charges, and allocated storage and infrastructure costs. Record the pricing date, allocation method, exchange rate where needed and VAT treatment. Compare configurations on the held-out set, keeping untouched cases for final sign-off.

Illustrative figures supplied in the brief - not measured results. Each configuration uses 250 cases and a store of 50,000 memories. Prompt tokens are per-case averages. Accuracy and faithfulness use the pass-rate definitions above; failure means any incorrect, incomplete, unfaithful or improperly non-abstaining answer.

Configuration Precision Recall@5 Accuracy Faithfulness Median / p95 latency Prompt tokens Failure rate Cost/case
Vector retrieval 78% 84% 81% 88% 420 ms / 1.2 s 1,900 12% £0.006
Hybrid plus reranking 86% 91% 89% 93% 610 ms / 1.7 s 2,100 7% £0.010
Larger context 73% 93% 88% 85% 760 ms / 2.4 s 4,800 9% £0.018

Use extra context only when the gain in answer quality justifies the cost and delay. Compare distant-fact, conflicting-fact, multi-fact and no-memory cases separately rather than relying only on overall averages. Use paired comparisons and confidence intervals before treating small differences as reliable. Choose the smallest context limit that meets the agreed quality thresholds.

4. Repeat Tests and Set Release Criteria

Replay Conversations and Record Results

Once the test set is fixed, rerun it unchanged to check that the gains hold across runs. Version the dataset, prompts, model, retrieval settings and scoring rules. Keep operating limits the same except for the change under test. Give each independent case its own memory namespace, retaining memory across turns and sessions where needed. Replay turns in order without revealing the expected facts.[17][18]

Add the run ID, software version, configuration hash and timestamp to the logs described earlier. Limit access to authorised evaluators, redact personal data where possible and encrypt sensitive records. Score retrieval and answer quality separately, then repeat each configuration under the same controlled settings. Report the run count and confidence intervals. Bootstrap by conversation to keep related turns together. Rerun tests after changes to the model, prompts, infrastructure, embeddings or memory schema.[16][17]

Group Failures by Cause

Break down results by turn distance, number of conversations, fact type and conflict status. Use the traces to find where each failure starts: a missing fact points to storage; a stored fact that is not returned points to retrieval or ranking; and a returned fact that goes unused points to generation. Track stale-fact errors separately from unsupported claims.[15][18]

Rank repeated failures by severity, frequency, affected users and reversibility. Give each an owner and a regression test. Address privacy breaches, unsafe answers and cross-user retrieval first. Strong same-session results must not hide poor recall across multiple sessions.[15][18]

Set Quality, Privacy and Cost Limits

After replaying the runs, set the pass/fail gates before reviewing the results. Treat the table below as a template, not a benchmark. Base thresholds on business risk and require every important subgroup to pass. Privacy-sensitive or safety-critical regressions override average gains. Any cross-user access blocks release.[18][19]

Illustrative release gates - not measured results.

Metric Baseline result Candidate result Change Release threshold
Memory recall@5 86.0% 91.5% +5.5 pp ≥90%
Retrieval precision@5 78.0% 81.0% +3.0 pp ≥80%
Answer accuracy 88.0% 92.0% +4.0 pp ≥90%
Faithfulness 93.0% 94.5% +1.5 pp ≥94%
False retrieval rate 3.2% 2.1% −1.1 pp ≤2.5%
p95 latency 1.80 s 2.05 s +0.25 s ≤2.20 s
Cost per interaction £0.014 £0.017 +£0.003 ≤£0.020
Cross-user memory access 0 0 No change 0 permitted

pp means percentage points. Reject narrow passes until confidence intervals clear the threshold. Keep every gate measurable, whoever implements it.

5. Conclusion: Test Memory Before Release

Once the gates are set, use the held-out suite as the final release check. Start with a small, representative set of conversations used with consent or anonymised. Give each case a stable ID, and label expected facts, excluded facts and acceptable answers. Keep these cases - and close paraphrases - out of development work.[9][5]

A correct answer alone doesn’t prove retrieval worked. Score retrieval separately from answer quality by measuring retrieval precision, missed memories and answer accuracy. Track p50 and p95 latency, plus cost in pounds per attempted case and per correct answer.[20][4][22]

Turn reviewed failures into new test cases that have also passed a privacy review. Keep earlier dataset versions unchanged.[9][5] Before releasing any model, prompt or memory changes, run the full workflow against the suite. Release only when every mandatory quality, privacy, latency and cost gate passes.[9][21][23]

FAQs

How many conversations do I need for reliable tests?

Start with 20 to 50 representative tasks, covering successful runs, failed attempts and complex edge cases. Use this set to define quality requirements and set routing thresholds [1].

Once you’ve tested your policy on the initial set, test it against a larger evaluation dataset. Then use a separate holdout set to check overall system reliability [1]. Keep monitoring and testing to ensure results stay grounded, accurate and cost-effective [2].

How do I set realistic memory quality thresholds?

Set memory quality thresholds based on business risk and observed failure patterns, not arbitrary similarity scores. Use a risk-coverage curve to weigh errors against unnecessary refusals. Compare memory performance against a fixed no-memory baseline, tracking retrieval precision, regressions and contradictions.

Require reliable evidence for material claims. Set clear limits for spending, latency and human review, using representative historical data where needed. Document who is responsible for each area, and review thresholds after changes to models, tools or workloads.

How do I calculate cost per correct answer?

Calculate cost per success by dividing the total delivery cost by the number of accepted, successful tasks. Include model inference, tool usage, infrastructure, retries, recovery work and human review time, plus all billable supervisor and worker calls. Count each accepted task only once, no matter how many retries it takes.

As an approximation, divide the cost per attempt by the success rate:

If 10% of attempts fail, divide the cost per attempt by 0.90.

Need help with your DevOps, cloud or AI plans?

Hokstad Consulting helps companies with DevOps transformation, cloud architecture and hands-on AI development — pragmatic consulting with measurable results.

Our services: DevOps on Retainer · Hosting & Cloud · AI Development & Strategy