AI Partner Track Records: Checking Post-Launch Support | Hokstad Consulting

AI Partner Track Records: Checking Post-Launch Support

AI Partner Track Records: Checking Post-Launch Support

Check an AI partner’s support record before signing - not just its launch promises. Speak to at least two clients with systems live for six months or more, then check their accounts against redacted tickets, monitoring records and invoices from the past 6–12 months.

The review should establish:

  • Support terms and costs: named owners, UK cover hours, separate response and resolution targets, and extra charges in £.
  • Incident handling: how failures were contained, service restored and repeat faults prevented.
  • Update controls: testing, approval, version records and rollback arrangements.
  • Team independence: whether internal developers can deploy changes and recover service using the supplied documentation and access.
  • Exit terms: rights to code and other materials, handover support, data return and deletion.

Mark findings as verified, partially verified, unverified or contradicted. Put unresolved gaps, deadlines and acceptance tests into the contract. Strong results elsewhere should not cancel out a critical gap in recovery, update control or independent handover.

::: @figure How to Verify an AI Partner’s Post-Launch Support{How to Verify an AI Partner’s Post-Launch Support} :::

Evaluating AI Vendors: The Red Flags, Questions, and Evidence You Need

2. Choose Client References and Plan Interviews

Use the written support commitments from Section 1 to check how the partner performs in live use.

Choose Clients with Similar Live Systems

Request at least two references whose systems have been live for six months or more. Match them on data sensitivity, integration complexity, user volumes and business importance - not brand recognition. Check each system’s go-live date, current production status, adoption trend and current support model. Confirm that the same team supports the live system.

Ask the partner to arrange permission for a direct conversation. Send an interview note explaining the purpose, duration, topics, recording plans, storage and access. Offer an unrecorded interview, and request only redacted or aggregate records. Get permission before recording or attributing quotations.

Use these conversations to check whether the promised support model works in production.

Ask About Support Requests, Failures and Changes

Follow the same sequence in every interview. Start with the most recent routine request and urgent incident. Record when each was reported, who responded, what they did and how long recovery took. Cover degraded outputs, integration failures and changing requirements.

Treat acknowledgement, restored service and permanent resolution as separate milestones. Ask whether the client had to chase updates, whether the first fix worked and whether the fault returned. Request the relevant ticket and root-cause report.

For the latest model or prompt change, ask what changed, how it was tested, who approved it and whether rollback was available. Check whether the documentation was updated too. Find out when internal developers last resolved an issue without help and which instructions they followed.

Finish by asking whether the client would choose the partner again for support, why, and what they would change in the support terms. Use the scorecard to compare each answer with dated evidence.

Record Interview Answers in a Scorecard

Create one scorecard per reference using the rows below. Record the speaker’s role and mark claims as verified, partially verified, unverified or contradicted. Keep opinions separate from documented facts.

Check conflicting dates or accounts with both parties rather than hiding them in an average score. Give every outstanding check an owner and a deadline.

Assessment area Reference identifier Answer Dated evidence Approved quotation Verification status Follow-up action
Production status Client code Go-live date; live now? Production or service report Permission and approved wording Record status Verify current operation
Live usage Client code Usage, trend and workarounds Dated usage report Permission and approved wording Record status Explain declining use
Response quality Client code Urgent and routine request handling Ticket timestamps Permission and approved wording Record status Check unexplained delays
Technical skill Client code Diagnosis and lasting fix Root-cause report; change record Permission and approved wording Record status Investigate repeat faults
Transparency Client code Disclosure of impact and limitations Incident communications Permission and approved wording Record status Resolve conflicting accounts
Handover Client code Tasks developers perform unaided Runbook; internal recovery record Permission and approved wording Record status Verify independent operation

3. Check Incident Response and Model Updates

Build a Recent Incident Timeline

Use the references gathered in Section 2 to trace one recent incident and one near miss for each client, from report to closure. Track acknowledgement, diagnosis, containment, restoration, permanent fix, validation and closure.

Assess post-launch support using live incident records. Record dates and times, noting whether timestamps use UTC or UK local time. Where records exist, cover outages, unsafe outputs, failed integrations, data-quality changes and performance degradation. Label tabletop scenarios separately: they show preparedness, not live performance.

Review the post-incident report for impact, root cause, contributing factors, missed alerts and corrective actions. Each action should have an owner and deadline. Check whether completed actions led to regression tests or monitoring rules. Repeated faults with the same cause should lower the score, even if every ticket was closed promptly.[4]

Compare Response and Resolution Times with Targets

Complete the table using anonymised records, rather than recollection alone. Compare observed times with the contractual targets set in Section 1. Record both total elapsed time and contract clock time, identifying exclusions for evenings, weekends, bank holidays, third-party dependencies and time awaiting client action.

Restored service is not a permanent fix. A fallback can make the system usable while the underlying fault remains.

Incident type Milestone Target Observed elapsed / contract clock time Impact Evidence Verification status
Availability outage Acknowledgement Contract target Time to acknowledgement Users affected; downtime Ticket; monitoring logs Pending evidence
Unsafe or incorrect output Containment Contract target Time to containment Exposure; manual review needed Incident report; disablement log Pending evidence
Integration failure Service restoration Contract target Time to restoration Queued or failed transactions Integration logs; recovery record Pending evidence
Performance degradation Permanent fix Contract target Time to permanent fix Delays; staff rework Change record; validation results Pending evidence

Check whether later releases were tested, approved and rolled back in line with the written support terms.

Check Update Testing, Approval and Rollback

Compare the latest completed release in each category below. Link version IDs to tests, approvals and release notes. Check who can reject or defer deployment, and whether release notes and notices reached the agreed recipients.

Change type Testing Approval Notification Rollback Evidence
Routine maintenance Regression, security and integration checks Named technical authority Maintenance window Previous package or configuration Test results; approval; release note
Emergency fix Focused safety, security and smoke tests Emergency authority; retrospective review Immediate incident update Tested bypass or last known-good release Emergency record; logs; review
Material model change Quality, safety, robustness, latency and cost against baseline Business, risk and technical owners Advance impact and migration notice Previous version where available; fallback or disablement Evaluation report; version record; monitoring results

Ask for the last rollback record. Check its duration, data-loss risk and whether it restored compatible prompts, retrieval indexes, configuration and schemas - not just application code.

Review upstream notice periods, the version retirement policy and approval limits. Record unavailable controls as residual risks. Require post-release monitoring with named owners and rollback triggers. Use the NIST AI RMF and its Generative AI Profile as voluntary references for monitoring, recovery and change control.[1][6]

4. Test Documentation and Developer Handover

After testing incident response and rollback, check whether internal teams can use the handover materials to run, recover and change the system without the partner’s help.

Review Handover Materials

Ask reference clients to show what their developers can do with the handover pack - not how many documents it contains. Check that versioned, searchable records cover purpose, permitted and prohibited uses, assumptions, limits, validation results, versions, datasets, metrics, thresholds and trade-offs. Compare the documented accuracy, latency, cost and failure rates with the last three months of monitoring and evaluation records.[3][8]

Use the table below to record a named owner, evidence-backed status and remaining dependency for each item. Start every status as Unverified and update it as you check the evidence. Accept an artefact only when it passes its test; written confirmation alone does not establish acceptance.

Artefact Owner Current status Acceptance test Remaining dependency
Architecture and data flows Technical lead Unverified Trace a request through retrieval, model invocation, logging and downstream action; explain limits and failure points Undocumented processing or failure points
Source code, configuration and infrastructure definitions Engineering lead Unverified Build a working test environment from the supplied files Supplier-only scripts or accounts
Model, prompt, retrieval and evaluation records AI lead Unverified Reproduce a recorded evaluation using the documented versions and dataset Missing versions or restricted datasets
APIs and third-party dependencies Integration owner Unverified Locate interface specifications, versions and failure-handling instructions Inaccessible subscriptions or connectors
Monitoring and incident runbooks Service owner Unverified Diagnose a controlled fault and restore service from the runbook Supplier-only dashboards or tacit know-how
Security, access and retention controls Security lead Unverified Verify permissions, credential rotation and retention settings Shared credentials or inaccessible controls
Test suites Test lead Unverified Run documented tests and interpret failures Missing fixtures or test data
Deployment and rollback procedures Release owner Unverified Deploy and reverse a change in a test environment using only transferred materials Missing packages or permissions
Ownership and licence rights Commercial owner Unverified Obtain written confirmation of rights to use, modify and maintain transferred materials Restricted components or unclear rights
Unresolved issues Service owner Unverified Check each issue has impact, severity, owner, workaround and target resolution date Untreated risks or overdue actions

Next, test whether internal staff can use those materials without help.

Test Internal Ownership and Exit Support

Ask whether the client’s developers can deploy a change, investigate an incident and restore service using only the handover materials. Request dated agendas or notes, attendance lists, exercises, training materials, open questions and acceptance sign-off. Attendance is not proof of capability. Ask two internal developers who did not lead the original implementation to repeat the agreed exercises independently.[5]

Run the exercises in a controlled, non-production environment. Put a small change through testing, approval and deployment, then introduce a safe fault for investigation and recovery.

Examples include updating a prompt or API configuration.

Record pull requests, test output, logs, elapsed time, blocked steps and any partner help. Compare detection, diagnosis and recovery times with the contractual target.[1]

Check that the organisation - not just the supplier - has administrator or suitably privileged access to the repository, deployment environment, accounts, monitoring tools, data stores and support portal. Test access in a controlled environment using named internal users, least-privilege roles and multi-factor authentication.

Confirm in writing who owns the bespoke code, prompts, datasets, documentation, infrastructure, test suites, dashboards and configuration. Check the relevant licences and use rights, including any restrictions that prevent independent operation after exit. Ask whether the client receives repository history, build scripts and the right to appoint another support provider.

Set out the delivered materials, access-transfer deadlines, knowledge-transfer sessions, transition response times, maximum assisted-support period and extra-work rates. Also specify data return and deletion, cooperation with a replacement supplier and final acceptance tests.[2][7]

Include a controlled exit rehearsal: the internal team or replacement provider must deploy a change, investigate a simulated incident and restore service using the transferred materials. Assign every remaining dependency an owner, workaround and target resolution date. Mark claims as verified, partially verified or unverified, and accept them only when the evidence supports doing so.

5. Conclusion: Record Findings and Contract Conditions

Rate Each Area by Verification Status

Turn the evidence from Sections 1–4 into a final contract decision. Mark each area verified, partially verified or unverified. These ratings cover only the records examined - not future performance. For every finding, record the source, date and any limits.

Base ratings on interview notes, tickets, incident logs and handover tests. Supplier claims alone aren't enough.

Assessment area Status Evidence note to complete Specific pre-contract action
Support commitments Unverified Signed service levels, escalation routes, support hours and exclusions Add response, restoration and resolution targets.
Response performance Unverified Client ticket logs showing acknowledgement and action times Review anonymised tickets from the past 6–12 months and compare times with targets.
Update control Unverified Change records, test results, approvals and rollback plans Require material updates to be tested, approved and reversible.
Fault resolution Unverified Incident timeline, root-cause analysis and corrective actions Specify severity-based recovery and reporting obligations.
Documentation Unverified Current architecture, runbooks, model information and data-flow records Set contractual documentation acceptance criteria.
Handover independence Unverified Internal team completing routine tasks without supplier intervention Make independent handover testing an acceptance condition.
Reference reliability Unverified Comparable production clients and corroborated answers Check references from comparable production clients and corroborate their answers.

One critical gap can't be offset by strong ratings elsewhere. If recovery targets, update controls or the internal team's ability to work without supplier help remain unsupported, keep approval conditional.

Agree Acceptance Conditions Before Signing

Use unresolved items from the scorecard and evidence review to set the conditions for signing. For each gap, record the missing evidence, a named owner, a deadline and what happens if the item isn't accepted. Attach this action register to the decision record.

State which gaps block signature and which need an explicit stage gate, a deadline for fixes or a withheld payment milestone. Put support targets, records access, update approval, documentation acceptance and exit assistance in the contract, not just in meeting notes.

Apply the same checks to every shortlisted provider, including Hokstad Consulting for AI strategy, implementation or custom agent work. This article does not verify its support record. Base approval on documented commitments, tested handover and production-client evidence, rather than broad support promises.

FAQs

What if client support records are confidential?

Redact personal or sensitive information before review, replacing it with consistent placeholders or hashes. Use correlation IDs to trace workflows, triggers and outcomes without keeping sensitive inputs or payloads.

Store logs in tamper-resistant, append-only systems. Apply strict role-based access controls so only authorised evaluators can access them. Encrypt sensitive records at rest, and clearly document data retention and anonymisation policies to meet UK GDPR requirements.

How do I set realistic AI recovery targets?

Measure current handling time, error rates and rework costs. Use RTO, RPO and RTA to spot gaps and set targets based on business risk, operational requirements and representative historical data - not arbitrary thresholds.

Define clear success metrics, assign owners and review the evidence regularly. Track both infrastructure health and the effect on the full workflow. Adjust targets after model changes or serious incidents.

How often should I retest developer handover?

Retest developer handovers and related controls after material changes to models, prompts, infrastructure, embeddings, memory schemas, regulation, policy or priorities. Check handover processes before launch, then monitor performance through regular governance reviews.

After a failure or breach, retest immediately to confirm that corrective actions work before resuming standard operations. Use controlled, consistent test settings and record the results to keep evidence ready for audit.

Need help with your DevOps, cloud or AI plans?

Hokstad Consulting helps companies with DevOps transformation, cloud architecture and hands-on AI development — pragmatic consulting with measurable results.

Our services: DevOps on Retainer · Hosting & Cloud · AI Development & Strategy