The 60-point go-live checklist

Architecture and boundaries

1. Write down the one job the system must do for the user, and the one thing it must never do — get both signed off by the business owner before anyone writes code.
2. Draw the request path end to end (client, gateway, orchestrator, model, tools, data) and mark every hop where a third party can fail, rate-limit or silently change behaviour.
3. Set a latency budget per hop and per request, then measure it under concurrent load instead of on a developer laptop.
4. Decide explicitly what must be synchronous: anything the user waits for needs a streaming or partial-answer path, or it will feel broken before it is.
5. Cap every agent loop in code — maximum steps, maximum wall-clock, maximum tokens — and never rely on the prompt to enforce a limit.
6. Pin the model and its version in config, and treat a provider-side model swap as a release that has to pass the eval suite, not as a background change.
7. Define the degraded mode now: which fallback model, which reduced feature set, and which single flag flips the system into it.
8. Document which part of each answer comes from the model, which from retrieved data, and which from a deterministic rule — otherwise a wrong answer cannot be attributed to anything.

Data and retrieval

9. Confirm you have the right to send every document in the corpus to the provider you picked — licence, data-processing terms and region included.
10. Version the corpus: every index carries a snapshot ID, and every logged answer records which snapshot produced it.
11. Make ingestion re-runnable: deleting and rebuilding the index from source has to be one command, not a week of manual work you are afraid to touch.
12. Test retrieval separately from generation. A wrong answer with correct citations and a wrong answer with invented citations are two different bugs with two different fixes.
13. Write down the chunking policy and why it was chosen, then verify that each known answer sits inside one chunk instead of straddling a boundary.
14. Keep document permissions travelling with the chunks and filter at retrieval time — filtering after the model has already seen the document is not access control.
15. Build the empty-result path: when nothing clears the relevance threshold the system must say it does not know, and that path needs its own test.

Evaluation

16. Freeze a golden set of real questions with known-good answers before tuning anything, so improvements are measured against a fixed target.
17. Score with two independent methods — deterministic checks plus a model grader — and read every case where the two disagree.
18. Track retrieval quality (recall at k, hit rate) separately from answer quality (faithfulness, correctness); a single blended score hides which half regressed.
19. Re-run the same suite on every prompt, model or index change. If a change cannot be measured, it does not ship.
20. Keep a regression list of failures found in production and add each one to the golden set the day it appears, while the details are still accurate.
21. Evaluate citation behaviour on its own: if a citation is required, an answer without one is a failure even when the text happens to be right.
22. Test the worst realistic input, not the clean one — typos, mixed languages, pasted tables, contradictory instructions and outright prompt-injection attempts.
23. Blind the grading order, or grade the two systems in separate sessions, so whoever scores is not anchored by having just read the previous answer.

Observability

24. Log one trace per request containing prompt version, model version, index snapshot, retrieved chunk IDs, token counts and total latency.
25. Give the user a request ID they can quote and put that same ID on every log line the request touches.
26. Retain the prompt and response pair for a sample of traffic, consistent with your privacy notice — you cannot debug a reply you never recorded.
27. Alert on error rate, p95 latency and empty-retrieval rate at the same time; a silent quality drop costs more trust than a loud outage.
28. Run a scheduled canary: the same question every day, asserting an expected answer shape, so a broken provider surfaces before a user finds it.
29. Track tokens, latency and cost per feature and per request, not only as a monthly total that nobody can attribute.
30. Give the front-line team a read-only way to open the trace of a bad request without waiting for an engineer.

Security and access

31. Keep every provider key server-side. A key shipped in a browser bundle is a published key, no matter how it is obfuscated.
32. Treat retrieved documents and tool output as untrusted input: strip instructions and active content before either reaches the prompt or the page.
33. Enforce authorisation for each tool call at the API layer, so the model cannot widen its own permissions by asking nicely.
34. Redact or hash personal data before it crosses your boundary, and write down which destinations are allowed to receive it.
35. Require explicit human confirmation for anything with a side effect: writing, sending, paying, deleting, changing a record.
36. Set rate limits and spend caps per key and per user, and make the cap a circuit breaker that stops calls rather than a dashboard someone watches.
37. Rotate credentials used during development and revoke them at go-live; a test key must never be able to reach production data.
38. Check that system instructions and internal rules cannot be extracted by a user, and re-check it after every prompt change.

Cost control

39. Compute the full cost of one request — retrieval, input tokens, output tokens, tool calls — and compare it against the value of one resolved request.
40. Set a monthly budget with alerts partway up, and decide in advance what shuts off when the ceiling is reached.
41. Cache where the inputs are genuinely stable (embeddings, retrieval results, repeated prompts) and measure the hit rate instead of assuming it works.
42. Route easy requests to a cheaper model and hard ones to the stronger one, but only after the eval shows the cheap path holds up.
43. Cap context size in code. Slowly growing prompts are the most common cost leak, and the hardest to notice.
44. Report cost per active user per month, not cost per million tokens — only the first number can be compared with a budget line.
45. Build the kill switch: one flag that stops all model calls and serves the degraded experience, tested before you need it.

Go-live process

46. Run the full eval suite against the exact build that ships, not against the branch that passed last week.
47. Get written sign-off on the acceptance criteria from the business owner before launch day, so success is not renegotiated afterwards.
48. Prepare the user-facing copy: what the system does, what it cannot do, what it costs, and where to complain when it is wrong.
49. Brief the front-line team with three real examples and three real failure modes before the first user arrives, not after the first complaint.
50. Launch to a limited audience — one team, one workflow, one country — and hold it there until the numbers stop being interesting.
51. Name the person who owns the system after launch, with the authority and the documentation to switch it off at 2am.
52. Confirm retention and deletion behaviour matches what your privacy notice, contract and DPA actually promise.
53. Freeze non-essential changes for a short window after launch so a regression can only have come from the launch itself.

Rollback and incidents

54. Version every deployable artefact separately — prompt, index, config and code — and make the running combination visible in one place.
55. Rehearse the rollback before launch, in production, executed by someone who did not build the system, following only the runbook.
56. Make rollback a flag wherever possible; a rollback that needs a rebuild takes longer than the incident it is supposed to contain.
57. Keep the previous index snapshot warm so switching back does not mean re-ingesting the corpus under pressure.
58. Define what degraded-but-safe looks like — a read-only mode with an honest message beats a mode that keeps answering confidently and wrongly.
59. Agree the incident triggers in advance: which metric, which threshold, who decides, and who is told.
60. After every incident, add the failure to the eval set and the canary before the ticket is closed, then re-read this list and fix the line that was skipped.