# The 60-point go-live checklist

The 60-point checklist for shipping an LLM feature: architecture, data and retrieval, evaluation, observability, security, cost, go-live and rollback. Free, no email required.

v1.0 · updated 2026-09-25 · 60 points in 8 sections
Free resource — https://fdenotes.com/go-live-checklist/ (no email required)

> This is opinionated engineering practice, not a benchmark report. Where a number appears it is a starting threshold to argue about, not a measured result of ours. Nothing here is gated: no email required, share it, copy it into your own runbook.

## Architecture and boundaries

- [ ] Write down the one job the system must do for the user, and the one thing it must never do — get both signed off by the business owner before anyone writes code.
- [ ] Draw the request path end to end (client, gateway, orchestrator, model, tools, data) and mark every hop where a third party can fail, rate-limit or silently change behaviour.
- [ ] Set a latency budget per hop and per request, then measure it under concurrent load instead of on a developer laptop.
- [ ] Decide explicitly what must be synchronous: anything the user waits for needs a streaming or partial-answer path, or it will feel broken before it is.
- [ ] Cap every agent loop in code — maximum steps, maximum wall-clock, maximum tokens — and never rely on the prompt to enforce a limit.
- [ ] Pin the model and its version in config, and treat a provider-side model swap as a release that has to pass the eval suite, not as a background change.
- [ ] Define the degraded mode now: which fallback model, which reduced feature set, and which single flag flips the system into it.
- [ ] Document which part of each answer comes from the model, which from retrieved data, and which from a deterministic rule — otherwise a wrong answer cannot be attributed to anything.

## Data and retrieval

- [ ] Confirm you have the right to send every document in the corpus to the provider you picked — licence, data-processing terms and region included.
- [ ] Version the corpus: every index carries a snapshot ID, and every logged answer records which snapshot produced it.
- [ ] Make ingestion re-runnable: deleting and rebuilding the index from source has to be one command, not a week of manual work you are afraid to touch.
- [ ] Test retrieval separately from generation. A wrong answer with correct citations and a wrong answer with invented citations are two different bugs with two different fixes.
- [ ] Write down the chunking policy and why it was chosen, then verify that each known answer sits inside one chunk instead of straddling a boundary.
- [ ] Keep document permissions travelling with the chunks and filter at retrieval time — filtering after the model has already seen the document is not access control.
- [ ] Build the empty-result path: when nothing clears the relevance threshold the system must say it does not know, and that path needs its own test.

## Evaluation

- [ ] Freeze a golden set of real questions with known-good answers before tuning anything, so improvements are measured against a fixed target.
- [ ] Score with two independent methods — deterministic checks plus a model grader — and read every case where the two disagree.
- [ ] Track retrieval quality (recall at k, hit rate) separately from answer quality (faithfulness, correctness); a single blended score hides which half regressed.
- [ ] Re-run the same suite on every prompt, model or index change. If a change cannot be measured, it does not ship.
- [ ] Keep a regression list of failures found in production and add each one to the golden set the day it appears, while the details are still accurate.
- [ ] Evaluate citation behaviour on its own: if a citation is required, an answer without one is a failure even when the text happens to be right.
- [ ] Test the worst realistic input, not the clean one — typos, mixed languages, pasted tables, contradictory instructions and outright prompt-injection attempts.
- [ ] Blind the grading order, or grade the two systems in separate sessions, so whoever scores is not anchored by having just read the previous answer.

## Observability

- [ ] Log one trace per request containing prompt version, model version, index snapshot, retrieved chunk IDs, token counts and total latency.
- [ ] Give the user a request ID they can quote and put that same ID on every log line the request touches.
- [ ] Retain the prompt and response pair for a sample of traffic, consistent with your privacy notice — you cannot debug a reply you never recorded.
- [ ] Alert on error rate, p95 latency and empty-retrieval rate at the same time; a silent quality drop costs more trust than a loud outage.
- [ ] Run a scheduled canary: the same question every day, asserting an expected answer shape, so a broken provider surfaces before a user finds it.
- [ ] Track tokens, latency and cost per feature and per request, not only as a monthly total that nobody can attribute.
- [ ] Give the front-line team a read-only way to open the trace of a bad request without waiting for an engineer.

## Security and access

- [ ] Keep every provider key server-side. A key shipped in a browser bundle is a published key, no matter how it is obfuscated.
- [ ] Treat retrieved documents and tool output as untrusted input: strip instructions and active content before either reaches the prompt or the page.
- [ ] Enforce authorisation for each tool call at the API layer, so the model cannot widen its own permissions by asking nicely.
- [ ] Redact or hash personal data before it crosses your boundary, and write down which destinations are allowed to receive it.
- [ ] Require explicit human confirmation for anything with a side effect: writing, sending, paying, deleting, changing a record.
- [ ] Set rate limits and spend caps per key and per user, and make the cap a circuit breaker that stops calls rather than a dashboard someone watches.
- [ ] Rotate credentials used during development and revoke them at go-live; a test key must never be able to reach production data.
- [ ] Check that system instructions and internal rules cannot be extracted by a user, and re-check it after every prompt change.

## Cost control

- [ ] Compute the full cost of one request — retrieval, input tokens, output tokens, tool calls — and compare it against the value of one resolved request.
- [ ] Set a monthly budget with alerts partway up, and decide in advance what shuts off when the ceiling is reached.
- [ ] Cache where the inputs are genuinely stable (embeddings, retrieval results, repeated prompts) and measure the hit rate instead of assuming it works.
- [ ] Route easy requests to a cheaper model and hard ones to the stronger one, but only after the eval shows the cheap path holds up.
- [ ] Cap context size in code. Slowly growing prompts are the most common cost leak, and the hardest to notice.
- [ ] Report cost per active user per month, not cost per million tokens — only the first number can be compared with a budget line.
- [ ] Build the kill switch: one flag that stops all model calls and serves the degraded experience, tested before you need it.

## Go-live process

- [ ] Run the full eval suite against the exact build that ships, not against the branch that passed last week.
- [ ] Get written sign-off on the acceptance criteria from the business owner before launch day, so success is not renegotiated afterwards.
- [ ] Prepare the user-facing copy: what the system does, what it cannot do, what it costs, and where to complain when it is wrong.
- [ ] Brief the front-line team with three real examples and three real failure modes before the first user arrives, not after the first complaint.
- [ ] Launch to a limited audience — one team, one workflow, one country — and hold it there until the numbers stop being interesting.
- [ ] Name the person who owns the system after launch, with the authority and the documentation to switch it off at 2am.
- [ ] Confirm retention and deletion behaviour matches what your privacy notice, contract and DPA actually promise.
- [ ] Freeze non-essential changes for a short window after launch so a regression can only have come from the launch itself.

## Rollback and incidents

- [ ] Version every deployable artefact separately — prompt, index, config and code — and make the running combination visible in one place.
- [ ] Rehearse the rollback before launch, in production, executed by someone who did not build the system, following only the runbook.
- [ ] Make rollback a flag wherever possible; a rollback that needs a rebuild takes longer than the incident it is supposed to contain.
- [ ] Keep the previous index snapshot warm so switching back does not mean re-ingesting the corpus under pressure.
- [ ] Define what degraded-but-safe looks like — a read-only mode with an honest message beats a mode that keeps answering confidently and wrongly.
- [ ] Agree the incident triggers in advance: which metric, which threshold, who decides, and who is told.
- [ ] After every incident, add the failure to the eval set and the canary before the ticket is closed, then re-read this list and fix the line that was skipped.

---

Print it, cut the items that do not apply, and put a name and a date next to every remaining line. A checklist nobody owns is a wish list.
