A model is paid to continue the text. It is not paid to know things. Confidence is a style choice.
If you need a working rule: do not ship a draft you would not sign. The model does not have a reputation. You do.
This is not a vibe check. It is a short review you can run in five minutes on anything that leaves the building.
The four-pass review
Do the passes in order. Stop at the first fail. Do not "fix the tone" on a draft that invented a fee.
Pass 1 — Source lock
Ask: where did each load-bearing fact come from?
Load-bearing means a customer, a regulator, or your future self could act on it: prices, dates, names, policy, medical or legal claims, "we will," "you must," citations, URLs, quotes.
For each load-bearing line, mark one:
- IN SOURCE: you can point to the pasted brief, ticket, or doc
- NEED: required, not in source, must ask a human
- INVENTED: sounds specific, not in source
One INVENTED is a fail. Not a polish item. A fail.
Common invented objects: case studies, statute numbers, paper titles, "according to our Q3 report," tracking numbers, "as you requested on March 3," customer logos, star ratings, user counts.
Pass 2 — Citation honesty
If the draft cites anything, open it. A citation the reviewer cannot open is decoration.
Rules:
- No URL? Treat the claim as unsourced.
- URL opens but does not support the sentence? Invented.
- "Studies show" with no study? Invented.
- The model offers to "add citations" after the fact? Those are usually worse. Find the source yourself or delete the claim.
Internal citations count. "See the refund SOP, section 2" is honest only if section 2 says that.
Pass 3 — The customer test (E2)
Read the draft out loud as if the customer is on speaker with your manager.
Would you send this as written, with your name?
- Yes, maybe a comma: pass
- Yes, after I change the facts: fail (you are rewriting, not reviewing)
- No: fail
This pass catches the polite disaster: accurate facts, wrong promise. "We can rush this today" may be factually possible and still a promise you do not make in writing.
Pass 4 — Shape and leftovers
Only if 1–3 passed:
- Required sections present
- No leftover prompt-speak ("as an AI," "certainly," "great question")
- No secrets from the source that should not be in the send (other customers, internal costs, personal data)
Copy-paste: reviewer prompt (you still decide)
Use this after you have the draft. It is a second reader, not a judge you obey.
Review the DRAFT using only SOURCE. You are a skeptical editor. You do not improve style until facts pass.
>
List:
1. Load-bearing claims. For each: IN SOURCE / NEED / INVENTED. Quote the SOURCE line for IN SOURCE.
2. Any citation or URL. For each: supports / does not support / cannot check.
3. Promises (we will, you will receive, guaranteed). Allowed by SOURCE or not.
4. Customer test: ship / do not ship, in one sentence.
5. If do not ship: the smallest edit that would make it honest, or "rewrite."
>
If SOURCE is missing, say so and stop.
>
SOURCE:
[the brief you actually gave the model]
>
DRAFT:
[the output]
Then you mark ship or no. The reviewer prompt is allowed to be wrong. You are not allowed to skip Pass 3.
Decision rules
- Customer-facing or legal-adjacent: all four passes, every time.
- Internal scratch: Pass 1 on any number, date, or name. Then think.
- Brainstorm lists: no review beyond "did it just suggest something illegal or cruel."
- Two models disagree on a fact: neither is a source. Go to the document or the person.
- You are tired: do not review. Queue it. Tired people ship invented tracking numbers.
Failure modes
- Fluency bias: it reads clean, so it feels true. Clean is cheap.
- Citation laundering: a real link next to an unrelated claim.
- Partial rewrite: you fix one invented price and miss the invented deadline.
- Reviewer loop: you ask the same model to "make sure this is accurate." It will agree with itself.
- Metric theater: a 1–5 "quality" score with no fail line. If everything is a 4, you are not evaluating.
Aside. The model will invent a policy number with the calm of someone who has never been asked for a receipt. That is the job it was trained to do. Your job is the receipt.
A one-week habit
For seven sends that used a model, keep a three-column log: what you asked, what almost shipped, what you caught. You will see your house pattern — usually dates, or promises, or fake structure ("three key pillars"). Put that pattern in the FAIL example of your prompt packet.
Download
Ship/no-ship scorecard: /downloads/output-eval-scorecard.md
You are done when
You can fail a fluent draft in under five minutes and explain the fail with a line from the source — or the absence of one.