MAX’S NOTES
How to check AI answers: facts, numbers and sources
A confident tone proves very little. Here is a way to turn a long answer into individual claims you can actually check.
To check an AI answer, separate it into verifiable claims, find support for each important statement and recalculate numbers independently. A link or a confident tone does not establish that a statement is correct.
OpenAI warns that ChatGPT can make errors and fabricate facts or references. Important information still needs verification when the answer sounds convincing. [1] The method below is my working checklist, not a promise to catch every possible mistake.
Separate the claims
Consider this fictional answer: “Reducing costs from 80 thousand to 60 thousand saves 25%, and the service is available in every country.”
It contains at least two different claims:
- The savings calculation.
- The service's availability.
Arithmetic can check the first. The second requires a service name, current terms, a country and a date. A correct calculation does not validate the sentence next to it.
Recalculate the numbers
The fictional saving is 20 thousand. Divide it by the original 80 thousand: 20 / 80 × 100 = 25%.
Also check that the periods and cost categories match. A monthly payment cannot quietly be compared with an annual one. The formula may be right while its inputs are not comparable.
Ask the model to list its inputs if they are missing. An unknown amount should remain unknown rather than appearing out of politeness.
Check three things about a source
Open the source and locate the passage supporting the claim.
- Does the page exist?
- Does it actually support what the answer says?
- Do its conditions match the relevant date, country and product version?
A similar article title is not enough. For a price, prefer the supplier's current pricing information; for a feature, use product documentation. An old review may help you find a source, but its date matters.
Ask for uncertainty to be marked
You can use this prompt:
Break the previous answer into individual factual claims. For each one, provide a source and the supporting passage when available. Separate information from the supplied text from your own assumptions. Mark a claim “unconfirmed” if support is missing. Do not invent links.
Review that list too. Asking a model to check itself is another pass, not independent evidence. A second chat may repeat the same mistake.
Decide what can be used
I would mark important claims as confirmed, needs clarification or unconfirmed. Then edit the answer so it does not present the latter two as established facts.
For the fictional sentence above, the calculation checks out. Availability cannot be established without knowing the service. Part of the answer is useful; part remains unfinished. That is more honest than declaring the entire response correct.