Methodology

Seven stages, per story

One run covers all of a story’s claims. The state of every run lives in the repo, so any session can pick up where the last one stopped, and so can you.

1. 1

   Framing

   A written brief fixes the scope of the story and states each claim precisely before any model runs. This is where the arguable words get pinned down — what “lost” means, gross versus net, whose money, and the as-of date that closes the accounting window. Any calculation the verdict depends on is named here. Since v1.7 the brief is written by Stew, the AI steward, from an intake record that says where each circulating form came from, and it is checked before the freeze by a model from a different vendor whose only question is whether the brief tests the claim people actually make, at its strongest fair reading, with definitions both sides would accept before seeing the result. That report is committed beside the brief. The brief is frozen only once the check passes, so nobody can move the goalposts once the answers start arriving.

2. 2

   Blind research round

   Three models — Claude, GPT and Gemini — receive the identical package: the brief, the public reviewer prompt, and a required JSON output schema. Each runs in its own scratch directory with no access to this repo and no sight of the others. Each returns a per-claim verdict, its confidence, the evidence supporting and challenging it, its limitations, what it could not find, and comparison cities with a judgement about whether those cases transfer to Edmonton.

3. 3

   Evidence staging and deterministic merge

   Cited URLs are fetched, hashed and snapshotted. Then a script — no model, no network — validates the three JSON files, de-duplicates the sources, and produces a combined evidence set plus an explicit list of where the models disagree. Accepted sources are added to the evidence registry deliberately, by a separate serialised script, never automatically.

4. 4

   Cross-review round

   Each model now sees the combined evidence and the other two sets of findings. The instruction is the opposite of consensus-seeking: find what you missed, find their errors, and state your final position. This is an error-documentation round, not a second vote — since v1.3 nothing said here can move the canonical finding, which removes the incentive to drift toward the majority. Every final position is published beside the round-one one, so a reader can see who moved and why. When this round catches something material — a fabricated citation, evidence that turns out to say something else — the answer is a fresh blind re-run of that claim, not a quiet patch inside the same run.

5. 5

   Deterministic synthesis

   A script computes the canonical finding from the three locked round-one verdicts, using a published lookup table over every possible combination. Round one is the only round in which the three reviewers are independent of each other, so it is the only round that can honestly be read as three separate judgements of the record. Model identity never enters the calculation, and the property is tested. A genuinely split panel produces Mixed, with the split shown. Beside the finding the site publishes panel agreement — Unanimous, Adjacent or Split — which replaced a canonical confidence score in v1.3, because agreement is what the method actually measures and confidence implied a probability of truth that nothing here computes. All published arithmetic lives in a tested script, never in prose.

6. 6

   Drafting and faithfulness check

   Claude drafts the story and the claim records from the merged evidence — a fixed choice, disclosed here. The other two models then check the draft against the evidence sentence by sentence, looking for anything asserted that the evidence does not carry. Flags are fixed before a human sees it.

7. 7

   Publication gate

   Before publication, a dedicated audit — separate from the models that produced the draft — verifies every statement of fact against the archived bytes of its cited sources (not the live web), and scans the raw review output for personal information and other material unsuitable for public release. Both audit reports are committed alongside the run. Since v1.1 this gate is performed by AI auditors under a standing delegation from Ildar Abdulin, who remains accountable for everything published and can revert any decision. A story may be deployed earlier in pending-review status; it is bannered as such and is not a finding.

All four currently published stories passed the AI audit gate under this delegation; no published story was gated by a per-source manual founder review.

The words we use

No TRUE/FALSE. No scores out of ten. A finding is a sentence in a controlled vocabulary, and it is reported alongside two separate things: the evidence basis (what kind of record answered the question) and panel agreement (Unanimous, Adjacent or Split). A claim can be Supported on a thin basis, or Not established by a panel that agreed unanimously that the record simply does not exist.

Supported

The evidence establishes the claim.

Partially supported

Part of the claim holds and part does not — usually because a true fact has been stretched past what it shows.

Not established

The evidence available does not settle it. This is not a synonym for false. It means the record cannot answer the question yet, and we say what is missing.

Contradicted

The evidence points against the claim.

Mixed

Reserved for a materially split panel. Reviewers cannot output Mixed; only synthesis produces it, and only when the three reviewers genuinely divide. Where you see Mixed, the disagreement is shown rather than smoothed over.

There are two vocabularies, deliberately. A reviewer may only return Supported, Partially supported, Not established or Contradicted. Mixed is a synthesis outcome, not a hedge a model can reach for — which is why a Mixed finding always comes with the panel split that produced it.

What stories carry

A story has no verdict of its own. Verdicts belong to claims — the atomic, testable propositions inside it — and each claim has its own anchor so it can be linked and quoted directly. Stories also record what the City promised, separately from whether the promise was kept: recording a commitment is transcription and needs no panel run, and it only becomes assessed when a real claim about it goes through the process above.

Every load-bearing source is registered with its publisher, retrieval date, and the SHA-256 of the bytes we archived. Where we are allowed to mirror a file, it is committed to the public repo. Where we are not, the hash and the original URL are published and the file is retained privately, so a missing document can still be proved to be the one we read.

Limits we set on ourselves

Version 1 of YEGFacts publishes no allegations about named individuals. The claims we check are about policies, programs, budgets and outcomes.

Reproducibility has a real limit and we would rather state it than paper over it. The panel runs through consumer CLI subscriptions, so an identical rerun cannot be guaranteed to produce identical model output. What is recorded instead is the manifest: which models, which CLI versions, which prompt hash, which methodology version, and when. The method is rerunnable; the model output is not bit-exact, and no finding depends on it being so — the evidence and the synthesis rule are both public.

The full audit trail behind this site is public at https://github.com/civicfacts/yegfacts. The prompts, the synthesis matrix, the validation rules, the raw review artifacts, the gate reports and the full edit history are all there.