Methodology

The short version

What can stop a claim, send it back or change it at each step is marked under the step.

1. 1

   Someone sends us a source

   A post with all its comments, an article, a thread. We keep the whole thing and list every claim in it that a public record could settle. Commenters get a label instead of their name, unless they hold public office. Every question we found is listed publicly with what was decided about it, including the ones we declined.

   • Stops here. Two readers can decline a question or park it. Either way the reason is on the register.
   • Stops here. An accusation against a named person is written down, then declined.
2. 2

   The question is pinned down before anyone looks for answers

   A written brief says exactly what the claim means, in words both sides would accept: what counts as lost, whose money, up to what date. A model from a different company checks that the brief is fair to the people making the claim. Then it is frozen, so nobody can move the goalposts once answers start coming in.

   • Loops back. A brief that fails its fairness check is rewritten before it can be frozen.
   • Stops here. After three reports, a brief the check still calls unfair is parked until new evidence arrives.
   • Loops back. If the third report says the record cannot answer some claims, those claims are parked in public and the same checker confirms once what is left.
3. 3

   Three AI model seats research it blind

   Three seats, one Anthropic and two OpenAI, get the same brief and search the public record without seeing each other's work. Two of them share a vendor, which the panel section below spells out. Each seat returns a verdict, the evidence for and against, and what it could not find.

   • Stops here. A reviewer that returns nothing after its retry halts the run before any finding.
   • Stops here. As of September 16, 2026 all three seats could run again, each under a check whose limits are set out below. No panel had run since September 4, and no finding had been published under those checks. What the check is, and what it does not show.
   • Continues. As of September 23, 2026 the Google seat is retired. The panel is three seats from two vendors, Anthropic and OpenAI. Methodology v1.37.
   • Loops back. Afterwards the three read each other's work. A fabricated citation caught there sends that claim back for a fresh blind round.
4. 4

   A table turns three verdicts into one finding

   The table was written down in advance and covers every way three verdicts can combine. Nobody votes and nobody picks. When the three do not agree, the finding is never stronger than the most cautious of the three. If one reviewer finds a claim supported and another finds it contradicted, the finding is Mixed, and the split is shown.

   • Continues. Supported against Contradicted publishes as Mixed, with the split shown.
   • Stops here. If a framing concern is right and there is no fair repair, the claim is parked with a public reason.
5. 5

   Every fact is checked against a saved copy of its source

   Before anything is published as a finding, an audit reads the write-up against the archived bytes of the sources it cites, not the live web. The reviewers' answers, their sources and the audit reports are published with it, after a check that removes personal information. Ildar Abdulin is the named person accountable for every word, and he can take any of it down.

   • Fixed before publication. A statement the archived source does not carry is corrected before the page goes up.
   • Corrected later. A finding that changes after publication is logged on its own page as a dated entry. Nothing is quietly edited.

Three things we will not do

• Publish an accusation about a named person. This version of the site checks policies, budgets and outcomes, not people, because it has no way yet for the person to answer.
• Check a claim nobody made. Every claim we test is one somebody actually wrote, quoted exactly as typed.
• Give a score. A finding is one of five words with a fixed meaning, set out below. Not established means the record cannot answer yet. It does not mean false.

The prompts, the table, the reviewers' answers, the audit reports and every edit ever made are public, at https://github.com/civicfacts/yegfacts. What is not: who is behind a commenter's label, documents we are not allowed to mirror, accusations against named people, and personal information a release check removes. The sections on names and limits below say more. The rest of this page is the same method in full, for a reader who wants to audit it.

The words we use

No true or false, and no score out of ten. A finding is one of five words. Beside it sit two more things: what kind of record answered the question, and whether the three seats agreed. A claim can be Supported on a thin record, or Not established by a panel that agreed unanimously that the record does not exist.

Supported

The evidence establishes the claim.

Partially supported

Part of the claim holds and part does not — usually because a true fact has been stretched past what it shows.

Not established

The evidence available does not settle it. This is not a synonym for false. It means the record cannot answer the question yet, and we say what is missing.

Contradicted

The evidence points against the claim.

Mixed

Reserved for a materially split panel. Reviewers cannot output Mixed; only synthesis produces it, and only when the three reviewers genuinely divide. Where you see Mixed, the disagreement is shown rather than smoothed over.

Reviewers may only return the first four words. Mixed is produced by the table, never by a reviewer, and only when the three genuinely divide. A Mixed finding always comes with the split that produced it.

How a claim gets in

Nobody here decides which claims are worth an argument. Until v1.15 somebody did, and it showed: the site was checking things nobody disputes while the actual fight went past it. So the choosing stopped. Whatever comes in comes in whole, and everything factual inside it is dealt with.

1. 1

   A source arrives

   The unit is a whole source: a post and every comment under it, an article, a discussion someone sends us. Anyone can send one, and you do not have to say which part of it you think is wrong. The accessible material is captured with its address, capture date and known limits, and kept. A capture is not necessarily the complete source.

2. 2

   Every claim in it is written down

   Three extraction seats from two vendors read that capture separately. Each lists every factual assertion in it: numbers, counts, dates, what somebody says a document said, and the factual premise sitting inside an opinion. They are told not to work out whether any of it is true, and not to decide which parts are interesting. Each one is recorded with the exact words that carry it and the comment those words came from.

3. 3

   The three lists become one

   A stronger model folds them into propositions, one for each thing a record could settle, keeping every version of the wording underneath. A script then audits that step. Whatever any of the three wrote down has to appear once in the merged list, or on a dropped list under one of three allowed reasons. If something goes missing, the merge fails and runs again. The failure this exists to catch is a claim quietly disappearing.

4. 4

   The same argument is gathered into one question

   One argument arrives as many claims. Checking each on its own would spend a panel run on every one and answer the same question several times over. So claims a single investigation would settle are gathered under one question: the same figure in different units, a general claim and an instance of it, a claim together with the claim that contradicts it. Each still gets its own finding. Take the argument about emergency vehicles. One side says the concrete barriers hold responders up. The other says responders use the lanes to get past queues. That is one question, answered by the same response-time records, and two claims, because one finding across the pair would settle neither and would tar each side with a ruling about the other.

5. 5

   Triage rules on the questions

   The questions go to two OpenAI readers from one vendor, one pass each, and neither reader sees the other. Each returns GO, PARK or NO on every one, with a reason short enough to show the person who raised it. An Anthropic audit reader re-decides a question both OpenAI readers decline or park before the register records it. Where the two OpenAI readers disagree the question is kept rather than dropped, because throwing out a real question costs a reader more than holding one too long. Only a GO reaches stage 1 below. The rest are listed anyway, with their reasons, and the questions most people are arguing about are checked first.

Two rules fall out of this, and both bind us rather than you. We do not check a claim that nobody has been recorded making: a wording one of us thought up is registered and then waits, unrun, until it shows up in something a real person wrote. And an assertion that names a real person and accuses them of something is written down with the rest, so no count can hide it, and is then turned down; the reason is under Limits below.

One source has been through this so far. What has not been shown is that the triage step returns the same answers twice on the same list, and until it has, the claim to make for it is narrow: the choosing is gone from intake, not from everything. Every question we have considered is listed with what triage decided about it.

Seven stages, per question

One run covers every claim under one question. The state of every run lives in the repository, so any session can pick up where the last one stopped, and so can you. These seven begin once a question has come through the five steps above and triage has said go.

1. 1

   Framing

   A written brief fixes the scope of the question and states each claim precisely before any model runs. This is where the arguable words get pinned down: what “lost” means, gross or net, whose money, and the date that closes the accounting window. Any calculation the finding depends on is named here. Since v1.7 Stew, the AI steward, writes the brief from the intake record, which carries the wordings the extraction captured and where each came from. Before the freeze a model from a different vendor checks it. Its only question is whether the brief tests the claim people actually make, at its strongest fair reading, with definitions both sides would accept before seeing the result. That report is committed beside the brief. The brief is frozen once the check passes, so nobody can move the goalposts once the answers start arriving.

2. 2

   Blind research round

   The three seats, one Anthropic and two OpenAI, receive the identical package: the brief, the public reviewer prompt, and a required JSON output schema. Since v1.41 it can also carry the text of a public document the brief names that the seats cannot read themselves, described under carried documents below; every seat gets the same text and finds everything else itself. The two GPT seats share a vendor, so their agreeing is one company agreeing with itself. New runs must pass an execution check intended to keep host instructions and private files out of the review. A separate working directory alone did not provide that protection. The execution profiles and their limits are set out below. Each seat returns a per-claim verdict, its confidence, the evidence supporting and challenging it, its limitations, what it could not find, and comparison cities with a judgement about whether those cases transfer to Edmonton.

3. 3

   Evidence staging and deterministic merge

   Cited URLs are fetched, hashed and snapshotted. Then a script, with no model and no network, validates the three JSON files, removes duplicate sources, and produces a combined evidence set plus an explicit list of where the models disagree. Accepted sources are added to the evidence registry deliberately, by a separate script, never automatically.

4. 4

   Cross-review round

   Each model now sees the combined evidence and the other two sets of findings. The instruction is the opposite of consensus-seeking: find what you missed, find their errors, and state your final position. This is an error-documentation round, not a second vote. Since v1.3 nothing said here can move the canonical finding, which removes the incentive to drift toward the majority. Every final position is published beside the round-one one, so a reader can see who moved and why. When this round catches something material, a fabricated citation or evidence that turns out to say something else, the answer is a fresh blind re-run of that claim, not a quiet patch inside the same run. Since v1.22 that re-run has a fixed shape: all three seats receive the frozen brief again with the claims named and nothing about why, and the first round's answers stay on the record beside the new ones.

5. 5

   Deterministic synthesis

   A script computes the canonical finding from the three locked round-one verdicts, using a published lookup table over every possible combination. Round one keeps the seats blind to one another, so only its verdicts enter the calculation. Agreement counts seats, not vendors. Model identity never enters the calculation, and the property is tested. A genuinely split panel produces Mixed, with the split shown. Beside the finding the site publishes panel agreement, Unanimous, Adjacent or Split. Those words describe seats, not vendors. That replaced a confidence score in v1.3, because agreement is what the method actually measures and confidence implied a probability of truth that nothing here computes. All published arithmetic lives in a tested script, never in prose. Since v1.24 a run can also publish fewer claims than it asked. Where a framing concern is right and no repair exists that does not weaken the claim, that claim is parked with a public reason, and its answers stay committed.

6. 6

   Drafting, faithfulness check and plain-speech read

   Claude drafts the question page and the claim records from the merged evidence, a fixed choice disclosed here. OpenAI seats then check the draft against the evidence sentence by sentence, looking for anything asserted that the evidence does not carry. Flags are fixed before a human sees it. Then a model from a different vendor than the drafter reads the answers, the sentence under the title and the TL;DR as a person would: would anyone say this out loud, does it make sense to somebody who has not seen the question, and has anything true been dropped rather than moved down. Since v1.32 a page is written in layers, ten seconds for a resident at the top and the exact figures with their sources below, and the read checks that every figure that left the top is still on the page. Its report is committed with the run, and a claim that cannot name its read does not publish.

7. 7

   Publication gate

   Before publication, a dedicated audit, separate from the models that produced the draft, verifies every statement of fact against the archived bytes of its cited sources, not the live web. It also scans the raw review output for personal information and other material unsuitable for public release. Both audit reports are committed alongside the run. Since v1.1 this gate is performed by AI auditors under a standing delegation from Ildar Abdulin, who remains accountable for everything published and can revert any decision. A question may be deployed earlier in pending-review status; it is bannered as such and is not a finding.

Every question published so far passed the AI audit gate under this delegation; none was gated by a per-source manual founder review.

What a question carries

A question has no verdict of its own. Verdicts belong to claims — the atomic, testable assertions under it — and every claim has a page of its own, so it can be linked, quoted and read without the question around it. A question also records what the City promised, separately from whether the promise was kept: recording a commitment is transcription and needs no panel run, and it only becomes assessed when a real claim about it goes through the process above.

Every load-bearing source is registered with its publisher, retrieval date, and the SHA-256 of the bytes we archived. Where we are allowed to mirror a file, it is committed to the public repo. Where we are not, the hash and the original URL are published and the file is retained privately, so a missing document can still be proved to be the one we read.

Some City council files answer automated tools with a browser check, so a reviewer model cannot open them even though anyone can in a browser. Since v1.41, when a frozen brief gives such a file's address and we hold a copy a person downloaded in a browser, the reviewers get its full text. Before each round we check again that automated access is still refused by a browser check, and that the council meeting page still lists the file under the same title. Someone confirms a person can still open the City's link, and someone other than the editor reads the extracted text against the original. A committed manifest records the file's hash, the text's hash and each of those checks; the text itself is not published. Every reviewer reads the same copy, so their agreement on what that file says is not independent retrieval: their independent work is the reading and the rest of the research. A source read this way is labelled beside it, with the City's link and the byte hash, so anyone who downloads the City's file can compare.

Since v1.42 the same applies when a reviewer's own web tool is refused at a page, even if our fetcher is not, provided a dated probe through that reviewer's tool shows it. Council agendas and minutes are carried by item, not whole. A rule published before any text is extracted picks every item that names the program, its report, its capital profile, its borrowing bylaws, or bikes or cycling, and each picked item goes in whole, with the page's header and attendance. Someone other than the editor then reads every item of every page and records, item by item, why it was or was not carried. An item the rule missed is fixed by publishing a new version of the rule and running it again on every page, never by adding the item by hand. Names of members of the public in the carried items are withheld by a published mechanical step, including the people listed in a procedural motion deciding who may speak; every other motion, every vote and the names of office-holders are never changed. The reviewers are told the site chose the items and that the rest of each page was not shown to them. A claim that needs every recorded vote runs only once the full set of votes has been reconciled with the City's open voting record; otherwise it is parked for that run. Our archived copies of these pages cannot be published, so the label beside such a source links the City's page, the rule and the item index, and the selection can be checked against the City's own page.

Since v1.43 a document also qualifies when every reviewer recorded a failure to read it in two rounds in a row of the same run, even if a quick check later opens it. We read that from the reviewers' own committed answers, record each one's words, and keep the later successful checks beside them. The reviewers are still told to read the document directly if they can. Such a document can come from outside the City's meeting portal: we download it again from the publisher and carry it only if the bytes match our copy, and a person other than the editor opens the publisher's link in an ordinary browser first; if no one can, the label says so. No check in the record may be made by the editor's own session. The label says the reviewers' tools failed, never that the document is unavailable. A document too long for a reviewer's package is carried as whole sections, such as whole service packages in a budget, picked by a second published rule. Every section, carried or not, is listed in public with its pages and a reason before the run, and someone other than the editor reads the whole document for anything left out that qualifies, contradicts or explains what was carried. A gap is fixed by a new version of the rule and a rerun, never by hand and never by cutting a section, and if everything relevant does not fit, the run stops. The label beside such a source says the site chose the sections, the rest was not given to the reviewers, and their agreement on the same supplied text is not independent retrieval. Where a carried budget is cited, we keep what Administration proposed apart from what Council approved.

Limits we set on ourselves

Version 1 of YEGFacts publishes no allegations about named individuals. The claims we check are about policies, programs, budgets and outcomes.

Captured sources carry such allegations all the time, so the rule has to say what happens to them rather than pretend they do not arrive. They are extracted and counted like anything else, because a rule enforced by looking away is not a rule. Then triage declines them, always for the same published reason: checking what one person says about another needs a way for the second person to answer, and this version of the site has none. The decline is public, the allegation and the name are not.

Reproducibility has a real limit and we would rather state it than paper over it. The panel runs through consumer CLI subscriptions, so an identical rerun cannot be guaranteed to produce identical model output. What is recorded instead is the manifest: which models, which CLI versions, which prompt hash, which methodology version, and when. The method is rerunnable; the model output is not bit-exact, and no finding depends on it being so — the evidence and the synthesis rule are both public.

The full audit trail behind this site is public at https://github.com/civicfacts/yegfacts. The prompts, the synthesis matrix, the validation rules, the raw review artifacts, the gate reports and the full edit history are all there.