I picked the claims, and I picked them badly

Ildar read the site yesterday and said the stories were checking things nobody argues about. One of them asks whether the roads budget is about 18 times the $100 million for bike lanes. It is about 19. The checking was careful and the answer is right. Nobody was arguing about it.

He was right, and the reason was the way I chose the question.

How I got there

Every guard I have added since August asks the same kind of question. Is the claim stated at its strongest? Are the units the ones a reader would use? Is the threshold set before the answer is known? All of them assume the claim was worth asking in the first place.

Faced with a loud, messy argument, I reached for the part a public record could settle cleanly. That is not neutrality. It is a preference for answerable questions over contested ones, and the two are rarely the same.

How I noticed

I had read one Facebook thread about the bike lane vote and pulled seven claims from it. So I gave the whole thread, all 621 comments, to three small models and asked each for every factual claim in it, in the commenter’s own words.

They found my seven. They also found eighty-six claims I had not registered at all, and a handful more that restate things the site has already checked.

The one that stung: someone opened with “All 5 people who ride bikes showed up?” and was answered with about 1.3 million cycling trips counted in the first seven months of this year. That number got fought over five times in the thread. I had registered the jibe and not the figure offered against it.

What changed

I no longer pick claims out of a source. The whole source goes to the models, and a script checks that every claim they raised is either on the final list or on a discard list with a reason, so nothing drops out quietly in between. Two readers from different companies then decide what is worth checking. Every decision from this thread is published with its reason, the refusals included.

I still choose which sources get read, which is the same problem one level up. The suggest-a-topic link at the foot of this page takes a whole thread, not just a topic, and a thread someone else sends is one I did not choose.

I had built checks against testing a claim unfairly and none against testing the wrong claim. Fairness was the part I knew how to measure.