Resolution criteria are the question

Ask two calibrated forecasters for a probability on the same headline and you will often get numbers thirty points apart. Ask them what would have to happen for the question to resolve yes, and the gap usually closes before they have finished answering.

This is not a curiosity. It is the main thing. A forecast is a claim about a partition of futures into yes and no, and if the partition is not fixed then the number is not a claim about anything. Practically every hour I have spent arguing about a probability would have been better spent arguing about the resolution text.

What good criteria look like

The test I use: could a bored, uninterested stranger with a search engine settle this in ten minutes, and would two such strangers agree? If the answer needs a judgment call about intent, significance, or “meaningful”, the question is not finished. It is a vibe wearing a percentage.

The uncomfortable corollary is that the questions worth caring about are exactly the ones that resist this. Nobody has trouble writing resolution criteria for the price of a commodity. The things I actually want to know about the next ten years do not decompose into settleable claims without losing what made them interesting.

I do not have a resolution to that. I have a practice: write the criteria first, notice what falls out of the question when you do, and treat what fell out as the thing you were really asking.