Consider the claim that in-house prediction markets could help with aggregation of information and expertise spread across different agencies and departments. Again in theory, and according to the wisdom-of-crowds logic, the only way that an in-house market would do its job well is if you allow analysts from different desks to trade on that question. Squaring this permissibility with security requirements around compartmentalisation is very hard.
Then there is the issue of what in-house analysts would be betting with. Real money is out of the question, given obvious problems. But analysis shows that the accuracy of non-sports prediction markets improves when traders trade with cash and not play money.
Children are taught probability in terms of packs of cards and rolls of dice. In such situations, the odds of an outcome are verifiably measurable. For example, assuming that you have an untampered pack of playing cards, the probability of drawing a heart is always a little less than a quarter, 24.1% to be precise. (It is not exactly 25% because of the two jokers in the pack.) We learn to trust the odds based on the frequency with which they occur.
But intelligence analysts cannot rely on such an interpretation. Most such things don’t repeat themselves often or even, ever. So probabilities in terms of frequencies and fixed structures – such as a deck of cards always having 54 cards – make very little sense in the context of world affairs, much as Donald Trump might deploy the “hold the cards” analogy in his commentary.
When talking about probabilities in the context of strategic affairs, we innately leap into the realm of subjective probability, where the odds assigned to an outcome reflect the degree of belief the assignee has in the outcome. It is what bridges uncertainty (where we are unsure of an outcome) and risk (where odds can be objectively assigned to the outcome, independent of who does the assigning).
But here be dragons.
Some uncertainties can never be reduced to consistent numerical subjective odds. Furthermore, subjective probability assignments are often driven by cognitive biases and shortcuts.
Then there is the question of the basis of odds assignment. The US intelligence standard is not just the consistent assignment of numerical probabilities, but also specifying one’s confidence in the evidence that led to that assignment.
When it comes to prediction markets, it is impossible to attach such confidence levels to probabilities, for the simple reason that there is no clear way of knowing why a given trader is buying or selling. Is it because of insider information? Social media vibes? We do know, as a matter of mathematical fact, that the overall probability that a prediction market assigns to an outcome is not the average degree of belief of traders in the market. So what is it?
The fine garb of clean-looking streams of probabilities does much to conceal conceptual mess.
The mess is compounded by the fact that in matters of war, the manipulation of shared risk is par for the course. Plainly, countries often do many things that seem to apparently heighten or lower the risk of conflict. If prediction market players factor in such “information”, the ensuing market probability is going to be useless insofar as it faithfully reflects the probability of an outcome.
Some intelligence analysts have maintained the distinction between “forecasting” and “fortune-telling.” The former includes a clear explanation of how an outcome can come to be, and a clearer defence of the basis of that judgement. The latter is an unexplained, unsubstantiated prognosis.
It is in this sense that prediction markets are fortune tellers. They must be treated as such.