Two answers to the same question. You ask someone you know whether to put money into something, and she says, without pausing, that yes, now is a good moment, she would do it herself. Then you ask a model, and it tells you there is a sixty-two per cent chance you are ahead in a year, and that the number itself is uncertain. Almost everyone finds the first answer more useful. I want to work out why, because the second answer is more honest and carries more information, and we throw it away anyway.
The usual explanation for why the machine’s answer lands badly is that people are bad with probability — that a number between zero and one is just harder to hold than a yes. I think that is partly true and not enough. The same people who glaze over at sixty-two per cent will argue happily about whether it is going to rain this afternoon. Chance is not the obstacle. Something else is being asked for and not supplied.
Here is what I think is happening, and I should say upfront that this is reasoning from a mechanism rather than evidence I can point to. Confidence was never mainly a claim about the world. It was a claim about the speaker.
Work through where the signal comes from. For most of human history you had no way to audit anyone’s record. You could not look up how often your uncle’s advice had worked out; nobody was keeping score. What you could observe was how much the person in front of you stood to lose by being wrong. Someone who says definitely, to your face, in front of people you both know, has attached themselves to the outcome. If it goes badly, you stop asking them, others hear about it, and they have to sit across from you afterwards. That cost is what makes confidence expensive to fake. So we learned to read it as a rough measure of how willing someone is to be held to a thing — and while the people advising you were also exposed to you, that was a decent stand-in for whether they were likely to be right.
A model breaks the link. It can produce any level of certainty at no cost whatsoever. It is not going to lose your business, or see you at a wedding, or feel anything at all in a year’s time. Nothing comes back to it.
This puts a careful model in an odd position. When it says sixty-two per cent and flags that it might be off, it is doing the most responsible thing available to it — being calibrated, which just means that of all the things it calls seventy per cent likely, about seventy per cent of them happen. And we hear that as weakness. We are running a very old test on it, a test that asks how much is this speaker risking by saying so, and the honest answer is nothing, so the test returns nothing. Being truthful about uncertainty does not help. It reads as the absence of the signal we were looking for.
Now the strongest case against me, which I find genuinely hard to dismiss. Maybe people are not making a mistake at all. A probability attached to a single event is awkward to use, because you do not get a hundred tries at your own life — you get one, and afterwards you are either ahead or you are not. And the confident person’s exposure is real information, not a bias. Someone who bears a cost for being wrong has told you something the number cannot: that they have something to lose. Preferring that is not a failure of arithmetic. It is using the only check you actually have.
I take that seriously. What makes me still worry is that the check stopped working a while ago, well before any of this. Most people now get financial advice from someone whose exposure is to the sale, not to how it turns out for you. The confidence is still there, practised and professional, and the consequence it used to stand for has quietly been taken out. The machines did not break the proxy. They arrived after it was already broken, and made it cheap to produce at scale.
So here is where I land. Confidence is not a forecast. It is a promise about consequences, and we read it fast, by feel, the way we always have. Asking a model to be well calibrated is asking it to be excellent at the one thing confidence was never doing — and that is worth doing anyway, but it will not buy the trust people think it buys.
What follows is not really a modelling problem. If the signal we instinctively look for is who pays when this is wrong, then the useful question about any automated advice is exactly that: when it is wrong, does anything come back to anyone? That is a question about how firms and regulators are arranged, not about how good the model is. I doubt we will learn to read a probability as a substitute. The old reading is too quick and too deep. It is easier to rebuild the consequence than to retrain the instinct.