Calibration of judgement

Calibration is the degree to which - in judgement situations - confidence matches accuracy. An actor would be considered well-calibrated if, for example, with judgements assigned a .75 probability of being correct, 70% of the actor's judgements were accurate. Perfect calibration in this example would mean 75% accuracy for this current decision set.

Remarkably, when subjects are given a general knowledge quiz comprised of simple, two choice, questions of the kind “is absinthe a drink or a precious stone?” overconfidence increases with the difficulty of the questions!

Incorrect calibration can be corrected with intense training, especially when the feedback is immediate and clear. For example, studies of expert weather forecasters have found that they are almost perfectly calibrated in their estimates of the likelihood of rain – meteorologists enjoy the distinct advantage of working under conditions in which they receive large quantities of prompt, unambiguous feedback with clear-cut events (rain or no rain). Similar opportunities are seldom available for most people in the majority of analytic judgements they are pressed to make concerning uncertain events, so there should be little surprise that generally people, including experts, do not learn how to make such judgements optimally.

However, it must be noted that intense training does not prevent nor guard against overconfidence, though it does tend to improve calibration.

(see also: Illusion of validity, Predictive value, Redundancy, and Confidence)

» Return to belief, judgement, and clear thinking »