Calibration

In a previous post, we saw how humans have an overconfidence bias, and how that can play havoc on financial forecast estimations. What is not immediately clear, however, is the increasing role overconfidence plays the more knowledge one acquires. That is, as expertise rises, so does overconfidence, resulting in the fact that the people with the most knowledge are likely to be the most miscalibrated, which can result in detrimental effects.

But knowledge alone does not doom someone into being overconfident. James Montier argues that it's a lack of feedback that pushes people to be overconfident. To demonstrate this, Montier cites a study performed by Scott Plous that compares the calibration of weathermen against that of doctors. Weathermen were asked to predict the weather, while doctors were asked to diagnose patients (based on case notes). Both were asked to provide confidence intervals, so that their calibrations could be measured.

Interestingly, weatherman were well calibrated, while doctors were very poorly calibrated. Montier argues that this is because weather forecasters benefit from the fact that they receive immediate evidence of their abilities as forecasters, while doctors do not.

While one may be inclined to believe that financial analysts should be like weathermen in that they can see whether their forecasts came true, similar calibration tests appear to show that analysts are calibrated to a similar extent as doctors! As a result, the industry is glowing with overconfidence. Further testing confirmed the opening hypothesis: experts in finance are more overconfident than lay people, as they apply too narrow a band around their estimates as compared to the general public.