Predictionmachine.com generated a statistical model for the 2013 Super Bowl that reports a 67% probability of the 49ers winning. The probability of this outcome is different than predicting that the 49ers in fact will win. Yet such simplistic, and incorrect, interpretations are common to Monte Carlo style models (the type of model run by Predictionmachine.com). For instance, CBS Philly, reporting on the model, comments: “This particular computer model has predicted the correct winner of seven of the last nine Super Bowls.”
Yet the Monte Carlo simulations do not predict winners. They merely report the probabilities of possible outcomes. For illustration, it’s possible that a model gives a 50% chance of one team winning and a 50% chance the other team wins. With such an outcome, it is obvious that the model is not predicting a winner. And it is not predicting a tie. It is merely assigning probabilities to possible outcomes.
Suppose the Ravens clobber the 49ers this Sunday. Predictionmachine.com’s model could nevertheless be perfectly valid even though it assigns a 67% chance that the 49ers win. This is because such a clobbering is possible and assigned positive probability by the model.
All this is important to remember because people frequently latch onto the average outcome as if it is the predicted outcome. In fact, the actual occurrence of an average outcome could be extremely unlikely. For instance, the average outcome in the Predictionmachine model (as reported by CBS Philly) is as follows:
The 49ers beat the Ravens, 28.6 to 21.3
San Francisco’s yards advantage: 85
Joe Flacco (Ravens QB): 20.9 of 35.3 passes, 1.3 touchdowns and 0.8 interceptions
Colin Kaepernick (49ers QB): 17.2 of 28.8 passes, 1.6 tochdowns, 0.8 interceptions
Even though this is the average outcome, the probability of precisely this outcome is extremely small or, for that matter, entirely impossible since no one ever has 0.8 interceptions or scores 28.6 points. Even if rounded numbers are used (e.g. final score: 29 to 21), the likelihood of precisely that outcome remains extremely small. The simulation is most useful when interpreted as a range of possible outcomes with assigned probabilities, not when interpreted as a prediction. Of course, Predictionmachine.com’s name does not help matters.
In terms of financial modeling and financial planning, clients need to be careful with their interpretation of statistical models. View the simulation in terms of what could happen, together with the probability of that occurrence, and not as a prediction of what actually will happen. For instance, a Monte Carlo simulation of a retirement portfolio might assign a 67% probability the portfolio and investment strategy will work. Are you comfortable with a 33% chance that your retirement strategy will not work (i.e. spending needs will surpass available wealth)? Probably not. Instead consider what is the probability of financial ruin as simulated by the model and whether you are willing to accept a strategy that has that probability.
