Predictionmachine.com ran 50,000 iterations of its statistical model to give a 67% chance the San Francisco 49ers will win the Super Bowl. Such a large number (50,000 iterations) and “computer model” may create an impression of validity and sophistication about the prediction. This could be a false impression.
The output from a statistical model is only as good as the model’s underlying assumptions and inputs. It’s also only as good as its interpretation. The number of iterations employed in what statisticians call a “Monte Carlo” simulation — whether ten thousand, fifty thousand or five million — is of secondary concern. If the model is poorly constructed or based on flawed data, the number of iterations does not matter at all.
Yet the huge number of iterations is what gets all the attention while the underlying data and assumptions (and related potential problems) go unreported. For instance, an article from CBSPhilly reports in its second paragraph:
“PredictionMachine.com…ran more than just ‘some’ numbers. they ran Sunday’s 49ers vs. Ravens match-up through their system 50,000 times to see what the most likely outcome of the game will be.”
Most people, sports fans included, make little if any effort to assess the validity of a statistical model that predicts sporting event outcomes. Such predictions make for fun speculation, but that’s about it (unless, of course, the prediction leads to a bet).
On the other hand, readers should pay close attention when the model involves predicting, say, the possibility of financial ruin, as arises in the management of a retirement portfolio. Monte Carlo simulation can be useful in predicting levels of wealth through retirement based on portfolio investment style and spending patterns. But, as with models of Super Bowl outcomes, financial models also have limitations that require consideration.
In particular, you should not be impressed simply because your financial advisor generated a Monte Carlo style model and ran it 50,000 times. How good is the model? What assumptions does it make? What does it fail to consider? Is the interpretation correct?
This series of posts reviews some basic factors to consider when assessing predictive models of the Monte Carlo variety. Since football is far more interesting than finance, they take modeling the Super Bowl’s outcome as their starting point.