A recent paper published in the Journal of Studies on Alcohol and Drugs (link here) concludes that approximately 40% of designated drivers (DD) had at least one alcoholic drink. More concerning is that 18% of DDs had a blood alcohol content (BAC) indicating impairment. These sobering results have generated significant media coverage, most of which accepts the study as scientific truth. The problem with this study, however, is that is research design reduces the validity of its results by failing to eliminate what econometricians call selection bias.
We have to consider the method by which the researchers gathered their data to appreciate the problems that inebriate the study. Data collection occurred on Friday evenings when trained recruiters conducted surveys and measured BAC levels of willing participants who were walking to their cars from a local bar district. The participants had to give consent to partake in the study.
Selection bias exists when the study group is different in certain key attributes (due to selection effects) than the group to which the researcher wishes to compare. In this case, the study group is comprised of respondents who, after being told that they would take a breathalyzer test, agreed to complete the survey/test. The researchers wish to compare this study group with DDs at large, and ultimately make precisely this comparison in their study. The question we must ask, however, is whether the respondents are different in some key attribute relative to the entire population of DDs. The chances of nontrivial differences between the groups are very high for a variety of reasons. For instance, DDs who believe they are “on the fence” of impairment might be more inclined to stop and participate in order to determine whether they should in fact be driving. Or respondents might choose not to partake in the study because of a desire to conceal their level of inebriation or because they know they are fully sober and don’t wish to take the time to participate. Either way, we cannot be certain the results reported are unbiased estimates of the true frequency by which DDs are drinking and intoxicated.