Jim Manzi, author of Uncontrolled and a leading expert on the design of business experiments, offers observations on the Oregon Health Experiment. Among other things, he notes that of the 90,000 Oregon residents who applied for Medicaid coverage under the state’s lottery, 35,000 won the lottery — yet only 60 percent of the lottery winners took the next step of actually signing up for coverage. As Jim goes on to explain, this raises a number of hard-to-answer questions about the experiment itself, as it is premised on the notion that we can differentiate between those who secured coverage via the lottery and those who lost the lottery, despite the fact that there is a large population of lottery winners (approximately 14,000) who chose not to accept coverage. In an ideal world, we’d want to know more about the rationality and the conscientiousness of participants in the experiment, as these qualities are not necessarily evenly distributed across this population. It turns out, for example, that while fewer than 25 percent of Americans between 19 and 64 smoke cigarettes, the same is true of 48.4 percent of those who received free coverage smoke cigarettes. If granting smokers coverage makes them somewhat more likely to quit smoking, the health benefits would be substantial. But if it made them somewhat less likely to quit smoking, the health benefits would be far more modest. The experiment doesn’t settle this particular question. Jim observes that advocates of Medicaid expansion have pointed to some of the observed health benefits of coverage that don’t pass the test of statistical significance. The problem, however, is that there are also negative effects that don’t pass the test of statistical signifiance, and there is no principled way to embrace the former findings while dismissing the latter.
Jim concludes that if nothing else, the Oregon Health Experiment demonstrates the value of randomized controlled trials in informing public policy decisions. One promising area for policymakers to explore is how we might embed the capacity for experimentation and analysis into social service provision more broadly.