"In Anon Ymous's latest response, it seems he feels he's being misunderstood. Yet, he keeps making statements that demonstrate exactly the view I'm arguing against.
"The Oregon study did not find evidence of statistically significant improvements in physical health. That is a pretty significant result."
No, this is exactly what I've criticized. If you run a test that has low power, it is NOT significant if you find no effect. Suppose you have a magic pill that stops you from dying, you give it to 10 undergrads and use 10 as a control group and measure the effect after one year. You find no statistically significant effect. Is that finding significant? NO!!! Even if the pill prevents 100% of all deaths, you'd have to give it to thousands of undergrads to get statistical significance, because undergrads just don't die very often."
I don't think Clarke understands that I am agreeing with his core point, that the Oregon study did not prove Medicaid does not improve physical health. Based on this study, we can't rule out the possibility that Medicaid might improve health. However, we can say, based on this study, that even if Medicaid improves health, it does not improve health dramatically enough to be captured in what was a fairly large study. Indeed, it was done, as Clarke suggests in his hypothetical, on thousands of individuals, and there was still not statistical significance.
It is possible that they would have found statistically significant improvements in physical health if there were more individuals in this study. It is possible they will find statistically significant improvements in physical health when looking at a larger time horizon. However, that is not the study we have. For now, we have a study that has a sample size in the thousands and a time horizon of a few years. In this study, there was not evidence found of statistically significant improvements in health.
Clarke would be well served in this argument to just come out and admit that there wasn't evidence, in this study, of significant improvements in physical health. Instead, Clarke resorts to arguing that the numbers provide just as much support for the view that Medicaid substantially improves physical health as to the view that Medicaid does not substantially improve physical health:
"For high blood pressure, there is a fall from 16.3% of people to 14.97% of people. That's an 8% reduction in incidence of high blood pressure. It's more consistent with a "bold claim" of up to a 16% reduction of high blood pressure than it is a zero percent reduction.
For diabetes, there is a fall from 5.1% to 4.17%. That's an 18% reduction in the incidence of diabetes. That's more consistent with a "bold claim" of up 36% reduction in the incidence of diabetes than no reduction in the incidence of diabetes."
For diabetes, there is a fall from 5.1% to 4.17%. That's an 18% reduction in the incidence of diabetes. That's more consistent with a "bold claim" of up 36% reduction in the incidence of diabetes than no reduction in the incidence of diabetes."
This is a point I responded to a few days ago in a previous post:
"The Oregon Medicaid Study did indeed show no statistically significant improvement in objective measures of physical health. I am a bit surprised that so/ many people have taken issue with this assertion. The common response is that there were positive results, but they just weren't positive enough to be statistically significant. In order to be statistically significant, results have to have less than a 5% chance of being statistical noise. In the Oregon study, the positive results on elevated blood pressure had a 65% chance of being statistical noise, the positive results on elevated blood sugar had a 61% chance of being statistical noise, and the positive results on high cholesterol had a 37% chance of being statistical noise. In other words, not only were the results not statistically significant. They weren't even close.
It's actually worse than that. If we just look at raw numbers and ignore statistical significance, we can also say that Medicaid increases smoking and decreases cardiovascular health of those that were already sick. The raw numbers suggest that. Of course, neither of these results are statistically significant, but they are closer than any of the other measures. There is only a 24% chance that the results showing Medicaid reducing cardiovascular health are statistical noise, and there is only an 18% change that the results showing Medicaid increasing smoking are due to statistical noise. These still are far from statistically significant, but if one is going to argue that the raw numbers that are statistically insignificant are useful in telling us the effect of Medicaid, then they should also be arguing that Medicaid increases smoking and worsens cardiovascular health of the already sick.
I would argue that we shouldn't read too much into any of the statistically insignificant results and instead focus on the statistically significant ones. By the way, that data is from Megan McArdle who had an excellent post on this very topic."
It's actually worse than that. If we just look at raw numbers and ignore statistical significance, we can also say that Medicaid increases smoking and decreases cardiovascular health of those that were already sick. The raw numbers suggest that. Of course, neither of these results are statistically significant, but they are closer than any of the other measures. There is only a 24% chance that the results showing Medicaid reducing cardiovascular health are statistical noise, and there is only an 18% change that the results showing Medicaid increasing smoking are due to statistical noise. These still are far from statistically significant, but if one is going to argue that the raw numbers that are statistically insignificant are useful in telling us the effect of Medicaid, then they should also be arguing that Medicaid increases smoking and worsens cardiovascular health of the already sick.
I would argue that we shouldn't read too much into any of the statistically insignificant results and instead focus on the statistically significant ones. By the way, that data is from Megan McArdle who had an excellent post on this very topic."
The raw numbers would also suggest that Medicaid increases smoking and decreases cardiovascular health for the already sick. Of course, these numbers aren't statistically significant, but they are closer to statistical significance than most of the positive results in the raw numbers.
Using Clarke's logic, we could say that the study provides more evidence that Medicaid increases smoking and reduces cardiovascular health among the sick than evidence that Medicaid improves any of the other measures of objective physical health.
For the sake of clarity, I want to lay out my exact position on this study:
1.) The Oregon study found statistically significant positive results when looking at self reported health, mental health, and financial strain. The Oregon study did not find statistically significant positive results when looking at objective measures of physical health.
2.) This does not rule out the possibility that a larger study over a longer time horizon would have found statistically significant positive results on objective measures of physical health.
3.) It does mean that this sample size of thousands over the given time horizon was not able to find evidence of statistically significant improvements in physical health.
4.) Now, while this does not rule out the possibility that a larger study would have found statistically significant improvements in physical health, it is also true that if there were very substantial improvements in physical health (anywhere near the size of the improvements seen in other areas) resulting from Medicaid, they would have likely been captured in this study.
5.) This study did capture significant improvements in other areas (financial strain, mental health, self reported health). This suggests that, at the very least, Clarke can admit that this study suggests that the improvements in objective physical health are not anywhere near as large as the improvements in the aforementioned areas.
As for the implications for public policy:
1.) This study does suggest that there is much value in having a program that provides health care for the poor. We can gather this from the results on mental health, financial strain, and even self reported health.
2.) However, this study does not tell us that Medicaid does a particularly good job of being this program. It does not tell us that a smaller Medicaid program that takes a very different form could not offer the same benefits without all the costs.
3.) This is why I say that the study, at the very least, slightly boosts the case for reform.
Now, let us go back to what Raj Chetty said:
1.) The Oregon study found statistically significant positive results when looking at self reported health, mental health, and financial strain. The Oregon study did not find statistically significant positive results when looking at objective measures of physical health.
2.) This does not rule out the possibility that a larger study over a longer time horizon would have found statistically significant positive results on objective measures of physical health.
3.) It does mean that this sample size of thousands over the given time horizon was not able to find evidence of statistically significant improvements in physical health.
4.) Now, while this does not rule out the possibility that a larger study would have found statistically significant improvements in physical health, it is also true that if there were very substantial improvements in physical health (anywhere near the size of the improvements seen in other areas) resulting from Medicaid, they would have likely been captured in this study.
5.) This study did capture significant improvements in other areas (financial strain, mental health, self reported health). This suggests that, at the very least, Clarke can admit that this study suggests that the improvements in objective physical health are not anywhere near as large as the improvements in the aforementioned areas.
As for the implications for public policy:
1.) This study does suggest that there is much value in having a program that provides health care for the poor. We can gather this from the results on mental health, financial strain, and even self reported health.
2.) However, this study does not tell us that Medicaid does a particularly good job of being this program. It does not tell us that a smaller Medicaid program that takes a very different form could not offer the same benefits without all the costs.
3.) This is why I say that the study, at the very least, slightly boosts the case for reform.
Now, let us go back to what Raj Chetty said:
"Other economic studies have taken advantage of the constraints inherent in a particular policy to obtain scientific evidence. An excellent recent example concerned health insurance in Oregon. In 2008, the state of Oregon decided to expand its state health insurance program to cover additional low-income individuals, but it had funding to cover only a small fraction of the eligible families. In collaboration with economics researchers, the state designed a lottery procedure by which individuals who received the insurance could be compared with those who did not, creating in effect a first-rate randomized experiment.
The study found that getting insurance coverage increased the use of health care, reduced financial strain and improved well-being — results that now provide invaluable guidance in understanding what we should expect from the Affordable Care Act."
Nothing Chetty says here is actually incorrect, as I said before. However, I would have to say that a vast majority of readers that know little about the actual Oregon study would finish reading this article with the impression that the Oregon study simply told us that Medicaid "reduced financial strain and improved well-being" and was therefore a vindication of Medicaid.
Clarke says that this a fair characterization of the study. I disagree. The study didn't prove that Medicaid did not improve physical health. However, it did look to see if there was any evidence that it did in any significant manner, and they did not find any. That result does matter.
Clarke is wrong to say that we should just leave out that part of the study. It may not prove that Medicaid doesn't improve physical health, but it does tell us that a study that apparently was large enough to allow us to conclude that Medicaid brings about significant improvements in financial strain, mental health, and self reported health was not able to detect any significant improvements in physical health.
That does tell us something about Medicaid. And, it is not unambiguously positive. An economist of Chetty's caliber writing about why economics should be considered a science should have made a stronger effort to convey this point. After all, that's what any good scientist would do.
No comments:
Post a Comment