Charlie Clarke and his co-blogger, Robert H, are both sure that I'm either committing some sort of statistical fallacy or suffering from some misunderstanding. The thing is that I'm not. Instead, both Robert H and Charlie Clarke are using very faulty logic and misunderstanding my position.
They are both convinced that the only relevant results from the Oregon Medicaid study are the positive results on mental health, self reported health, and financial hardship. In other words, the only relevant results are the ones that reinforced their preexisting beliefs about Medicaid. The results that are less convenient for their worldview (the ones on physical health) should be thrown out.
The justification for this ideologically convenient analysis of the study is that the lack of positive results on physical health supposedly tells us absolutely nothing about the effect that Medicaid has on physical health. Obviously, that is a rather absurd view. In fact, a correct reading of the results on physical health do make claims of Medicaid improving physical health dramatically much weaker.
Typically, people are guilty of overstating their case. Oddly enough, as I read back on this exchange and look at the evidence, it seems that I have been understating my case. The lack of positive results on Medicaid is indeed a rather poor result for Medicaid. This was not a small sample size by any means. The study was not, by any reasonable measure, underpowered. If Medicaid made substantial improvements in health, it should have shown up this study.
As I have said, this does not mean that this experiment disproves significant health improvements stemming from Medicaid. It does not. However, it does significantly weaken the justification for belief in such effects.
The more I read Clarke and H, the more it seems that what they are really saying is "the results don't matter unless positive results show up". In other words, heads they win, tails we can't draw any conclusions from.
If these results did turn up significantly positive, there is only one likely conclusion that we could come to:
1.) Medicaid significantly improves physical health relative to being uninsured.
If they turned up with no statistically significant effects, there are really only two likely conclusions:
1.) Medicaid does not significantly improve physical health relative to being uninsured.
2.) Medicaid does significantly improve physical health relative to being uninsured, but not by enough to have significant results in this study.
Clarke and H seem to have come to the rather absurd conclusion that the results would only have mattered if they came out statistically significant in some direction. That is incorrect. Virtually nobody would have predicted that Medicaid would have significantly reduced physical health. Instead, the critics of Medicaid would predict a lack of major improvement.
I'll also add that I have let Clarke off the hook for ignoring the raw numbers that actually show Medicaid reducing certain measures of health. Of course these numbers are not statistically significant, but they are closer to statistical significance than most of the positive results on physical health. If Clarke is going to claim that these raw numbers suggest improvements in physical health, he needs to also claim that they suggest Medicaid hurts the cardiovascular health of the already sick and increases smoking. (more on that here). Of course, I think it is obviously wrong to suggest that this qualifies as evidence that Medicaid hurts cardiovascular health among the sick or increases smoking, but that is the logical conclusion that Clarke's arguments should lead him to.
By the way, there's been another round of results from the Oregon study that don't find statistically significant changes in employment resulting from Medicaid. I'm waiting for Clarke and H to explain to the NYtimes that they really shouldn't be reporting on these results as they don't really tell us anything about the effects that Medicaid has on employment. I would agree with the NYtimes here. But, if the folks at Bad Outcomes are to be logically consistent, they should be accusing them of committing all kinds of statistical fallacies. Don't hold your breath.
No comments:
Post a Comment