In a prior posting, the Disease Management Care Blog made a statistically based argument about the plus-or-minus distribution or 'spread' of cancer treatment-related survival around an average. It referred to a specific Erbitux research paper, where there were two treatment groups, each with an 'average' survival with a standard deviation. The DMCB compared and contrasted the two standard deviations, not the average results, and went on to state that cancer patients and their oncologist physicians are more interested in the upper and lower bounds of possibility, not what happened to most patients in the middle.
Well, wouldn't you know, a statistician emailed the DMCB with some insights about its faulty logic. It likes to hear from statisticians almost as much as from actuaries, especially when there's learning to be done. This reminds the DMCB of a key difference between statisticians and actuaries: the former tells you how you were wrong, while the latter tells how you're going to be wrong.
Cody L. Custis is employed by the State of Montana and also teaches mathematics at UM Helena. He points out that he is speaking as an individual and not for any of his employers. The DMCB points out it doesn't feel so bad if it took these kinds of credentials to uncover a mistake.
Here' the email:
As a professional statistician, I wanted to raise two objections about the conclusions in your blog's posting on Erbitux. In it, you say:
“Check out this real life clinical trial that is available on line. It showed Erbitux resulted in a median survival of 12 months with a confidence interval ranging from about 8 ½ to just over 15 months versus just over 9 months of survival with a confidence interval extending from about 7 ½ months to just under 12 months without Erbitux. This means the real bottom line in this trial is that getting Erbitux may result in a life expectancy as high as 15 months versus a life expectancy as low as 7 ½ months without Erbitux.”
Based upon the study referenced, I assume that these conclusions come from the following statement in the Butts et. al. paper:
"Median survival time was 11.99 months in the cetuximab arm (95% CI, 8.80 to 15.18) and 9.26 months in the platinum/gemcitabine arm (95% CI, 7.43 to 11.79)."
First, if a clinical trial involving two groups results in two sets of data, there will be two separate averages or means. If one group’s average result is compared to the other group’s average, it is important to not only know the difference between the two means, but also the confidence interval for that difference. In contrast, your blog compared two separate confidence intervals. In the Butts et. al. paper, the authors did not construct a confidence interval for the increase in life expectancy in the Cetuximab study; rather, they calculated two separate intervals for two treatments. Because confidence intervals are given for the two treatments separately, rather than for the difference, cancer patients and their physicians cannot know if a difference in the life expectancy of 1.5 months is really significant. Mathematically, if the confidence interval extends to zero, the 1.5 months could be the outcome of random chance. As the authors of the Butts et. al. paper state in their conclusion : 'The major limitation of the study was its noncomparative design, not statistically powered to demonstrate significant differences between treatment arms.'
Second, your blog's conclusion takes the worst possible case for one treatment and the best possible case for another treatment. Thus, while your blog is correct to focus on the importance of confidence intervals rather than point estimation, the conclusions are based on unfair assignment of best and worst case outcomes of two treatments. A skeptic could just as fairly state the conclusion: this means the real bottom line in this trial is that getting Erbitux may result in a life expectancy as low as 8 months versus a life expectancy as high as 12 months without Erbitux. Both conclusions unfairly take extreme outcomes.
The DMCB says good points. In the paper referenced, patients would be better served by knowing the distribution around the average difference in survival and should also be made aware of the up and down sides of any treatment option.
That being said, the DMCB also thinks, based on experience, that most patients and their oncologists tend to believe in the most optimistic treatment scenarios. If those scenarios fall with the reasonable (plus or minus) bounds of possibility, it's hard for insurers, policy makers, comparative effectiveness researchers, regulators and legislators to say no.
The DMCB thanks Cody Custis for the insights.
Well, wouldn't you know, a statistician emailed the DMCB with some insights about its faulty logic. It likes to hear from statisticians almost as much as from actuaries, especially when there's learning to be done. This reminds the DMCB of a key difference between statisticians and actuaries: the former tells you how you were wrong, while the latter tells how you're going to be wrong.
Cody L. Custis is employed by the State of Montana and also teaches mathematics at UM Helena. He points out that he is speaking as an individual and not for any of his employers. The DMCB points out it doesn't feel so bad if it took these kinds of credentials to uncover a mistake.
Here' the email:
As a professional statistician, I wanted to raise two objections about the conclusions in your blog's posting on Erbitux. In it, you say:
“Check out this real life clinical trial that is available on line. It showed Erbitux resulted in a median survival of 12 months with a confidence interval ranging from about 8 ½ to just over 15 months versus just over 9 months of survival with a confidence interval extending from about 7 ½ months to just under 12 months without Erbitux. This means the real bottom line in this trial is that getting Erbitux may result in a life expectancy as high as 15 months versus a life expectancy as low as 7 ½ months without Erbitux.”
Based upon the study referenced, I assume that these conclusions come from the following statement in the Butts et. al. paper:
"Median survival time was 11.99 months in the cetuximab arm (95% CI, 8.80 to 15.18) and 9.26 months in the platinum/gemcitabine arm (95% CI, 7.43 to 11.79)."
First, if a clinical trial involving two groups results in two sets of data, there will be two separate averages or means. If one group’s average result is compared to the other group’s average, it is important to not only know the difference between the two means, but also the confidence interval for that difference. In contrast, your blog compared two separate confidence intervals. In the Butts et. al. paper, the authors did not construct a confidence interval for the increase in life expectancy in the Cetuximab study; rather, they calculated two separate intervals for two treatments. Because confidence intervals are given for the two treatments separately, rather than for the difference, cancer patients and their physicians cannot know if a difference in the life expectancy of 1.5 months is really significant. Mathematically, if the confidence interval extends to zero, the 1.5 months could be the outcome of random chance. As the authors of the Butts et. al. paper state in their conclusion : 'The major limitation of the study was its noncomparative design, not statistically powered to demonstrate significant differences between treatment arms.'
Second, your blog's conclusion takes the worst possible case for one treatment and the best possible case for another treatment. Thus, while your blog is correct to focus on the importance of confidence intervals rather than point estimation, the conclusions are based on unfair assignment of best and worst case outcomes of two treatments. A skeptic could just as fairly state the conclusion: this means the real bottom line in this trial is that getting Erbitux may result in a life expectancy as low as 8 months versus a life expectancy as high as 12 months without Erbitux. Both conclusions unfairly take extreme outcomes.
The DMCB says good points. In the paper referenced, patients would be better served by knowing the distribution around the average difference in survival and should also be made aware of the up and down sides of any treatment option.
That being said, the DMCB also thinks, based on experience, that most patients and their oncologists tend to believe in the most optimistic treatment scenarios. If those scenarios fall with the reasonable (plus or minus) bounds of possibility, it's hard for insurers, policy makers, comparative effectiveness researchers, regulators and legislators to say no.
The DMCB thanks Cody Custis for the insights.
Posts
No comments:
Post a Comment