More Transparency Is Needed on Comparative Effectiveness Research

During the last of the Federal Coordinating Council for Comparative Effectiveness Research’s “listening sessions,” the folks from the Population Health Impact Institute (PHII), in partnership with ABQAURP, AHIMA, CMSA, and IHPM, advocated on behalf of better transparency of Federally funded comparative effectiveness research (CER). Its statement argues that consumers of the information need to be able to assess the merits of any and all conclusions that purport to show that one treatment approach is truly better than another. They also pushed the idea that researchers who accept CER funding should sign a transparency pledge and supported reimbursing experts who perform ‘peer review.’ You can listen in on all the proceedings here. Don't take the DMCB's word on how this is noteworthy. The PHII comments gained some national press coverage here and here

The Disease Management Care Blog agrees these are all good ideas. Here’s why:

Open up any medical-scientific journal and you, along with the majority of other readers, will be tempted to skip past the ‘methods’ section, scan the results and jump right to the conclusions. So will much of the media. However, if the topic is really important to you, your patients or your business, the methods may be the most important art of the mansucript. As a rule of thumb, if the DMCB doesn’t understand the methods section because it is lengthy and littered with complex jargon that involves combined/irrelevant endpoints involving subgroups and strained comparisons, it generaly believes the results are suspect and moves onto the next article. PHII’s recommendations, if carried out, could go a long way toward reassuring readers of CER that complex methods aren’t being used to obscure weaknesses in the conclusions.

As we know, strings are being attached to other sources of Federal funding. If it is good enough for business and finance, it is certainly good enough for health-related research. PHII thinks any researcher should be willing to put their pledge where his or her month is. In fact, PHII is perfect organization to offer such a pledge document.

All the ‘peer review’ medical-scientific journals maintain a bullpen of ‘go to’ clinicians/scientists/researchers who agree to read submitted manuscripts that are being considered for publication. The DMCB has served as a peer reviewer for a number of top flight journals and it can be a tedious task: the writing is seldom polished and the conclusions are often not valid. As a result, it's often up to the peer reviewers to serve as an ‘outsourced’ editorial staff to help correct the English and point out the flaws in the underlying science. Peer reviewers generally agree to do this because its flattering to be asked to do it, breaking information is available before it makes it to print, it makes for good ballast on a CV or resume and sometimes it leads to being asked to write an editorial. However, ultimately, reviewers typically do this on their own dime and some may not give their reviews the attention they deserve. PHII argues its time for the peer review process – and that part of the U.S. economy devoted to health care - to get what it pays for.

As noted in prior posts, the DMCB would go one step further and go ‘open source’ on CER. Publicly funded research should generate publicly available data. Ultimately, anyone should be able to download the data and - assuming they adhere to privacy standards and use deidentified data - keep the original researchers and their conclusions honest. Who knows, they may find something that was missed. If no one can attack the integrity of the original data, the federal sponsors of CER will be that more confident that the conclusions are valid.