Cécile Bour, MD, May 14, 2020
In the era of EBM (proof-based medicine), the headlines of last few weeks during the Covid 19 pandemic have highlighted the drifts that undermine this approach of a modern medicine.
We should not only blame the recent context; over the last decade and perhaps more, there are three main problems in modern medicine that Jeanne Lenzer, an investigative medical journalist, and Shannon Brownlee, Vice President of the Lown Institute, « a nonpartisan think tank advocating bold ideas for a just and caring system for health in the United States », have denounced in "Issues" column of their site.
These three drifts seriously weaken evidence-based medicine and jeopardize the benefits owed to patients, and they are currently exacerbated in this period of crisis.
Three main drifts of today's medicine
- The main drugstore in the city where I practice has the motto "primum non nocere" above the counters. But, according to Lenzer and Brownlee, doctors are insufficiently trained to discern good science from poor studies, and prefer to use molecules they know and whose effect appears biologically plausible. Fear and haste have sounded the death knell for the Hippocratic maxim, a pillar of medicine. Beliefs and unproven faith in medications used for other medical conditions have been considered, without concern for their real usefulness against an emerging and unknown virus, or their possible harmfulness.
- The second pitfall identified by the authors is the prominent role of the media today. On the one hand, convictions and opinions of non-physicians are following in the footsteps of politicians, to whom the floor is widely and generously given. On the other hand, many of media that echo this are ignorant and incompetent in scientific method or in search for evidence, and also have a commercial interest in hype around these well-known and popular, but null, town criers.
We can only agree with this observation, having ourselves been confronted with a very unpleasant episode of attacks from a TV host, who mediated his experience of prostate cancer at will to convince people who asked him nothing, to undergo prostate screening test, in spite of all the non-recommendations of such screening [1].
The naive crowds that "follow" these characters, fan clubs generally very well supported on social networks, do not facilitate the expression of cautious doctors or independent collectives like ours, which are inevitably a killjoy in the general hope. The word of the Cassandras is hardly a media-friendly one...
●The third calamity denounced by Lenzer and Brownlee concerns poor quality studies that do not include what is called a "control arm", i.e. inclusion in the study of a comparison group that has not been given the drug being tested. The control arm is the cornerstone of a strong randomized controlled trial. It aims to formally establish the true efficacy of a product, highlighting possible biases in the tested protocol. For example, a drug may appear to be effective in reducing the risk of a given disease when the population to which it was administered is already naturally less at risk, because they are younger, healthier or have easy access to medical care. A control arm with a wide range of subjects can show ineffectiveness of the test drug in certain groups of population and thus point to errors in reasoning or biases that would go unnoticed. In hastily constructed studies, on the other hand, potential harms may be underestimated.
But even outside of critical health periods, such as in the field of cancer for example, proclamation of "miracle" drugs has been legion in recent years, with questionable studies, selling hope of so-called "revolutionary" chemotherapies. On this subject read the post of Annette Lexa, our toxicologist.
We add to this last point on the poor quality of medical researchn a particular type of study that is currently spreading, it is the "non-inferiority" test, all the more fraudulent and perfidious that nobody understands a drop.
Doshi and all. 2] studied informed consent forms from non-inferiority studies on antibiotics. Their observations are generalizable, however, because there are many applications of this type of study, in diabetes, cardiology, infectiology and cancerology. Doshi et al. found that often neither the methodological experts nor the members of personal protection committees were able to define the true objective of the study from the information forms given. For the methodologists, only 1 out of 50 trials, according to them, correctly restituted the objective of the study; for the patients, 7 out of 50 studies succeeded in doing it. These results raise the question of whether consent is truly an informed one, and thus whether the trial is even ethical.
But what are we talking about?
Non-inferiority trials
We have given a summary explanation about the MyPEBS study for personalised breast cancer screening which concerns the theme of our site, a study that we have analyzed in depth here: https://cancer-rose.fr/my-pebs/.
In the non-inferiority trial it is a question of comparing two elements (two medical devices or two procedures, or two drugs) to check whether the device or procedure or treatment being tested would not be worse than what is already in use, accepting a certain loss of effectiveness within a certain tolerated margin, known as the non-inferiority threshold.
It is not a question of verifying whether the system, procedure or drug tested would be superior to former one. This is often how the press reports it and how doctors and public understand it, but it is not the case!
For example, for breast cancer screening, intended purpose is normally to reduce severe forms of cancer. In the MyPEBS non-inferiority study, if the new personalized screening tested does not appear to favor a rate above 25% (arbitrarily set threshold) for additional severe cancers, the trial will be declared a success. You have read correctly. There is no control group there either to test what would happen without screening, which would be legitimate though since recent studies on this screening no longer manage to demonstrate a positive benefit/risk balance.
It is true that in non-inferiority trials in cancerology the control arm is rarely considered, this being judged to be unethical given seriousness of disease, and patient cannot be deprived of any care. But here, in the MyPEBS study, healthy and non-complaining women are being tested, we should remember.
What must therefore be understood, in short, is that it is possible, with non-inferiority trials, that a new health procedure can be accepted as effective, even if its therapeutic or beneficial effect is slightly inferior to current standard. In a non-inferiority trial, the new procedure or drug is not supposed to make the participant healthier than she would have been outside the trial, since superiority of the procedure or drug is not sought.
The only assumptions are :
- In the best configuration, participants randomized into the study's test group could fare as well as if they had not participated in the trial,
-or then potentially worse within an arbitrarily accepted margin, in the wrong scenario.
And everyone is happy. The public because they are not well informed and believe that what was tested on them is 'superior', the journalists who did not grasp the subtleties of the methodology and write laudatory articles, and above all the designers of the study. Why is such an approach necessary? Because with this set-up, the study is subtly biased towards the result desired by the promoter, which is to obtain or safeguard a market share much more than to answer a scientific question whose stake is the well-being of the patient. For MyPEBS, the aim is to establish breast cancer screening, since women will only have a choice between two options: old screening or personalized screening, but screening in all cases.
To caricature, the patient or population will be worse or not less well, but the good news is that the study is a success..
Two interesting articles to help the practitioner in the critical analysis of the studies that published
Vinay Prasad, in an editorial in the JGIM[3] (Journal of General Internal Medicine) where he actually quotes the results of Doshi et al, refers to the publication of Aberegg, Hersh and Samore who analyzed 183 non-inferiority comparisons of 163 clinical trials published in the five major impact medical journals.
Aberegg and colleagues found that only 70% of the non-inferiority studies explicitly stated why new therapy would have an advantage, and in 11% of cases no advantage could be inferred. This suggests to these authors that many of these studies should not have been conducted.
What should the practitioner beware of and what trials of non-inferiority can he trust? According to V.Prasad, it is necessary to :
- Consider whether new therapy is less expensive, more convenient, less invasive or less toxic than old one. If answer is no, stop looking, he says! There must be a positive compensation for the patient for loss of effectiveness of the new procedure or new drug being tested. If not, there is no point in learning about the "novelty".
- Ask yourself how much loss of effectiveness of the new procedure or new treatment you would be willing to accept in order to adhere to it. 5% loss of effect, or 10%? More or less?
- Be concerned about the margin of inferiority accepted. E.g. for MyPEBS the non-inferiority threshold is very generous. This means that if, at the end of the new screening, 25% additional serious cancers are found, the study is "successful". This margin would have to be justified, and this justification would have to appear at least in the study protocol, which is not the case.
- Finally, consider whether the new intervention was in fact 'inferior', not 'non-inferior'.
The French independent journal Prescrire[4] examined problems of these particular trials in 2006. Their advice is similar to that mentioned above. For the journal, one must be critical of the threshold of non-inferiority that is chosen a priori, arbitrarily by trial designers. For the patient, this threshold is equivalent to the loss that is consented to in relation to the reference treatment or device. It is therefore necessary to be sure that the result is not in fact a true inferiority.
For Prescrire, in short, these tests are intended to simply exclude the possibility that a treatment or process is massively worse than what already exists. When the treatment or procedure is slightly less effective, and within a certain accepted margin, the new treatment or procedure is only of interest to the patient if it brings other benefits in compensation. And this is where the stick can hurt, as in this study we regularly criticize, and not just us.
To conclude:
The urgency and willingness to obtain results quickly justifies dispensing with well-conducted controlled studies and exposes us, because of flawed and poorly executed studies, to biased medical judgments and erroneous conclusions.
Improper, sometimes excessive, communication by lay people or ignorant media aggravates media coverage of drugs or procedures that we do not know what they actually induce, whether they do more harm than good.
These problems pre-existed before the context of the Covid19 pandemic which only brought them to light.
Research in science is important, yet it must meet quality standards, be done by scientists who are concerned about respecting proven methodologies and meeting standards, and this in a serene media environment.
This is currently not the case, as damage and physical inconveniences to health of populations are aggravated by misinformation and pollution of scientific debate by quarrels of opinion.
If we want to make good science that is useful to people, we must develop a critical spirit, constantly verify results of studies promoted as revolutionary by media, see by which methods conclusions have been reached.
We have come full circle; if we want to make good medicine we will always return to what is proclaimed at the front of the drugstore, the basic principle that must underpin our medical practice: "primum non nocere" .
References
[1] https://cancer-rose.fr/2020/02/06/ah-mais-quelle-aubaine-ce-cancer/
[2] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5710221/
[3] EDITORIAL
Non-Inferiority Trials in Medicine: Practice Changing or a Self-Fulfilling Prophecy?
Vinay Prasad, MD, MPH
Division of Hematology and Medical Oncology, Knight Cancer Institute, Oregon Health and Science University, Portland, OR, USA; Department of Preventive Medicine and Public Health, Oregon Health and Science University, Portland, OR, USA; Center for Health Care Ethics, Oregon Health and Science University, Portland, OR, USA.
J Gen Intern Med 33(1):3–5
DOI: 10.1007/s11606-017-4191-y
© Society of General Internal Medicine 2017
[4] La Revue Prescrire avril 2006/Tome 26 N°271, page 249