LYMEPOLICYWONK: New Study Reveals Fatal Flaws in NIH Klempner Trial Statistical Analysis. Is this Error Human, Incompetence or Worse?
A new study by Allison DeLong, Barbara Blossom, Dr. Elizabeth Maloney, and Dr. Steven Phillips entitled “Antibiotic Retreatment of Lyme disease in Patients with Persistent Symptoms: A Biostatistical Review of Randomized, Placebo-controlled, Clinical Trials” examined the quality of the studies, statistical analysis and conclusions reached by the four NIH-funded clinical trials for persistent Lyme disease. For this blog, I am going to focus on one of them, the Klempner trial. It is the trial most commonly cited by the Infectious Diseases Society of America to support the IDSA guidelines recommendation that patients who remain ill after treatment should not be retreated. The trial involved both a seropositive arm (patients who tested positive for Lyme disease) and a seronegative arm (patients who tested negative for Lyme disease).
So what was the fatal error in the Klempner study?
Researchers who design clinical trials define when a treatment is considered a success. Does the treatment work or not? How much does a patient have to improve before you can say the treatment helped? Do they have to return to perfect health? Clinical trials usually require that patients improve, but not that they return to perfect health. DeLong found that the Klempner trial set the level for determining treatment success excessively high. For instance, in the seronegative arm of the trial, treatment success required that patients not simply return to the norm for the general population, but instead surpass the norm by essentially one full standard deviation. That type of success is unheard of in clinical trials.
More specifically, the Klempner study used too high a level to determine success on something called the SF-36 scale. It used a measure that was not clinically meaningful. Let’s look at this in context. While other chronic diseases set the bar for improvement on the SF-36 between 2 to 5, the Klempner trial set the bar at roughly 7 and 9 in the seropositive arm and between 9 and 13 in the seronegative arm. This amount of improvement is far higher than the amount of improvement that could reasonably be expected among the chronically ill.
If the Klempner trial had used a clinically meaningful measure of success, say between 2 and 5, it would have had to have used much, much larger sample sizes, which it did not have. For instance setting the clinically meaningful measure of success at 2 would have required a sample of 400 or more per treatment arm. The sample sizes in Klempner were only 78 (seropositive) and 51 (seronegative)—a sample size which would not yield statistically significant results. When researchers select an excessively large treatment effect required for success, as was done here, the size of the sample required is reduced. The trade-off, however, is that the results are not clinically meaningful.
Many researchers say that small sample size trials are unethical because they can literally steer scientists down the wrong path. That’s what the Klempner study did. It was a waste of money, a waste of time, and it has led research down the wrong path for the last 10 years.
Was it a matter of researcher incompetence? DeLong and colleagues point out that studies on the level of improvement that is clinically meaningful for other chronic diseases had not been done at the time the Klempner trial was conducted. So the Klempner researchers did not know that they were using an excessively large treatment effect at the time they conducted the study. Does that take the researchers off the hook? Not really. First of all, once the studies on the way to measure success in chronic disease had been conducted, the Klempner team could have acknowledged a possible error.
Second, the Klempner team could have avoided “overstating” its findings and highlighted the potential for error. The ethical implications of exaggeration in science are understood by other scientific societies. For example, the code of conduct for the American Chemical Society states: “Public comments on scientific matters should be made with care and accuracy, without unsubstantiated, exaggerated, or premature statements.”
Instead, the Klempner study concluded flatly that retreatment was ineffective. The NIH press release headline read: “Chronic Lyme Disease Symptoms Not Helped by Intensive Antibiotic Treatment.” It further concluded that “ it is unlikely that a longer course of treatment or different antibiotic combination would result in greater improvement than what we found in these studies.” This conclusion was clearly beyond the scope of the study since it did not test other antibiotics or other durations of antibiotic therapy.
In short, rather than acknowledging the potential weaknesses of its findings and encouraging further research, the Klempner study overstated its conclusions and firmly shut the door on future science. Dr. Klempner himself then sat on the 2006 IDSA Lyme disease guidelines panel, which used his study to deny seriously ill patients any access to care. Research of fellow panel members on a guideline panel may be less likely to be critically challenged, and the IDSA guidelines are riddled with references from the panel members.
This guideline recommendation has caused untold suffering by patients, who are essentially told to simply live with the symptoms of chronic Lyme, which can cause disability equivalent to that of congestive heart failure. For example, 25% of over 4,000 patients who responded to a survey by LymeDisease.org have been on disability at some point in their illness. If a guidelines panel is going to preclude further care of patients this ill, they should be absolutely certain that further treatment will not be beneficial. And, they should err on the side of treatment. Denying patients access to care when treatment options are available results in unnecessary suffering.
All of this brings us back to what Congressman Smith referred to as the “lost decade” of research in Lyme disease. He was referring to the delay in publishing the Embers research trial which found persistent infection in monkeys after treatment using the Klempner protocol. The lack of scientific rigor in the Klempner trial coupled with the delay in publication of the Embers trial has caused the Lyme community enormous harm. Both studies were NIH funded, and, again we are left to wonder why the NIH and Dr. Phil Baker, who headed up Lyme research for the NIH, failed to properly oversee and follow-up on this research. Why aren’t they setting the record straight now that they have the information?
It brings to mind a comment that Dr. Stephen Barthold made at the congressional hearing chaired by Congressman Smith:
There is overwhelming evidence in a variety of animal species as well as humans that B. burgdorferi persists without treatment, but the crucial question is does it survive following treatment, and if so, do surviving spirochetes cause “chronic” Lyme Disease or PLDS? These questions cannot be answered by speculative and expensive human clinical trials motivated by firmly held dogmatism.
Hear! Hear!
References:
Delong, A. K., B. Blossom, et al. (2012). “Antibiotic retreatment of Lyme disease in patients with persistent symptoms: A biostatistical review of randomized, placebo-controlled, clinical trials.” Contemp Clin Trials.
Klempner, M., L. Hu, et al. (2001). “Two controlled trials of antibiotic treatment in patients with persistent symptoms and a history of Lyme disease.” N Engl J Med 345(2): 85-92.
Embers, M. E., S. W. Barthold, et al. (2012). “Persistence of Borrelia burgdorferi in Rhesus Macaques following Antibiotic Treatment of Disseminated Infection.” PLoS ONE 7(1): e29914.
My previous blog postings regarding the Embers monkey trials and their importance for Lyme patients can be found here.
The LYME POLICY WONK blog is written by Lorraine Johnson, JD, MBA, who is the Chief Executive Officer of LymeDisease.org, formerly CALDA. Contact her at lbjohnson@lymedisease.org.
Lorraine, these findings are SO meaningful. Thank you for your review of this!
Jill
I have always wondered why the NIH is so unscientific when it comes to lyme disease. Their handling of this disease, during Phil Baker’s tenure and even after he left, has been terrible. The people who are in charge of infectious diseases at that institute have got some explaining to do, hopefully on their way out the door, making way for more honest and competent people.
Who at NIH has set this policy? We want to know.
Another great finding to further patients’ needs. Thank you for sharing it! This among other late-breaking info will hopefully encourage my community to change their current policy of following IDSA guidelines, causing a plethora of patients to simply take their business elsewhere AND get better. The nerve.
Quite embarrassing if they want a reputation of competence, and pretty soon it will be hard for any place using only IDSA guidelines to back-pedal without looking stupid and/or totally uncaring towards their patients. And when kids are involved … that leads to some pretty big scowls aimed their way.
Interesting article. I had always wondered, given all the noise about evidence-based medicine, how this underpowered study of less than 150 patients came to be the justification used to deny every lyme patient the world over, any treatment of any antibiotic for any tick borne infection. Plus the trial was halted before its completion leading to more questions. With no other disease could they have gotten away with this.
I have to think that there was a lot of force applied to make sure this one underpowered trial was made into the end-all be-all of all tick borne illness treatment.
I hope the chickens come home to roost before all the farmers who raised them are long dead.
So, any chance that the NIH will now fund a study similar that is BETTER DESIGNED so that we can have an ACTUAL double blind scientific study that yields usable data that doesn’t lie? The truth about what is happening isn’t going to change no matter HOW much the IDSA and then NIH resist… so let’s see a real study… let’s see them admit their mistakes and utter failure to address concerns. Let’s see them step up and agree to REALLY investigate chronic and persistent lyme disease! Let’s see someone who has POWER inside the NIH and the IDSA put their egos aside and give this an honest look. The truth will out… which side of it do they want to be on when it does? The side that tried to HIDE it or the side that tried to HELP shed light? Well, I think we already know what their answer to that has been in the past, but what about now? Who was it that said that pretty soon it’s going to be “what did you know and when did you know it”…. because that time is getting closer by the second.
My case ! Several tick bites 26 years ago I pulled them off, rash, all Lyme symptoms save arthritis, including Lyme rage. 9 years late first clinical Dx but negative Elisa Doxy for two months nevertheless. 14 years of infections, antibiotics a minimum of 4 weeks each year, plus 3 ” related” surgeries. Igenex test 2 clinical Dx 6 years of ILADS heavy protocols . Now symptom free, save from babesia air hunger and fatigue. IDSA is still for now into Perception Management. The mounting pressure from the infected people the public and the Dr’s market as we can see now will eventually prevail. IDSA will fold and blame somebody else or find an excuse.