When studies fail to take heterogeneity into account, researchers can leap to the wrong conclusion. In a recent open-access article, Dr. Fallon and colleagues describe the four NIH trials, the Krupp Stop-LD study, the Klempner seronegative study, the Klempner seropositive study, and Fallon neurologic study and make a key point. There are considerable differences between chronic Lyme patients (so-called patient heterogeneity) that need to be addressed in study design to improve what he calls the signal to noise ratio. If a study does not address heterogeneity, the results may simply reflect “garbage in, garbage out.”

It’s well known that patients with chronic Lyme disease are heterogeneous.  Some are mildly affected, some significantly impaired.  Some have profound pain, others profound fatigue.  Dr. Fallon and colleagues point out that in study design this variation matters especially when treatment effects are averaged across the sample population that participates in the study.  For instance, if you are measuring response of fatigue to treatment, it is important that each of the patients in the group actually suffer from fatigue.  Otherwise, you cannot expect improvement from treatment for fatigue.  Averaging the results of a patient with fatigue who improves on this score with those of a patient without fatigue who doesn’t improve does not tell us anything useful.  It simply generates noise.

The way researchers account for this type of variation between patients is by selecting patients who are impaired on the primary outcome being measured a the beginning of the study.  So if the study is looking at fatigue, all the patients suffer from fatigue.  If the study is looking at cognitive impairment, all the patients suffer from cognitive impairment and so forth.  By selecting like patients on the outcome measure, the researcher makes the patient population that is being studied more alike (or homogeneous).  This reduces the noise to signal problem that heterogeneous patient populations create in research.

Studies that fail to account for heterogeneity in their sample populations on the outcome they are measuring, are more likely to show no difference or no effect when the results of patients who vary on this outcome are averaged.  Their results essentially cancel each other out. This is called a Type II error – which happens when these studies fail to detect a difference between the treatment and placebo group in heterogeneous groups.

Dr. Fallon and colleagues point out that of the four NIH trials, the two studies that didn’t address heterogeneity issues (the two Klempner trials) failed to detect a treatment difference.  In contrast, the two studies that were designed to address the issue of heterogeneity (Krupp and Fallon) demonstrated a treatment effect.

This is part two of a three-part blog post focusing on very important open access article published recently by Dr. Fallon, director of the Columbia Lyme Center, and colleagues.  My first blog post on this article focused on sample size. My next post will cover Dr. Fallon’s discussion of the importance of separating the analysis of the effectiveness of a treatment from the analysis of whether the benefits of treatment exceed the risks of treatment.  These are separate issues that should not be blended together.

References:
Klempner M, Hu L, Evans J, Schmid C, Johnson G, Trevino R, et al. Two controlled trials of antibiotic treatment in patients with persistent symptoms and a history of Lyme disease. The New England Journal of Medicine. 2001 Jul 12;345(2):85-92.

Krupp LB, Hyman LG, Grimson R, Coyle PK, Melville P, Ahnn S, et al. Study and treatment of post Lyme disease (STOP-LD): a randomized double masked clinical trial. Neurology. 2003 Jun 24;60(12):1923-30.

Fallon BA. Lyme borreliosis: Neuropsychiatric aspects and neuropathology. Psychiatric Annals. 2006;36(2):120-28.

The LYME POLICY WONK blog is written by Lorraine Johnson, JD, MBA, who is the Chief Executive Officer of LymeDisease.org, formerly CALDA. Contact her at lbjohnson@lymedisease.org.