- Is the study design of high quality (includes control group, placebo, adequate randomization)
- Are the statistical methods appropriate?
- Is the sample size large enough?
While there are tons of reasons that a study can be faulty, these 3 factors are the things that tend to make a study pass a cursory sniff test.
In an ideal world, all of the things that doctors and therapists use in practice would be supported by a well designed randomized clinical trial, with sample sizes into the hundreds, if not thousands. Unfortunately, this isn’t the case for a lot of what happens in a doctor or therapist’s office on a day to day basis. Myself included.
If health care isn’t supported by good evidence, what is actually happening in a doctor’s office?
Dr. David Eddy is a former surgeon turned medical economist. In 1991, He turned the world of health care on it’s head when he was able to show that just 15-20% of medical interventions are supported by hard evidence. That means that 80% of medical interventions were UNSUPPORTED by rigorous science.
This caused a stir in science and medicine, and many institutions rushed to address the problem. Although there’s been improvement, even the most optimistic views in medicine estimate that 46-70% of of practices are evidence-based depending on the setting (primary care, specialty, emergency room).
So if roughly 50-60% of health care interventions are based in hard evidence, what does that mean for everything else that a doctor is doing? It means that we are just doing our best at integrating our knowledge about the human body, our knowledge about you as an individual, and our past experiences with other similar patients to develop a solution to your problem.
In other words we have to rely on anecdote aka n=1, to make a clinical judgement.
Why n=1 is important (sometimes)
Evidence-based medicine relies on high quality studies to develop treatment algorithms so that doctors can decide what is most likely to work for various conditions. It also helps us find out if a specific intervention is more likely to be effective, more likely to be junk, or more likely to be dangerous.
It’s very literally playing a game of probability and statistics. When used correctly, it can be a powerful tool to help save people time, money, aggrevation, or even their life. The first line of treatment and management should be based on good evidence.
It may seem odd to hear that from a chiropractor because for many of the cases I manage on a regular basis; the things that I’m doing are not based on a rigorous clinical trial.
Here’s the caveat. Even the best treatments that are supported by the best evidence have people that fail to respond. We see this all of the time when patients have chronic pain syndromes. If you have a chronic pain syndrome, you’ve probably gone to your primary care doctor for routine anti-inflammatory meds. If that didn’t work, you likely went to a neurologist or orthopedist to see if injections or higher level narcotic medications would help. If that didn’t work, then maybe you’ve also seen a physical therapist for exercise therapy.
At each level, the doctor prescribed an intervention that is based in good evidence, but you didn’t get a great result. Does that mean that those drugs, shots, or exercises don’t work? Of course not. It just means that it didn’t produce a response for YOU.
That’s why n=1 matters, because the most important n=1 in the world is YOUR case and YOUR situation.
When Traditional Methods Fail, You Must Become Your Own Case Study
In clinical research, the lowly case study sits near the bottom of the evidence pyramid, and for good reason. Just because something worked once for someone else is a poor representative that it will work for you.
In fact, one of the biggest critiques of alternative approaches is that most of their research on these lower levels of evidence.
But what should you do when your doctor has done everything he’s supposed to recommend for you based on high quality evidence? You can either assume that your case is done and nothing can help you, or you can start exploring a little bit outside the realm of what’s considered evidence-based medicine.
This is where treating yourself as a case study can come in handy. You are now the scientist in your own self-experiment with the goal of making your body function better. Here are some of the more common things people experiment with:
- Will a Paleo diet help me with MS?
- Will going gluten-free or dairy-free help me with my allergies?
- Will these probiotic supplements help make my irritable bowel syndrome go away?
- Will this new athlete programming make me a better/faster athlete?
Thousands of people are doing these very things every single day because their self-experiment shows that it has made a meaningful and important contribution in their lives.
Here’s the thing:
You aren’t going to find great evidence to support these interventions, but it doesn’t mean that these interventions are useless. Sometimes the study hasn’t been done to prove or disporve a treatment. Other times, we just have to acknowledge that our unique DNA and life circumstances mean that some of us will be more responsive to a treatment where others will not.
You Can Also Test How You Respond to Unproven Treatments
This is pretty much how most people end up in the offices chiropractors, acupuncturists, and other practitioners that are tossed into the bucket of alternative medicine. People are experimenting with these alternative approaches because they want to see if one of these things will finally work for them.
People generally DO NOT see one of these providers as their first option. People care about whether the treatment or doctor that they seek is proven, and they rely on their primary care doctor to guide them through evidence-based treatments first.
No one ever says, “Man, this vertigo is terrible. I need to go google a chiropractor in the area”. Why would they?
People want to just go to their doctor, take a medicine, and have their problem dealt with. With these ancillary approaches, they will usually have to pay money out of their own pocket and go for several visits to get a desired result.
Anecdotes and case studies only have real power when you are frustrated or desperate for answers to problems that have been plaguing you for a long time. For better or worse, and despite the fact that most of alternative medicine has poor evidence, it has found a niche in helping people where proven conventional treatments have come up short.
A Guide for Navigating the Unproven
Just because I’ve made a case for the importance of anecdotes and treating yourself like a case study doesn’t mean I believe people should blindly jump into a quagmire of unproven treatments. That would be silly, a waste of money, and possibly dangerous.
One of my favorite authors is Tim Ferriss for the very simple reason that he does so much in the world of self-experimentation. His book The 4-Hour Body is one of the definitive guides for using experimentation to make your body better. His blog is a tremendous resource, and he wrote a great article about the value of self-testing and you can read that here.
The Value of Self-Experimentation
Whether it’s an off label use of a supplement/medication or you’re exploring alternative medicine for the first time, you need a filter to decide if it’s right for you. Here are some things to think about when you’re faced with a decision to do something that’s not necessarily.
How risky is it? – first and foremost, what are the risks of performing this self-experiment? Obviously a surgical procedure is going to have greater inherent risks than say…..giving up gluten for a month. However, there are a ton of interventions that live in a gray area. Certain herbs may have adverse reactions with medications that you need to survive, while stopping certain medications cold turkey can have a terrible impact on your day to day life.Make sure you have a frank and honest conversation with a doctor before you start experimenting because your doctor may know more things about your health history that may complicate matters in a negative way.
Is there biological plausability? – While not many interventions have a large randomized clinical trial to support it’s use, many doctors and therapists use their knowledge of anatomy and physiology to determine if something is worth trying. This concept is called biological plausability. Is there a likley physiological mechanism that can explain how a treatment or intervention can work?Biological plausability can be rooted in animal studies, or based on what is known about how the human body functions. It’s not a replacement for a high quality clinical trial, but it can serve as a temporary stop gap until a good study is performed. Chiropractic frequently relies on biological plausability for many of the conditions that it is known to help.
There are no randomized clinical trials that show that the NUCCA approach is effective for migraines….yet.However there is a biologically reasonable explanation that gently correcting the neck can help reduce migraines.
When I combine the biology with a past history of success with migraine cases, I can still have a lot of confidence in saying that this approach can help many migraine patients.
Can You Measure the Change? – The placebo effect can be powerful, especially when it comes to a subjective symptom like pain. While I don’t have a big problem with placebos, I certainly understand that people don’t want to potentially waste time, energy, or money on something that is likely not a real treatment effect.
One of the ways we can account for potential placebo effects is by measuring an objective physiologic change. One of the great things about living in 2015 is that you can measure so many health metrics through your phone. Things like blood pressure, heart rate variability, pulse, eye movements, posture, and sleep tracking is all available in app form. Companies like uBiome and WellnessFX have made it easier than ever to have lab tests done by bringing it directly to the consumer.
When you measure some of these physiological changes that happen in your body, you can quantify how much an intervention is impacting your life. You can even use these metrics to fine tune dosages, or to see if your body is continuing to falter despite the fact that you FEEL better.
Is the Measurement Valid and Reliable- This fact mainly applies to selecting a practitioner. Most doctors and health practitioners are going to have some type of tool to measure your improvement. However, there are some people that use tools in inappropriate ways or use tools that are borderline absurd.
I remember going through a seminar hall one day and the sales rep put a crown shaped electronic tool on my head for 2 minutes. From this, his computer said that I my adrenal gland was dysfunctioning, my thyroid hormones were out of balance, and my shoulder was enflamed. He recommended a a line of supplements to address my ails, and said that this device would be a great tool for my practice.
While I have a history of shoulder pain, there’s not really any evidence adrenal or thyroid problems symptomatically or on blood work. Needless to say I was a bit skeptical. I asked the rep, how does the device work? How could it possibly know what’s circulating in my blood stream? Is there any study at all about the accuracy of this device?
Of course there were limited answers to these questions, but you probably already knew that just from reading that story. It’s just crazy to think that an electronic helmet plugged into a computer could know those things, so trust your intuition if things look and feel a little crazy.
Enjoy the Process
When you’re sick or in pain, your illness tends to rob you of most of the things you once enjoyed, this is especially true of physical tasks. However, one of the things I’ve noticed about patients with chronic illness or chronic pain is that they still maintain a lot of their mental sharpness.
Treating yourself as your own experiment can be scary and desperate, but some of the people I know find it stimulating and exciting to try to be the detective of their own medical case. If you can find some joy in the process of problem solving, solving your own n=1 may be a rewarding experience.
