June 28 2020
Here are 4 important points that I learned about scientific studies that may help you to interpret a study. There are more, but these are the ones that most people miss or ignore, including researchers.
Years back I thought that any outcome related to the condition or the question is appropriate as long as it comes from a scientific study. Little did I know that there is a hierarchy of outcomes.
What are surrogate outcomes? These are outcomes/endpoints in scientific studies that are just laboratory/physiological markers or substitutes for clinically meaningful outcomes (or outcomes that people actually care about). If you are confused, these examples might help:
What is the harm? We have 100’s of studies that show how surrogate outcomes can be very misleading. In other words, treatment might show positive results in surrogates, but you might see no change in meaningful outcomes or even increased harms. For example, drugs lowered irregular heartbeat, a surrogate outcome, after a heart attack, which is great; we all thought. But tripled deaths. Antioxidants decrease oxidative stress and inflammation which is great but also reduced muscle mass. Testosterone has shown to increase muscle mass in the elderly, but shows little changes in function.
So why are they still used? The majority of the studies use them because these can be gathered rather quickly and cost much less than looking at long-term outcomes such as a disease or death that could take years requiring large samples. They serve well as preliminary data or to generate hypotheses or proof-of-concept trials. But that’s about it; they cannot be used to make a recommendation. That been said, some surrogates are strongly correlated and in the causal pathway, but very few (blood pressure, but still not 100% reliable).
I don’t know who said this, but most patients only care about two simple questions: will I live longer or feel better?
If you think every scientific study is equally true, then this one is for you. I used to think this way too. And I will name this the “scientific study bias”.
Simply put, there are different types of study designs and how much trust or credibility we can place on the study depends on the type of study design. As we go down the pyramid, the less we can trust the evidence.
Just to give an example of how little we know about human body: It takes years of in-vitro and animal research or basic science studies by some of the best scientists to come up with a new drug. And once they finally get there, the drug will be permitted to be tested in humans in 4 phases that could take a decade or so. And guess how many drugs make to Phase 1? Just 10%. Or 90% of the drugs fail before even getting to just Phase 1. How is that for the failure rate? For cancer drugs, this is around 3-4%. And this gets worse. Out of the 10%, just 10-20% make it to Phase 3. And even after the drug gets approved and is in the market, they still can be rejected or pulled in Phase 4. And with all this testing, the majority of these drugs are just slightly more effective than a placebo. Also, this “success rate” is considering all the shady things drug companies are notorious for. And in breaking news. we just successfully shot a rocket to space last month, but we still haven’t found a cure to Covid which is killing thousands all over the world (rocket science is hard; medical science is just harder). The bottom line is that the evidence from these studies is not very trustworthy for making health/medical decisions. We may get there one day where we can solely rely on physiology/mechanisms, but we are not there yet for sure. So beware of claims based on these type of evidences.
I was very familiar with statistical significance or the p-value (are the results due to chance?) since you always see those in an abstract, and it is always taught in our stats classes.
What is it? However, clinical significance is less talked about, and you will never see it in an abstract. It asks, “is the improvement ‘big enough’ to make a meaningful impact in people’s lives (which is why we do research in the first place, right)?”
Any examples? For example, my study shows a 10% statistically significant (P less 0.05) increase in strength or the time taken to rise from a chair in older adults after 12 weeks of exercise training. And the study is statistically significant; hence positive and everyone is happy. But what difference does it make in the lives of my participants? Or in other words, how much of an increase in strength is enough to show an improvement in the quality of life or their everyday functioning? In the multi-million dollar LIFE study, it was considered clinically significant if participants could walk 400 m in 15 min by walking between 2 cones in a long hallway- measure of walking disability. Indeed, more people in the exercise group could do it at the end of three years compared to the non-exercise group. But does it really tell us they had a meaningful improvement? Can they now take the bus or walk a quarter-mile outside and go meet their long-time friend? Maybe. Maybe not. For weight loss, a clinically significant weight loss is around 5-7% of your body weight. But majority of the weight loss studies are often statistically significant, but not clinically significant. So always look at the clinical importance of the outcomes of a study, even if the outcomes are meaningful.
Researchers are humans and almost all, including myself, are “biased” towards a favorable, positive or interesting findings. Most people are familiar with the financial conflict of interest (COI), but there is an intellectual conflict of interest that is very hard to disentangle. This was something I never really understood until I started research and publishing (I am not talking about misconduct or fraud or falsifying data, which is cheating and is quite rare).
Why this bias? Study results that are positive or interesting or exciting get quickly published, in better journals, get more “likes” and ‘praises’, gets shared more, and lead to new papers and overall makes the research team very happy. Also indirectly lead to other important incentives like tenure, promotions, and preliminary results for the next big grant. Most of the big researchers in most major universities in US are funded by grants. No grants, find another job. Also, our reward system in research incentivizes publication numbers or quantity and exciting results, but not the methodology or the quality of the paper. Quality suffers when quantity is the focus. The constant mantra in research is “productivity”; quality is the forgotten child. For instance, just look at the number of studies on HCQ against Corona, but the majority of them are just low quality and useless and only serve to create more confusion.
In fact, quality or risk of bias measures like blinding of assessors, concealed allocation (selection bias), pre-registration (reduce selective reporting and p-hacking), blinding data analysis, funnel plot analysis (publication bias), data transparency are all measures to prevent or minimize researcher bias. High-impact journal in medicine are more often than not higher quality and covers most of these biases. Then again, these are million dollar studies, so it is expected. And all that I wrote above is known for years and is not just from my personal opinion/experience. The graph from a recent study shows how registered reports affects the number of “positive” studies.
There are a few more points I wanted to write, but I will end here to give ‘quality’ the last word.
So next time you read a study, ask yourself:
Anoop, congrats again on another well thought out post. Your review on study design and is so much needed today. As an aside, one thing I noticed during the COVID press briefings is how often the scientists had to stop and explain to reporters and the public what a quality research study looked like (RCTs).
Your words go a long way to helping people understand these things.
Anoop | Sun July 05, 2020
Thank you Joe for the comment.
Unfortunately, none of this is taught in schools or colleges. Even some doctors have trouble with all this information.
Anoop, I agree completely. I dont remember it being addressed when I was in college either. Your words are definitely making a difference!
Great article! Breaking down the myths! Would be nice if all advice was evidence based.
Anoop | Wed August 12, 2020
Thank you Rob. Appreciate the feedback. If everything was evidence based, I will have to close the site!! 😊
Add us on Facebook
Get tweets
Subscribe to RSS feeds
Get email updates
© 2008 - 2024 | Exercise Biology. All rights reserved.
