Why Anecdotes/Testimonials Are Unreliable?
March 16 2010
If you are familiar with this website, I always insist on an evidence-based approach and do not put much value in anecdotes or testimonials. Many people just say science is important without fully understanding why evidence-based approach or science evolved.
Here are some of the reasons for why anecdotes/testimonials are considered the lowest form of evidence and why an evidence-based approach evolved.
Confounding variables
There are too many variables like diet, age, trainee level, genetics, sex that can significantly affect the results of the training program.
“A muscular guy says he does this special program, so I am going to try his program.”.
The problem with this testimonial is that most muscular guys are just genetically gifted. They can do just whatever and eat whatever and still be under 8% body fat.
“That exercise program didn’t work for me”
Here is a guy who has given a decent program but didn’t make any progress because his diet was still the same.
“That program just killed me”
Here is example of 40 yr old guy who just copied a workout from a 20 year old who got great results.
In research, these confounding variables are eliminated by random selection from the specific population and randomizing people into groups, restricting, and matching. The randomization eliminates the confounding variables and hence is very important in research.
Hidden Data
In testimonials and anecdotes, you only read about positive results. The people who didn’t gain much never bother to talk about it.
“Program X worked worked for 3 of my friends”.
Does that mean X works? What if I show you 5 people (hidden data) for who the program X didn’t work or who dropped out?
In research, the negative hit and the number of people who dropped (attrition rate) out are also recorded.
Cognitive Biases
These psychological biases distort reality and judgment and has been extensively studied and confirmed in the psychology field. A few of the biases which can skew reality are memory recall biases, expectancy effect, exposure effect, bandwagon effect and so on. There are over 35 or more of them which becomes relevant depending on the specific context.
“I do high reps with light weights to tone”
This is classic case of bandwagon effect or herd effect where people just follow what others do without analyzing the underlying evidence.
“I know it works. I have been doing this for almost 20 years and there is some research to show it works”
This is an example of the confirmation bias and is one of the most common biases out there. Here people tend to prefer, seek, and interpret information which confirms their hypothesis and neglects the opposing data.This bias is so often seen among fitness guru’s who have a product or idea to sell.
“That diet doesn’t work. I didn’t eat too much ”
According to the social desirability bias, there is a tendency to provide answers which is more socially acceptable /desirable. This type of bias has now been widely accepted in nutritional studies which include dietary recalls.
“I was really excited about the program and it worked great”
This is called the hawthorne effect where just by believing in the program the person was doing all the right things that he never did with other programs. He is eating all the right foods, getting enough sleep and so on.
In research, these biases are taken into account and eliminated by having a control group and blinded designs. For example, in double blinded trials, neither the researchers nor the participants are aware of who belongs to which group.
By Chance
Hypothetically, if every bias and confounding variable is taken into account, there is still a possibility that the results were due to mere chance than the program/diet.
In science, the chance is taken care by the p value (p less than .05) as often seen in studies. If the result are statistically significant, it means there is only a 5% chance the results were due to chance or accident.
Subjective Outcome Measures
Anecdotal Outcome measures are extremely subjective. For example , judging muscle size by looking in the mirror, waist size, and so on.
“The program worked. I look a lot bigger than I was”
This could be just due to increased glycogen, fat and water accumulation than just actual muscle growth.
“My weight hasn’t changed at all”
They might be losing fat and gaining muscle or just holding water.
This can only be measured with more sensitive & accurate body composition measures.
