We need your help now
Support from readers like you keeps The Journal open.
You are visiting us because we have something you value. Independent, unbiased news that tells the truth. Advertising revenue goes some way to support our mission, but this year it has not been enough.
If you've seen value in our reporting, please contribute what you can, so we can continue to produce accurate and meaningful journalism. For everyone who needs it.
Sign in. It’s quick, free and it’s up to you.
An account is an optional way to support the work we do. Find out more.
By continuing, you are indicating that you accept our Terms of Service and Privacy Policy.
THE HEADLINES ANNOUNCED a dream come true: Scientists Say Eating Chocolate Can Help You Lose Weight!
They were reporting the results of a study published in the impressive-sounding International Archives of Medicine, an online open-access journal. The study’s authors claimed eating chocolate along with a low-carb diet had helped people lose weight and increase their general well-being, compared to people who just ate a low-carb diet.
The problem is, the whole study was an elaborate hoax.
Journalist John Bohannon and others designed, ran, published, and publicised a diet study of purposely dubious scientific merit to make a point about how claims with little to no scientific backing make headlines and drive diet fads, Bohannon reveals in io9.
The study was designed to be part of a documentary about the junk-science diet industry, and the filmmakers had a doctor (to run the trial) and a statistician (to play with the numbers) all lined up. Bohannon, who has a PhD in the molecular biology of bacteria and a known history of successful scientific pranks, was brought on to help get the study published and publicized.
They recruited 15 participants, who were paid €150, screened for any health problems, and told they were going to be in a documentary film about dieting — which was true.
One set of participants ate a low-carbohydrate diet, another followed the same diet with the addition of a 1.5 oz bar of dark chocolate, and a third set acted as controls. They all weighed themselves daily for three weeks, while the “researchers” measured everything from sleep to BMI.
Lo and behold, the group that ate chocolate plus a low-carb diet lost weight 10% faster than the group on the low-carb diet without chocolate. The difference was statistically significant, and Bohannon had a hook to sell.
It wasn’t hard for Bohannon to get the study published — journals have widely varying standards for peer review, and a lot of them accepted the study. Once the study was accepted, Bohannon wrote a press release, created a fake site for his made-up research center, and — sure enough — reporters started writing about the sham study.
It seemed they’d pulled it off.
In Bohannon’s story on io9, he lists major German daily newspaper “Bild,” the German “Cosmopolitan” website, and the German and Indian “Huffington Post” websites among the news outlets that fell for the ruse. He also includes screenshots of coverage from the “Prevention” website, a few British tabloids, and the June issue of “Shape” magazine.
Bohannon seems pretty pleased that these publications with large audiences picked up his study, “fooling millions,” the io9 headline claimed. But as journalist Daniel Engber pointed out on Twitter, maybe it wasn’t such a feat after all:
Still, it’s worth taking the warning to be a little more skeptical about reports that a certain diet or food has major effects on your health — especially when those reports show up in a tabloid.
This study wasn’t so different from many other diet studies that also use small sample sizes. It’s unfortunately not common for scientists to tweak their data by running an experiment multiple times or excluding outliers to get statistically significant results, mostly innocently.
A good study typically has a hypothesis and then tests it. This study had no real hypothesis and tested 18 different things — not one thing — to see if any statistically significant associations popped up. In poorly designed diet studies, that’s not actually so unusual. If enough things are measured in such a small group, that approach is bound to yield… something. Bohannon explained it this way:
Think of the measurements as lottery tickets. Each one has a small chance of paying off in the form of a “significant” result that we can spin a story around and sell to the media. The more tickets you buy, the more likely you are to win. We didn’t know exactly what would pan out—the headline could have been that chocolate improves sleep or lowers blood pressure—but we knew our chances of getting at least one “statistically significant” result were pretty good.
In other words, while the study did indeed find a statistically significant effect on weight loss, it was designed to do exactly that. They were statistically more likely to find what looked like a significant effect but was actually a fluke.
Read Bohannon’s full account of the hoax here.
To embed this post, copy the code below on your site
Maybe it’s just another trick and they want all the chocolate.
I’ll carry out my own research.
No 5hit.
This hoax highlights some of the serious challenges faced by science and science publication. This topic is actually going to be the focus of the annual Robert Boyle Summer School taking place in Lismore June 25th-28th. There’s more info on http://www.robertboyle.ie
Well duh.
every days a school day.
My dream diet. Thought it was too good to be true :’(
Hardly elaborate in fairness. It’s a press release…. ooooh, sneaky!
The Irish Examiner was another one of those who regurgitated it. I’d say if you gave The Journal another few days, they’d have had it too. :)
The Professor doesn’t care! He’s sticking to the strictly chocolate diet anyway! All hail the chocolate promoting scientific study!
I’m no statistician, but I’m pretty sure that by definition the threshold of statistical significance becomes more demanding as you test for more things, since the chance that at least some of your horses will come in obviously increases as you put more in the race. If I’m right about that, then it’s really not so easy to fake “statistical significance” unless you just don’t mean what a (competent) statistician would mean by that term. But if you’re willing to claim statistical significance when your spoof experiment doesn’t actually meet the accepted definition then why even go through the whole charade in the first place – just make the whole paper up out of thin air.
The phrase “statistically significant” should always have include the percentage likelihood of a true result. Most of them say that 95% is an appropriate number, some require 99%. In other words either 5% or 1% of all statistically significant results will be flukes.
No need to fake the results, just keep creating new studies.
have your say