Dr Wright with partner, Exercise Ehysiologist Morgen Smith. Both of whom were involved in the BROAD Study.
Part 2: The fallout
As an example of previous research I have been involved with, for example the BROAD study, which showed the largest significant changes in body weight in free-living individuals for a study of its kind cost (estimating off the top of my head) around 150k USD. Going by the estimates at least 20 times less than this research.
The large funds available allow the researchers essentially unlimited ability to run a trial whatever way they wish. This is huge money by any standards for diet research, although many times more money is put into pharmaceutical research.
This is unfortunate in many ways as diet questions have potential to be much more interesting, applicable, and clinically useful to patients. They however do not tend to make as much money as there is not a patentable research outcome at the end.
All in all, the funding model and approach used by this organisation is extremely rigorous. The article is highly detailed. But there are some problems appearing very quickly.
What did the study do?
They took people, starved them a bit by forcing them to all eat a bit less for a couple of months, then randomly allocated them to one of three groups. These groups are high in carbohydrate, medium for carbohydrate, and low for carbohydrate. This means that the quantity of fat changed, as the amount of protein was kept the same.
How real world is this study?
The participants were provided real food (not liquid meals) which is great. However they did not cook it themselves, and it was free. It is not realistic to have someone’s meals always provided for them (i.e. we cannot generalize this to the public everywhere), but these studies can still be important.
Real-world studies of free living individuals are more interesting, but there are cases in which tightly controlled adherence to a specific diet is necessary. You may wish to test the effect of a diet, not the ability of people to follow a diet, for example. Real world studies with free living individuals test the ability to comply with instructions as well as the effects of when they follow said instructions.
So, if you design a diet great for weight loss but have a group of people who won’t follow it then it might be the group was not right, or it could be that you have made a diet plan that too difficult to follow.
What were they measuring?
The primary outcome is how much energy is being used by a person. They did this by getting what is called doubly labelled water. A patient drinks liquid containing water with a chemical agent added, and this water then is included in the body in different metabolic reactions. When we observe a peron breathing out carbon dioxide, and the rate at which they breathe it out, we can then calculate aspects of their metabolism.
It is generally considered an accurate way to measure aspects of the metabolism of the body. The researchers also used accelerometers (e.g. fitness watch) to measure movement to assess other forms of activity.
What did they find?
The outcomes from this research were that the group who ate more fat used more energy. They suggested that this was as much as 209 kcal per day more than the low fat group. This is the first study to show this. If correct, this would be very interesting indeed.
What are the problems with this research?
Multiple. If there was a genuine difference between the amount of energy being used and the research groups are all being fed the same amount, then we would expect one group to start weighing less. We should have seen that the group which was “using more energy” begin to start weighing less than the group that did not. This is a glaring problem.
Let’s think a bit more widely. Has anyone else noticed this problem?
Here, I go wide and start to collect a bit more context for this piece of research, so I google the trial. I have read through the methods, and the original article 2-3 times now, including looking through their charts, reviewing quickly their results and made some notes on questions or points I would like to see answered.
This step is simple: google the title of the trial + critiques / criticisms, then skim through the top ten or so, and read through any opposing views that appear to have merit.
Where did all of the calories go?
Let’s look at some quotes from the study author: “However, even through [sic] the people in the three test diet groups had the same average weight, their metabolism differed remarkably” and “So, the study showed that the type of calories consumed affect the number of calories burned, challenging the long-standing dogma that all calories are alike to the body” and “all calories are not alike to the body“. These quotes from the primary investigator need further analysis.
If one group is losing more energy than another then how did they finish with the same weight? The participants did not differ in weight, and yet the authors have claimed the low-carbohydrate diet would lead to a weight loss of 10 kg over three years.
This is completely unsupported from the data. The participants were the same weight, and now the researchers are saying this is going to help with weight loss? You can see expert reaction to this article here, in which all but one of the experts bring up the missing calorie difference, the same point I have.
I tried to download the dataset which the authors have made available in order to analyse it however I would require a several hundred dollar programme so I did not end up reviewing it. Kudos to the authors for making it available. I was very keen to see what the weight was, and what the total energy intake was. I wanted to know where the calorie difference was coming from.
It was not reported in the article in graph form or table form, which for me was very suspiscious. One last critique comes from the former colleague and this is the most damning of all, Dr Kevin Hall was cited previously. I read the critique here and my eyeballs just about popped out of my head. Incredibly, “the authors did not report that the total energy intake provided to the subjects was substantially less than TEE during the weight loss maintenance period such that subjects in the intention to treat group consumed (mean ± SE) 460 ± 46 kcal/d less than they expended”.
In English, the total amount that a person was eating did not match with their total energy expenditure (TEE) during the phase when they were supposed to be maintaining their weight. How can people have a stable weight when there is energy leaving the body? The author continues “Such large amounts of missing calories, in violation of the law of energy conservation, indicates that the measurements of intake and TEE were inaccurate and imprecise for many subjects”.
To exapnd on this point, a person can be using up their stores of energy, say glycogen sugars from the body and might not lose weight if they were, for example, drinking lots of water (sugar out, water in = no change in weight). Here, they might maintain the same weight over a few hours, but over time it is essentially impossible for groups of people to be not eating as many calories as they are using and still maintain the same weight.
Because people cannot keep using more energy than they are bringing in without a change in weight, this can only mean that the amounts of energy they are bringing in is incorrect (not too likely as it’s easy enough to measure energy of food) or the measurement of energy used is incorrect. I see no other explanation. The author of this review article states “The significant effects of low carbohydrate diets on TEE reported by Ebbeling et al. failed to materialize when data were analyzed according to the original pre-specified plan.”
As per this report, the authors denied the BMJ (publishing the article) the original analysis of their data, and changed the primary question of the research just prior to publication. This is not good science. Whilst there can be genuine reasons to adjust the study question, like if you realise that the research will not be able to answer it at all. At worst, it is akin to kicking a ball in soccer and then when you miss moving the goalposts.
If you move the posts when you are running up to the ball, but still prior to kicking it, it might not be a big deal. Part of what we look at is “if we had not moved the goalposts would we still have scored?” and if the answer is yes, then it did not matter. When the data was reanalysed by this group the research claims were found to be not significant. This can be another indicator of a big problem. At worst here, it is like saying they missed the goal, moved the posts in mid flight and then claimed the victory.
I will finish with one quote from the correspondence section of the BMJ website, which allows for feedback from experts to comment on articles. Once an article is published with sensationalistic headlines, the criticisms rarely see the light of day. One author states: “I would urge extreme caution in interpreting the results of this study, especially given the high profile, in the lay media, of irrational dietary recommendations presenting carbohydrate as only slightly better than poison.”
What are the conclusions?
There is a mismatch here between the headlines: “Low-Carb Diets Help Maintain Weight Loss”,
The author conclusions:
“lowering dietary carbohydrate increased energy expenditure during weight loss maintenance. This metabolic effect may improve the success of obesity treatment”
The press release for the article claimed results which were not in keeping with the results, and implied greater significance of findings. This is a shame as several changes to the research would have made it an interesting read. These results are probably not significant at best, and very misleading at worst.
The experts who reviewed the article do not appear to have delved into the statistics or methods in great depth, and this article should not have been published in a journal like the BMJ. Even with this rudimentary analysis from the cited expert commenters, there were large problems with the article in multiple cases. I do not think these went far enough, and in the analysis I have cited which looked in great depth at this work, it was far more damning of the results.
There are unaccounted for calories, and based on my rudimentary analysis I am leaning towards simply disregarding the findings of this research. This trial does not prove high fat diets are superior in any way, and as pointed out has major inconsistencies which make it clinically not useful.
Who cares? What is the point of all this?
The media cares. The point here is on first read this looks like an excellent study, then on weak analysis it crumbles. They appear to have shifted the goalposts to score, but even then they have not done got the goal.
Sadly, this trial also received more media attention than almost any other trial published in 2018, (top 100 trials of 2018 of any published research). No wonder the public is skeptical.
1. Ebbeling CB, Feldman HA, Klein GL, Wong JMW, Bielak L, Steltz SK, et al. Effects of a low carbohydrate diet on energy expenditure during weight loss maintenance: randomized trial. BMJ. 2018 Nov 14;363:k4583.
2. Wong JM, Bielak L, Eddy RG, Stone L, Lakin PR, Sandman M, et al. An Academia-Industry Partnership for Planning and Executing a Community-Based Feeding
Study. Curr Dev Nutr [Internet]. 2018 Sep 1 [cited 2019 Apr 6];2(9). Available from: https://academic.oup.com/cdn/article/2/9/nzy060/5049472
3. Molteni M. The Struggles of a $40 Million Nutrition Science Crusade. Wired [Internet]. 2018 Jun 18 [cited 2019 Apr 6]; Available from: https://www.wired.com/story/how-a-dollar40-million-nutrition-science-crusade-fell-apart/
4. Hall KD, Chen KY, Guo J, Lam YY, Leibel RL, Mayer LE, et al. Energy expenditure and body composition changes after an isocaloric ketogenic diet in overweight and obese men. Am J Clin Nutr. 2016 Aug 1;104(2):324–33.
5. Yoni Freedhoff. Dr. Kevin Hall declares the insulin hypothesis of obesity “falsified” [Internet]. [cited 2019 Apr 6]. Available from: https://www.youtube.com/watch?v=atfBiEsxG5o
6. The Case for a Low-Carb Diet is Stronger Than Ever [Internet]. Dr. David Ludwig. 2018 [cited 2019 Apr 6]. Available from: https://www.drdavidludwig.com/case-low-carb-diet-stronger-ever/
7. Carbs versus fat: does it really matter for maintaining lost weight? | bioRxiv [Internet]. [cited 2019 Apr 6]. Available from: https://www.biorxiv.org/content/10.1101/476655v5