Nearly forty years ago, Western governments, after consulting with the world’s top nutrition scientists, told us to change the way we eat. If we wanted to stay healthy, they said, we needed to cut back on foods rich in saturated fats and cholesterol.

By and large, we did as we were told. Steak and sausages were replaced with pasta and rice, butter with margarine and vegetable oils, and eggs with muesli.

But instead of becoming healthier, we grew fatter and sicker. In the decades that followed the issuing of these dietary guidelines, a public health catastrophe unfolded. Obesity, which until then had been relatively stable, rose dramatically, as did the incidence of related diseases, like diabetes.

What went wrong?

In a recent article for the Guardian, entitled, The Sugar Conspiracy, I argue that this was a foreseeable disaster, and that it could have been avoided – were it not for the arrogance and myopia of the experts in whom we placed our trust.

The article tells the story of Professor John Yudkin, a British scientist who, back in 1972, warned the world that the real threat to our health was not fat, but sugar, in a book called Pure, White and Deadly. “If only a small fraction of what we know about the effects of sugar were to be revealed in relation to any other material used as a food additive,” wrote Yudkin, “that material would promptly be banned.”

The book did well enough, but Yudkin paid a high price for it. Prominent nutrition scientists combined with the food industry to destroy his reputation. His career never recovered. He died, in 1995, a disappointed, largely forgotten man.

By the time Yudkin wrote his book, most nutrition scientists had signed up to a different theory; one that had emerged, a decade or so before, from the United States. Its central tenet was that a healthy diet is a low fat diet. Yudkin, a professor of nutrition at Queen Elizabeth College in London, led a diminishing band of dissenters who believed that sugar, not fat, was the more likely cause of maladies such as obesity, heart disease and diabetes.

Yudkin noted that sugar has been a major part of western diets for just 300 years; in evolutionary terms, it is as if we have, just this second, taken our first dose of it. Saturated fats, by contrast, are so intimately bound up with our evolution that they are abundantly present in breast milk. To Yudkin’s thinking, it seemed more likely to be the recent innovation, rather than the prehistoric staple, making us sick. He also believed that the evidence that fat is bad for us was relatively weak.

But by the time he wrote his book, the commanding heights of his field had been seized by proponents of the fat hypothesis. The American scientists who made up the world’s nutritional elite were skilled at political combat, and didn’t hesitate to take down the reputation of anyone who challenged their ideas. On their advice, American and British governments told their citizens to cut back on fat and cholesterol-rich foods.

Yudkin, whose ideas about sugar had previously been considered perfectly mainstream, quickly found himself marginalised. He was disinvited to scientific conferences, and shunned by scientific journals. His research eventually fell out of circulation altogether, and the story of his swift demise scared off any other scientists interested in challenging the consensus that fat was the chief problem with our diets.

When people cut back on fat, they usually increase their consumption of carbohydrates. It is now becoming increasingly apparent that by turning saturated fats into our number one dietary enemy, we missed the greater threat of the most versatile, palatable and unhealthy carbohydrate of all: sugar.

Only in the last ten years has it become acceptable again, in scientific circles, to even research what sugar does to our bodies. The paediatrician and obesity expert Robert Lustig has led the way, after studying sugar’s effects on the metabolic system. A 90-minute talk he gave in 2009, titled Sugar: The Bitter Truth, has been viewed more than six million times on YouTube.

Lustig now champions Yudkin’s research, but it was so well buried that Lustig only came across it by accident, when a fellow scientist mentioned it to him at a conference. Lustig was astonished to find that it anticipated his own work. When I asked him why he was the first scientist in years to focus on the dangers of sugar, he told me: “John Yudkin. They took him down so severely – so severely – that nobody wanted to attempt it on their own.”

The response to my article has been explosive. It was viewed over a million times in its first week, and many thousands of people have shared it on social media. I receive emails almost every day from readers telling me how much it meant to them.

I admit I was surprised by this response. After all, it’s not exactly news that sugar is bad for us. Thanks in part to a high-profile campaign led by Jamie Oliver, public awareness is high. The British government recently recognised the urgency of the problem, by introducing a tax on sugary drinks. On reflection, however, I think the article tapped into something very powerful: a sense that the scientists and government officials, on whom we all rely for good advice, got it dreadfully wrong.

The obesity epidemic is sometimes talked about as if it was an act of nature, or an unfortunate accident. If anyone is blamed for it, it’s usually the food industry and certainly, the food companies have a lot to answer for. But if the nutritional advice we have followed for all this time was profoundly flawed, this is not a mistake that can be laid at the door of corporate villains. Nor can it be passed off as simple error; what happened to John Yudkin suggests otherwise.

No – this is something the scientists and officials did to themselves, and, consequently, to us.

For the full story of The Sugar Conspiracy, click here.

About Ian Leslie

Ian Leslie is a writer who contributes to the Guardian, among other publications, and the author of Curious: The Desire to Know and Why Your Future Depends On It. You can follow him on Twitter: @mrianleslie