Are we Underrating the Anecdotal?

There is an ancient British superstition which says if a child rides on a bear's back it will be protected from whooping-cough. Perhaps it’s just as well we destroyed the bears and their habitat centuries ago or we’d have a serious health and safety issue on our hands.

Such absurd beliefs can arise because of how we are programmed. There is a survival advantage to making associations between things that turn out to be true (such as poisonous berries and death) but no particular disadvantage to making associations that turn out to be false. This is how superstition develops. If I rub olive oil into a wound and it doesn’t get infected, why not do it every time I get a wound – just in case that’s what provided the protection. If you are interested in a full explanation, see this article in Scientific American.

Evolution has taught us to hedge our bets. So when someone tells their doctor about something they are convinced is curing their condition, the doctor is right to be cautious.

On the other hand, you sometimes read about a study finding evidence to support a piece of received wisdom and can almost hear the cries of “Well I could have told them that!” being uttered in kitchens up and down the country. I guess if you have an infinite number of monkeys randomly making superstitious connections, eventually they will come up with some genuine cures.

As someone with a science degree, I do have a reasonable grasp of the meaning and importance of controlled experimentation and empirical measurement – but as someone with a keen interest in nutrition who has experimented extensively with my own diet I am aware that anecdotal evidence often contradicts the advice given by the establishment; when that happens, it’s hard not to question whether sufficient value is attached to the potential value of that evidence.

Some years back a friend of mine was diagnosed with ulcerative colitis. It came as no surprise, since it ran in his family. Purely by chance, he was introduced to a ‘juicing’ diet by someone. This involved living, more or less, on the juice of various fruits and vegetables.

My friend decided to do some research. Naturally, he did not rent a medical facility and conduct controlled clinical trials with a recruited sample of colitis sufferers. Instead he read the testimonies of fellow sufferers who had tried juicing.

There are concerns around adopting radical diets like this which a doctor would quite rightly raise. Nevertheless, my friend went ahead with it for several months, over the course of which his symptoms started to disappear. I know, I know – what else was he doing during this period? It’s likely that his whole outlook on life had changed. Maybe as well as adopting the diet he had stopped sleeping for 3 hours a night and pouring half a litre of scotch on his cornflakes.

Nevertheless, whilst the symptoms were gradually disappearing for whatever reason, his specialist was suggesting fairly powerful drugs that would help slow down, but not cure the illness and telling my friend to prepare for the possibility of a life with a colostomy bag; he was not very interested in my friend’s supposed miracle cure.

Disillusioned by the advice he was getting from his specialist, but excited by his apparent reprieve from a life with a colostomy bag, my friend spoke to someone he knew in the medical profession, who laughed and said “Don’t expect them to be interested in anything that’s not in the textbooks.”

I understand why medical professionals would feel like this. If they took the time to fully explore every patient-reported treatment, they would barely have time to see a fraction of the patients they are expected to handle. Moreover, both practically and ethically, doctors can only recommend treatment for which there is clinical evidence and can only sanction lifestyle choices the establishment generally accepts to be safe. My friend’s specialist would have been irresponsible to say “Wow, sounds like that worked for a bunch of people and it sort of makes sense – so I would stick with the juicing instead of taking these clinically proven drugs.”

Don’t get me wrong, clinical researchers have looked into (and no doubt continue to look into) the connection between diet and ulcerative colitis and it would be a mistake for me to try to establish just how much was known when my friend was advised; but for me this story is important because it illustrates how a collection of anecdotes can be a powerful indicator that clinical research is required.

It’s possible my friend’s story was one of many that percolated into the medical research community and contributed to the anecdotal ‘noise’ driving clinical research on the condition – but given the reaction of his specialist, I doubt it; and this is what concerns me. The percolation process feels inherently inefficient. Do all the anecdotes get through? Many anecdotes may not even get as far as a specialist if patients have the impression there is no interest.

When my friend was researching his condition, the internet was embryonic and certainly lacked the kind of community-based interaction that now abounds. 15 years later there are millions of voices talking about the things that matter to them most – and health comes pretty high up the list. The internet is a huge database of case studies – the question is whether we are taking full advantage of it. The market research industry (obvious candidates to leverage this medium) is only just beginning to tap into the potential of social media as a research tool. It seems a safe bet that the medical profession is no further forward. The difference is, the stakes are so much higher for the latter. If market researchers don’t have a big push on harnessing social media, Acme sells fewer widgets. If the medical profession don’t, progress in researching important health issues is slower, so more people suffer.

But as well as being under-used in aggregate, are individual anecdotes sometimes given less importance than they deserve?

There was recent controversy over Dr John Biffa’s criticism of a diabetes organisation. He implied they are wilfully giving bad dietary advice to diabetics and subsequently received a grilling here and elsewhere. It was a pity that Biffa so poorly dealt with an issue that has been widely debated on the web – are low carbohydrate diets a better way to treat diabetes than the low fat diet recommended by most diabetes organisations?

If you trawl the web for testimonies on the subject, it is possible to find many from people who say they have self-treated type 2 diabetes with a low carbohydrate diet - and people appear to have been talking about it for a long time. Indeed I was unable to find any testimonies from people for whom such a diet had failed. I came away with the impression that this is the best diet for diabetics.

Of course this means nothing – it’s circumstantial, anecdotal and subject to recall bias and the placebo effect (not to mention my own ability with Google.) Plus, there are also testimonies from diabetics whose lives have been improved by the recommended low fat diet. Nevertheless, it seems that one organisation is indeed changing their advice and it would be fair to say that there is now a fair bit of research taking place in the area.

My question is this: can’t a single anecdote be considered powerful in its own right if it contradicts advice positioned as being for everyone in a certain group? For example, if an organisation is recommending a diet for people with a certain condition, yet one person is shown to improve significantly on another diet, doesn’t that prove the advice is flawed? Surely the advice should either be re-positioned or withdrawn, pending further research? If we believe it does then Biffa’s insinuations of chicanery sullied a valid criticism about the quality of advice to diabetics by this organisation.

To return to the question about anecdotes driving research, I would like to offer a final thought. Trials to evaluate the effects of diet are often expensive and difficult to run, and for potential funders like pharmaceutical firms there is rarely a pot of gold at the end of the rainbow; but this is all the more reason why we should make sure the research that is done is the right research. Instead of ignoring people who publish internet testimonials about themselves and others on the basis that it’s ‘purely anecdotal’, why not find a way to proactively engage with them? This would create a double benefit.

First, there would be the potential for more case studies to drive research in the right direction. The internet is teeming with people documenting their own experiences and supposed cures. Just because they are talking about it on the web instead of in a consulting room, does it make their story any less valid as a case study?

Second, by engaging with people who are potentially disillusioned with the medical establishment, a better understanding between all concerned could be fostered – it is a PR opportunity not to be missed. After all, in this case it may turn out they had a point.