Homeopathic clairvoyant dogs. Er, barking.
Hill PB, Hoare J, Lau-Gillard P, Rybnicek J, Mathie RT. Pilot study of the effect of individualised homeopathy on the pruritus associated with atopic dermatitis in dogs. Vet Rec. 2009. 164(12):364-70
I am not a vet but I enjoy sniffing around The Veterinary Record on a weekly basis. After reading this article I was found wandering around the house guffawing loudly in pleasure at the insanity of the world. This is probably not the professional approach to critical appraisal so I turned to Trish for help.
Trisha Greenhalgh on ‘the science of “trashing” papers’ suggests that :
‘Strictly speaking if you are going to trash a paper, you should do so before you even look at the results.’
This seems to be one of the most singularly useful and regularly ignored pieces of advice in the whole of critical appraisal. Ultimately, if the methodology of the trial is rubbish then stop reading. Stick your fingers in your ears and go nahnahnah.
So given Trisha G’s comments let us consider the methodology for a moment. They took a load of itchy scratchy dogs and gave ‘em homeopathy. There were two phases:
Phase 1
Open phase. Owners with itchy pooches were invited to take part. There were 20 dogs in this phase and they were given individualised homeopathic remedies by a veterinary homeopath. They had their pruritus scores measured on a validated scale.
If the owners felt there had been a substantial response to homeopathy they were invited to take part in Phase 2. Only 5 dogs showed any response but dog 15 was euthanased for an unrelated illness and dog 6 got better.
Phase 2
Blinded, randomised, placebo-controlled phase. There were 3 dogs left in this phase. This was actually a cross-over study where dogs received both a placebo and a homeopathic remedy in random order. They had their pruritus scores measured again during placebo and homeopathy phases.
Why could they not just put dogs straight into Phase 2 and what purpose does the open phase possibly have? The main issue that screams at me from this is bias. BIAS! Volunteers are never an ideal sample for any trial but they have ratcheted it up another notch by further sub-selecting those that think there is an effect. There is also an issue with a cross-over design in a condition that is not stable. My understanding of atopic canine dermatitis is that is a disease that will wax and wane.
However, one of the few advantages of only having 20 subjects is that it is possible to give details of the doggie themselves. Whatever the concerns with the methodology this paper might be worth reading for these vital scientific details alone. Some highlights: dog 4 had an aversion to onions, dog 11 was very sensitive but did not console his owner, dog 15 has loud flatus when excited (who doesn’t) and dog 16 fears thunderstorms, desires chicken and is clairvoyant (I’m not making this up).
Oh, one other thing. This study received funding from the British Homeopathic Society. Poor sampling, massive bias introduced systematically into the methodology, inadequate numbers and conflict of interest issues. Amusing doggie vignettes aside; my fingers are now in my ears.
nahnahnah!
The Veterinary Record has been publishing weekly for vets since 1888. The British Veterinary Association and the Vet Record are the vet equivalent of the BMA and the BMJ. One has to wonder what on earth happened to the peer-review process here.
Martin Alder. Editor, Vet Record.
Interestingly, the provision of homeopathy to animals is very well-regulated – it can only be provided by gen-u-ine vets. This is both reassuring and alarming in equal measure. It might ensure that animals do not receive ineffective therapies for serious illnesses but it does professionalise one of the most scientifically ludicrous alternative treatments in history. Homeopaths for humans could claim (although inexplicably they tend not to) that they are providing some kind of psychosocial intervention for the benefit of patients. After all, as Bob Hoskins once said: it’s good to talk. Of course, this potential benefit is somewhat diluted for vets who are not actually Dr Dolittle himself. What is left is a sugar pill coated with an active ingredient diluted to 200C. My advice is to cut out the middle dog and give the psychological intervention directly to the owner.
The British Homeopathic Association couldn’t resist trumpeting this trial and released a press release.
…a small, rigorously designed, research study at the University of Bristol’s Department of Clinical Veterinary Science.
The other four dogs that responded well in this first phase were then put forward into a blinded randomised trial in which they received their homeopathic prescription at some times and placebo at other times. The three dogs that completed this phase of the study improved more with the active remedy than with placebo, and owners were able to distinguish correctly which pill was which.
As mentioned in the methodology there was just three dogs in Phase 2. If there was a 50/50 chance of the owners guessing correctly then there would be a one in eight chance (12.5%) of getting the results by chance alone. Statistical significance it ain’t. It is not too difficult to imagine tossing a coin three times and getting three heads in a row.
Dr Peter Hill, who was lead clinician on the study, said “These preliminary data indicate the need for a large randomised controlled trial of homeopathy in canine atopic dermatitis.”
In no way does this study constitute any evidence that there is some effect of homeopathy on pruritus in dogs. It certainly does not justify the “need for a large randomised controlled trial of homeopathy in canine atopic dermatitis.”
The discussion is a masterpiece of obfuscation. The authors trot out some miserable canards as they grasp for explanations as to why the homeopathy didn’t reach the success rate of ’60-70 per cent predicted’.
- The dogs remained on the immunosuppressants corticosteroids and ciclosporin and this interferes with the action of the homeopathic remedy.
- Because the cases were volunteers these were cases which were more difficult to treat.
- These cases were from a referral population so were more severe than usual.
They finally suggest:
- “If the beneficial responses were due to spontaneous recoveries and chance occurrence, the failure rate would merely reflect the fact that homeopathy is not an effective treatment for canine atopic dermatitis.”
Bingo. The authors also seem to be confused about their motivations for the study. On one hand:
The authors’ interpretation of the results is that they provide data to justify a larger study to determine whether the findings are repeatable.
Two paragraphs later and they are suggesting:
…the authors consider that the overall success or failure rate in this study is somewhat irrelevant. The objective of this study was to determine whether homeopathic remedies appeared to be beneficial in enough cases of canine atopic dermatitis to justify a larger trial. Even with a cautious interpretation, the preliminary data appear to support that view…
This is all a touch circular. Surely the way to establish benefit is to conduct a proper randomised trial in the first instance. We are right back where we started. Labelling this study as a ‘pilot’ is a ruse to throw the scent off the appraisal hounds. If I may paraphrase Blackadder, this study has about as much power as an asthmatic ant with some heavy shopping. It is methodologically flawed with no reasonable conclusions.
The British Veterinary Voodoo Society have the measure of those that would pervert the evidence process.
Clinical trials are still in their early stages, however we are confident that by performing a sufficient number of small, poorly-controlled investigations we will easily generate enough p<0.05 outcomes to be able to claim with absolute assurance that the method is well proven by properly conducted double-blind research.
So why publish it? You could drive a coach and horses through the holes in the methodology. I usually treat with deep suspicion any published ‘pilot’ trials. Pilots might be essential pieces of work to facilitate research projects but rarely justify independent publication. Imagine for one minute if this was a Big Pharma sponsored ‘pilot’ study. It wouldn’t get past the editor’s shredder. Yet alternative medicine seems to have a special dispensation to exert influence over journals of which Big Pharma can but dream.
You weren’t nearly as woof (geddit?) on them as I would have been…but then I have less logic and more fury.
Nice work, Euan.
Indeed. I have, over the last couple of years, tried to popularize the term / description “stealth psychotherapy” for this aspect of homeopathy, though it doesn’t seem to have caught on.
I’m a vet, and no fan or homeopathy. I’m also not aware of the finer points of the methodology in the paper in question. However, it occurs to me that even if only one owner could tell the difference with 100% accuracy between when their dog received a placebo and when it received the remedy on sufficient occasions, (eg 15 or more) this would be signicant. Of course, the blinding and randomisation would have to be absolutely rigorous…
I’m not sure how many therapy comparisons were made in the paper, but one hopes prof hill doesnt end up looking like Bienveniste did with inadequete blinding controls.
Hi Geraint, thanks for the comments. Without looking at the paper again, I am fairly certain that the dogs only received one round of homeopathy and one round of placebo. In the discussion, the authors themselves acknowledged and quoted the 12.5% figure of these results happening by chance alone. To a certain extent, I felt the discussion tried to represent both sides of the argument while tying itself in knots as it justified its own existence. In itself, it could be argued, that the 12.5 % figure is justification for not publishing this trial. Not because the results were positive or negative but simply because the methodology doesn’t stand up.
To use crossover studies the condition really needs to be stable ie non-fluctuating. This simply isn’t the case with canine atopic dermatitis and it looks like it has been used as a mechanism to cope with tiny numbers rather than it being the best choice of methodology per se. It’s not like it is some super-rare condition and there is a shortage of canine atopic dermatitis is it?
Brilliant blog piece.
Please have a look at the letters published last week, of one of which I am an author (while also being Vice-President of the British Veterinary Voodoo Society, so thanks for the namecheck).
http://veterinaryrecord.bvapublications.com/current.dtl
Letters:
George W. Tribe
Pilot study of the effect of homeopathy on pruritus associated with atopic dermatitis in dogs
Vet Rec. 2009 164: 634. [Full Text] [PDF]
Simon J. Baker and Gordon J. Baker
Pilot study of the effect of homeopathy on pruritus associated with atopic dermatitis in dogs
Vet Rec. 2009 164: 634-635. [Full Text] [PDF]
Morag G. Kerr, Christopher A. Hebbern, Paul Wilson, and Joseph J. Magrath
Pilot study of the effect of homeopathy on pruritus associated with atopic dermatitis in dogs
Vet Rec. 2009 164: 635. [Full Text] [PDF]
P. B. Hill, J. Hoare, and R.T. Mathie
Pilot study of the effect of homeopathy on pruritus associated with atopic dermatitis in dogs
Vet Rec. 2009 164: 635-636. [Full Text] [PDF]
Oh, that did me good! Thanks for finding this, Simon.
Northern Doctor, you have no idea how hard it was to write letters about this abortion of a paper while trying to stay within shouting distance of the Vet Record’s word limits. When the barrel is as stuffed full of fish as this one, and you only have a limited number of shots, you can’t get them all….
And then of course the authors get the last word. The thing is farcical. It says more about the Vet Rec’s scrutineering standards than anything else, but I doubt if we’ll ever get Martin Alder to understand that.
“In itself, it could be argued, that the 12.5 % figure is justification for not publishing this trial.”
What the authors seemed to be doing was attempting to use that 12.5% in a manner analogous to p<0.05, which begs all manner of questions.
Most importantly, would they have done that calculation if they did not already know it would come out with the "right" answer? Unless it has been stated as a prior hypothesis, all a calculation like this does is show the danger of data dredging. Doing it post hoc is makes it in no way comparable to testing a prior hypothesis at p<0.05, and you don't even need to get all Bayesian to see this as being obvious.
While mentioning Bayes, what is very difficult to get across (because it sounds like sour grapes), is that frequentist statistical methods have essentially no applicability to hypotheses that have close to zero prior probability. This is where the appeal for "more research" is simply misplaced, especially in a field as small as veterinary medicine where it is impossible to tun a trial with adequate numbers to provide anything like enough statistical power. A great big heap of weak trial data lends no effective support to homeopathy. It all adds up to a picture of these trials simply being done for political ends, to give the appearance of playing the scientific gain.
As I said, though, the problem is that this can just sound like sour grapes especially to the statistically illiterate, so I try to refrain from using this argument, because there are only so many days left in one's life to try to teach statistics to those who do not want to learn. Much more efficiently fatal to homeopathy is to take their many claims at face value then highlight the internal contradictions and inconsistencies. As with so much fringe medicine, because they have no proper basis for any of their beliefs, their system becomes a chaotic accretion of all sorts of ideas. One I am focusing on at the moment is the use of nosodes instead of licensed vaccines in dogs. Many homeopaths might argue that nosodes are not 'orthodox' homeopathy. Many other homeopaths use them without a quibble. Their representative bodies have remained silent when asked for evidence of efficacy or to respond to examples where puppies left unprotected by the use of nosodes die of preventable disease.
Simon and Morag,
Thanks for your comments – I haven’t read the letters yet but thanks for highlighting them to me. I have just grubbed out the issue from a pile of Vet Records in the lounge and I will take a gander later.
I wrote about twice as much in my first draft and I felt I could have dredged this paper for about 3 weeks and still been able to come up with ‘internal contradictions and inconsistencies’. It really was a shocker.
BTW Top effort by the BVVS – I noted Simon Singh namechecked you a few weeks ago in the nationals. Nice one – you deserve the recognition given the scale of the task when articles like this are waltzing into your peer-reviewed journal.
Also – are you blogging anywhere?
I don’t blog. Too much of a Luddite.