Wednesday, August 12, 2015

Evidence and Propaganda

The Trouble With "Evidence" - Across the Fence

No, "evidence-based" is not something new under the sun, though it would certainly seem to be the way the term is being bandied about now.  Everyone seems to have an evidence-based whatnot they'd like you to buy into.  The horse world is always ripe for cultism.  The horse life is as close to organized religion as you can get and still pay taxes on your farm.  We just love belonging to a group and get all puffed up when we think we've found the Latest And Best.

"Evidence-based" could be replaced by "statistically significant results", but that's not sexy.  Math, after all..!  "Research-based" means the same thing as well, but it sounds too science-y.

It's all about trying something and seeing the results.  Then it's about trying it again and again and again and comparing the results with the first trial.  If the results are the same or nearly identical, then chances are that maths would prove the numbers to be statistically significant.  They matter.

The linked article points out very nicely that yes, statistics can be forced to tell lies.  It's not hard to do, and in the horse world it happens constantly.  The biggest fudge factor is in the n--the number of test subjects--and the r--the number of times the test was repeated.  If n = "tons", "bunches", "every horse in my herd", "lots and lots" or something similar, you needn't read any farther to know that it's not a legitimate report of results of a study done using classic Scientific Method.

Anybody remember high school chemistry enough to recall what Scientific Method is?  Here's a nifty pictorial representation I found on Pinterest:

Image result for scientific method worksheet 
The method that applies to pretty much anything you'd like to prove starts with a question:  "How can I keep my horse from sniffing his girlfriend's butt?"

From the question is derived a hypothesis:  "Smearing lemon balm on a male horse's nose will prevent him from sniffing the mares."

Then come the trials and the measurable results.  The diagonal arrows that grow out from "I am wrong" and "I am right" are the important part.  Those are the trials that either succeed or fail in replicating (not just approximating) the desired result that would prove your hypothesis.

Obviously, this is a really bad example because it would be impossible to track the gelding's sniffing 24/7, so no viable numbers would result.  That's when we get the "lots" or "tons" answer.  "Yep, tons of times he walked right past a mare without stopping to sniff, so the evidence says...."

Nope.  Nope, nope, nope.  No evidence here.

I've mentioned before how easy it is to get people to buy into unproven "facts" by using propaganda like testimonials from famous somebodies.  Four out of five dentists recommend Crest toothpaste.  Have you ever asked for a report on that study?  How was the question worded?  How many dentists were queried?  Were they paid?  Were they actual dentists?  No one knows and no one cares.  Propaganda rules!

We just witnessed the craziness perpetrated by an "expert testimony" regarding vaccines as a root cause of autism in children.  We've been watching Natural Horsemanship run its course when plain old Horsemanship stopped being alluring.  In those cases and a multitude of others, there's no replicable evidence to prove the original hypothesis.

As stated in the article above, real research--with "tons" of test subjects involved and a real set of verifiable numbers resulting--is expensive and time-consuming.  We horse people often get all testy about the racing industry part of our world, but without them and their Big Dollars, there would be no research into many of the ills that befall our equine partners, so we need to not do that anymore.  We need to fund research and we need to demand results, not just catch-phrases, before we buy into something new and different.

You can run your own study and see how it goes for you so you'll really understand the downfalls of the current approach to marketing horse-cult items.  Here's a form you can print out an use to record the results:  Scientific Method worksheet  Start with the problem you want to solve, move on to a theory about how to solve it, then have at it.  Do the work; run the numbers.  Report back when you're finished.  Use as many horses as you can find, and note immediately that they mostly belong to your friends or all live at the same farm or work with the same trainer.  Recognize that that automatically makes your sample unacceptable because it's no more random than asking your family at Thanksgiving whether they like your new haircut.  If you can predict the results before you do the trials, you can throw out the whole mess as being nothing worth wasting time on.

You might also want to notice that you can't undo the experiment. You can't roll it back to before you put that lemon balm on his nose and see what would have happened if you hadn't.  That's because a living creature has an infinite number of variables going into his daily existence, and living just one day longer creates even more.  You can't undo that. The best you can hope for is something that works exactly the same way on your horse every time.  There's your evidence-based whatever.  Put a label on it and you can sell it at the next Expo.

Bottom line:  Don't confuse evidence with conjecture or wishful thinking or even random accident.  If it's real, someone can prove it.  Look for the proof before you buy into the evidence.

No comments: