Thursday, May 14, 2015

Doctor Garry XIX: Nullius in Verba

Much like Bruce Forsyth, the Royal Society has a catchphrase: Nullius in verba or 'On the word of nobody’. Science isn’t about assertions on what is right, handed down from authority figures. It’s about clear descriptions of studies, and the results that came from them, followed by an explanation of why they support or refute a given idea.

Ben Goldacre:

How time flies. It’s a year to the day since Doctor Garry was first in. A year and a day since I Am Garry decided that

was a message that he wanted to send out into the world.

Alas for IAG - fortunately for the rest of us - science isn’t about assertions handed down from authority figures. I think you’ll find it’s a bit more complicated than that. If somebody - even an ex-World Chess Champion - says there are "studies" to support their argument it means precisely nothing unless we know what those studies are. Unless we can check for ourselves should we choose so to do.

This is fundamental. It’s a point that Goldacre repeats over and over again in the book of his collected writings.

Science has authority, not because of white coats or titles, but because of precision and transparency: you explain your theory, set out your evidence, and reference the studies that support your case. Other scientists can then read it, see if you’ve fairly represented the evidence, and decide whether the methods of the papers you’ve cited really do produce results that meaningfully support your hypothesis.

... science isn’t about authoritative utterances from men in white coats, it’s about showing your working.


Academic papers are filled with ideas and evidence to be read, weighed up, and critically appraised by people with the motivation and skills to do so, whoever they may be. Science is not, and should not be, about arguing from authority. The idea that conclusions of a published paper are automatically true was never helpful. 

As so often, this is about transparency, which is ultimately the only source of authority in science: we want the methods and result of scientific research to be formally presented, and accessible by all, so that we can see what was done, and what they found. If a government report on anything relies substantially on unpublished and inaccessible research, then we are correctly concerned. 

science is about clarity and transparency

It is possible in healthcare to do great harm, while intending to do good, and so medicine thrives on criticism: this is how ideas improve ....

If science has any credibility, it derives from transparency: when you make a claim about how something works, you provide references to experiments, which describe openly and in full what was done, in enough detail for the experiment to be replicated. You explain what was measured, and how. Then people can freely discuss what they think this all means in the real world. 

The entirety of science is built on transparency, on giving your evidence and engaging with legitimate criticism. If you hear of a company refusing to hand over the evidence it says it supports its claims - whether it is a drug company, or some dismal cosmetics firm - all you know is that you are being deprived of information, and that vital parts of the picture are missing.

This is not difficult to understand and conclusions are not difficult to draw. Anybody who doesn’t get it is simply not to be trusted.

And here we are again.  "It's been shown in studies ...." I Am Garry told us just a few weeks ago.

For what it’s worth, the 'potential for distraction is as disruptive as actual distraction' thesis strikes me as more than a little plausible. That I find it so is entirely besides the point. What if I wanted to check? What if I wanted to prove my gut feeling/suspicion/opinion based on personal experience of  two years trying to get some bloody work done at the god-awful LSE Library/etc was actually correct?

What could be less clear, less transparent, more inaccessible than "studies" that are not named? Since it’s impossible to know what research is being referred to here, there is no possible way to discover whether it exists and if it does whether it says what it’s claimed to say. We cannot look it up. We cannot check.

In truth there is nothing here of any substance at all. Nothing but, "believe on my word for I Am Garry."

That, a year on, is where we still are with Garry Kasparov. He implies 'scientific method' but scratch the surface and it turns out that what he’s up to is indistinguishable from the behaviour of a Chessbase bullshitter.

Chess and Dementia Index


an ordinary chessplayer said...

I must admit I didn't give this whole "studies show" business much thought before. Probably because I buy into the old adage "don't believe anything you hear and only half of what you read". Needs updating for the internet age of course.

I did a little research into scholarly vs non-scholarly references. Apparently newspapers and such don't have to give their references in the rigorous format, they just ought to give sufficient information that the reader can hunt down the study. The only thing I could find about twitter was how to make references *to*; apparently nobody ever thought of putting references *in*, except for twitter referencing twitter of course.

Then I did a "study" of my own. I googled "studies show", and looked at the first page of news links (10), and the first page of web links (8), to see how many of them actually give references [y] or not [n]. One difference would be that news is ordered more by date, whereas the web is ordered more by popularity. So I think the news might actually be more representative.

=== news ===
[n], [n], [n], [y], [y], [n], [y], [y], [y], [y]

=== web ===
[y], [y], [n], [y], [y], [y], [y], [y]

I found these two links amusing:

That book by Martha Stout I mentioned the other day does give end-notes -- sometimes referencing studies, sometimes referencing scholarly third-party sources which presumably further reference some studies.

Anonymous said...

I have to say that this article strays into overkill territory for me. Yes, it is bullshit, but I have a nose and I can smell that for myself. :)

David R said...

The 'Daily Mash' blog often uses a spoof authority, Dr Henry Neustader, Director of Studies, for comment when ripping the proverbial out of newspaper reports which begin "Studies show...".

It's a well-worn path for those who fail to follow the proper scientific way, yet seek to buttress their claims with the appearance of 'evidence'.

Gary has form in this regard. His multi-volume, MGP, quickly gave up its initial pretensions to academic-standard historiography when Vol 1 was trashed by erudite friend and foe alike for making absurd claims that Botvinnik, Tal, Petrosian and Spassky could be regarded as embodiments of changing phases in Soviet politics.

These things happen when great chessplayers confuse what they are good at with genuinely intelligent insight

Jonathan B said...

Afternoon all.

Newspapers are of course notoriously unreliable as sources of evidence. For anything at all. Particularly so with regard to academic research.

See DG V, for example.

More on this - particularly with regard to Ben Goldacre’s views on the press releases that sometimes accompany publication of research - at some point in the future.

Definitely lots of different types of evidence that you can cite. I don’t particularly have a problem with any. It all depends on context. Just as long as the reader knows what you’re referring to they can make a judgement as to how vaild/reliable it is.

Which is why it matters that I AM GARRY said, "There are many studies showing ..." when he actually meant "My mate Leanxto reckons ...."

@Blue Weasel:
I’m not entirely sure if you mean this particular post or the series that has now reached 19 posts.

Either way, I can only say that I wish all our readers were like you. It’s a cause of some distress to me, but I’m afraid some of our readers don’t see the problem at all. We do what we can to shake off these feckin eejits - and I’ll be having another go on Monday - but I’m afraid they do rather hang around.

Campion said...

Off-topic for this post, but perhaps not for this blog...

I've been a little lukewarm on Goldacre for a while, perhaps unduly prejudiced by (The Sun article is obviously garbage, but Goldacre jumps to massive conclusions about the "Cambridge mathematician" who the Sun quote, without bothering to type the name of said "Cambridge mathematician" into a search engine.)

an ordinary chessplayer said...

I meant newspapers don't have to follow the rigorous form when citing *studies*. See "deft textual attribution" at this link.

Jonathan B said...

Hello OCP.

I think that referencing protocols and different forms of evidence are different sides of the same coin.

No, I certainly wouldn’t expect newspapers to follow the standard form of reference that you would expect in academic articles (i.e.. as outlined in DG X: Making a Difference).

In print media, I absolutely would expect enough information to be able to track down the specific source. From internet publications, however, there’s no excuse for not providing a link to the journal abstract (if it’s an article being studied) or the particular book. Links to a news story or press release are worthless.

ejh said...

In print media, I absolutely would expect enough information to be able to track down the specific source

As not provided by one Nigel Short, for instance, when writing recently in New In Chess. (See THING TWO here.)