Wednesday, July 16, 2014

DG X: Making a Difference

Today I’m going to offer the chess world two rules that will improve the quality of debate about our game and dementia by several orders of magnitude.

  1. Writers and publishers: don’t cite 'research’ or 'science’ or 'evidence’ without providing a full reference to that research;
  2. Readers: dismiss out of hand anything and anyone that breaks Rule One.

To see how much of a difference it would make, I’m going to take a look at two pieces of writing. One, a manual used by staff in the NHS working with people who have been diagnosed with dementia; the other, an article which appeared on Chessbase.

The subject of both concerns the provision of cognitively stimulating exercises. In the circumstances, inviting readers to guess which one of the two is of negligible value seems a little out of place.





Making a Difference is a manual for running cognitive stimulation therapy programmes (known in the trade as CST) aimed at people who are living with dementia. It claims it is "effective". It claims to be an "evidence-based" programme that was evaluated in a "large-scale research project".

Describing the research that they say justifies their programme, the authors of Making a Difference write,

Over 200 people with dementia in care homes and day care centres took part, and were allocated randomly to attend the programme or to receive usual care. Those attending the groups reported increased quality of life compared to those receiving usual care. They actually improved on tests of memory and other abilities, and the conclusion was that the programme was just as effective as the drugs that have been developed to help people with dementia.

Next they do something that’s rather important when you’re discussing 'science' and 'research': they give a reference. The reference for the research that the authors of Making a Difference cite looks like this,

Spector A, Thorgrimsen L, Woods B, Royan L, Davies S, Butterworth M, Orrel M (2003) Efficacy of an evidence-based cognitive stimulation therapy programme for people with dementia: randomised controlled trial. British Journal of Psychiatry, 183 248-254

That’s a bit of a mouthful, I grant you, but if you’re serious about the 'science' thing, giving full references is what you do. Every last bit of them.

You give the authors’ names

Spector A, Thorgrimsen L, Woods B, Royan L, Davies S, Butterworth M, Orrel M (2003) Efficacy of an evidence-based cognitive stimulation therapy programme for people with dementia: randomised controlled trial. British Journal of Psychiatry, 183 248-254


the year of publication

Spector A, Thorgrimsen L, Woods B, Royan L, Davies S, Butterworth M, Orrel M (2003) Efficacy of an evidence-based cognitive stimulation therapy programme for people with dementia: randomised controlled trial. British Journal of Psychiatry, 183 248-254


the title of journal article that you are citing

Spector A, Thorgrimsen L, Woods B, Royan L, Davies S, Butterworth M, Orrel M (2003) Efficacy of an evidence-based cognitive stimulation therapy programme for people with dementia: randomised controlled trial. British Journal of Psychiatry, 183 248-254


the title of the journal in which that article appears

Spector A, Thorgrimsen L, Woods B, Royan L, Davies S, Butterworth M, Orrel M (2003) Efficacy of an evidence-based cognitive stimulation therapy programme for people with dementia: randomised controlled trial. British Journal of Psychiatry, 183 248-254


the volume number of that journal

Spector A, Thorgrimsen L, Woods B, Royan L, Davies S, Butterworth M, Orrel M (2003) Efficacy of an evidence-based cognitive stimulation therapy programme for people with dementia: randomised controlled trial. British Journal of Psychiatry, 183 248-254


and the page numbers of the volume of the journal in which the article can be found.

Spector A, Thorgrimsen L, Woods B, Royan L, Davies S, Butterworth M, Orrel M (2003) Efficacy of an evidence-based cognitive stimulation therapy programme for people with dementia: randomised controlled trial. British Journal of Psychiatry, 183 248-254


You go to all that trouble for one simple reason: so that anybody who happens to be reading what you’ve written can find your primary source(s) for themselves should they decide that they want to read it/them too.

Yes it’s a hassle, but if you’re claiming something based on somebody else’s words, the burden of enabling people to find those words falls on you and not your readers. Since the authors of Making a Difference did take the trouble to fully reference their claim that their programme is "evidence-based" and "effective" we can find the relevant Abstract (summary) on Pub Med here and even read the whole article ourselves here should we choose so to do.

Like I say, if you want your writing to have any credibility it’s what you do.






The articles that chess players write about dementia are invariably somewhat different. On Chessbase, that world famous repository of wisdom on health matters, the introduction to Michael Ciamarra’s second article about chess and dementia - the first really, since the 2013 one was cut and pasted from sundry other sources and can’t really count as his: Research is Vital - describes chess as a

"therapy"

and a

"treatment"

that

"seems to work".

So far so Making a Difference. How does Ciamarra proceed from there?

In his third paragraph* he writes,

Studies have shown that older adults with hobbies that actively engage their brains are two-times less likely to contract Alzheimer’s.

The general point, that people who live a mentally stimulating life are less likely to acquire a diagnosis of dementia is not in dispute. Ciamarra has made a very specific claim here, though. A claim which gives rise to a number of interesting questions:-


  • How old does a person have to be before they are considered an 'older adult'?
  • What does 'actively engage' look like when we see it? How is it defined?
  • What counts as a hobby?
  • How often does one have to engage in such activity before it makes a difference?
  • How frequently?
  • To get the protective benefit of this kind of hobby, do we have to have been doing it for many years, perhaps from our youth, or can we take it up in older age?
  • Do all of these "studies" come up with the same "two-times" conclusion, is this an average, or something else? (And anyway, two time less likely than what?)
  • Which studies?
  • Written by whom?
  • Published where?
  • Was the methodology used reliable?
  • Were the conclusions drawn valid?
  • Were the authors of these studies saying engaging with this kind of hobby causes the reduced risk of Alzheimer’s Disease or is the link one of association?


Ciamarra doesn’t answer any of these questions and neither can we because he doesn’t just fail to give a proper reference to the 'studies' which he cites, he doesn’t give any reference whatsoever. Neither, therefore, can we answer the most basic question of all: is Ciamarra’s summary of this research reasonable and accurate?

"Studies have shown …." Or, to put it another way:

I claim there is research that justifies what I say, but I’m not actually going to give you any details of that research so you’re going to have to take my word for it.

Is this an aberration? No, it what passes for Ciamarra’s scientific method. The very next sentence begins, "A recent study ….", and similar phrases -

"… according to a variety of scientific studies", " … the breakthroughs of science …", "… a study of older people …" - are littered throughout his article.

Nowhere will you find anything that enables the reader to track down anything that resembles a justification of what he writes.

To be fair, Ciamarra isn’t alone in his lamentable attitude to proper referencing. It’s what passes for normal when the chess world writes about dementia. Such low standards presumably going some way to explaining why Chessbase were prepared to publish such a mediocre article.

In no way does that make any of it OK.






Proper referencing matters. Without it the only justification for accepting an author’s claims is his or her say so. Beyond that, there is the proposition - very likely to be true, to say the least - that authors who cite research without providing full (or any) reference are more likely to misrepresent that research than those that do.

Ultimately, referencing matters because evidence and proof matter. Because something doesn’t become true just because you really really believe that it is, nor because your intentions are good and not even if what you’re claiming isn’t a million miles away from something that should be welcomed.

Provide opportunities for older people to learn and play chess or other hobbies of their choice - you’re alright by me. Claim that 'science' says that chess is an effective intervention to tackle dementia without any reference to the research that you cite - you’re a joke, albeit not an especially funny one.

The NHS doesn’t do jokes. That’s why if you go to one of their Memory Clinics where they diagnose and engage with people who have dementia (the one in Mile End Hospital, say, where I worked for two years before leaving a couple of months ago) and ask about Cognitive Stimulation Therapy you’ll find that they’re using the programme outlined in Making a Difference or something very similar. Not chess.

Why? Because Making a Difference has a genuine research base and as a result it’s approved by the National Institute for Clinical Excellence [but see comments box - JMGB]. If we want claims for the health benefits of chess to have the same credibility, we’re going to have to create a research base of our own. A culture of zero tolerance for citing 'research' or making allusions to 'science' without providing a full reference to that research is a pretty good way to begin building one.

Let’s make a start.


Chess and Dementia Index








* It’s a story for another day, but the fact that Ciamarra’s opening sentence demonstrates that he doesn’t understand the meaning of "early onset" in relation to Alzheimer’s Disease - basic terminology for anybody who works in the field - is very much Not A Good Sign.


5 comments:

Unknown said...

A couple of comments and one correction.

Just because a reference is given, that doesn't mean that it actually supports the author's thesis. Quacks will often cite journal articles knowing that they contradict their claims, but trusting that looking sufficiently 'sciencey' will be enough to convince people.

Second, a forthcoming paper in Lancet Neurology lists the risk factors for Alzheimer's as 'diabetes, midlife hypertension, midlife obesity, physical inactivity, depression, smoking, and low educational attainment'. No mention there of anything that could be improved by playing board games of any kind.

The correction: NICE was once the National Institute for Clinical Excellence (not 'Evidence'), but it's now the National Institute for Health and Care Excellence.

Jonathan B said...

Thanks Ian,

I’d actually meant to type Excellence - not sure how evidence got in there. Either autocorrect or typo on my part. Either way, I’ve changed it to what i’d intended and left a note guiding readers to your comment.

RE: NICE’s new name ... it was NICE when I was a Social Worker. I see it is still "NICE" although the C has been changed to 'Health and Care' as you say. A bit strange that - I suppose they wanted to keep the 'brand identity' as we say these days.

Anyway, thanks for the correction.


You’re quite right of course that a full reference doesn’t guarantee support for what the author claims. It does at least allow for the author’s claims to be checked, though. And thus disproved and debunked where they need to be.

I’m also pretty sure that full referencing would drastically reducing the misrepresentation of research even if it wouldn’t totally eliminate that misrepresentation. Kasparov would be much less likely to claim existence of research that doesn’t actually exist if he was in the habit of providing full references for any research he cites, for example.

Funnily enough, I was discussing that Lancet Neurology article with somebody yesterday. I was mentioned in the Guardian last week (without a full reference of course). I found the absence of any 'cognitive activity' category a bit strange because I think it’s definitely worth discussing as a risk factor, but, yes, the fact that it is absent is rather striking isn’t it.

Jonathan B said...

More on "scienceyness" next week, btw.

There Lancet Neurology article I’m intending to come back to in due course. Although ['m going to be a bit busy over the next month or so, so the chess and dementia stuff will have to take a little bit of a back seat.

Jonathan B said...

*It* was mentioned

John Townsend said...

"The Chesse of all games, wherein is no bodily exercise, is most to be commended, for therein is righte subtile engyne, whereby the wit is made more sharp and remembrance quickened."

(Words attributed to Sir Thomas Elyot, 1534: The British Miscellany, 1841, p. 133)