Chatting with John Hattie – Pt. 1
Late last year I wrote a blog post called, Is John Talking Through His Hattie?
The subject of the post was a series of posts and articles I had come across that called into question the validity of the statistical analysis that is the backbone of John Hattie’s Visible Learning work.
To save you the time of reading that post, the key point raised was:
John Hattie had admitted half the statistics in Visible Learning are wrong.
Within a couple of days I had a reply on the post from John Hattie himself suggesting that I was “Retweeting a lie.”
I decided that this might be an opportunity to directly engage with John regarding the post and the issues raised. He agreed, and what followed was a great 60min chat about all things education, despite a few technical issues that this song brilliant captures…
The interview will actually be the basis for a series of posts, with this being the first.
So… to that blog post.
Did Hattie ever say half the statistics in Visible Learning are wrong?
Well, not according to him, he says,
“The message is that someone tweeted from a conference in London a few months ago, that apparently I said ‘half the statistics are wrong’ and I never said that.”
But that’s not to say there weren’t issues with the statistics. The main argument centres around something called the Common Language Effect Size, which allows you to compare effect sizes in simple terms. For example, Strategy A has a 90% chance of working better than Strategy B.
And it is here where Hattie admits there were some errors.
He says, “Unfortunately at the very last minute I put the wrong column up on the – as part of the confidence levels rather than the actual common language exercise.” It wasn’t until about three years later when some students in Norway alerted Hattie to the errors that the correct data was added to reprints of Visible Learning.
But as and when the next edition is published Hattie says, “I’d take all that stuff out because it didn’t work [anyway].”
Hattie says that whilst “there are some minor errors” in Visible Learning – errors that he says he and others have picked up – these are corrected quickly. And though he is disappointed that such errors occur he is confident that it doesn’t change the overall message of the book, as he says, “Not one iota.”
No doubt debate will continue in educational circles and the blogosphere, but Hattie says,
“The nature of academia is that you live off critique and so I thrive off that and so that’s why I’m happy to talk to you and anyone [but] I’m not going to tell you right now everything’s perfect.”
Hattie says he’s been doing this work and publishing Visible Learning work since about 1989 and what fascinates him is, that in the twenty or thirty years since he started working on it, “No one – not a single person – has critiqued the idea and come up with an alternative explanation for the data.”
In this series of Chatting with John Hattie posts I’ll share with you – amongst other things – our discussions around the use (and mis-use) of Visible Learning, 21st Century Learning, AITSL, Teacher Quality, and advice for new teachers.
Pingback: Chatting with John Hattie – Pt. 1 | The Echo Chamber
The CLE are wrong throughout the book, in many different sections, so it is not just a case of putting the wrong column in.
Remember that his first explanation, when it was pointed out to him that all the CLEs were wrong, was that he used a different version to calculate the CLEs (but didn’t tell anyone or refer to it in his book). He really needs to make up his mind.
They weren’t put in as part of the confidence levels, he printed the z levels instead of the probabilities you get from the probabilities. A mistake no-one with any knowledge of Statistics would make.
It is not a minor mistake. He relies on two statistics for his argument, the CLE and the Effect Size. The CLE appears throughout the book finishing off in 50 pages of tables devoted to it.
Far from correcting mistakes after they have pointed out to them, he has denied they exist or given them no publicity whatsoever.
It is not acceptable to release academic works that are riddled with errors. A Scientist or Mathematician who did so would be ridiculed and their career ruined.
Nobody critiqued the idea until someone with a proper Maths degree (me) looked at it and then pointed out that the Effect Size isn’t a real statistical tool and is unheard of by Mathematicians. Remember, John Hattie has no mathematical training.
Ollie. I’m not a statistician and far from a mathematician of any note. I was just examining Hattie’s book and was interested in how he arrived at the effect size of 0.59 for direct instruction. He cites 4 meta-analyses; 1 of 25 studies an an ES of 0.83, another of 37 studies and an ES of 0.75, one of 232 studies and an ES 0.21, and another of 10 studies and an ES of 0.55. Wouldn’t you give more weight to the larger study than just add up all the effect sizes for the 4 studies and work out the mean? They’re also all on different aspects of education. Special Ed, reading, algebra and comprehensive school reforms. Would be good to get a mathematicians perspective.