Is John talking through his Hattie?
John Hattie, the author of the much quoted Visible Learning was recently appointed by the Federal Government to the Chairmanship of the Australian Institute for Teaching and School Leadership.
In today’s Australian Newspaper, an article by Jennifer Buckingham from the Centre for Independent Studies said,
Last week, Hattie indicated things were going to change in teacher education under his watch. He talked of “tougher”, “harder”, “standards”, “outcomes” and “impacts”. This is a harbinger of his approach, grounded in the requirement for evidence that reforms are working.
It went on:
Hattie would like the accreditation and evaluation standards for teaching degrees to be much tougher. If this results in some courses being scrapped because they don’t meet the standard for the academic rigour of courses or for graduate teachers’ impact on students, then so be it.
And who better to have at the helm than the guru of effect size himself, John Hattie?
Well it would appear that just as Hattie would like teacher education to be subject to tougher standards, there are some who suggest his research should be subject to a more rigorous analysis.
Despite having been challenged as early as 2011 in the British Journal of Educational Studies, Hattie’s work has been seen as the guiding light in educational reform, but of late it has come under scrutiny, not least because of this blog post and it’s subsequent follow up which asserts that half the statistics in Visible Learning are wrong.
I’ll let you follow those threads if they interest you.
What worries me is that a great deal of educators, regional leaders, keynote speakers and politicians quote Hattie to further assert their position, particularly if that position is at odds with common sense – the class size debate for example.
Don’t get me wrong, I do believe that educational research is vital. But it is equally vital that we view it through an appropriate lens. As Dylan Wiliam says*,
Educational research can only tell us what was, not what might be. Moreover, in education, “What works?” is rarely the right question, because everything works somewhere, and nothing works everywhere, which is why in education, the right question is, “Under what conditions does this work?”
Besides I’ve always been skeptical that anyone could determine a numerical value for the impact any given teacher might have with any given student in any given classroom.
*If you want to access the quote from Wiliam you can get his PPT slides from a ResearchED event here.
[EDIT] Through the magic of Twitter, it was made apparent to me by Greg Thompson (seriously… follow him) that some of the issues raised here were first addressed by Snook et al in 2009 in NZ Ed Studies Journal.
[EDIT 2 – 19th Nov] John Hattie has agreed to an interview as per the comment thread below. I’ll let you know how things progress.
Dan
I NEVER said, anywhere, that half the statistics in my work are wrong — this is 100% incorrect, and has never said by me – so be careful what you relay what is not so. Retweeting a lie does not make it true. John Hattie
Hi John,
Thanks for taking the time to comment.
I’m interested to know if you’ve contacted the author of the blog post which makes that assertion given he appears to quote you directly from your responses to a Swedish blog.
I’m also interested as to whether your issue with my post is that I’m “retweeting a lie” – ie. that you said the stats are inaccurate, or whether the issue is that you disagree with the premise that your stats could be inaccurate.
As an aside I particularly enjoyed reading Visible Learning and got a lot out of it, and I think this is why I’m so keen to explore this further with you, because people like me rely on people like you.
Indeed many politicians, regional directors, principals, head teachers and teachers in general rely on you.
At the pointy end of things some teachers are struggling mentally and physically because their bosses insist they teach in a particular way based on your work while politicians use your stats to skew the education debate in their favour.
I’d welcome the opportunity to chat more with you about this, perhaps with the view to writing an article or interview?
All the best,
Dan
Dr HAttie how do you feel about the staff at AITSL posting ads for your book on twitter? I felt this was an extreme conflict of interest and AITSL should not be being used to promote your book (personal financial gain). What do you think? THe people at AITSL didn’t like it when I called them on it. #notagoodlook
Hi Dan,
This is a big deal because Visible Learning is so widely used, it’s well respected and there seems to be a dearth of related research (please correct me if I’m wrong).
But…
I think Ollie Orange is being disingenuous claiming that half of Hattie’s statistics are wrong. The CLE is based entirely on the effect size statistic. Nobody uses the CLE or references it in discussions – it has always been about effect size. It’s a bad miss but it hardly affects any of the conclusions of his thesis. You don’t close a bar because the beer nuts are off. No one’s eating them anyway.
Hi Rob,
Thanks for the comment and I admire anyone who can incorporate beer nuts into an education research debate!
Having said that – and to draw the analogy of rancid beer nuts out a little further – in the realm of educational research, Hattie’s work is hardly seen a drinking establishment. Rather it is akin to a Michelin starred restaurant, and as the accolades increase so do the expectations…
You wouldn’t settle for little – seemingly inconsequential – issues if you were sitting down at l’Auberge du Vieux Puits (apparently it’s quite nice there.)
Cheers again,
Dan
No! I always thought meta analyses were for the ‘common man’, bringing it all together while being accessible. It’s Starbucks. We know it’s not the most pure blends of coffee – it can’t be when you’re combining research from different contexts.
Most of us love that there’s a lot to choose from and it’s a great place from which to discuss the big picture of teaching.
Okay, I broke the metaphor.
Teachers need unifying models and principles to direct them, even if they’re personal. Meta analyses provide the breadth of research such models require.
The question is then, does Visible Learning provide that with appropriate rigour? I think we all need more information, including from social sciences researchers, not just mathematicians. I’m concerned that this is a beat up but I’m equally concerned it’s not.
I am still astounded, how 500,000 were ‘meta-analysed’ without taking into account the various levels influencing both students and teachers!!
Mmmmm. I must admit that when reading Visible Learning I was struck with how this wouldn’t normally be OK in the sector I have worked in (I have a PhD in evidence-based practice/knowledge translation in the health/disability field). A meta-analysis just wouldn’t be done if the outcome measures used were different in each study. This is why there are so many inconclusive findings in health and disability review articles.
I guess the extremes are to never do a meta-analysis unless it is all perfect OR to meta-analyse everything. I would suggest that both can be unhelpful and potentially misleading. The strength of Hattie’s book is that he has made research accessible. Good knowledge translation. The concern is that the precision of the information may be misleading or incorrect (due to methodological issues of doing meta-analyses on everything). This is however, an ongoing difficulty in translation of research into practice. A very simple message for educators (or others) often sacrifices the nuances and detail of the research behind it.
The need for numbers and effect sizes is only important if you are looking at whether or not something is effective. The question ‘Is x effective or not?’ is often fraught, and in education (and I would suggest in many areas..including knowledge translation) the answer is often ‘most things work a little bit, and more in some circumstances than others’….so maybe the research translation needs to focus on context, circumstance and creating healthy school environments so that these key teacher behaviours are ‘easy’ or ‘natural’. Anyway, just my random thoughts……
Dan one would think you worked for the state education system and was reading yammer…we have been talking about the exact same think for several weeks. What concerns me is that a meta analysis of a series of meta analyses is always going to be a bit of a fudge, let alone trying to produce statistics from what are often qualitiative studies. What if one or two of each of the original studies in each of the metaanalyses had mathematical errors and this is then compounded with further errors…the house of cards comes falling down. I have never seen educational research that is repeatable and valid and reliable enough to convince me of anything unless I already agreed with it and it was giving me the evidence to support my beliefs. To do a meta analysis of meta analyses of this kind of research just sounds like a compunding of what on the whole is often poor research (from a sciencetif point of view). I know as a teacher class size makes a huge difference to kids getting what they need to be nurtured and feel secure to then learn (thank you MAslow)…anyone who undermines this is looking through a kaleidescope of studies that are clouding the vision of children in a classroom looking for nurture and to be valued as people before they can learn.
I find the worship of visable learning worrying. There are lots of fantastic researchers out there but they didn’t get a book deal and a publisher. MArketing should not determine what evidence is accepted in education for evidence based!
Anyway love your work as usual.
This is a great article on the HAttie stuff and does some heavy lifting on the stats.
https://academiccomputing.wordpress.com/2013/08/05/book-review-visible-learning/
CJF
Interesting. I think that the social-media crowd create the exact story they want in order to engage the exact debate they want. Prof. Hattie has fallen foul of the consumer-culture that is so pervasive in ‘online educational debate’. When people put themselves aggressively into markets, either as a sophisticated, published media presence or as a ‘have a go Twitter’ then there are a group of true patriot’s and others are put in the ‘out group’. In this case, can it be possible that the Twitter-Heads so quick to latch on to the attractive idea and title of “Visible Learning” skipped the lectures, didn’t do the readings … but just listened to the crowd?
As Hattie is from the academy (largely the villain of the online debate due to elitism, producing ‘the wrong teachers’ etc), this dialogue is mostly interesting (to me) as an event in the growing competition in media cultures for authority. It’s a power struggle — what a shock. I am sure that his critics, once they have acquired the skills and reputation it takes to be a professor, will also present them to a global audience. As it is, this is just another tweet-storm, made interesting by Prof. Hattie leaving a comment, which is a kind of discursive currency I guess.
Some great points as usual! Cheers for the comment.
I’m pleased to say that he has agreed to an interview when he gets back to Sydney…
I’ll keep you posted! 🙂
Dan
He also agreed to an interview for TER Podcast at the beginning of 2014.
Dude, that’s a scoop!
As a person whose knowledge of statistics could be potentially expressed as a negative number, could someone more knowledgeable help me with this question…
Even if it is true that the measurement of effect size used in Visible Learning were based on a flawed calculation, if that flawed method is applied consistently across all the evaluated elements of education, would the comparative ratings between and among effects still be of value? Or would a more precise method of calculation render existing comparisons meaningless?
I wrote the post quoted in an earlier comment, which I still stand by: http://academiccomputing.wordpress.com/2013/08/05/book-review-visible-learning/ I know that post might be a bit dense, but it does go into a lot of the issues with the book, and not just the more trivial problems mentioned by others. My response to Capitan Typo’s question is that the flaws do not cancel out. Apples seem to be too often compared to oranges, and the problem is that as a reader, you don’t know when this is occurring without going to read all the studies included in the analysis.
As a side note, it’s quite interesting how often this post of mine gets pointed to. I only read the book as I’d seen it mentioned by some teachers, and it falls it bit outside my main area of interest. But I regularly get pingbacks from posts discussing Visible Learning — the book seems to have provoked a lot of discussion, even several years after its release.
Hi Neil,
Thanks for the the comment, I appreciate it! The book not only provokes a lot of discussion but guides a great deal of professional learning in schools…
Cheers again,
Dan
Then my follow up question is, how much of Visible Learning was subject to peer review before publication?
Hi Dan,
The one thing that I love about education is the polarity of views.
For me, the beauty of the work that John Hattie, through his Visible Learning meta analysis, has created is the increased focus on the role of the classroom teacher as a key determiner of student success. We need to raise the focus on the role of the TEACHing and how quality teacher choose the most effective strategy to meet the specific needs of a student.
As a principal of many years, and in many diverse communities, the choices a teacher makes in creating the learning experience makes the difference for the student. Hattie’s work provides teachers with the strategies to make a difference. Like Marzano and others have offered in the debate on quality teaching, Hattie’s synthesis has provided the catalyst for robust discussions in staffrooms all around the world.
I look forward to the interview and post commentary.
Great work. Cheers
Jake
I too think that Hattie’s great work has created dialogue for teachers and educators internationally. He is making the most positive difference and positive impact in classrooms everywhere. He allows teachers and educators to ask the questions and seek out how they can maximise impact for students.
It really annoys me when I find people who say how concerned they are about teaching and learning, only to bring down those who are committed to making a difference- like John Hattie. He should be congratulated for the exceptional work he has done and continues to do for teaching and learning- an inspiring professor committed to student growth and achievement and how to assist teachers in maximising their impact.
Why create a hype to bring those who are making a difference down? Tall Poppy Syndrome Perhaps? His work IS creating growth for both teachers and in doing so our students……. You should rethink slamming those who are creating such a positive influence in our schools……..
You reference the ‘class size debate’ as an instance where we’re left scratching our heads (or our beer nuts?)… It’s a no-brainer. Yes, my school leaders always quote Hattie when we push for lower class sizes each year, making statements such as “As Hattie says, rather than lower classes sizes – which have little effect – focus on practices around providing quality feedback – one of the highest effects – to your students”. Isn’t it obvious that a teacher’s ability to provide quality feedback to students increases as class sizes decrease? I could rush around giving 30 students some ill-considered, half-assed feedback; or give 18 students informed, carefully-considered and planned, quality feedback. Which would Hattie prefer? I hate when ‘research’ goes against common sense.
I am searching for the peer review of Hattie’s research. As a mathematics teacher I am concerned deeply about the damage flawed reseach results can bring about. I find it difficult to understand any educator who is blindly following research without looking for peer review. Most of his results have been no brainers but as he is calling for more rigour in teacher training shouldn’t his research be subject to the same rigour?
Robbo, It maybe a bit late but i also am trying to get together peer reviews, this is what i’ve found so far – http://visablelearning.blogspot.com/
Why is it that no one ever comments on the overall effect size comparison in the data. That is there seems to be a trend for each of the strategies / effects that Hattie has in his meta data to move to the middle as more research effect sizes are recorded. i.e Once very high effect size strategies with more research data added in over time sees a lowering in that strategies effect size. The highest and lowest effect sizes have the fewest number or data samples. as an effect size has more data samples recorded then they tend to move to the middle. Is this not troubling when we determine which effect sizes we choose to implement as school leaders / systems?
As I have learned to teach in the ways that Hattie recommends, which are more explicit and clear, and I have learned new ways to get and keep everyone’s attention, and understand at any given time what each student knows and needs to learn through good assessment, the size of my group doesn’t matter. This year I have 26 second graders in my class. We do some smaller groupings for reading but not math. My students are in the top of the district in math because of how I’m teaching it.
Remember, he was looking at academics, not neccesarily relationships. Academically there isn’t a basis for class size reduction. During noisy transitions and report cards, I would take a smaller class any day. But a smaller class wouldn’t be learning more.
If you look at the 3 papers that Hattie uses for class size they do say academic achievement will improve, particularly for younger children – have a look at their graphs here –
http://visablelearning.blogspot.com/p/class-size.html
Everything has some good effect, as Hattie acknowledges. The question is, what would be smarter: helping teachers learn how to be incredibly effective (including with behavior problems, which lower considerably with effective teaching) or hiring thousands of inexperienced teachers just to lower the number of kids in the class? Old style teaching is less effective and there is a lot of wasted time and boredom. Having more 1-1 time with a kid will probably help that one, but if you need lots of 1-1 time helping kids then you aren’t teaching the material properly to begin with.
It would be interesting to compare a small class size using old methods with a large class size where the teacher uses the best teaching techniques.
My teaching has transformed over the past several years, not because I am trained in “visible learning” at all or because I read the book. This all happened through my own searching, going to trainings, and trying different things and an open-minded principal. When I read the book it just confirmed what I had already experienced. I have proved at least in my own classroom which makes the biggest difference.
If we’re going to spend money – let it be to get us some instructional aide time (which I do have, some of the time) That’s really all I need to target some of the instruction.
if it works for you great; but what this blog and others are trying to do is look at Hattie’s evidence in detail. There are many complications and Hattie makes many claims that need to be checked, firstly by reading the original studies and then checking his calculations – that’s what i tried to do.
As a result, I and many others, including the 12 professors, who have analysed his work, have found many significant errors that people should be aware of.
It is interesting that the latest Meta-analysis from Monash university shows behavior improves with lower class size.
*its subsequent
Sure Hattie’s book provides material for discussion but to use his work to condemn so called “old fashioned” teaching approaches is a bit rich. Do people not remember the fuss over Whole Language where old fashioned teachers were condemned for using a phonics approach?
It is good to see people questioning “evidence” and research. It is also good to see school staff have points for discussion.
But don’t condemn the sceptics or the poeple who teach using what has worked and continued to do so!
I’ve come a different industry, and i find it hard to reconcile Prof Hattie’s statements ‘that class size doesn’t matter’. I’m not sure if he’s just being selectively quoted. You must take a holistic view of such a statement. It’s a time and labour consideration, surely? if I have 38 students rather than 18, to teach in 45 minutes, how much additional work pre- and post- class time is generated for a teacher? Should that be a consideration? If I’m contracted for a 38 hour week, should I be working 70 hours, as I found in my first year out? Which leads me to dissent…. compared to the industry I came from, education in Australia doesn’t seem to allow much dissent. Maybe there’s no room for it if we’re trying to transform this research into applied teaching and learning practice, but that take me back to the ‘class size doesn’t matter…’ paradigm.
No Hattie has not been selectively quoted he has made a number of public presentations from 2004-2015, where he calls the focus on class size “a disaster” and he recently published with Pearson -“Class Size is one of the major distractions”. Although, this year he is retreating form that polemic, probably because many other scholars are calling into question his research, e.g. Prof Peter Blachford, Prof John O’Neill. Details of the 3 studies Hattie uses can be found here (I think it will surprise you what the studies actual conclude) -http://visablelearning.blogspot.com.au/p/class-size.html
I think he is being selectively quoted. He doesn’t argue that class size doesn’t matter – he argues that when class sizes are reduced, teachers rarely change their approach to take advantage of this, and as a result reducing class size hasn’t worked to improve student achievement. It could work if when class sizes were reduced, teaching strategies were changed to make sure the learning experience for a student in a small class was different to their experience of being in a large class. Class size obviously does matter, in terms of pre- and post- work for teachers, but if we are pushing for reducing class sizes because it will improve student outcomes without changing how we teach (likely teaching a smaller class with the aim of greatly improving outcomes would require putting in similar hours), the evidence, according to Hattie, is just not there.
All I know is the Hattie is slowly killing us teachers! So many have left due to the workload and stress.
I sympathise as another teacher who Hattie is slowly killing Patty. I fear I have little time left as a classroom teacher due to the Hattieisation of the Australian school system, including our teacher union in my state. This is very sad for me. I appreciate and thank Dan H, George Lilley, Higgins, Simpson, Kelvin Smythe, OllieOrange2, Darcy Moore, Scott Eacott and others who have spoken out against Hattie. Where I work if you’re not on the Hattie train you’re just not cool enough for school. Unfortunately PD gluttony and me and my big career greed is the rule of our education system. So much of hardworking taxpayer money is wasted.
Dan,
Thank you for your article. You are onto important considerations. Could someone kindly provide a page/site reference to a description of the underlying measures of ‘effect size’, as in effect on what? Is the impact derived from the aggregation of loads of test performance based studies or is a wider set of holistic learner outcomes (as distinct from learning outcomes) included ? I’m interested in the definition of rigour and ‘student performance’ in this context, as I have at times found these terms to be incorrectly or narrowly assumed to mean exam performance.
Put another way, does “effect’ mean effect on test scores or a wider set of learner outcomes, for example, learners’ growth in their propensity for self learning – heutagogy?
Where I’m coming from is that we ought to be wary of seeking to maximise ‘effect’ if that effect is merely the appropriation of test scores, as this begs the question ‘effect on what and at what cost?’
I hear educators speak of their professional intuition as to what learning strategies have best effect in diverse situations. I’d appreciate deeper analysis and dialogue about the juxtaposition of an empirical meta research analysis as against the artistry of learning professionals who knows their students and have a capacity to develop students in ways which no test can fully measure. Learning is most visible through the life outcomes of learners. By this I mean the extent to which students develop their lifelong learning inventory and engage in enterprising pursuits based on sound literacy, numeracy, soft and life skills.
What do others say about this?
Kind regards,
Mark Lamont
I had the same questions as you Mark. I’ve found a number of peer reviews which look at Hattie’s methods in detail. Also, I decided to read the meta-analyses of the controversial influences like “class size” , “reducing disruptive behaviour” and a number of others. I found most of the meta-analyses weren’t measuring effect size but correlations. Where studies calculated effect sizes they were often NOT of achievement but something else., e.g. behaviour, attention, IQ. When achievement was used used it ranged from a mother’s rating of their child on a scale of 1-5 to the number of times a tennis ball could be hit continuously against a wall. Hattie jumbles and morphs all this together.
I agree with Professors Higgins and Simpson (2011) who were the first to publish many of Hattie’s misrepresentations and mistakes,
“We argue the process by which this number has been derived has rendered it practically meaningless“ (p199).
A number of other Professors say much the same thing, I’ve got the details here-
http://visablelearning.blogspot.com.au/