Is John talking through his Hattie?

Research QuoteJohn Hattie, the author of the much quoted Visible Learning was recently appointed by the Federal Government to the Chairmanship of the Australian Institute for Teaching and School Leadership. 

In today’s Australian Newspaper, an article by Jennifer Buckingham from the Centre for Independent Studies said,

Last week, Hattie indicated things were going to change in teacher education under his watch. He talked of “tougher”, “harder”, “standards”, “outcomes” and “impacts”. This is a harbinger of his approach, grounded in the requirement for evidence that reforms are working.

It went on:

Hattie would like the accreditation and evaluation standards for teaching degrees to be much tougher. If this results in some courses being scrapped because they don’t meet the standard for the academic rigour of courses or for graduate teachers’ impact on students, then so be it.

And who better to have at the helm than the guru of effect size himself, John Hattie?

Well it would appear that just as Hattie would like teacher education to be subject to tougher standards, there are some who suggest his research should be subject to a more rigorous analysis.

Despite having been challenged as early as 2011 in the British Journal of Educational Studies, Hattie’s work has been seen as the guiding light in educational reform, but of late it has come under scrutiny, not least because of this blog post and it’s subsequent follow up which asserts that half the statistics in Visible Learning are wrong.

I’ll let you follow those threads if they interest you.

What worries me is that a great deal of educators, regional leaders, keynote speakers and politicians quote Hattie to further assert their position, particularly if that position is at odds with common sense – the class size debate for example.

Don’t get me wrong, I do believe that educational research is vital. But it is equally vital that we view it through an appropriate lens. As Dylan Wiliam says*,

Educational research can only tell us what was, not what might be. Moreover, in education, “What works?” is rarely the right question, because everything works somewhere, and nothing works everywhere, which is why in education, the right question is, “Under what conditions does this work?”

Besides I’ve always been skeptical that anyone could determine a numerical value for the impact any given teacher might have with any given student in any given classroom.

*If you want to access the quote from Wiliam you can get his PPT slides from a ResearchED event here.

[EDIT] Through the magic of Twitter, it was made apparent to me by Greg Thompson (seriously… follow him) that some of the issues raised here were first addressed by Snook et al in 2009 in NZ Ed Studies Journal.

[EDIT 2 – 19th Nov] John Hattie has agreed to an interview as per the comment thread below. I’ll let you know how things progress.

32 thoughts on “Is John talking through his Hattie?

  1. John Hattie Reply

    Dan
    I NEVER said, anywhere, that half the statistics in my work are wrong — this is 100% incorrect, and has never said by me – so be careful what you relay what is not so. Retweeting a lie does not make it true. John Hattie

    • Dan H Reply

      Hi John,

      Thanks for taking the time to comment.

      I’m interested to know if you’ve contacted the author of the blog post which makes that assertion given he appears to quote you directly from your responses to a Swedish blog.

      I’m also interested as to whether your issue with my post is that I’m “retweeting a lie” – ie. that you said the stats are inaccurate, or whether the issue is that you disagree with the premise that your stats could be inaccurate.

      As an aside I particularly enjoyed reading Visible Learning and got a lot out of it, and I think this is why I’m so keen to explore this further with you, because people like me rely on people like you.

      Indeed many politicians, regional directors, principals, head teachers and teachers in general rely on you.

      At the pointy end of things some teachers are struggling mentally and physically because their bosses insist they teach in a particular way based on your work while politicians use your stats to skew the education debate in their favour.

      I’d welcome the opportunity to chat more with you about this, perhaps with the view to writing an article or interview?

      All the best,
      Dan

    • back2teaching Reply

      Dr HAttie how do you feel about the staff at AITSL posting ads for your book on twitter? I felt this was an extreme conflict of interest and AITSL should not be being used to promote your book (personal financial gain). What do you think? THe people at AITSL didn’t like it when I called them on it. #notagoodlook

  2. Rob McTaggart Reply

    Hi Dan,
    This is a big deal because Visible Learning is so widely used, it’s well respected and there seems to be a dearth of related research (please correct me if I’m wrong).
    But…
    I think Ollie Orange is being disingenuous claiming that half of Hattie’s statistics are wrong. The CLE is based entirely on the effect size statistic. Nobody uses the CLE or references it in discussions – it has always been about effect size. It’s a bad miss but it hardly affects any of the conclusions of his thesis. You don’t close a bar because the beer nuts are off. No one’s eating them anyway.

    • Dan H Reply

      Hi Rob,

      Thanks for the comment and I admire anyone who can incorporate beer nuts into an education research debate!

      Having said that – and to draw the analogy of rancid beer nuts out a little further – in the realm of educational research, Hattie’s work is hardly seen a drinking establishment. Rather it is akin to a Michelin starred restaurant, and as the accolades increase so do the expectations…

      You wouldn’t settle for little – seemingly inconsequential – issues if you were sitting down at l’Auberge du Vieux Puits (apparently it’s quite nice there.)

      Cheers again,
      Dan

      • Rob McTaggart Reply

        No! I always thought meta analyses were for the ‘common man’, bringing it all together while being accessible. It’s Starbucks. We know it’s not the most pure blends of coffee – it can’t be when you’re combining research from different contexts.
        Most of us love that there’s a lot to choose from and it’s a great place from which to discuss the big picture of teaching.
        Okay, I broke the metaphor.
        Teachers need unifying models and principles to direct them, even if they’re personal. Meta analyses provide the breadth of research such models require.
        The question is then, does Visible Learning provide that with appropriate rigour? I think we all need more information, including from social sciences researchers, not just mathematicians. I’m concerned that this is a beat up but I’m equally concerned it’s not.

        • Steve Reply

          I am still astounded, how 500,000 were ‘meta-analysed’ without taking into account the various levels influencing both students and teachers!!

        • limegreenwombat Reply

          Mmmmm. I must admit that when reading Visible Learning I was struck with how this wouldn’t normally be OK in the sector I have worked in (I have a PhD in evidence-based practice/knowledge translation in the health/disability field). A meta-analysis just wouldn’t be done if the outcome measures used were different in each study. This is why there are so many inconclusive findings in health and disability review articles.

          I guess the extremes are to never do a meta-analysis unless it is all perfect OR to meta-analyse everything. I would suggest that both can be unhelpful and potentially misleading. The strength of Hattie’s book is that he has made research accessible. Good knowledge translation. The concern is that the precision of the information may be misleading or incorrect (due to methodological issues of doing meta-analyses on everything). This is however, an ongoing difficulty in translation of research into practice. A very simple message for educators (or others) often sacrifices the nuances and detail of the research behind it.

          The need for numbers and effect sizes is only important if you are looking at whether or not something is effective. The question ‘Is x effective or not?’ is often fraught, and in education (and I would suggest in many areas..including knowledge translation) the answer is often ‘most things work a little bit, and more in some circumstances than others’….so maybe the research translation needs to focus on context, circumstance and creating healthy school environments so that these key teacher behaviours are ‘easy’ or ‘natural’. Anyway, just my random thoughts……

  3. back2teaching Reply

    Dan one would think you worked for the state education system and was reading yammer…we have been talking about the exact same think for several weeks. What concerns me is that a meta analysis of a series of meta analyses is always going to be a bit of a fudge, let alone trying to produce statistics from what are often qualitiative studies. What if one or two of each of the original studies in each of the metaanalyses had mathematical errors and this is then compounded with further errors…the house of cards comes falling down. I have never seen educational research that is repeatable and valid and reliable enough to convince me of anything unless I already agreed with it and it was giving me the evidence to support my beliefs. To do a meta analysis of meta analyses of this kind of research just sounds like a compunding of what on the whole is often poor research (from a sciencetif point of view). I know as a teacher class size makes a huge difference to kids getting what they need to be nurtured and feel secure to then learn (thank you MAslow)…anyone who undermines this is looking through a kaleidescope of studies that are clouding the vision of children in a classroom looking for nurture and to be valued as people before they can learn.
    I find the worship of visable learning worrying. There are lots of fantastic researchers out there but they didn’t get a book deal and a publisher. MArketing should not determine what evidence is accepted in education for evidence based!
    Anyway love your work as usual.
    This is a great article on the HAttie stuff and does some heavy lifting on the stats.
    https://academiccomputing.wordpress.com/2013/08/05/book-review-visible-learning/
    CJF

  4. dskmag Reply

    Interesting. I think that the social-media crowd create the exact story they want in order to engage the exact debate they want. Prof. Hattie has fallen foul of the consumer-culture that is so pervasive in ‘online educational debate’. When people put themselves aggressively into markets, either as a sophisticated, published media presence or as a ‘have a go Twitter’ then there are a group of true patriot’s and others are put in the ‘out group’. In this case, can it be possible that the Twitter-Heads so quick to latch on to the attractive idea and title of “Visible Learning” skipped the lectures, didn’t do the readings … but just listened to the crowd?

    As Hattie is from the academy (largely the villain of the online debate due to elitism, producing ‘the wrong teachers’ etc), this dialogue is mostly interesting (to me) as an event in the growing competition in media cultures for authority. It’s a power struggle — what a shock. I am sure that his critics, once they have acquired the skills and reputation it takes to be a professor, will also present them to a global audience. As it is, this is just another tweet-storm, made interesting by Prof. Hattie leaving a comment, which is a kind of discursive currency I guess.

  5. Capitan Typo Reply

    As a person whose knowledge of statistics could be potentially expressed as a negative number, could someone more knowledgeable help me with this question…

    Even if it is true that the measurement of effect size used in Visible Learning were based on a flawed calculation, if that flawed method is applied consistently across all the evaluated elements of education, would the comparative ratings between and among effects still be of value? Or would a more precise method of calculation render existing comparisons meaningless?

    • Neil Brown Reply

      I wrote the post quoted in an earlier comment, which I still stand by: http://academiccomputing.wordpress.com/2013/08/05/book-review-visible-learning/ I know that post might be a bit dense, but it does go into a lot of the issues with the book, and not just the more trivial problems mentioned by others. My response to Capitan Typo’s question is that the flaws do not cancel out. Apples seem to be too often compared to oranges, and the problem is that as a reader, you don’t know when this is occurring without going to read all the studies included in the analysis.

      As a side note, it’s quite interesting how often this post of mine gets pointed to. I only read the book as I’d seen it mentioned by some teachers, and it falls it bit outside my main area of interest. But I regularly get pingbacks from posts discussing Visible Learning — the book seems to have provoked a lot of discussion, even several years after its release.

      • Dan H Reply

        Hi Neil,

        Thanks for the the comment, I appreciate it! The book not only provokes a lot of discussion but guides a great deal of professional learning in schools…

        Cheers again,
        Dan

  6. Jake Madden Reply

    Hi Dan,

    The one thing that I love about education is the polarity of views.

    For me, the beauty of the work that John Hattie, through his Visible Learning meta analysis, has created is the increased focus on the role of the classroom teacher as a key determiner of student success. We need to raise the focus on the role of the TEACHing and how quality teacher choose the most effective strategy to meet the specific needs of a student.

    As a principal of many years, and in many diverse communities, the choices a teacher makes in creating the learning experience makes the difference for the student. Hattie’s work provides teachers with the strategies to make a difference. Like Marzano and others have offered in the debate on quality teaching, Hattie’s synthesis has provided the catalyst for robust discussions in staffrooms all around the world.

    I look forward to the interview and post commentary.

    Great work. Cheers

    Jake

  7. Passionate Teacher Reply

    I too think that Hattie’s great work has created dialogue for teachers and educators internationally. He is making the most positive difference and positive impact in classrooms everywhere. He allows teachers and educators to ask the questions and seek out how they can maximise impact for students.

    It really annoys me when I find people who say how concerned they are about teaching and learning, only to bring down those who are committed to making a difference- like John Hattie. He should be congratulated for the exceptional work he has done and continues to do for teaching and learning- an inspiring professor committed to student growth and achievement and how to assist teachers in maximising their impact.

    Why create a hype to bring those who are making a difference down? Tall Poppy Syndrome Perhaps? His work IS creating growth for both teachers and in doing so our students……. You should rethink slamming those who are creating such a positive influence in our schools……..

  8. Teachling Reply

    You reference the ‘class size debate’ as an instance where we’re left scratching our heads (or our beer nuts?)… It’s a no-brainer. Yes, my school leaders always quote Hattie when we push for lower class sizes each year, making statements such as “As Hattie says, rather than lower classes sizes – which have little effect – focus on practices around providing quality feedback – one of the highest effects – to your students”. Isn’t it obvious that a teacher’s ability to provide quality feedback to students increases as class sizes decrease? I could rush around giving 30 students some ill-considered, half-assed feedback; or give 18 students informed, carefully-considered and planned, quality feedback. Which would Hattie prefer? I hate when ‘research’ goes against common sense.

  9. Robbo Reply

    I am searching for the peer review of Hattie’s research. As a mathematics teacher I am concerned deeply about the damage flawed reseach results can bring about. I find it difficult to understand any educator who is blindly following research without looking for peer review. Most of his results have been no brainers but as he is calling for more rigour in teacher training shouldn’t his research be subject to the same rigour?

  10. Brian Reply

    Why is it that no one ever comments on the overall effect size comparison in the data. That is there seems to be a trend for each of the strategies / effects that Hattie has in his meta data to move to the middle as more research effect sizes are recorded. i.e Once very high effect size strategies with more research data added in over time sees a lowering in that strategies effect size. The highest and lowest effect sizes have the fewest number or data samples. as an effect size has more data samples recorded then they tend to move to the middle. Is this not troubling when we determine which effect sizes we choose to implement as school leaders / systems?

  11. Sally Bergquist Reply

    As I have learned to teach in the ways that Hattie recommends, which are more explicit and clear, and I have learned new ways to get and keep everyone’s attention, and understand at any given time what each student knows and needs to learn through good assessment, the size of my group doesn’t matter. This year I have 26 second graders in my class. We do some smaller groupings for reading but not math. My students are in the top of the district in math because of how I’m teaching it.
    Remember, he was looking at academics, not neccesarily relationships. Academically there isn’t a basis for class size reduction. During noisy transitions and report cards, I would take a smaller class any day. But a smaller class wouldn’t be learning more.

      • Sally Bergquist Reply

        Everything has some good effect, as Hattie acknowledges. The question is, what would be smarter: helping teachers learn how to be incredibly effective (including with behavior problems, which lower considerably with effective teaching) or hiring thousands of inexperienced teachers just to lower the number of kids in the class? Old style teaching is less effective and there is a lot of wasted time and boredom. Having more 1-1 time with a kid will probably help that one, but if you need lots of 1-1 time helping kids then you aren’t teaching the material properly to begin with.
        It would be interesting to compare a small class size using old methods with a large class size where the teacher uses the best teaching techniques.
        My teaching has transformed over the past several years, not because I am trained in “visible learning” at all or because I read the book. This all happened through my own searching, going to trainings, and trying different things and an open-minded principal. When I read the book it just confirmed what I had already experienced. I have proved at least in my own classroom which makes the biggest difference.
        If we’re going to spend money – let it be to get us some instructional aide time (which I do have, some of the time) That’s really all I need to target some of the instruction.

        • George LILLEY Reply

          if it works for you great; but what this blog and others are trying to do is look at Hattie’s evidence in detail. There are many complications and Hattie makes many claims that need to be checked, firstly by reading the original studies and then checking his calculations – that’s what i tried to do.

          As a result, I and many others, including the 12 professors, who have analysed his work, have found many significant errors that people should be aware of.

          It is interesting that the latest Meta-analysis from Monash university shows behavior improves with lower class size.

  12. Nick Reply

    Sure Hattie’s book provides material for discussion but to use his work to condemn so called “old fashioned” teaching approaches is a bit rich. Do people not remember the fuss over Whole Language where old fashioned teachers were condemned for using a phonics approach?
    It is good to see people questioning “evidence” and research. It is also good to see school staff have points for discussion.
    But don’t condemn the sceptics or the poeple who teach using what has worked and continued to do so!

  13. Bongo Reply

    I’ve come a different industry, and i find it hard to reconcile Prof Hattie’s statements ‘that class size doesn’t matter’. I’m not sure if he’s just being selectively quoted. You must take a holistic view of such a statement. It’s a time and labour consideration, surely? if I have 38 students rather than 18, to teach in 45 minutes, how much additional work pre- and post- class time is generated for a teacher? Should that be a consideration? If I’m contracted for a 38 hour week, should I be working 70 hours, as I found in my first year out? Which leads me to dissent…. compared to the industry I came from, education in Australia doesn’t seem to allow much dissent. Maybe there’s no room for it if we’re trying to transform this research into applied teaching and learning practice, but that take me back to the ‘class size doesn’t matter…’ paradigm.

    • George LILLEY Reply

      No Hattie has not been selectively quoted he has made a number of public presentations from 2004-2015, where he calls the focus on class size “a disaster” and he recently published with Pearson -“Class Size is one of the major distractions”. Although, this year he is retreating form that polemic, probably because many other scholars are calling into question his research, e.g. Prof Peter Blachford, Prof John O’Neill. Details of the 3 studies Hattie uses can be found here (I think it will surprise you what the studies actual conclude) -http://visablelearning.blogspot.com.au/p/class-size.html

  14. Patty Lennon Reply

    All I know is the Hattie is slowly killing us teachers! So many have left due to the workload and stress.

Leave a Reply