Categories
Education Leadership

Acting on Evidence

This blog post is in response to a piece written by Dr Deborah Netolicky. It would be worth reading it to give you the context of my response below. But if you want the quick version:

  • Social Ventures Australia, The Commonwealth Bank and the Education Endowment Foundation released the Aussie Teaching & Learning Toolkit that looks at loads of different research in an attempt to rank effect or non-effect educational practices by cost and by the “security” of the research.
  • Fairfax Media published a piece about the Toolkit entitled The 10,000 Pieces of Research That Will End the Homework Wars.
  • I said, “Nonsense” in my section on the TER Podcast, and lots of others said similar, and Dr Deborah Netolicky blogged about it far more eloquently than I did.
  • SVA responded in a blog post saying, “Chill out” (it was far more reasoned than that, but you get the gist).
  • Dr Deborah Netolicky then responded with more words of caution around the use of meta-analyses on education research as well as the value of being publicly challenged.

Phew… you with me?

As has already been pointed out there are many who hold reservations about the veracity of meta-analyses. Dylan Wiliam pointed out several issues in a comment on a post I wrote last year about Visible Learning in the Aussie Documentary Revolution School.

But even if we assume – just for a moment – that we could place 100% faith in the “padlock” system, we are then presented with how we act according with evidence.

There are countless examples in society of where, even when presented with fairly substantial evidence, people still make “interesting” decisions – whether they be jurors in a courtroom, parents who choose to run the gauntlet with measles, smokers, or dare I say it, leaders of the free world.

So it’s interesting to note in his E4L blog post, John Bush states:

“We do not envision the Toolkit as a resource that should dictate or direct professional decisions in schools. Instead, we hope school leaders and teachers will use it to start discussions with their peers and to help inform their professional judgement with research evidence.”

For what it’s worth I find these words encouraging – not that E4L – or anyone else for that matter – need or want my blessing – but I believe these words also serve to highlight one of the fundamental issues in education.

In my current research I’m reflecting on the point that schools have never had more research, policies and programs aimed student well-being, yet the NSW Centre for Education Statistics & Evaluation present findings that suggest these are simply not being implemented or if they are, they are missing the mark.

This could be because as Stephen Ball suggests in his 2006 book, Education Policy and Social Class, that the disparity between policy, programs and student experience could because whilst providing goals or outcomes, such documents rarely tell you what to do. They may provide links to further resources or options for action, but a response from individual schools still needs to be put together. The contextual nuance means that the success or otherwise of these responses are hard to predict. He states that the enactment of such texts, “relies on things like commitment, understanding, capability, resources, practical limitations, cooperation and (importantly) inter-textual compatibility (Ball, p.47, 2006).

Furthermore Ball suggests the more ideologically abstract – which one might argue describes the concept of well-being and perhaps learning too – the less likely it is to be accommodated into the practice of a school.

To be clear, I’m not anti the toolkit, just as I’m not anti Visible Learning or meta-analyses per se, rather I’m urging – as I thinking most are now – a careful, contextually appropriate and nuanced approach to school improvement.

 

Categories
Education Media

The Revolution Won’t Necessarily Be Televised

keep-calm-and-start-a-revolution-6Over the past month Australia’s national broadcaster, the ABC, aired a four-part documentary called Revolution School in which it followed the staff and students of Kambrya College in Victoria throughout the course of 2015. The premise of the doco was that Kambrya was a struggling school – in 2008 its Year 12 results put it in the state’s bottom 10% of schools – and that by applying “cutting edge research developed by Professor John Hattie at the University of Melbourne’s Graduate School of Education, [Kambrya] undergoes a dramatic transformation. Ultimately [Revolution School] is a lesson for all schools in Australia, identifying what they can do to improve standards at this critical time.”

It was set against the backdrop of an Australian education system that is, “letting down our kids and the nation” and that’s compounded by the ‘fact’ that most of the things educators and parents think matter in education actually don’t.

John Hattie states from the outset that reducing class size, private education and giving parents choice do not “make a difference to the quality of education.”

The show concluded this week and it’s fair to say it received a mixed response – from 43 people on Twitter at least.

Screen Shot 2016-06-23 at 1.40.20 pm

Here, for what it’s worth, are my thoughts on it.

THE TITLE – REVOLUTION SCHOOL

revolution

noun: revolution; plural noun: revolutions

a dramatic and wide-reaching change in conditions, attitudes, or operation.

I can only assume that this was a decision made by TV execs who needed something catchy, and to be fair it certainly caught my attention when I saw it advertised. I was genuinely curious to see what kind of revolution was taking place, but I was left feeling a little underwhelmed.

To be clear – I’m not dismissive of the efforts of the teachers, the consultants and – importantly – the students. Any school, teacher or student who works to improve standards is worthy of acknowledgement. From the outset I commend the staff and students of Kambrya for allowing the cameras in to give an insight into how a dedicated team of teachers (as well as the kids) can address some of the daily challenges faced in a typical school. For some watching it would have proven insightful.

Rather I was left underwhelmed because there was very little in the show that could be seen as being revolutionary. Whilst it might have documented a wide-reaching change in conditions, attitudes or operations at Kambrya, the claim that Revolution School would “serve as a lesson for all schools in Australia,” might be seen as a tad patronising.

It should be noted that Revolution School  had at one time been titled Making The Grade, which in my opinion would have been a more appropriate (if less appealing to TV execs) title for the documentary.

THE STRATEGIES

Many of the strategies for improvement were a result of the school’s partnership with Melbourne Graduate School of Education (MGSE), with its impressive team of educators and experts. Teachers were able to work with the likes of John Hattie, Lea Waters, Bill Rogers and Di Snowball to refine their approaches in class and the wider community. However the coverage of these working relationships was superficial at best, limited to soundbites and the obligatory rounds of applause at the end of teacher PL sessions. Bill Rogers, for example is a legend in the realm of classroom climate and handling behaviour issues, but all we got from his appearance was the idea that you could chart on a whiteboard how on-task the class are over the course of a lesson. Now I appreciate that TV execs might not think that a more in-depth exploration of classroom climate would be compelling viewing, but if Revolution School really was going to “serve as a lesson for all schools in Australia” then these are the paths they needed to go down.

Similarly with regard to Lea Waters’ work on wellbeing, I cringed at the superficial nature in which it was presented to the TV audience, with no reference to what Positive Psychology is or the potential significance of character strengths.

However one soundbite I did enjoy was literacy expert Di Snowball’s reflection of the way in which many schools teach reading using literacy exercise books. In doing this kids only read small passages in isolation – something that was implied did little to encourage a love of reading and as Di pointed out:

What’s the point of improving reading through these programmes if you aren’t then reading?

John Hattie was a common voice throughout the series and in Episode 4 introduced the idea of teachers having their lessons transcribed live, and their words projected onto screens around the classroom. A programme from MGSE called “The Visible Classroom.” (As an aside, this is on top of Hattie’s Visible Learning approach, and Lea Waters’ Visible Wellbeing approach. I’m spotting a theme.) One of the main reasons for doing this is so that the data from the class can be evaluated by MGSE to ascertain how much a teacher talks and what kind of dialogue with regards to questions and interaction is happening. Hattie argues that teachers should be talking for around 50% of the time in class, but in reality most teachers spend 80-90% of the time talking. Assuming this premise is correct, the episode went on to show teachers who had in fact reduced their talk in class, but again tellingly for a series that was to “serve as a lesson for all schools in Australia,” it didn’t explore the changes in practice/planning that teachers undertook to make this happen. I’d imagine for some teachers these changes would have presented significant challenges, and it would have been good to see how these challenges were addressed.

Towards the end of the series the concept of Clinical Teaching (the approach taught at MGSE) was introduced as having “the potential to revolutionise our classrooms” but again I was – along with many teachers I’d expect – left underwhelmed. The core principles of Clinical Teaching as explained by Hattie are:

  • Diagnose: Ascertain the areas in which kids need to improve, learn etc.
  • Intervene: Deploy strategies, questions and activities that address these areas.
  • Evaluate: Ascertain whether or not kids have improved their knowledge, skills or understand.

In other words… teach.

Look, I know I’m being a little flippant here, and I do know that this might seem revolutionary to some… but seriously, I recommend you read pretty much anything by Dylan Wiliam, because formative assessment has been addressing this stuff for yonks.

AS FOR THE OTHER STUFF…

An outdoor ed camp, a class for disengaged boys, a kid who left under a cloud of drug use, another who left after Year 10, kids getting into strife for fighting and (alleged but ultimately unproven) theft, a school production of Aladdin, a formal ball, a kid who didn’t get into medicine, stressed out G&T kids, a girl arguing with her mother, school captain elections and a deputy who was a little skeptical of consultants… are the ingredients of a typical day in a typical school and I’m unsure as to what lessons they taught us, other than apparently after going on a four-day hike, some kids who hated school and had previously all failed a maths test all of a sudden aced it on their return… Again, not underestimating the impact of the outdoor education or the commitment of the staff, but come on… the superficial nature in which it was covered left me cold.

I was left wondering if police officers are equally underwhelmed after watching an episode of RBT? Or what about doctors and nurses after watching Trauma: Life in the ER?

But then again RBT & Trauma aren’t broadcast on the premise that the policing or health professions are failing to such an extent that they require such TV shows to “serve as a lesson.”

Now to be fair, the series finished by highlighting the gains the school has made which are impressive and for which everyone’s efforts should be applauded, and it’s worth noting that this has been an 8 year journey of which MGSE were a part for the past 12 months. Clearly they’ve made great gains but there are still some questions that remain particularly given the premise upon which the series was set.

Screen Shot 2016-06-22 at 6.16.27 pm

QUESTIONS THAT REMAIN

  1. When Hattie says it’s not the school that make a difference, but the teachers, what does that really mean?
  2. When Hattie says, “Class size doesn’t matter,” do you think he should clarify that by finishing that statement with, “if we don’t change our practice.” As you may be aware I’ve covered this in the past in my blog posts and in an interview I had with John Hattie himself. 
  3. Aren’t the things advocated for in Clinical Teaching (more student interaction, better feedback etc) impacted by class size?
  4. What was the impact on staff & student engagement of having cameras in the school?
  5. How many adults (teachers, consultants, camera & sound operators) were in each classroom and what impact would this have had?
  6. Given the series website states, “By applying simple low cost ideas in the classroom Kambrya undergoes a dramatic transformation,” how much would it cost your school to do something similar with MGSE? Having looked at the MGSE’s School Network page, a quick calculation would put it at around $50-60K
  7. How did the airing of students struggling with drugs and family pressures serve as a lesson for schools around Australia?
  8. What have you learned (with regard to pedagogy, wellbeing, classroom management, professional growth, enhancing student outcomes) as a result of watching #RevolutionSchool?
Categories
Chatting with John Hattie Education

Chatting with John Hattie – Pt. 3

This is the third in a series of posts based around an interview I had with John Hattie earlier this year.

Whilst many education conferences around the world issue a call to arms – of sorts – to embrace 21st Century Skills, it’s worth pointing out that the cohort of kids that started Kindy in the year 2000 are now – for whom those the system worked – into their second year of an undergraduate course.

Bit late for them.

More than that though, the phrase – to me at least – seems divisive.

SugataWe all know that the Y2K bug came to nothing, but did something happen at 11:59pm on Friday 31st December 1999 that meant all that went before in classrooms was now obsolete?

Certainly some high profile thinkers have come to this opinion.

2013 TED Talk Prize winner Sugata Mitra actually states:

“Schools as we know them are obsolete.”

Now of course I’m not saying that schools shouldn’t be moving with the times. Rather, my problem is the way in which the need to move with the times has been communicated to teachers. Talk around 21st Century Skills is vague; it means different things to different people.

Having previously discussed Direct Instruction v Inquiry Based Learning with John Hattie, our conversation naturally turned to the notion of 21st Century Skills.

“I have a problem with this notion of twenty first century skills,”

says Hattie, “I’m just finishing a synthesis on learning strategies, it’s not as big [as others he’s done] there’s only about 15 – 20 million kids in the sample, and one of the things that I’ve learnt from the learning strategies, and a lot of them include the 21st Century skill strategies is that there’s a dirty secret.”

And the dirty secret is that not only does Hattie believe that these strategies are very simplistic, but he is skeptical of classes that focus solely on critical thinking or creativity – something I’m seeing some (but not many) schools adopt.

“These strategies only work within the subject area,”

Hattie says, “They’re not generic. So one of my problems with the 21st Century skills is schools are running critical thinking and creativity classes as if those things can generically cross over. [But] you can think that way in Maths but not necessarily in physics or in English.”

Of course most schools who embrace 21st Century Skills are seemingly embedding them in their teaching of the curriculum. But then this leads us back to the Content v Skills debate and the teaching style most appropriate.

“You’re right, it polarises the debate in terms of strategies versus content. I bet in every school that you go into, you look at the assessments – and I do that because that’s what kids look at in terms of what teacher’s values – you’re still dominated by content.”

It’s a fair point, consider how much your school focuses on kids ability to digest content compared to how much it genuinely assesses critical or creative thinking.

I wonder if this in some way might explain why Hattie can’t find evidence of Inquiry Based Learning, or 21st Century Skills having much of an impact.

In many cases, maybe what we say we value isn’t what we end up assessing?

Hattie’s main worry is that some schools are over privileging 21st Century Skills, because what he calls “surface learning” is – for the most part – over-privileged by many.

His argument is we need a balance of surface to deep transfer. He says,

“We’ve over-obsessed on content.”

But in response to this comes a “generic notion [of 21st Century Learning], and my terror is that schools will be teaching 21st Century Skills. I don’t think you can do that. I don’t think it’s a viable way to go. New Zealand went down that line of enquiry thinking schools and schools were very proud to say ‘we’re deep thinkers, we’re enquiry thinkers’ – it didn’t work.”

He then went on to discuss what had worked in New Zealand, which I found interesting…

Hattie says, “There needs to be a constructive alignment particularly in high schools between what we say we’re going to do and what we actually assess.”

One of the reasons New Zealand have significantly improved their retention rate is because they changed the upper high school assessment system.

“We got away from this notion that you can only be excellent in physics and chemistry and we said you can be excellent in many things,” he says, “And we showed that even kids who wanted to be water polo coaches or baristas could be excellent at those things and could learn skills right till the end of high school.”

Sounds good to me I thought… how did that go down?

“[Because] it was privileged along with physics and everything else it made many of those teachers very upset. It’s been a traumatic transformation but it’s been a very positive one; it hasn’t been without its problems,” he says.

“Its critics are still out there claiming that we’ve got rid of Shakespeare and calculus and chemistry. And for some kids we have, as we should have done.”

I wonder if that’s the key to a 21st Century education – having diversity and a flexibility in the curriculum that is equally valued based on the needs of students and communities?

Categories
Chatting with John Hattie Education Leadership

Chatting with John Hattie – Pt. 1

Visible LearningLate last year I wrote a blog post called, Is John Talking Through His Hattie?

The subject of the post was a series of posts and articles I had come across that called into question the validity of the statistical analysis that is the backbone of John Hattie’s Visible Learning work.

To save you the time of reading that post, the key point raised was:

John Hattie had admitted half the statistics in Visible Learning are wrong.

Within a couple of days I had a reply on the post from John Hattie himself suggesting that I was “Retweeting a lie.”

I decided that this might be an opportunity to directly engage with John regarding the post and the issues raised. He agreed, and what followed was a great 60min chat about all things education, despite a few technical issues that this song brilliant captures…

The interview will actually be the basis for a series of posts, with this being the first.

So… to that blog post.

Did Hattie ever say half the statistics in Visible Learning are wrong?

Well, not according to him, he says,

“The message is that someone tweeted from a conference in London a few months ago, that apparently I said ‘half the statistics are wrong’ and I never said that.”

But that’s not to say there weren’t issues with the statistics. The main argument centres around something called the Common Language Effect Size, which allows you to compare effect sizes in simple terms. For example, Strategy A has a 90% chance of working better than Strategy B.

And it is here where Hattie admits there were some errors.

He says, “Unfortunately at the very last minute I put the wrong column up on the – as part of the confidence levels rather than the actual common language exercise.” It wasn’t until about three years later when some students in Norway alerted Hattie to the errors that the correct data was added to reprints of Visible Learning.

But as and when the next edition is published Hattie says, “I’d take all that stuff out because it didn’t work [anyway].”

Hattie says that whilst “there are some minor errors” in Visible Learning – errors that he says he and others have picked up – these are corrected quickly. And though he is disappointed that such errors occur he is confident that it doesn’t change the overall message of the book, as he says, “Not one iota.”

No doubt debate will continue in educational circles and the blogosphere, but Hattie says,

“The nature of academia is that you live off critique and so I thrive off that and so that’s why I’m happy to talk to you and anyone [but] I’m not going to tell you right now everything’s perfect.”

Hattie says he’s been doing this work and publishing Visible Learning work since about 1989 and what fascinates him is, that in the twenty or thirty years since he started working on it, “No one – not a single person – has critiqued the idea and come up with an alternative explanation for the data.”

In this series of Chatting with John Hattie posts I’ll share with you – amongst other things – our discussions around the use (and mis-use) of Visible Learning, 21st Century Learning, AITSL, Teacher Quality, and advice for new teachers.

Categories
Chatting with John Hattie Education Leadership

Is John talking through his Hattie?

Research QuoteJohn Hattie, the author of the much quoted Visible Learning was recently appointed by the Federal Government to the Chairmanship of the Australian Institute for Teaching and School Leadership. 

In today’s Australian Newspaper, an article by Jennifer Buckingham from the Centre for Independent Studies said,

Last week, Hattie indicated things were going to change in teacher education under his watch. He talked of “tougher”, “harder”, “standards”, “outcomes” and “impacts”. This is a harbinger of his approach, grounded in the requirement for evidence that reforms are working.

It went on:

Hattie would like the accreditation and evaluation standards for teaching degrees to be much tougher. If this results in some courses being scrapped because they don’t meet the standard for the academic rigour of courses or for graduate teachers’ impact on students, then so be it.

And who better to have at the helm than the guru of effect size himself, John Hattie?

Well it would appear that just as Hattie would like teacher education to be subject to tougher standards, there are some who suggest his research should be subject to a more rigorous analysis.

Despite having been challenged as early as 2011 in the British Journal of Educational Studies, Hattie’s work has been seen as the guiding light in educational reform, but of late it has come under scrutiny, not least because of this blog post and it’s subsequent follow up which asserts that half the statistics in Visible Learning are wrong.

I’ll let you follow those threads if they interest you.

What worries me is that a great deal of educators, regional leaders, keynote speakers and politicians quote Hattie to further assert their position, particularly if that position is at odds with common sense – the class size debate for example.

Don’t get me wrong, I do believe that educational research is vital. But it is equally vital that we view it through an appropriate lens. As Dylan Wiliam says*,

Educational research can only tell us what was, not what might be. Moreover, in education, “What works?” is rarely the right question, because everything works somewhere, and nothing works everywhere, which is why in education, the right question is, “Under what conditions does this work?”

Besides I’ve always been skeptical that anyone could determine a numerical value for the impact any given teacher might have with any given student in any given classroom.

*If you want to access the quote from Wiliam you can get his PPT slides from a ResearchED event here.

[EDIT] Through the magic of Twitter, it was made apparent to me by Greg Thompson (seriously… follow him) that some of the issues raised here were first addressed by Snook et al in 2009 in NZ Ed Studies Journal.

[EDIT 2 – 19th Nov] John Hattie has agreed to an interview as per the comment thread below. I’ll let you know how things progress.