In part 2 of this 4-part interview, Harvard GSE researcher Heather Hill tells us why to cancel your school’s data team meetings.
– Welcome to another PLtogether Lounge Talk. I’m Adam Geller, founder and CEO of Edthena, the video observation platform. Today we’re talking with Heather Hill. She is a researcher at the Harvard Graduate School of Education. And she’s also the co-author of a column in Ed Week that appears regularly with Susanna Loeb called “What Works, What Doesn’t” Heather, thanks so much for joining us.
– Yeah. Thanks Adam. It’s nice to be here.
– Well, let’s start with something that feels like Ripley would have put it in the believe it or not museum, which is, you’re ready to tell us. Teachers don’t need to have meetings where they pull out last week’s assessment and talk about that data. Tell us why?
– I will actually go even further and say they should not have those meetings where they pull out the data and talk about how kids are doing. So let me back up though, and I’ll tell you about what I did to sort of come to this conclusion. So, teachers studying student data has been around for quite a while when data’s started being available to schools and districts, a lot of people said, “oh, well teachers should study this data and then they’ll know what their kids don’t know, And they’ll be able to adjust instruction accordingly.” And a lot of academics went off and did pretty rigorous studies of programs that were designed to get teachers to study student assessment data. And what we found 10 studies that were conducted when I was doing this little review I found 10 studies that were conducted over the last 10 years or so. And they did 23 sort of statistical models where they tried to find an effect sort of 23 chances for one of these programs to have an impact across all of these 23, they found two positive impacts one negative impact and 20 zero impacts. So in my field that is considered to be a very “precisely estimated zero”. So like the positive and the negative pretty much balance each other out, and there’s a lot of zeros there. So across 10 different programs, 10 different shots at getting teachers to really be productive and like see this theory of action through like teachers know what kids don’t know they change their instruction, Kids learn more, 10 tries, zero impacts. And so that convinced me that we’re doing something wrong in schools that convinced me that it’s time to really rethink this practice and rethink the time that we that teachers spend on it because there’s an opportunity cost, teachers could be doing other things that may be more productive in terms of helping them meet their kids’ needs.
– So there’s a lot there. And if I’m a building leader and I’m convinced that teachers should be talking about data in this time and you know, okay, great. You have some studies, but was there a theme about what’s missing in those discussions that you could tell us about that that could help convince me that like, oh, okay. Yeah, talking about data isn’t the right way to spend our time.
– Yeah. So actually I supervised a couple of dissertations where people watched these data team meetings. I’ve been to data team meetings and I think two things tend to happen. So first I think there’s a kind of tendency in some social settings for people to say, you know with good intention, oh, like “Adam did not great on that assessment, the interim assessment last week or the assessment that we gave at the end of the unit but he was having a bad day or he was having a bad week.” And that resulted in the sort of performance that he produced on the assessment. So that may be true, but it doesn’t signal to the collective that anything has to change in terms of the teaching that’s going on. The second is so teachers then would say, well we need to remediate this problem. Adam has with division. And what I saw a lot of was I think because teachers are under a lot of time pressure, a lot of kind of quick fixes where somebody would say, oh like “Pinterest has this dinosaur division worksheet. That’s really gonna help Adam out. Why don’t you go download it?” And that I’m always a little worried about Pinterest worksheets, A and B, that is not a coherent sort of way to address whatever’s going on with Adam. It isn’t getting inside. Like, so what part of the procedure don’t you understand? Is it sort of like a basic conceptual misunderstanding? Is it something about place value and how that plays into division? So it’s not, it’s not really tuned up to what, you know this particular hypothetical child named Adam isn’t able to do.
– Well, it sounds like we’re getting out the red marker. We’re canceling the data team meetings but we could still spend that time in other ways maybe reflecting on practice and talking about what’s happening in our classrooms and other things. Right?
– That’s right. Well, I think that’s, that’s sort of two things about this. Like I can cancel, I, if I had a red marker, I would cancel teacher data team meetings, two things though. First, this is not to say like administrators shouldn’t look at student data because I think that’s an important part of sort of helping a school is to say where are our kids not doing okay. And what do we have to do to meet those needs? The second thing is the exception to this rule that I found these sort of largely no results studies was programs that are more prescriptive about how to help the kids who are missing key pieces of knowledge, right? So like Adam, you don’t understand this thing about long division here is here is a little bit of personalized instruction whether that be something that’s online or something that the teacher does that helps you, Adam sorry, Adam, the student then master whatever is missing. And so when you had that level of specifications so it gave teachers literally something to do with Adam that wasn’t like they had to invent from home cloth or figure out on their own. It was supplied to them. Those programs actually did show positive impacts. So two pieces of better news for it before we canceled the whole thing.
– Well, thank you Heather, for sharing the data about data team meetings and reviewing data in schools. And if you’re just finding this conversation on some corner of the internet or shared on social media join us @ pltogether.org to find the rest of my conversations with Heather as well as with many others. Heather, thank you so much for being part of PLtogether.
– Yeah. Thanks Adam. This was fun.
