Goal Setting is Often an Act of Desperation: Part 5

In this episode, John Dues and Andrew Stotz apply lessons five through seven of the 10 Key Lessons for implementing Deming in classrooms. They continue using Jessica's fourth-grade science class as an example to illustrate the concepts in action.

TRANSCRIPT

0:00:02.2 Andrew Stotz: My name is Andrew Stotz and I'll be your host as we continue our journey into the teachings of Dr. W. Edwards Deming. Today I'm continuing my discussion with John Dues, who is part of the new generation of educators striving to apply Dr. Deming's principles to unleash student joy in learning. This is episode five about goal setting through a Deming lens. John, take it away.

0:00:23.2 John Dues: Yeah, it's good to be back, Andrew. Yeah, like you said, for the past few episodes we've been talking about organizational goal setting. We covered four healthy conditions, or four conditions of healthy goal setting and 10 key lessons for data analysis. And then what we turn to in the last episode is looking at an applied example of the 10 key lessons for data analysis and in action. And, if you remember from last time we were looking at this improvement project from Jessica Cutler, she's a fourth grade science teacher, and she did the improvement fellowship here at United Schools Network, where she learned the tools, the techniques, the philosophies, the processes behind the Deming theory, continual improvement, that type of thing. And in... And in Jessica's specific case, in her fourth grade science class, what she was settled on that she was gonna improve was, the joy in learning of her students. And we looked at lessons one through four through the eyes or through the lens of her project. And today we're gonna look at lessons five through seven. So basically the next, uh, the next three lessons of those 10 key lessons.

0:01:34.8 AS: I can't wait. Let's do it.

0:01:37.3 JD: Let's do it. So lesson number five was: show enough data in your baseline to illustrate the previous level of variation. Right. So the basic idea with this particular lesson is that, you know, let's say we're trying to improve something. We have a data point or maybe a couple data points. We wanna get to a point where we're starting to understand how this particular concept works. In this case, what we're looking at is joy in learning. And there's some different rules for how many data points you should, should have in a typical base baseline. But, you know, a pretty good rule of thumb is, you know, if you can get 12 to 15, that's... That's pretty solid. You can start working with fewer data points in real life. And even if you just have five or six values, that's gonna give you more understanding than just, you know, a single data point, which is often what we're... What we're working with.

0:02:35.6 AS: In, other words, even if you have less data, you can say that this gives some guidance.

0:02:40.9 JD: Yeah.

0:02:41.1 AS: And then you know that the reliability of that may be a little bit less, but it gives you a way... A place to start.

0:02:46.9 JD: A place to start. You're gonna learn more over time, but at least even five or six data points is more than what I typically seen in the typical, let's say, chart where it has last month and this month, right? So even five or six points is a lot more than that. You know, what's... What's typical? So I can kind of show you, I'll share my screen here and we'll take a look at, Jessica's initial run chart. You see that right?

0:03:19.3 AS: We can see it.

0:03:21.2 JD: Awesome.

0:03:22.3 AS: You wanna put it in slideshow? Can we see that? Yeah, there you go.

0:03:24.9 JD: Yeah, I'll do that.

0:03:25.4 AS: Perfect.

0:03:26.3 JD: That works better. So, you know, again, what we're trying to do is show enough data in the baseline to understand what happened prior to whenever we started this improvement effort. And I think I've shared this quote before, but I really love this one from Dr. Donald Berwick, he said "plotting measurements over time turns out in my view to be one of the most powerful things we have for systemic learning." So what... That's what this is all about really, is sort of taking that lesson to heart. So, so you can look at Jessica's run chart for "joy in science." So just to sort of orient you to the chart. We have dates along the bottom. So she started collecting this data on January 4th, and this is for about the first 10 days of data she has collected. So she's collected this data between January 4th and January 24th. So, you know, a few times a week she's giving a survey. You'll remember where she's actually asking your kids, how joyful was this science lesson?

0:04:24.4 JD: Mm-hmm.

0:04:27.2 JD: And so this is a run chart 'cause it's just the data with the median running through the middle, that green line there, the data is the blue lines connected by, or sorry, the blue dots connected by the points and the y axis there along the left is the joy in learning percentage. So out of a hundred percent, sort of what are kids saying? How are kids sort of evaluating each of these science lessons? So we've got 10 data points so far, which is a pretty good start. So it's starting to give Jessica and her science class a decent understanding about, you know, when we, you know, define joy in science and then we start to collect this data, we really don't have any idea what that's gonna look like in practice. But now that she started plotting this data over time, we have a much better sense of what the kids think of the science lessons basically. So on the very first day...

0:05:25.4 AS: And what is the... What is the median amount just for the listeners out there that don't see it? What would be the... Is that 78%?

0:05:33.8 JD: Yeah, about 78%. So that very first day was 77%. The second day was about 68%. And then you sort of see it bounce around that median over the course of that, those 10 days. So some of the points are below the median, some of the points are above the median.

0:05:50.4 AS: And the highest point above is about 83, it looks like roughly around that.

0:05:54.4 JD: Yeah. Around 82, 83%. And one technical point is at the point that it's a run chart we don't have the process limits, those red lines that we've been taking a look at and with a run chart and, you know, fewer data points, we only have 10. It's fairly typical to use the median, just so you know, you can kind of better control for any outlier data points which we really don't have any outliers in this particular case but that's just sort of a technical point. So, yeah, I mean, I think, you know what you start to see, you start to get a sense of what this data looks like, you know, and you're gonna keep collecting this data over an additional time period, right? And she hasn't at this point introduced any interventions or any changes. Right now they're just learning about this joy in learning system, really. Right.

0:06:51.8 JD: And so, you know, as she's thinking about this, this really brings us to... To lesson six, which is, you know, what's the goal of data analysis? And this is true in schools and it's true anywhere. We're not just gonna look at the past results, but we're also gonna, you know, probably more importantly, look to the future and hopefully sort of be able to predict what's gonna happen in the future. And, you know, whatever concept that we're looking at. And so as we continue to gather additional data, we can then turn that run chart from those initial 10 points into a process behavior chart. Right. You know, that's a, sort of a, you know, it's the run chart on steroids because not only can we see the variation, which you can see in the run chart, but now because we've added more data, we've added the upper and lower natural process limit, we can also start to characterize the type of variation that we see in that data.

0:08:00.1 AS: So for the listeners, listeners out there, John just switched to a new chart which is just an extension of the prior chart carrying it out for a few more weeks, it looks like, of daily data. And then he's added in a lower and upper natural process limit.

0:08:18.9 JD: Yeah. So we're still, we're still plotting the data for joy in science. So the data is still the blue dots connected by the blue lines now because we have 24 or so data points, the green line, the central line is the average of that data running through the data. And we have enough data to add the upper and lower natural process limit. And so right now we can start to determine do we only have natural variation, those everyday ups and downs, that common cause variation, or do we have some type of exceptional or special cause variation that's outside of what would be expected in this particular system. We can start making...

0:09:00.7 AS: Can you...

0:09:02.2 JD: Go ahead.

0:09:02.8 AS: I was gonna... I was gonna ask you if you can just explain how you calculated the upper and lower natural process limits just so people can understand. Is it max and min or is it standard deviation or what is that?

0:09:18.3 JD: Yeah, basically what's happening is that, so we've plotted the data and then we use that data, we calculate the average, and then we also calculate what the moving range, is what it's called. So we just look at each successive data point and the difference between those two points. And basically there's a formula that you use for the upper and lower natural process limits that takes all of those things into account. So it's not standard deviation, but it's instead using the moving, moving range between each successive data point.

0:09:52.9 AS: In other words, the data that's on this chart will always fall within the natural upper and lower. In other words it's... Or is, will data points fall outside of that?

0:10:05.7 JD: Well, it depends