Goal Setting Is Often An Act of Desperation: Part 2

Do you struggle to meet your goals or targets? Find out how you can change your thinking about goals and your process for setting them so you can keep moving forward. In this episode, John Dues and host Andrew Stotz discuss the first five of John's 10 Key Lessons for Data Analysis.

TRANSCRIPT

0:00:03.0 Andrew Stotz: My name is Andrew Stotz and I'll be your host as we continue our journey into the teachings of Dr. W. Edwards Deming. Today I'm continuing my discussion with John Dues, who is part of the new generation of educators striving to apply Dr. Deming's principles to unleash student joy in learning. This is episode two of four in a mini-series on why goal setting is often an act of desperation. John, take it away.

0:00:32.3 John Dues: Hey, Andrew, it's good to be back. Yeah, in that last episode, that first episode in this mini-series, we talked about why goal setting is often an act of desperation and I basically proposed these four conditions that organizations should understand prior to setting a goal. So it's not the goals in and of themselves that are bad, but it's with this important understanding that's often lacking. So those four things that organizations should understand, one, what's the capability of a system under study? So that's the first thing, how capable is the system or the process? The second thing is what's the variation within that system or process under study? So that's the second thing we talked about last time. The third thing is understanding if that system or process is stable. And then the fourth thing was, if we know all of those things, by what method are we going to approach improvement after we set the goal, basically? So you gotta have those four things, understanding the capability of the system, the variation of the system, the stability of the system, and then by what method, prior to setting a goal. And so I think I've mentioned this before, but absent of an understanding of those conditions, what I see is goals that are, what I call it, arbitrary and capricious.

0:01:48.8 JD: That's a legal characterization. You look that up in the law dictionary. And it basically says that an "arbitrary and capricious law is willful and unreasonable action without consideration or in disregard of facts or law." So I'm just now taking that same characterization from a legal world and applying it to educational organizations and accountability systems, and I just switched it to "a willful and unreasonable goal without consideration or in disregard of system capability, variability, and/or stability." And we see these all over the place for education organizations, for schools, school districts, teachers, that type of thing.

0:02:31.6 JD: And so what I tried to do in the book and tried to do here in my work in Columbus is develop some sort of countermeasures to that type of goal setting and develop the 10 key lessons for data analysis. An antidote to the arbitrary and capricious goals seen throughout our sector. And this process behavior chart tool, looking at data in that format is central to these lessons. So what I thought we would do in this episode and the next is outline those 10 key lessons. So five today and then do another five in the next episode. And in the fourth episode of the series, what we would do is then apply those examples to a real life improvement project from one of our schools. It's helpful, I think too, to sort of, to understand the origin of the key lessons. So there's the lessons that I'll outline are really derived from three primary sources.

0:03:36.0 JD: So the first two come from Dr. Donald Wheeler, who I've mentioned on here before, a lot of Deming folks will, of course, have heard of Dr. Wheeler, who's a statistician in Tennessee, a colleague of Dr. Deming when Dr. Deming was alive and then has carried on that work to this day. The two books, two really great books that he wrote, one is called Understanding Variation, a thin little book, a good primer, a good place to start. And then he's got a thicker textbook called Making Sense of Data, where you get in really into the technical side of using process behavior charts. So I'd highly recommend those. And then the third resource is a book from a gentleman, an engineer named Mark Graban called Measures of Success. And I really like his book because he has applied it, the process behavior chart methodology, to his work and he's really done it in a very contemporary way. So he's got some really nice color-coded charts in the Measures of Success book and I think they're really easy to understand with modern examples, like traffic on my website, for example, in a process behavior chart, really easy to understand modern example. But all three of the books, all three of the resources are built on the foundation of Dr. Deming's work. They're, you know, Graban and Wheeler are fairly similar and I think Graban would say he's a student of Wheeler.

0:05:00.4 JD: He learned of this mindset, this approach to data analysis by finding a Donald Wheeler book on his own dad's bookshelf when he was in college and starting down that path as a young engineer to study this stuff. And basically what I've done is take the information from those three resources and make some modifications so they can be understood by educators, basically. I think it's also worth noting again that process behavior chart methodology is right in the center of this, really for three reasons. One, when you plot your data that way, you can start to understand messages in your data, I think that's really important. Second, you can then start to differentiate between special and common causes, special and common causes, translate that into regular language. I can translate between something that I should pay attention to and something that's not significant basically in my data. And then in so doing, I know the difference between when I'm reacting to noise versus when I'm reacting to signals in my data, so I think that's really important. So the process behavior chart is at the center of all this. So we'll go through five of these lessons, one by one, I'll outline the lesson and then give a little context for why I think that particular lesson is important.

0:06:25.4 AS: That sounds like a plan. So capability, variation, stability and method. You've talked about Donald Wheeler, excellent book on Understanding Variation, that's the one I've seen. And of course, Mark Graban's book, Measures of Success, very well rated on Amazon and a podcaster himself, too.

0:06:49.6 JD: Yeah. And if I was a person studying this and wanting to get into process behavior charts and really knowing how to look at data the right way, I would read Understanding Variation first because it's a good primer, but it's fairly easy to understand. And then I would read Measures of Success 'cause it's got those practical applications now that I have a little bit of a baseline, and then if I wanna go deep into the technical stuff, the Making Sense of Data, that's the textbook that drives everything home. Yeah. So we'll dive into the lesson then.

0:07:19.5 AS: Let's do it.

0:07:20.0 JD: Yeah. Okay. So the first lesson, and I've talked about this in various episodes before, but lesson one, the very first lesson is, "data have no meaning apart from their context." So this seems commonsensical, but I see this all the time where these things aren't taken care of. And what I'm talking about is answering some basic questions. So for anyone looking at my data, they should be able to answer some basic questions, very simply, anybody that looks at my data. First thing is who collected the data? That should be apparent. How were the data collected? When were the data collected? Where were the data collected? And then what do these values represent? So oftentimes I see data either in a chart or in some type of visualization and almost none of those things are known from looking at the data, all important questions.

0:08:18.6 JD: The second question would be, well, that first set goes together. The second question is what's the operational definition of the concept being measured? So we have to be on the same page about what it is exactly being measured in this data that I've collected. I also wanna know how were the values of any computed data derived from the raw inputs? That's important. And then the last thing is, have there been any changes made over time that impact the data set? For example, perhaps the operational definition has changed over time for some reason. Maybe there's been a change in formula being used to compute the data.

0:09:05.4 JD: So an example would be, from my world, high school graduation rates. You know, 20 years ago there was one definition of how you calculated a high school graduation rate, now there's a different definition. So when you compare those two sets of data, you've gotta be careful because you're actually, you're actually working from different definitions and I think that happens all the time. More recently here in Ohio, what it means to be proficient on a state test, that definition changed about 10 years ago. And so if you look at test results from 2024 and try to compare them to 2014, you're really comparing apples and oranges 'cause there's two different definitions of proficiency, but no one remembers those things a decade later. So you have...

0:09:52.3 AS: And then a chart will be presented where the different methodologies are shown as one line that says...

0:10:00.8 JD: Yes.

0:10:00.8 AS: That no one's differentiated the fact that at this point it changed.

0:10:04.6 JD: Yeah, at this point it changed. So first lesson, data have no meaning apart from their context. Second lesson is we don't manage or control the data, the data is the voice of the process. What we control is the system and the processes from which the data come. There's a differ