Making Data Meaningful: Deming in Schools Case Study with John Dues (Part 3‪)‬ In Their Own Words

Education is often touted as data- or evidence-driven. But in this discussion, John Dues contends that educational data is often fiction, given how easy it is to distort, both via the inputs and outputs and through manipulation.
0:00:02.6 Andrew Stotz: My name is Andrew Stotz and I'll be your host as we continue our journey into the teachings of Dr. W. Edwards Deming. Today, I'm continuing my discussion with John Dues, who is part of the new generation of the educators striving to apply Dr. Deming's principles to unleash student joy in learning. The topic for today is Data is Meaningless Without Context. John, take it away.
 
0:00:28.3 John Dues: Yeah, thanks for having me back, Andrew. I'm thinking a lot about educational data, and I think about how it's often presented, and I think so often, what we're actually doing with our educational data is what I call writing fiction, which is taking a lot of liberties with the data that comes into our system, whether it's state testing data or some other type of data that gets reported out to the public, and we often sort of manipulate that data or distort that data in a way that paints our organization or our school system or our state in a positive light, and I think we do that sometimes at the detriment of actually working to improve those organizations or those systems because we spend so much time trying to paint this positive picture instead of just putting the data out there.
 
0:01:26.3 AS: And it's interesting you talked... One of the interesting things about what you're saying is that it could be accurate and good data, but it's just the context or the structure of how it's presented makes it meaningless.
 
0:01:39.9 JD: Yeah, we try so hard to sort of paint it in this positive light to make it look like we're doing a good job. Everybody wants to do a good job, but I think we often do that at the detriment of our systems.
 
0:01:54.7 AS: One of the things that made me think about it, in the financial world, we have a code of ethics, and that is basically that... Particularly for CFA charter holders, financial analysts, that you have to present a complete picture of your performance. So if you have 10 customers that you're managing their money and one of them, you really bombed out and you decide you're gonna do the average of the nine that you did well on and then go out to your clients and say, "This is my performance," that's a very... You have an obligation to accurately represent your performance. And when I think about it in all the charts and graphs that people are making in education all around, I would say that most people probably are just, I would call it CYA, cover your ass type of charts [laughter] of, "How do we make this look good?"
 
0:02:43.9 JD: Yeah, I think... I read somewhere that there's sort of three ways you can respond to your data. You can actually work to improve the system, which would be a positive, and then the other two ways are two forms of a negative, one is you could distort the system itself, or you could actually distort the data. And a lot of times, there's not sort of a nefarious motivation underneath that distortion, but there's sort of, again, this desire to paint your organization or your system in this positive light. So sometimes they're straight up unethical behavior or cheating, but most of the time, that's not what I'm seeing and that's actually not what I'm talking about here today. It's more of this sort of taking liberties, writing fiction. "Okay, we declined from two years ago, but it's up from last year." Those types of sort of distortions of the data that I think are fairly common in education sector, probably all sectors too, so.
 
0:03:49.9 JD: I think... Maybe I'll share my screen for the folks that have video and I'll talk through it for the listeners that don't have video, but one of the things I often think of and focus on is state testing data, because so many people are looking at that data all