When Measurements Warp Outcomes

H ave you noticed the proliferation of measurements? It seems that we have a metric for everything. From intelligence to weight, performance to objectives, and beyond, we like to quantify every possible aspect of our world. There is benefit in these measurements. We are able to set benchmarks against which we can measure progress or regress. Metrics enable us to evaluate people, organizations, policies, decisions, and effects. Much good comes from our ability to measure.

However, there is a downside to our addiction to measurement. It tends to warp outcomes.

One of the most blatant examples of this occurs in our schools.  We have become obsessed with standardized tests. PSAT, SAT, Core 40, ACT, CAT, and so on – we have a test for everything. The goal for our testing is noble: we want to hold our schools accountable to producing results in our children and make sure that our children are in fact learning all that they should.  We want to make sure that all students receive a certain minimum of education during their school years so they are able to advance, prosper, and become good citizens. There are numerous risks and problems with this approach:

1. Someone, somewhere becomes the arbiter of what should be known. This may happen by committee, board, or fiat but someone has to decide what is a minimally acceptable level of knowledge for a student completing their educational process. There is inherent risk in this approach: who decides what is worth knowing and how it should be known? Now it becomes political.
2. Next, we have to decide how to measure that minimum baseline of knowledge. A standardized test is created and we’re in business. Now we ask: tell us again why we’re testing students? Oh, right, we need to make sure they have a certain amount of knowledge so they can become productive citizens. Of course, does the test verify that they have that knowledge or does it affirm those who are best able to memorize that knowledge for a test? How is the test to be administered? In the most efficient and expeditious way of course – a sit down, hours long, multiple choice extravaganza. In the end, we find that this approach doesn’t work well for all students but rationalize that it is the best option available.
3. In anticipation of these tests, schools adjust their curriculum to prepare for them. Now we enter the “death march” to knowledge accumulation: we have so many days to drill X amount of information into X number of students before they take the next test. For some reason, this process reminds me of an assembly line moving students down a hall full of pieces and parts that have to be bolted onto their brain. Same size, same volume, same tools, ad nauseam.
4. In the process, we eliminate superfluous items from our educational curriculum: shop, home economics and anything that is not measured on our standardized test. Why? Because we need to put everyone through college preparatory classes right? What does college want? Oh, yeah, a certain score on a certain standardized test to show that you are ready to begin taking their tests.
5. What happens along the way? We reward those effective at memorization. We reward those whose learning style fits the standard. We reward those who can sit still and be compliant for long periods of time in the classroom and for lengthy standardized tests. We reward those who aspire to college or higher learning of a form that fits our measurement philosophy.
6. The ultimate result is a skewed focus in the classroom. The elimination of “non-critical” elements of the school academic experience: recess and non-college preparatory distractions. A focus on memorization rather than comprehension. A generation that cannot write in cursive and that is accustomed to multiple choice tests graded by a computer. Where do we learn to think? Where do we learn to reason? Where do we learn to cope when we don’t fit in the system? Where do we turn when the system fails to help us be all we can be? Our good intentions end up disenfranchising groups of students who don’t fit well within our measurements and then handicap the remaining students who fared well in the system only to discover that the same memorization-testing model doesn’t actually apply to the real world at all. Prepping for standardized tests is not the same as finding a job and making a living.

Perhaps the school example is too big, too complex. How about healthcare? Small and manageable, right? We’ve moved, and continue to move, toward a system of measurements and rewards based on outcomes. I’m all for getting good outcomes from our healthcare. As consumers of healthcare, we should be able to tell which practitioners are more skilled, more successful in helping their patients optimize their health. However, the same warping effect we see in our education system is brought to bear in our healthcare system. Someone, somewhere sets the measurements – we assume with good intentions. From there, we enter the world of unintended consequences.

Alas, in healthcare, the model is “follow the money” and metrics give us the path.  With metrics like “readmission rates” and “patient satisfaction” used to reward and punish providers we create a twisted system that obsesses on gaming the metrics rather than truly making patients better. Of course individual health care professionals care about their patients – the problem lies with the system. Want to decrease readmission rates? Make sure patients don’t return to the hospital by catching them in another care net. Of course, the metric doesn’t factor the readmissions that occur because patients aren’t taking ownership of their own health. Does patient satisfaction actually correlate to better health outcomes? That doesn’t matter. It is a metric so we’ve got to coddle patients who refuse to follow our instructions and take care of themselves properly. We adjust our approach to manage to the metric; even if the adjustment isn’t really solving the underlying problem.

A recent article in the New York Times reminded me of the truly nefarious nature of measurement, especially when trying to quantify human behavior. The author was supporting a premise that people basically chose to believe what they want even when confronted with facts to the contrary. She referenced several studies (Stanford University) that described test subjects being given false information that brought them to certain conclusions. The researchers then went back into the groups with factual information and found that most of the subjects did not change their original conclusions. I must confess to not being a professional researcher but it seems to me that if I were in a study that gave me one set of data so I would draw a conclusion, then gave me new information saying that they just wanted to see how I reacted to the first set of data, I might remain a bit incredulous as to the reliability of the new information or pretty much anything they were telling me.

My conclusion? What we ask, how we ask it, and the receiver’s trust of the one asking are all pretty substantial variables when trying to measure a person’s response. The measurement itself actually affects the outcome.

Start looking around and you’ll see it everywhere. A few more examples:

• Performance evaluations – measurement affects behavior. That is why we keep changing our KPI’s – we discover unintended consequences when people really work to hit their bonuses. Sometimes the desired results follow, sometimes they don’t.
• Comp plans – have you ever seen a sales person game a comp plan? This is often quite artful.
• Kids – clean your room and I’ll pay you an allowance. In this situation, you will discover the difference between your measurement of “clean” and your child’s.
• Polls – do we drive the poll results or do the questions drive our responses?

Should we stop measuring things? That is probably not the answer. I realize that there are correct and incorrect ways to measure. However, we should be aware of the dangers of quantifying everything in our lives. Human beings are naturally subjective. We are prone to hyperbole and conjecture. We remember things differently and, given enough time, completely differently than they may have occurred. Facts do exist. Often, we can objectively determine the truth but in many cases we’re asking the wrong question or we’re asking the right question in the wrong way.

The bottom line is that when we measure something, we place a value on it. We are saying: “This is important.” At that point, things change because people respond to it. They recognize that we are keeping score. There are times when we get the desired change and the measure is successful. Often, we get other changes that we did not anticipate. Our job is to be aware of the benefits and pitfalls of what we measure and maintain a certain skepticism when presented with the statistics, measurements, or conclusions. Most importantly, we need to look for the warping effects that will inevitably occur when money is attached to our measurements.