Your metrics are not your goals
I enjoyed reading Haider Rafique’s blog SLAs are for suckers: Why ITSM orgs should be allowed to be more human, so I tweeted a quote, with a link to the blog.
This is an example of Goodhart’s law in action. Goodhart was an economist and his law can be summarised in a few words. "When a measure becomes a target, it ceases to be a good measure." This is because when people know what is being measured, they adapt their behaviour accordingly, and the results are unlikely to be what was originally intended. And it’s a law applies that applies just as well to IT service management as it does to economics.
I was surprised at the amount of discussion my tweet generated. It quickly became clear that some people thought that I was advocating anarchy. How could I dare suggest not using service level agreements (SLAs) to document services! Madness!
And, of course, they have a point. IT services are almost always documented in SLAs, precisely because SLAs can and should be a useful tool for helping to frame a discussion between service providers and their customers. With SLAs in place the customer knows what service they’re going to get, and the service provider can plan to put in place the resources needed to meet customer expectations.
Unfortunately, that’s often not the way it turns out in practice. Many SLAs don’t actually help customers understand what to expect in terms of service delivery, and they don’t help providers with their planning either. In fact, the worst SLA’s I have seen set out a list of numbers that relate to technical targets that customers don’t really understand, and that are, in any case, not relevant to their business needs; and then the service provider works to achieve these numbers, regardless of the impact on their customers.
I had a personal experience of this some years ago, in a very different context. I had to take one of my children for treatment at a dental hospital. We researched local hospitals and selected one that had low waiting times for appointments, as well as good outcomes. Our doctor wrote to the hospital to request an appointment. Two weeks later I phoned the hospital to ask why we hadn’t heard and they said that the letter hadn’t arrived yet, so I spoke to the doctor who sent a second letter to the hospital. When the hospital told me that this second letter hadn’t arrived after a further two weeks, I got the doctor to write a third letter, which we hand delivered to the hospital. We talked to the receptionist at the hospital, who showed us a large box of unopened letters, and explained that this was the box of “letters that haven’t arrived yet”.
The hospital had a metric of “Number of days from receiving doctor’s letter till patient has an appointment”, but letters didn’t count as having arrived until they were opened and date stamped. So the hospital achieved its target by keeping letters in a box until they had the capacity to process them.
This may seem like an extreme case, but I have seen similar things with IT metrics too. One outsource contract included metrics:
- 98% of P2 issues will be resolved within 8 hours
- 95% of P3 issues will be resolved within 24 hours
There were significant penalties for failing to meet these targets, so the service provider worked very hard to meet them. One month a senior manager sent an email to service desk staff saying that the target for P2 (highly significant issues) had been breached that month, and that the P3 target (somewhat less important issues) was also at risk; to avoid breaching both targets staff should ignore any P2 issues and focus on the P3 ones. You can imagine how the customer felt when someone showed them this email!
We all know that we need to have SMART KPIs in our agreements, so we focus on making sure they are specific, measurable, achievable, relevant and time-bound; but the thing we forget is that KPIs influence behaviour, and, as Goodhart’s Law tells us, they may drive behaviour that we really don’t want. Each of my examples happened because people treated the things that were being measured as if they were the targets, and forgot to consider the fact that they were performance indicators. To avoid this, we need to remember that the measurement IS NOT the target, it is just an indicator.
I would be the last person to discourage measurement per se. Measurement is essential. It tells us about trends,it helps us to identify improvement opportunities, and it provides a framework for discussion with customers. But we must never forget that achieving the KPI is not an end in itself, and that our work is not done until we meet our customers’ expectations.
So it’s fine to use KPIs for planning, understanding, and discussion, just remember to maintain a focus on your real goals and don’t confuse them with your KPIs!