First, we obviously would like to create metrics that lead to the right kind of behavior. To illustrate this, consider a call center's duration-of-hold-time metric. This metric makes good sense relative to assessing customer satisfaction; that is, long hold times would probably correlate to customer dissatisfaction. However, in isolation, this metric can drive the wrong behaviors unless safeguards are implemented to prevent abuse. Consider the focus that operators will give to achieve the targeted metric objective during understaffed peak call periods. Operators might simply answer the phone within the allotted time period, ask, "Can you hold, please?" and then quickly place the caller on hold for a much longer period. It is not bad that the operator answered the phone and responded, asking that the caller wait; however, this type of action should not be simply the result of wanting to make the overall duration-of-hold-time metric look good. Relative to recording the actual hold time for future customer satisfaction analyses, it would be better to capture the total hold time from initial connection until the incoming caller is connected to the appropriate person.
Second, we need data presentation and assessment formats that lead to the right kind of behavior, with appropriate reward systems in place to encourage this behavior. If an organization is measured solely on the meeting of goals, which might be arbitrary, bad things can occur. For example, Krispy Kreme shipped doughnuts that executives knew would be returned so that they would meet quarterly targeted objectives. Additionally Enron and, more recently, Dell made some decisions that enabled them to meet quarterly objectives but were poor in the long run. According to press reports, the senior management of Dell regularly falsified quarterly returns from 2003 through 2006 to create the appearance that the company had met their goals.
Characteristics of a Good Metric
We have all heard the clichés:
- You get what you measure.
- What you measure is what you get.
- If you don't measure it, you can't manage it.
- Tell me how I'm going to be measured and I'll tell you how I'll perform.
- You cannot improve what you can't measure.
- Garbage in, garbage out.
- If you don't measure it, it's just a hobby.
These clichés are true! Measurements need to be the processes' eyes, which stimulate the most appropriate behavior. Measurements need to provide an unbiased process performance assessment. When process output performance is not accurately seen and reported relative to a desired result, there is not much hope for making improvements. Generic measures for any process are quality, cost and delivery. Most processes need a balance measurement set to prevent optimizing one metric at the expense of overall process health. Metrics can also drive the wrong behavior if conducted in isolation from the overall enterprise needs. When appropriate, the addition of a people measure assures balance between task and people management.
As an illustration, consider the last customer satisfaction survey form that you received. Do you think that a summary of responses from this survey truly provides an accurate assessment of what you experienced in your purchase process? My guess is that your response is no. It seems that often surveys are conducted so that the responses will be satisfactory but don't truly provide insight into what actually happens in a process.
Writing an effective survey and then evaluating the responses is not easy. What we would like to receive from a survey is an honest picture of what is currently happening in the process, along with providing improvement direction. A comment section in a hotel guest survey might provide insight to a specific actionable issue or improvement possibility.
Good metrics provide decision-making insight that leads into the most appropriate conclusion and action or non-action. The objective is the creation of an entity that is measurable, auditable, sustainable and consistent. Effective and reliable metrics require the following characteristics:
- Business alignment: Metrics consume resources for both data collection and analyses. Metrics need to provide insight to business performance, its issues and its needs. Metrics surrounding your business alignment can be found by looking at your value chain.
- Honest assessment: Creating metrics so that the performance of someone or an organization will appear good has no value and can be detrimental to the organization. Metrics need to be able to provide an honest assessment, whether good, bad or ugly.
- Consistency: Identified components in any metric need to be defined at the outset and remain constant. Criteria and calculations need to be consistent with respect to time.
- Repeatability and reproducibility: Measurements should have little or no subjectivity. We would like for a recorded measurement response to have little or no dependence on who recorded the response and when the response was recorded.
- Actionability: Often measures are created for the sake of measuring, without any thought as to what would be done if the metric were lower or higher. Include only those metrics that you will act on; that is, either remove a degradation problem or hold the gain. When the metrics response is unsatisfactory, organizations need to be prepared to conduct root-cause analysis and corrective or preventive actions.
- Time-series tracking: Metrics should be captured in time-series format, not as a snapshot of a point-in-time activity. Time-series tracking can describe trends and separate special-cause from common-cause variability in predictable processes.
- Predictability: A predictability statement should be made when time-series tracking indicates that a process is predictable.
- Peer comparability: In addition to internal performance measurements, benefits are achieved when comparisons can be made between peer groups in another business or company. A good peer comparison provides additional analysis opportunities, which can identify improvement possibilities.