Two Important Tips for Gathering Metrics
Before you dive into reports and spreadsheets, you need to define exactly what data you want to collect and analyze. Be careful not to fall into the trap of gathering an overwhelming number of metrics. You need to make sure that every metric you monitor provides actionable insight.
“Here's the TenStep guest blog post "Two Important Tips for Gathering Metrics":
Metrics are needed to make more fact-based decisions about the nature of the project. However, metrics are tough and require work to implement successfully. Here are two tips to ensure that you can be successful.
Tip #1. Beware Unintended Consequences When Establishing Metrics
Collecting metrics will drive certain behaviors. In fact, this is exactly what you want to have happen or you would not be collecting the metrics to begin with. Therefore it is critical that the metrics drive the correct behaviors and not drive unintended behaviors.
Look at the following example. A team was being measured on the length of time that it took to close customer inquiries. The target was set to close all open customer inquiries within two business days. Team members realized that they needed to close the tickets quickly or they would be viewed as unsuccessful. The team members also realized that if they could not close the inquiry quickly, they should simply guess at the cause and then close the open ticket. If the guess did not solve the problem, a new problem ticket was opened and quickly resolved again. The result was repetitive thrashing of problem tickets, wasting much more time than required. In other words, the target was being met by driving bad behavior.
In this example, one of the problems was the metric. This team looked good on paper, since they were closing problem inquiries quickly. However, in reality they were performing poorly, generating extra work and causing the client to be dissatisfied. The idea was fine, but perhaps a better metric would have been to consider the inquiry “closed” when the client agreed and approved the resolution.
When you are considering a metric for your project, think about how the metric might drive unintended consequences. Be sure that you set up the metric in a way that clearly drives the desired behaviors.
Tip #2. Gather a Baseline if no Target is Available
Collecting metrics information by itself provides only limited value. Most of the value comes when you can compare your metrics against some type of standard or target. For many metrics, there is an implicit target, even if you don't always specifically state it. For instance, the collection of actual effort, duration and cost information is used to compare against the estimated effort, deadline and budget to see how the project stands.
For many metrics, however, there is not necessarily an implicit target to attain. For instance, if you track your client's satisfaction with the project, they may rate your team an average of 3.8 out of a 5.0 scale. However, is that a good number or a bad number? Since you have nothing to compare it with, it's hard to say. The way to gain more value from the metric is to use the first measurement as a baseline - that is, a reflection of where you are today. In the survey example, you would want to consider your first results to be the baseline, and then you would strive to improve upon your baseline numbers. For instance, after collecting a baseline of 3.8 out of 5.0, you may choose a target of achieving a 4.2 out of 5.0 before the project is over. Another option is to look for a 10% improvement in the baseline, once the original metrics have been collected.