Problem in a nutshell: Metrics can be misused. Metrics can (and will) be gamed. This doesn’t mean we should avoid using any quantitative measures for team and project decision making – we just need to know why and what we are measuring, and interpret the results accordingly

“Just like dynamite, it would appear that metrics can be used for good as well as evil. It all depends on how you use them.”

1. Don’t embarrass people

Embarrassing people is easy to do when showing metrics they feel responsible for. This causes data to be hidden, obscured, and mis-reported. This leaves you with an incomplete and inaccurate picture even with data. Once you embarrass someone, thats the last time they will trust any metric, and the last time you have an accurate metric.

Do

  • Focus on trends rather than single point values.
  • Leave axis values off charts where possible; focus people on trends.
  • Exclude any name information Its OK for that team to identify themselves, but NOT for others to point out another team.

 

Figure 1 – Its the trend that matters. No team names or axis values help compare “trend”

2. Focus on Trends Not Individual Values

Trends are charts of the same measure over time. Trends help make sense of noisy data by helping see relative direction of change. Figure 1 shows a trend-line applied to cycle time data. The orange line is the team looking at its data, the grey line is the trend of the same measure of the rest of the company. This chart shows that the team is driving down its cycle time average over time, whereas the company trend is level over time.

Do

  • Capture data that helps show trend values over time
  • Add linear trend-line to data to help see the big picture of change
  • Help teams see how their trend tracks against “others” in similar situation
  • “other” means teams in SIMILAR situations, don’t compare apples versus oranges. Eg. sustainment teams versus production support teams.

3. Use Balanced Metrics

Tracking just one metric promotes overdriving that metric at the loss of everything else. Multiple opposing metrics should be equally shown with the emphasis that trade something you are above the trend with for something that is trending worse than others. Changing one metric is easy; changing that metric without decimating another is much harder.

Larry Maccherone in his “Software Development Performance Index” uses a metric from multiple quadrants –

Responsiveness – Time in Process average (often called cycle time).

Productivity – Throughput / team size (team size is to help normalize team size, making bigger teams and smaller team trends comparable)

Predictability – variability of throughput / size values. Helps teams identify they have peaks and troughs rather than smooth flow

Quality – How ready to release is the codebase? Could be number of open blocking P1 or P2 defects, or a score based on passing tests, number of un-merged feature branches, performance regressions. This is always the most difficult to find for each company. Avoid defect counts alone. Find ways to make quality mean improved customer experience.

Do –

  • Look for opposable measures. NO team should be able to be BEST at all, just one or two
  • Being BEST in a measure is an alarm! It means that they may be overdriving one measure at the sacrifice of others
  • Always show the measures together so people can see the tradeoffs they are making
Always show balanced metrics together. Avoids focus on just one.

Always show balanced metrics together. Avoids focus on just one.

4. Use Sampling – Track some metrics just sometimes

Some metrics are expensive to capture. You don’t need every metric all of the time. Sampling allows data to be captured for a short period of time to get a snapshot of how high or how low the metric is compared to estimate. For example, how much interrupt driven work is the team fielding requests for? Get the team to stick a post-it note on a whiteboard every time they do a “small job.” over the week you will get a good indication of percentage and make appropriate process changes. You can repeat one week next month and don’t track for the other three. This has made the cost of getting this metric 1/4 of the original cost and given the same result! Sampling is a powerful and underused technique.

Do

  • For measure that rely on people to do extra work to capture; use sampling. For example, track one week a month.
  • It takes less data than you think. 11 samples give a representative picture of a measure, by 30 samples you are almost certain the result is similar to every sample.

5. What, So What, Now What – Help people see the point

There has to be a reason for tracking and showing a metric. Make it clear how a metric trend aligns to a better decisions and improvement. If people don’t know why a metric is being tracked, they will assume its to track them personally! Help them see its about the work and the system, not the worker and their livelihood!

Do

  • Promote system metrics rather than personal metrics
  • Promote team metrics rather than personal metrics
  • Share how a trend of a metric has led to a better decision or improvement
  • Be vigilant about dropping metrics that are just available to capture – have a reason

In summary

Metrics aren’t evil. Although they are often mis-used, they don’t need to be. Make people responsible for determining actions on their own metrics. Send ideas and stories on what you have seen work and fail.