When planning, either on the back of a napkin or using more statistical methods, some estimation of “how long” is often needed. Precision will differ depending on needs, for example it might just be good enough to know the calendar quarter to confirm that a new website will be in place well in advance of a promotional period. Other people might want more clarity because they are making trade-offs between multiple options. We often get asked to confirm or build a forecasting model for these organizations and see some common errors that cause erroneous forecasts.
This post and subsequent posts will outline some of the common ones we see.
Assuming uniform throughput, and that team performance is the only factor for throughput
Whether forecasting using velocity or story-count over time, the amount of work being completed is a measure of throughput or in plain terms, a rate of completed work over time. We often see organizations consider this rate as within the teams control, and used as a measure of progress and performance. Sometime it is, but if we plot the throughput history, notable areas of instability and step-changes are evident. Knowing the sources of these, and how to adjust throughput forecasts knowing this in advance is key to improving any method of delivery estimation.
Figure 1 – Throughput run charts like this show discontinuities that are other than team completion rate and need to be considered when forecasting
Common causes we see and adjust for –
1. Team forming stage and other phases – Teams often take on new technologies and team members at the beginning of a new piece of work. It should be expected that the early storming phases for tams and new investigative development will be slower than when a team is long-running and stable. We start with an adjustment of minus 50% for the first 20% of a project.
2. Calendar events – Differing by regional geography and countries, often there are whole calendar periods where throughput drops dramatically and recovers slowly. Depending on the granularity of our forecasts, consider known holiday days where no-one will work, long weekends where people take vacation to extend the single days holiday to a week or more, and the biggest factor of all, December. If an organization has a “use or lose” vacation policy, a lot of technical staff end up using that vacation in December, and some combine it with new vacation and extend to January. We see roughly 4 weeks of almost no progress in some organizations. Impact is cascading if teams have tight dependencies. Forecasting over these periods is challenging.
3. Organizational changes – Employee concerns and stress during leadership changes and re-orgs is another step-function factor in throughput impact. Even rumors can be seen in throughput run charts. Expect a -20% decrease recovering over one to two months depending on how well the change is accepted and communicated. For large companies, we assume there will be at least one of these for every six month period.
4. Changes in the way work is sub-divided or described – this is an obvious one, but often overlooked. New processes or constraints or motivations will impact the way work is sub-divided. Throughput or velocity being captured at one granularity is not going to forecasts work in a different granularity. We often adjust for this by taking a sample of prior work and getting teams to break-down using the new process to find a multiplying factor. Performing this regularly for samples of work in each quarter going back 12 months helps normalize a throughput run-chart back to “real rate of progress.” This process re-plots the historical throughput to a similar rate of that being used today with the aim of isolating team process improvements rather than work-size anomalies.
These are just a start of the factors that influence throughput in ways that make it a poorer predictor of the future than it could be. These factors apply no matter what unit of completion rate used, be it velocity, or story count.