The argument for agile gets even more appealing…

The Standish Group’s October 2015 CHAOS report is based on a study of thousands of technology projects worldwide. Whereas past CHAOS reports had considered project success as achieving the triple constraint, their research is now utilizing a more current definition taking into account factors such as on Target, on Goal, Value and Satisfaction. Especially in the case of technology-intensive initiatives, this approach is critical to ensure that an outcome of the operation was a success, but the patient died wouldn’t be considered a success. In addition to using this approach on the projects assessed for the 2015 report, the Group also updated their statistics for 2011-2014 using a similar protocol to ensure that year-to-year comparisons would be feasible.

Some of their findings aren’t astounding such as smaller projects tending to have a higher likelihood of success than larger ones. This is likely to always be the case as size is a contributing factor to project complexity. Outcomes for projects have also not improved dramatically over the past five years with overall success rates continuing to hover around 29%. This is also not a surprise when one considers that while global knowledge of how to improve project outcomes has increased, lessons haven’t been learned. Increased economic constraints and increasing complexity may have also conspired to throttle gains.

However things get more intriguing when we compare the results for agile and waterfall.

Out of a sample of over 10,000 software projects, 39% of those which were managed using agile approaches succeeded as compared with only 11% of those using a traditional approach. It is possible that differences in sample size between agile and waterfall could certainly affect the numbers, but with such a large overall sample size, this still appears to be a meaningful difference.

But things get really interesting when we look at the differences in performance taking project size into consideration. For small sized projects, agile projects had a 58% success rate as compared to a 44% success rate for waterfall projects. Combining the successful and challenged categories gives us 96% partially successful for agile versus 89% for waterfall which is a difference of 7%. But for large sized projects, 77% of those managed using agile approaches are partially successful versus only 58% for waterfall – a difference of 19%!

What could account for a 12% increase in the difference between failure rates for the two approaches when we move from small to large sized projects?

Three which come to mind are:

• While agile projects can fail faster and hence be shutdown quicker, the principle of early and frequent delivery of business value should result in a higher likelihood of achieving at least partial success even if not all scope can be delivered within cost and time constraints.
• A higher degree of user involvement throughout the project lifecycle in requirements definition and decision-making avoids realizing the What the Customer Wanted cartoon which is more likely to occur on bigger projects.
• Sustained executive sponsorship is critical for large projects which can span multiple years. With traditional approaches, executives might have to wait a long time to reap the benefits of their financial and political investments and this could cause their attention to drift. With regular value delivery, there is less likelihood of this risk being realized.

The changes required to be successful at an enterprise portfolio level with agile are not for everyone as it does require sustained commitment from all levels and is a major cultural transformation. But studies such as the CHAOS report are starting to bolster the case for all organizations with significant investments in technology projects to take a serious look at agile.