When we hear of the difficulties of making decisions in the presence of uncertainty, especially about software features and capabilities, there are straightforward ways to solve this problem.
Decision Analysis is a principle, technique, and application to address complex decisions in a structured manner. One approach to decision making utilizes a form of multi-criteria decision analysis to evaluate multiple conflicting criteria in decision making. This method is the Analytic Hierarchy Process (AHP). AHP was developed in the 1970s by Dr. Thomas Saaty. AHP has been studied, refined, and applied for over 40 years to be an effective approach to complex group decision making. [7]
In our agile software development world, AHP is rarely found. I came to AHP through a seminar by Dr. James T. Brown and his book The Handbooks of Program Management: How to Facilitate Project Success with Optimal Program Management
AHP structures decision problems as a hierarchy, shown below. At the top is the decision objective. The next level consists of decision criteria. This is followed by any number of options or alternatives. AHP can be used as a Value Management System to organize the criteria and assess trades off costs and benefits in considering total value.
AHP is based on the principle that all measurements are relative. People are generally very good at comparing things relative to other things. AHP provides a framework to make relative comparisons using a rational decision structure based on scaled pairwise comparisons (Borda Ranking) using a scale that converts stakeholder preferences and priorities into ratio measures. Using this method, the performance, cost, time, and risks of alternatives can be articulated as ratios that can then be compared with one another. This decision model for software development projects addressed: performance, cost, time, and risk.
Mathematically, the value equals performance over cost plus time. The risks related to performance, cost, and time must also be considered.
The relative measures of these values are Ordinal numbers. This is one role for Story Points in agile development. One relative measure used in our space and defense domain of technical maturity is the Technical Readiness Level.
This is an Ordinal Measure used to make decisions, in the same way, Story Points can be used, if properly formed and properly used
Technology Readiness Levels (TRL) are a type of measurement system used to assess the maturity level of a particular technology. Each technology project is evaluated against the parameters for each technology level and is then assigned a TRL rating based on the progress of the project. There are nine technology readiness levels. TRL 1 is the lowest and TRL 9 is the highest.
When technology is at TRL 1, scientific research is beginning and those results are being translated into future research and development. TRL 2 occurs once the basic principles have been studied and practical applications can be applied to those initial findings. TRL 2 technology is very speculative, as there is little to no experimental proof of concept for the technology.
When active research and design begin, technology is elevated to TRL 3. Both analytical and laboratory studies are required at this level to see if a technology is viable and ready to proceed further through the development process. Often during TRL 3, a proof-of-concept model is constructed.
Once the proof-of-concept technology is ready, the technology advances to TRL 4. During TRL 4, multiple component pieces are tested with one another. TRL 5 is a continuation of TRL 4, however, a technology that is at 5 is identified as a breadboard technology and must undergo more rigorous testing than the technology that is only at TRL 4. Simulations should be run in environments that are as close to realistic as possible. Once the testing of TRL 5 is complete, technology may advance to TRL 6. A TRL 6 technology has a fully functional prototype or representational model.
TRL 7 technology requires that the working model or prototype be demonstrated in a space environment. TRL 8 technology has been tested and "flight qualified" and it's ready for implementation into an already existing technology or technology system. Once a technology has been "flight-proven" during a successful mission, it can be called TRL 9.
Risk assessment can be made with Ordinal value as well. Scoring the severity of each risk factors can be rated on an Ordinal scale. The ordinal risk values are then combined with additive weighting or by multiplication to compute an aggregate measure of overall risk. [1]
The Core Issue with Ordinal Measures
First some definitions and a prior post on The Story Point Problem
- A Cardinal number says how many of something there is
- An Ordinal number tells the position of something relative to something else
Measurement theory is a well-debated topic and has been going on long before agile software developers started using Story Points [2], [4]. The core issue is Ordinal measures are subjective and provide little predictive value. This issue starts with the process of assigning numbers to objects so that the unique relationships of the objects are reflected in the unique numbers themselves.
One side of the conversation (at least in Stroy Points) maintains that the scales on which the measurements rests are both representative and unique, and the nature of the scale determines the statistics that should be used. The opposing viewpoint is that within the measures there is at best only loose relationship to both representatives and uniqueness, and therefore there is no relationship to the scales and the statistics that should be used [3].
Business Operates with Cardinal Numbers
The business makes decisions with Cardinal numbers. Dollars, Hours, weight, throughput, all the other ...ilities.
The Wrap Up
The use of Ordinal numbers in TRL, AHP, and similar decision-making processes is useful to make relative trade-offs.
Monetizing those choices can be part of that decision-making process. Or it can be applied after the relative measures of value, for example, have been made
But in the end, for the business to make a decision a Cardinal value is needed if that decision impacts the balance sheet.
And of course
No Credible Decision Can Be Made in the Presence of Uncertainty with making Estimates of the Impact of those Decisions. To suggest otherwise, willfully ignores the principles of Managerial Finance, Microeconomics of Software Development, and Probabilistic Decision Making.
References
[1] "Problem with Scoring Methods and Ordinal Scales in Risk Management," Douglas Hubbard and Dylan Evans, IBM Journal of Research and Development, May 2010.
[2] "Evaluating Risk: A revisit of the Scales, Measurement Theory, and Statistical Analysis Controversy," J. D. Solomon, Daniel Vallero, and Kathryn Benson, International Reliability and Maintenance Symposium, 2017.
[3] “Measurement Scales and Statistics: The Misconception Misconceived,” J. T. Townsend and F.G. Ashby, Quantitative Methods in Psychology, 1984.
[4] “On the Theory of Scales of Measurement,” S.S. Stevens, Science, Vol. 103, No. 2684. June 7, 1946, pp. 677-680.
[5] How to Measure Anything: Finding the Value of "Intangibles" in Business, Douglas Hubbard, John Wiley & Sons, 2014.
[6] Decision Analysis for the Professional, Fourth Edition, Peter McNamee and John Celona (download this book and read how to make decisions in the presence of uncertainties and stop listening to the nonsense of #Noestimates)
[7] "Application of the AHP in Project Management," Kamal M. Al-Subhi Al-Harbi, International Journal of Project Management, 19 (1002), pp. 19-27.
[8] "A Methodology for Project Selection Using Economic Analysis and the Analytical Hierarchy Process," Jeffrey L. Battin, Captain USAF, Air Force Institute of Technology, September 1992.