Misunderstanding Making Decisions in the Presence of Uncertainty (Update Part 2)

There was a Tweet a few days ago from one of the founders of eXtreme Programming, that said...

What happens if you shift focus from "accurate estimation" to "reliably shipping by a date"? 

This quote shows the missing concept of the processes for making decisions in the presence of uncertainty and the processes and events that create uncertainty that impact the reliability of making the date to ship value.

The answer is ... You can't shift focus from accurate estimate to reliably shipping by a date ...

Accurate and precise estimates (to predefined values as shown in the target picture below) are needed before you can reliably ship products by a date. Because you can't know that date with any needed level of confidence without making estimates about the reducible and irreducible uncertanties that inpact that date.

So the answer to the question is.

In the presence of uncertanity, You can't reliably ship by a date without estimating the impact of those uncertanties on the probability of making the date.

Since uncertainty creates risk, managing in the presence of uncertainty requires Risk Management, we can now answer the question, with:

• If you want a reliable shipping date, you have to discover and handle the uncertainties in the work needed to produce the outcomes to be delivered on that date.
• You have to estimate the needed schedule, cost, and technical performance margins needed to protect that date from the Aleatory uncertainties.
• You have to estimate the probabilistic occurrence of the epistemic uncertainties that will impact that date and provide a Plan B an intervention, or some corrective action to protect that date.

Each of these uncertainties creates risk to meeting that reliable shipping date. And as we all know

Risk Management is How Adults Manage Projects - Tim Lister

Details of the Answer to the Question

First, let's establish a principle. All project work has uncertainty. Uncertainty comes from the lack of precision and accuracy about the possible values of a measurement of a project attribute.

There is naturally occurring variability from uncontrolled processes. There is a probability of the occurrence of a future event. This absence of knowledge (Epistemic uncertainty)  can be modeled as a probability of occurrence or a statistical distribution of the natural variability. If your project has no uncertainty, there is no need to estimate. All outcomes are certain, occurring with 100% probability, and with 0% variance. Turns out in the 

These uncertainties come in two forms. Naturally occurring variances (Aleatory uncertainty) and Event based probabilities (Epistemic uncertainty).

The naturally occurring variability comes from uncontrolled and uncontrollable processes. This uncertainty is modeled as a statistical distribution from past performance or an underlying statistical process model, usually stochastic (stationary or non-stationary). The probability of a random event is the absence of knowledge. This uncertainty is modeled as a probability of occurrence or a statistical distribution of the natural variability. If your project has no uncertainty, there is no need to estimate. All outcomes are uncertain, occurring with 100% probability, and with 0% variance.

A formal definition of uncertainty in the project decision-making paradigm is ...

Situation where the current state of knowledge is such that (1) the order or nature of things is unknown, (2) the consequences, extent, or magnitude of circumstances, conditions, or events is unpredictable, and (3) credible probabilities to possible outcomes cannot be assigned. 

If your project has no uncertainty, there is no need to estimate. Then the planned ship date is deterministic. All outcomes are certain, occurring with 100% probability, and with 0% variance.

Turns out in the real world there is no such project.

When we say uncertainty, we speak about a future state of the system that is not fixed or determined. Uncertainty is related to three aspects of the management of projects:

1. The external world - the activities of the project itself.
2. Our knowledge of this world - the planned and actual behaviors of the project.
3. Our perception of this world - the data and information we receive about these behaviors.

Let's revisit the two flavors of uncertainty - uncertainty that can be reduced (Epistemic) and uncertainty that cannot be reduced (Aleatory)

Aleatory uncertainties are unknowns that differ each time we assess them. They are values drawn from an underlying population of possible values. They are uncertainties that we can't do anything about. They cannot be suppressed or removed. My drive from my house to my secret parking spot on the east side of Denver International Airport is shown at 47 minutes by Google Maps. If I ask Google what's the duration at a specific time of day, 3 days from now, I'll get a different number. When I get on the farm road to I-25 and get to I-25 I may find a different time. This time is the random variances of distance and traffic conditions. I need margin to protect me from being late to the parking spot.

The naturally occurring work effort in the development of a software feature - even if we've built the feature before - is an irreducible uncertainty. The risk is created when we have not accounted for this natural variances in our management plan for the project. If we do not have a sufficient buffer to protect the plan from these naturally occurring variances, our project will be impacted in unfavorable ways.

The notion (as suggested in the quote) of shifting from accurate (what ever that means) ways of estimating to reliability shipping be a date is not physically possible since the irreducible and reducible uncertainties are always present.

Dealing with Aleatory (irreducible) uncertainty and the resulting risk requires we have margin. Aleatory uncertainty is expressed as a process variability. Work effort variances, productivity variances, quality of product and resulting rework variances.  Epistemic uncertainties are systematic, caused by things we know about in principle. The probability of something happening. An aleatory risk is expressed as a relation to a value. A percentage of that value. This is the motivation for short work intervals found in agile development. 

Epistemic uncertainties are systematic, caused by things we know about in principle. The probability of something happening. This uncertainty is introduced by a probabilistic event, rather than a naturally occurring process. Epistemic uncertainty is introduced by an assumption about the world in which the system is embedded. This assumption can be from the lack of data - an ontological uncertainty. Epistemic uncertainties have probabilities of occurrence. The likelihood of a failure for example. Epistemic uncertainty can also occur when there is a subjective evaluation of the system - a risk from a rare event or an event with little or no empirical data. Epistemic uncertainty can also occur from the incompleteness of knowledge - a major hazard or condition not identified or a causal mechanism the remains undetected. And epistemic uncertainty can also occur from undetected design errors, introduced by ontological uncertainties into the system behavior. 

Epistemic uncertainty can also occur when there is a subjective evaluation of the system - a risk from a rare event or an event with little or no empirical data. Epistemic uncertainty can also occur from the incompleteness of knowledge - a major hazard or condition not identified or a causal mechanism the remains undetected. And epistemic uncertainty can also occur from undetected design errors, introduced by ontological uncertainties into the system behavior. 

Before completing this post, let's look quickly at procession and accuracy as mention in the original quote. All estimates have precision and accuracy. Deciding how much precision and accuracy is needed for a credible estimate is critical to the success of that decision. One starting point is the value at risk. By determining the value at risk, we can determine how much precision and accuracy is needed and how much time and cost we should put into the estimating process.

Let's go back to the original quote.

What happens if you shift focus from "accurate estimation" to "reliably shipping by a date"? 

With our knowledge of Epistemic and Aleatory uncertainty, we now know we cannot reliably ship by a date, without knowing the extent of the reducible and irreducible uncertainties, that protect that date with margin or reserve for the irreducible uncertainties and specific actions, redundancies, or interventions for the reducible uncertainties. To know how much margin or reserve for irreducible uncertainties and performance of the of the redundancies we now need to know. 

For our credible estimate, we must have a desired and measurable:

• Precision - how small is the variance of the estimate?
• Accuracy - how close is the estimate to the actual value?
• Bias - what impacts on precision and accuracy come from human judgments?

So in the end, if we are to make a decision in the presence of uncertainty, we MUST make estimates to develop a reliable shipping date while producing an accurate and precise estimate of the cost, schedule, and technical performance of the product shipped on that date.

So it comes down to this, no matter how many times those claiming otherwise, so I'll shout this to make it clear to everyone...

YOU CANNOT MAKE A DECISION IN THE PRESENCE OF UNCERTAINTY (reducible or irreducible) WITHOUT MAKING ESTIMATES

A Short List of Resources for Managing in the Presence of Uncertainty

Risk Management is essential for development and production programs. Information about key project cost, performance, and schedule attributes is often uncertain or unknown until late in the program.
Risk issues that can be identified early in the program, which will potentially impact the program later, termed Known Unknowns can be alleviated with good risk management. in Effective Risk Management 2^nd Edition, Edmund Conrow, AIAA, 2003

 

Papers on Risk Management

1. “Quantifying Uncertainty in Early Lifecycle Cost Estimation (QUELCE),” Robert Ferguson, Dennis Goldenson, James McCurley, Robert Stoddard, David Zubrow, and Debra Anderson, Technical Report, CMU/SEI-2011-TR-026 ESC-TR-2011-026
2. “The Development of Progress Plans Using a Performance–Based Expert Judgment Model to Assess Technical Performance and Risk,” Justin W. Eggstaff, Thomas A. Mazzuchi, and Shahram Sarkani, Systems Engineering, Volume 17, Issue 4, Winter 2014, Pages: 375–391
3. “Using the Agile Methodology to Mitigate the Risks of Highly Adaptive Projects,” Dana Roberson and Mary Anne Herndon, 10^th Annual CMMI Technology Conference And User Group, November 5 – 8, 2012, Denver, CO
4. “Hybrid–Agile Software Development Anti–Patterns, Risks, and Recommendations,” Paul E. McMahon, Cross Talk: The Journal of Defense Software Engineering, July/August 2015, pp. 22–26.
5. “Using the Agile Methodology to Mitigate the Risks of Highly Adaptive Projects,” Dana Roberson and Mary Anne Herndon, 10^th Annual CMMI Technology Conference And User Group, November 5 – 8, 2012, Denver, CO.
6. “Assessment of risks introduced to safety critical software by agile practices — A Software Engineer’s Perspective,” Janusz Górski Katarzyna Łukasiewicz, AGH University of Science and Technology, University in Kraków, Poland, Computer Science, Vol 13, No 4.
7. “Ready & Fit: Understanding Agile Adoption Risk in DoD and Other Highly Regulated Settings,” Suzanne Miller and Mary Ann Lapham, 25^th Annual Software Technology Conference, Salt Lake City, 8-10 April 2013.
8. “Architecting Large Scale Agile Software Development: A Risk–Driven Approach,” Ipek Ozkaya, Michael Gagliardi, Robert L. Nord, CrossTalk: The Journal of Defense Software Engineering, May/June 2013.
9. “Risk Management Method using Data from EVM in Software Development Projects,” Akihiro Hayashi and Nobuhiro Kataoka, International Conference on Computational Intelligence for Modelling, Control and Automation, Vienna, Austria, Dec. 10 to Dec. 12, 2008.
10. “Analyse Changing Risk of Organizational Factors in Agile Project Management,” Shi Tong, Chen Jianbin, and Fang DeYing, The 1st International Conference on Information Science and Engineering (ICISE2009).
11. “Modeling Negative User Stories is Risky Business,” Pankaj Kamthan and Nazlie Shahmir, 2016 IEEE 17th International Symposium on High Assurance Systems Engineering.
12. “Project Risk Management Model Based on PRINCE2 and Scrum Frameworks,” Martin Tomanek, Jan Juricek, The International Journal of Software Engineering & Applications (IJSEA), January 2015, Volume 6, Number 1, ISSN: 0975-9018
13. “How to identify risky IT projects and avoid them turning into black swans,” Magne Jørgensen, Ernst & Young: Nordic Advisory Learning Weekend, Riga, 2016.
14. “A Methodology for Exposing Software Development Risk in Emergent System Properties,” Technical Report 11-101, April 21, 2001, Victor Basili, Lucas Layman, and Marvin Zelkowitz, Fraunhofer Center for Experimental Software Engineering, College Park, Maryland.
15. “Outlining a Model Integrating Risk Management and Agile Software Development,” Jaana Nyfjord and Mira Kajko-Mattsson, 34th Euromicro Conference Software Engineering and Advanced Applications.
16. “Towards a Contingency Theory of Enterprise Risk Management,” Anette Mikes Robert Kaplan, Working Paper 13–063 January 13, 2014, AAA 2014 Management Accounting Section (MAS) Meeting Paper
17. “Agile Development and Software Architecture: Understanding Scale and Risk,” Robert L. Nord, IEEE Software Technology Conference, 2012, Salt Lake City, 23-26 April, 2012.
18. “Using Risk to Balance Agile and Plan Driven Methods,” Barry Boehm and Richard Turner, IEEE Computer, June 2003.
19. “Does Risk Management Contribute to IT Project Success? A Meta-Analysis of Empirical Evidence,” Karel de Bakker, Albert Boonstra, Hans Wortmann, International Journal of Project Management, 2010.
20. “A Model for Risk Management in Agile Software Development,” Ville Ylimannela, Communications of Cloud Software.
21. “Product Security Risk Management in Agile Product Management,” Antti Vähä-Sipilä, OWASP AppSec Research, 2010
22. “A Probabilistic Software Risk Assessment and Estimation Model for Software Projects,” Chandan Kumar and Dilip Kumar Yadav, Eleventh International Multi-Conference on Information Processing-2015 (IMCIP-2015)
23. “Risk: The Final Agile Frontier,” Troy Magennis, Agile 2015.
24. ”Risk Management and Reliable Forecasting using Un-Reliable Date,” Troy Magennis, Lean Kanban, Central Europe, 2014.
25. “Management of risks, uncertainties and opportunities on projects: time for a fundamental shift,” Ali Jaafari, International Journal of Project Management 19 (2001) 89-101.
26. “On Uncertainty, Ambiguity, and Complexity in Project Management,” Michael T. Pich, Christoph H. Loch, and Arnoud De Meyer, Management Science © 2002 INFORMS, Vol. 48, No. 8, August 2002 pp. 1008–1023
27. “Risk Options and Cost of Delay,” Troy Magennis, LKNA 2014.
28. “Transforming project risk management into project uncertainty management,” Stephen Ward and Chris Chapman, International Journal of Project Management 21 (2003) 97–105.
29. “Risk-informed decision-making in the presence of epistemic uncertainty,” Didier Dubois, Dominique Guyonnet, International Journal of General Systems, Taylor & Francis, 2011, 40 (2), pp. 145-167.
30. “A case study of risk management in agile systems development,” Sharon Coyle and Kieran Conboy, 17th European Conference on Information Systems (Newell S, Whitley EA, Pouloudi N, Wareham J, Mathiassen L eds.), 2567-2578, Verona, Italy, 2009
31. “Risk management in agile methods: a study of DSDM in practice,” Sharon Coyle, 10th International Conference on eXtreme Programming and Agile Processes in Software Engineering, 2009.
32. “Distinguishing Two Dimensions Of Uncertainty,” Craig R. Fox and Gülden Ülkümen, in Perspectives on Thinking, Judging, and Decision Making, Brun, W., Keren, G., Kirkeboen, G., & Montgomery, H. (Eds.), 2011.
33. “Two Dimensions of Subjective Uncertainty: Clues from Natural Language,” Craig R. Fox and Gülden Ülkümen.
34. “An Essay Towards Solving a Problem in the Doctrine of Chance,” By the late Rev. Mr. Bayes, communicated by Mr. Price, in a letter to John Canton, M. A. and F. R. S.
35. “Playbook: Enterprise Risk Management for the U.S. Federal Government, in support of OMB Circular A-123.”
36. “Joint Agency Cost Schedule Risk and Uncertainty Handbook,” Naval Center for Cost Analysis, 12 March 2014.
37. “Quantitative Risk ‒ Phases 1 & 2: A013 ‒ Final Technical Report SERC-2013-TR-040-3,” Walt Bryzik and Gary Witus, November 12, 2013, Stevens Institute of Technology
38. “Distinguishing Two Dimensions of Uncertainty,” Craig Fox and Gülden Ülkumen, in Perspectives of Thinking, Judging, and Decision Making
39. “Using Risk to Balance Agile and Plan Driven Methods,” Barry Boehm and Richard Turner, IEEE Computer, June 2003
40. “Commonalities in Risk Management and Agile Process Models,” Jaana Nyfjord and Mira Kajko-Mattsson, International Conference on Software Engineering Advances(ICSEA 2007).
41. “Software Risk Management in Practice: Shed Light on Your Software Product,” Jens Knodel, Matthias Naab, Eric Bouwers, Joost Visser, IEEE 22^nd International Conference on Software Analysis, Evolution, and Reengineering (SANER), 2015
42. “Software risk management,” Sergey M. Avdoshin and Elena Y. Pesotskaya, 7^th Central and Eastern European Software Engineering Conference, 2011.
43. “A New Perspective on GDSD Risk Management Agile Risk Management,” Venkateshwara Mudumba and One-Ki (Daniel) Lee, International Conference on Global Software Engineering, 2010
44. “Using Risk Management to Balance Agile Methods: A Study of the Scrum Process,” Benjamin Gold and Clive Vassell, 2^nd International Conference of Knowledge-Based Engineering and Innovation, November 5-6, 2015
45. “Using Velocity, Acceleration, and Jerk to Manage Agile Schedule Risk,” Karen M. Bumbary, 2016 International Conference on Information Systems Engineering
46. “The Risks of Agile Software Development Learning from Adopters,” Amany Elbanna and Suprateek Sarker, IEEE Software, September/October 2016
47. “Software Delivery Risk Management: Application of Bayesian Networks in Agile Software Development,” Ieva Ancveire, Ilze Gailite, Made Gailite, and Janis Grabis, Information Technology and Management Science, 2015/18.
48. “Lightweight Risk Management in Agile Projects,” Edzreena Edza Odzaly, Des Greer, Darryl Stewart, 26^th Software Engineering Knowledge Engineering Conference (SEKE), November 2015.
49. “A Method of Software Requirements Analysis Considering the Requirements Volatility from the Risk Management Point of View,” Yunarso Anang, Masakazu Takahashi, and Yoshimichi Watanabe, 22^nd International Symposium on QFD, Boise, Idaho.
50. “Analyse Changing Risk of Organizational Factors in Agile Project Management,” Shi Tong, Chen Jiabin, and Fang DeYing, The 1^st International Conference on Information Science and Engineering (ICISE2009)
51. “Outlining a Model Integrating Risk Management and Agile Software Development,” Jaana Nyfjord and Mira Kajko-Mattsson, 34^th Euromicro Conference Software Engineering and Advanced Applications. 2009.
52. “How Do Real Options Concepts Fit in Agile Requirements Engineering?,” Zornitza Racheva and Maya Daneva, Eighth ACIS International Conference on Software Engineering Research, Management and Applications, 2010..
53. NASA Risk Informed Decision Making Handbook, NASA/SP-2010-576, Version 1.0 April, 2010.
54. “Managing Risk Within A Decision Analysis Framework,” Homayoon Dezfuli, Robert Youngblood, Joshua Reinert, NASA Risk Management Page.
55. “NASA Risk Management Handbook,” NASA/SP-2011-3422, Version 1.0, November 2011.
56. “Risk Management For Software Projects In An Agile Environment – Adopting Key Lessons From The Automotive Industry,” Oana Iamandi, Marius Dan, and Sorin Popescu, Conference: MakeLearn and TIIM Joint International Conference 2015: Managing Intellectual Capital and Innovation for Sustainable and Inclusive Society, Bari, Italy, 2015
57. “Role of Agile Methodology in Software Development,” Sonia Thakur and Amandeep Kaur, International Journal of Computer Science and Mobile Computing, Volume 2, Issue 10, October 2013, pp. 86-90
58. “Agile Risk Management Workshop,” Alan Moran Agile Business Conference, 08.10.2014, London England
59. “Embrace Risk! An Agile approach to risk management,” Institute for Agile Risk Management, 2014.
60. “Risks in distributed agile development: A review,” Suprika Vasudeva Shrivastava and Urvashi Rathod, Procedia - Social and Behavioral Sciences 133 ( 2014 ) 417 – 424, ICTMS-2013
61. “Risk and uncertainty in project management decision-making,” Karolina Koleczko, Public Infrastructure Bulletin, Vol. 1, Issue. 8 [2012], Art. 13.
62. “Uncertainty and Project Management: Beyond the Critical Path Mentality,” A. De Meyer, C. Loch, And M. Pich, INSEAD Working Paper, 2001.
63. “Proposal of Risk Management Metrics for Multiple Project Software Development,” Miguel Wanderleya, Júlio Menezes Jr., Cristine Gusmãoa, Filipe Limaa, Conference on ENTERprise Information Systems / International Conference on Project Management / Conference on Health and Social Care Information Systems and Technologies, CENTERIS / ProjMAN / HCist 2015 October 7-9, 2015.
64. “Outling a Model Integration Risk Management and Agile Software Development,” Jaana Nyfjord and Mira Kajko-Mattsson, 34^th Euromicro Conference Software Engineering and Advanced Applications, 2008.
65. “The Impact Of Risk Checklists On Project Manager’s Risk Perception And Decision-Making Process,” Lei Li, Proceedings of the Southern Association for Information Systems Conference, Savannah, GA, USA March 8th–9th, 2013.
66. “Identifying The Risks Associated With Agile Software Development: An Empirical Investigation.” Amany Elbanna, MCIS 2014 Proceedings. Paper 19.
67. “Uncertainty, Risk, and Information Value in Software Requirements and Architecture,” Emmanuel Letier, David Stefan, and Earl T. Barr, ICSE ’14, May 31 – June 7, 2014, Hyderabad, India
68. “Software project risk analysis using Bayesian networks with causality constraints,” Yong Hu, Xiangzhou Zhang, E. W. T. Ngai, Ruichu Cai, and Mei Liu, Decision Support Systems 56 (2013) 439–449.
69. “Risk Based Scrum Method: A Conceptual Framework,” Nitin Uikey and Ugrasen Suman, 2015 2nd International Conference on “Computing for Sustainable Global Development, 11th - 13th March, 2015.
70. “Implementation of Risk Management with SCRUM to Achieve CMMI Requirements,” Eman Talal Alharbi, M. Rizwan Jameel Qureshi, International Journal Computer Network and Information Security, 2014, 11, 20-25.
71. “Risk, ambiguity, and the Savage axioms,” Daniel Ellsberg, The Quarterly Journal of Economic, Vol. 75, No. 4, pp. 643-669, Nov 1961.
72. “Analytical Method for Probabilistic Cost and Schedule Risk Analysis: Final Report,” Prepared for NASA, 5 April 2013.
73. “Risk, Ambiguity, and the Savage Axioms,” Daniel Ellsberg, August 1961, RAND Corporation, Report P-2173.
74. “Dealing with Uncertainty Arising Out of Probabilistic Risk Assessment,: Kenneth Solomon, William Kastenberg, and Pamela Nelson, RAND Corporation, R-3045-ORNL, September 1983.
75. “Using Risk to Balance Agile and Plan-Driven Methods,” Barry Boehm and Richard Turner, IEEE Computer, June 2003.
76. “Epistemic Uncertainty Analysis: An Approach Using Expert Judgment and Evidential Credibility,” Patrick Hester, International Journal of Quality, Statistics, and Reliability, Volume 2012, Article ID 617481, 8 pages
77. “Representation of Analysis Results Involving Aleatory and Epistemic Uncertainty,” J. C. Helton, J. D. Johnson, W. L. Oberkampf, C. J. Sallaberry, Sandia Report SAND2008-4379, 2008.

Books on Risk Management

78. Probabilistic Risk Assessment and Management for Engineers and Scientist 2^nd Edition, Ernest J. Henley and Hiromitsu Kumamoto, IEEE Press, 2000.
79. Agile Risk Management and Scrum, Alan Moran, Institute for Agile Risk Management, 2014.
80. Managing the Unknown: A New Approach to Managing High Uncertainty and Risk in Projects 1^st Edition, Christoph H. Loch, Arnoud DeMeyer, and Michael Pich, Wiley, 2006.
81. Identifying and Managing Project Risk: Essential Tools for Failure-Proofing Your Project, Tom Kendrick, AMACOM, 3^rd Edition, March 2015
82. Integrated Cost and Schedule Control in Project Management, Second Edition 2nd Edition, Ursula Kuehn, Management Concepts, 2010.
83. Effective Risk Management, 2nd Edition, Edmund Conrow, AIAA, 2003.
84. Effective Opportunity Management for Project: Exploiting Positive Risk, David Hillson, Taylor & Francis, 2004.
85. Project Risk Management: Process, Techniques, and Insights, 2nd Edition, Chris Chapman and Stephen Ward, John Wiley & Sons, 2003.
86. Managing Project Risk and Uncertainty: A Constructively Simple Approach to Decision Making, Chris Chapman and Stephen Ward, John Wiley & Sons, 2002
87. Technical Risk Management, Jack Michaels, Prentice Hall, 1996.
88. Managing Risk: Methods for Software Systems Development, Elaine Hall, Software Engineering Institute, Addison Wesley, 1998.
89. Software Engineering Risk Management: Finding your Path Through the Jungle, Version 1.0, Dale Karolak, IEEE Computer Society, 1998.
90. Risk Happens: Managing Risk and Avoiding Failure in Business Projects, Mike Clayton, Marshall Cavendish, 2011.
91. Waltzing with Bears: Managing Risk on Software Projects, Tom Demarco and Timothy Lister, Dorset House, 2003.
92. Software Engineering Risk Management, Dale Karolak, IEEE Computer Society Press, 1996.
93. Practical Project Risk Management: The ATOM Methodology, David Hillson, Management Concepts Press, 2012.
94. Risk Management in Software Development Projects, John McManus, Routledge, 2003.
95. Department of Defense Risk, Issue, and Opportunity Management Guide for Defense Acquisition Programs, June 2015, Office of the Deputy Assistant Secretary of Defense for Systems Engineering Washington, D.C.
96. Project Risk Management: Process, techniques, and Insights, 2^nd Edition, Chris Chapman and Stephan Ward, John Wiley & Sons, 2003.
97. Technical Risk Management, Jack Michaels, Prentice Hall, 1996.
98. Software Engineering Risk Management, Dale Walter Karolak, IEEE Computer Society, 1996.
99. Software Engineering Risk Management: Finding Your Path Through the Jungle, Version 1.0, Dale Walter Karolak, IEEE Computer Society, 1998.
100. Managing Risk: Methods for Software Systems Development, Elaine Hall, Addison Wesley, 1998.
101. Risk Happens!: Managing Risk and Avoiding Failure in Business Projects, Mike Clayton, 2011.
102. Probability Methods for Cost Uncertainty Analysis: A Systems Engineering Perspective, Paul Garvey, CRC Press, 2000.
103. A Beginners Guide to Uncertainty of Measurement, Stephanie Bell, National Physics Laboratory, 1999.
104. Practical Risk Assessment for Project Management, Stephen Grey,
105. Assessment and Control of Software Risks, Capers Jones, Prentice Hall, 1993.
106. Distinguishing Two Dimensions of Uncertainty, Craig Fox and Gülden Ülkumen, in Perspectives of Thinking, Judging, and Decision Making

 

 

Related articles

Making Decisions In The Presence of Uncertainty

The Flaw of Empirical Data Used to Make Decisions About the Future

Managing in Presence of Uncertainty

Herding Cats: Decision Making On Software Development Projects

Making Decisions in the Presence of Uncertainty

Managing in the Presence of Uncertainty