There is a misuse of the terms of statistics and probability in many domains. Politics being one. The #Noestimates advocates are another of the most prolific abusers of these terms. Here are the mathematical definitions. Not Wikipedia definitions. Not the self-made definitions used to support their conjectures, but definitions from professional estimating organizations, handbooks from the US Government, papers, and books from the estimating profession.
The search for credible estimates, forecasts, and predictions are never-ending. There are many sources of error, misunderstanding, misuse and abuse, and sometimes outright manipulation of these numbers. But never are these undesirable outcomes the motivation for removing the need to know something, to some level of confidence about the future cost, schedule, and performance of project work when we are spending other people's money. [8]
The following definitions are used in our Software Intensive System of Systems domain. A number of definitions that can be found in dictionaries are broad and many times not applicable to this domain. The misuse and manipulation of these definitions are just as plentiful as the misuse and abuse of the numbers themselves. So when you hear a manipulated definition used to support some conjecture, ask why do you need to change an established definition for an established domain? The answer can usually be found by following the money - someone is trying to sell you something - a book, a conference, a consulting engagement - that is counter to the established principles of Microeconomics of decision making in the presence of uncertainty.
Estimates
- An Estimate is a value inferred for a population of values based on data collected from a sample of data from that population. The estimate can also be produced parametrically or through a model simulation (Monte Carlo is common, but Method of Moments is another we use).
- Estimates can be about the past, present, or future.
- We can estimate the number of clams in the Pleistocene era that are in the shale formations near our house.
- We can estimate the number of people sitting in Folsom Field for last night's game against Utah. The Buff's won and are now the PAC-12 South Champs.
- We can estimate the total cost, total duration, and the probability that all the Features will be delivered on the program we are working for the US Government. Or ANY software project for that matter.
- Estimates have precision and accuracy.
- These precision and accuracy values have estimates as well.
- The estimated completion cost for this program is $357,000,000 with an accuracy of $200,000 and a precision of $300,000.
- Another way to speak about the estimated cost is This program will cost $357,000,000 or less with 80% confidence.
- An estimate is a statistic about a whole population of possible values from a previous reference period or a model that can generate possible values given the conditions of the model.
In the project domain, an estimate is a calculated approximation of some desired measurement.
This is usually a cost, a completion date, a performance measure used in a closed-loop control system to keep the project GREEN while delivering the needed Capabilities to produce the Value for the customer at the needed time for the needed cost.
Showing up late, over budget, and with missing Capabilities is not what the customer paid for.
Forecasts
- Forecasts speculate future values for a population of possible values with a level of confidence, based on the current and past values as an expectation (prediction) of what will happen:
- This is the basis of weather forecasting.
- If you listen carefully to the weather forecast it says there is a 30% chance of snow next week over the forecast area.
- We live at the mouth of a canyon at 5,095' and if there is a 30% chance of snow in Boulder (8 miles south), there is a much lower chance in our neighborhood.
- Business forecasts, weather forecasts, traffic forecasts are typical. These forecasts usually come from models of the process being forecast. NOAA and NCAR are in our town, so lots of forecasting going on. Weather as well as climate.
- Not so typical to Forecast the cost of a project or forecast the delivery date. Those are estimated values.
- For example, a financial statement presents, to the best of the responsible party's knowledge and belief, an entity's expected financial position, results of operations, and cash flows. [2]
- In a forecast, the assumptions represent expectations of actual future events.
- Sales forecasts
- Revenue growth
- Weather forecasts
- Forecasts of cattle prices in the spring
A forecast is a prediction of some outcome in the future. Forecasts are based on estimating the processes that produce the forecast. The underlying statistical models (circulation, thermal models) of weather forecasting are estimates of the compressible fluid flow of gases and moisture in the atmosphere (way oversimplified).
Projections/Predictions
- Projections indicate what future values may exist for a population of values if the assumed patterns of change were to occur. Projections are not a prediction that the population will change in that manner.
- Projected revenue for GE aircraft engine sales in 2017 was an article in this week's Aviation Week & Space Technology.
- A projection simply indicates a future value for the population if the set of underlying assumptions occurs.
Prediction is important because it connects subjective and objective reality.
— Nate Silver, The Signal and the Noise
Project cost, Schedule, and Technical Performance Estimates
All projects contain uncertainty. Uncertainty comes in two forms - aleatory (irreducible) and epistemic (reducible). If we're going to make decisions in the presence of these uncertainties, we need to estimate what their values are, what the range of values are, what the stability of the underlying processes that generate these values are, how these values interact with all the elements of the project and what the impact of these ranges of value will do to the probability of success of our project.
Project decisions in the presence of uncertainty cannot be made without estimates. Anyone claiming otherwise does not understand statistics and probability of outcomes on projects.
As well anyone claiming estimates are a waste, not needed, misused by management, or any other dysfunction, is doing them wrong. So as we said at Rocky Flats - Don't Do Stupid Things On Purpose. This means when you do hear those phrases, you'll know they are Doing Stupid Things on Purpose.
And when you hear, we don't need estimates we need a budget, remember:
In a world of limited funds, as a Project Manager, Product Owner, or even sole contributor, you’re constantly deciding how to get the most return on your investment. The more accurate your estimate of the project cost is, the better able you will be to manage your project’s budget.
Another example of not understanding the probability and statistics of projects and the businesses that fund them is, there are two estimates needed for all projects that operate in the presence of uncertainty:
- Estimate at Completion (EAC)
- EAC is the expected cost of the project when it is complete.
- This can be calculated bottom-up from the past performance and future projections of performance for the work of the project - which in the future will be an estimate.
- Estimate to Complete (ETC)
- ETC is the expected cost to complete the project.
- The ETC used to calculate the EAC
- EAC = Actual Costs to Date (AC) + Estimated Cost to Complete (ETC).
- EAC = Actual performance to date / Some Index of Performance.
This last formula is universal and can be used no matter the software development method.
- Agile has such a formula - it's called the Burn Down Chart. We're burning down story points at some rate. If we continue at this rate, we will be done by this estimated date.
- Same for traditional projects. We're spending at a current rate - the run rate - if we keep spending at this rate, the project will cost that much.
- Earned Value Management provides the same EAC and can also provide an Estimated Completion Date (ECD).
- Earned Schedule provides a better ECD.
Wrap-up
Nothing in progression can rest on its original plan — Thomas Monson [11]
All project work is driven by uncertainty. Uncertainty is modeled by random variables. These variables can represent aleatory uncertainty or epistemic uncertainty. This uncertainty creates risk and as stated here often
Risk Management is How Adults Manage Projects - Tim Lister
So if you hear the conjecture that decisions can be made in the presence of uncertainty without estimates, you'll now know that is a complete load a crap, run away.
If this topic interests you here's a Bibliography of materials for estimating and lots of other topics in agile software development that is updated all the time. Please read and use these when you hear unsubstantiated claims about estimating in the presence of uncertainty. Making estimates is our business and this resource has served us well over the decades.
Other Resources
Cost is a driving consideration in decisions that determine how systems are developed, produced, and sustained. Critical to these decisions is understanding how uncertainty affects a system's cost. The process of identifying, measuring, and interpreting these effects is known as cost uncertainty analysis. Used early, cost analysis can expose potentially crippling areas of risk in systems. [13]
- Australian Bureau of Statistics
- Assuring Credibility in Cost Estimates
- Assuring Credibility in Cost Estimates (Presentation)
- Financial Forecasts and Projections, AT §301.06, AICPA, 2015
- Earned Value Management in EIA-748-C
- Earned Schedule uses the same values as Earned Value, to produce an estimated completion date, www.earnedschedule.com I started using ES at Rocky Flats to explain to the steelworkers that their productivity as measured in Budgeted Cost for Work Complete (BCWP or EV) means they are late. ES told them how late and provides the date of the projected completion of the planned work
- Project Management Analytics: A Data-Driven Approach to Making Rational and Effective Project Decisions, Harjit Singh, Pearson FT Press; 1st Edition, November 12, 2015.
- The Psychology of Cost Estimating, Andy Prince NASA/Marshall Space Flight Center, Engineering Cost Office, June 10, 2015
- Parametric Estimating Handbook, Fourth Edition
- Bottom-Up Methods for Estimating Software, James R. Black, 2017 ICEAA Professional Development & Training Workshop, June 2017
- Project Management Analytics: A Data-Driven Approach to Making Rational and Effective Project Decisions, Harjit Singh, FT Press Project Management, Nov 12, 2015
- Forecasting: Methods and Applications, 3rd Edition, Spyros Makridakis, Steven Wheelwright, and Rob Hyndman, John Wiley and Sons, 2005.
- Probability Methods for Cost Uncertainty: A Systems Engineering Perspective, Paul Garvey, CRC Press, 2000.
- Managing Project Risk and Uncertainty: A Constructively Simple Approach to Decision Making, Chris Chapman and Stephen Ward, 2002.
- Decision Analysis for the Manager, Rex Brown, Andrew Kahr, and Cameron Peterson, Holt, Rinehart, and Winston, 1974 International SEries in Decision Analysis.
- Decision Analysis for the Professional, 4th Edition, Peter McNamee and John Celona, SmartOrg, Inc., 2007.
