The software industry has had to deal with the issue of software estimation for years. There are many different opinions on how software estimates should be handled, and there is no common understanding of what works best. This post discusses why software estimations often fail and provides advice on how organizations can use them more effectively to make better decisions when it comes time to plan projects.
Project Estimating: What You Need to Know
The aim of this essay is to explain how to use estimation before a project begins. We don't address techniques for projects that are already underway.
What is an estimate?
An estimate should be an unbiased analytical process. The purpose of estimation is to provide an understanding of the effort and time required to complete a project or task. This information should be used to make better decisions about which direction to take.
"It is very difficult to make a vigorous, plausible, and job-risking defence of an estimate that is derived by no quantitative method, supported by little data, and certified chiefly by the hunches of the managers." — Fred Brooks
An estimate is not a target. It's not a commitment or a goal that needs to be met. An estimate is not a quote; it's an analytical judgment, not a negotiation tool. The usage of estimates as a tender offer and selecting the smallest is a typical executive's antipattern. It should not be used for this purpose.
Most of the predictions are incorrect, and the problem is not neutral. The software business has a bias toward underestimation, according to industry statistics.
Using the tiniest estimate's size as a reason for selecting a development partner merely results in making a decision based on incorrect information. There are many options to pick before the project begins, ranging from trade-offs over features even to project cancellation if the most realistic estimate isn't acceptable. If the predicted budget has already been wasted by lousy forecasting, it will be too late to make such judgments.
"No plan survives contact with the enemy." – Field Marshal Helmuth Graf von Moltke
How do estimates go wrong?
There have been numerous researches on the subject during decades of software development. Let's go through them one by one in chronological order.
The cult book from Fred Brooks, released in 1974, identifies famous and fundamental Brook's law, according to which "adding manpower to a late software project makes it later." He also says that he has a rule of thumb for successfully scheduling the project: planning 1/3, coding 1/6, and testing 1/2. According to the book, only coding is often correctly estimated, which makes up 1/6 of the overall project. Of course, it's for the outdated heavy Waterfall approach, but many organizations still use it. Modern Agile, on the other hand, does not promise quicker development; instead, it encourages focusing on critical features and tiny releases while reducing waste.
According to fundamental work published in 1992 by Albert L. Lederer and Jayesh Prasad, approximately two-thirds of all projects significantly exceed their estimates.
The Standish Group has published yearly reports on software project success and performance since 1994. The first study caused a stir in the programming world: only 16% of projects were completed on time and budget, while over half were under-estimated, and nearly one-third completely failed and abandoned. The average cost overrun was approximately 100%, and the average schedule overrun was 120%. But the accuracy is undoubtedly worse than those figures suggest. The Standish Group discovered that late projects frequently discarded a lot of functionality to fulfill their deadlines and budgets.
We see 30% of successful projects and 20% completely failed ones in CHAOS Report 2015. It's incredible: 70% of all software projects fail to satisfy expectations. This is not all; according to the study, only 11% of Waterfall projects were successful, with a 29% failure rate, which is nearly verifies the 1994 research. On the other side, we have Agile projects with a success rate of 40% and a failure rate of 9%. Imagine if processes that don't even require upfront estimation perform almost four times better?
The fascinating thing is that both reports stress the importance of user and executive management engagement. The essential success factor in 1994, the presence of clear requirements, was eliminated in 2015. Software developers sought to work with uncertain and changing requirements using Agile to tackle this problem.
Steve McConnell, the author of a software estimation bestseller published in 2006, offers fascinating data from one company.
A diagonal line symbolizes perfect scheduling accuracy. The graph should ideally show data points clustering closely around the line. Rather, the overwhelming majority of the data points displayed are above the line, indicating project overruns. "The common definition of estimate is the most optimistic prediction that has a non-zero probability of coming true." - Steve McConnell refers to Tom DeMarco's "Waltzing with Bears," another software estimation bestseller.
One of the book's central themes is that targets can only be achieved on controlled projects. Removing non-essential features, redefining criteria, replacing less-experienced personnel with more-experienced people, and so on are typical project control activities. To do so, the project manager must be able to alter the project to match the estimate.
According to a 2012 study from the global consulting firm McKinsey, software development projects have an average cost overrun of 66%, a timetable delay of 33%, and a benefits shortcoming of 17%. They discovered that overall, the investigated IT projects had a cost overrun of $66 billion, more than the GDP of Luxembourg.
What if the reason for missed expectations is poor management and estimating skills? Let's look at NASA, which indeed uses excellent practices and has good estimating abilities. The Government Accountability Office (GAO) has conducted an annual review of the status of NASA's key projects. According to the researchers, the average cost increase was up to 47 percent. In 2004, the Congressional Budget Office reviewed the first and revised budgets of 72 NASA projects stretching from 1977 to 2000 in order to compare them. The original budgets for these initiatives were $41.1 billion, whereas the amended budgets were $66.3 billion, a 61 percent increase.
I'm not sure if NASA's exact cost overrun is relevant data for a typical software project. The why behind the increase is far more fascinating than its actual cost. NASA has never found an answer to the initial overrun problem since the 1970s; they've simply discovered new causes.
The causes are as follows: faulty definitions before the budget was made, overly optimistic cost predictions, inability to execute the initial schedule, technical difficulties, and scope changes. These reasons are generally seen on each project and at every company.
There's no need to waste time and money attempting to fight these issues; far better to accept them, minimize damage, and turn them into advantages.
What estimate is good?
What is an acceptable estimate when estimates are frequently incorrect, and we can't eliminate the causes of error? It should be honest, accurate enough, and precise enough to enable a decision to be made.
If an estimate isn't honest, if it's made to match the executive's preconceived notion, it's pointless and harmful. It does not let you make a decision; rather, it supports a false one.
If the estimate is inaccurate and the error is measured in many orders of magnitude, the decision will be incorrect. The primary objective is to make an accurate call regarding whether or not to start a project. A good estimate also may help to determine which features are not essential to complete in order to fulfill a financial goal. The first estimate should not be relied on to decide what to do, simply because the requirements will alter.
If an estimate is not precise enough, and if the range is too broad, it simply cannot be used to make a decision owing to high uncertainty. Such an estimate is of no use.
But how will the executive know whether or not the estimate is sufficient?
We can't read minds, but businesses cannot influence honest predictions. An honest estimator will always be willing to cut the scope to fit the budget but never offer to finish features in fewer hours.
In his book, McConnell offers a simple test to evaluate estimation abilities. Fill in the upper and lower bounds that, in your opinion, give you a 90% chance of containing the correct value for each question. Please do not search for answers—the purpose of this quiz is to test your forecasting abilities, not your research ability. The answers will be listed at the end of this article.
The average number of correct answers for the test takers is 2.8. Only 2 percent of quiz takers get 8 or more answers correct. Most people's intuitive feeling of "90 percent confident" is comparable to "30 percent confident."
Instead of improving our guessing skills, eliminating guessing as much as possible is much more effective. We could count and then calculate instead of guessing. A simple formula will help:
Effort = NumberOfRequirements * AverageEffortPerRequirement
If we have more than ten requirements in a project, the law of large numbers is effective and estimating each one is unnecessary. If the project isn't first for the organization, historical data is already available to determine the average number of hours per requirement.
Of course, some activities may require much more time than the average job. Having a previous history of similar work would be fantastic.
If we don't have enough historical data and the task cannot be characterized as "average," we may utilize a guess. Using a PERT formula, a three-point estimate is a good practice in this situation:
PERT = (OptimisticEffort + 4*MostLikelyEffort + PessimisticEffort) / 6
Three points are an optimistic scenario, when we anticipate that everything will work out; a most likely scenario, when something can go wrong; and a pessimistic case when everything is certain to fail. Of course, we must keep in mind the typical estimating quiz result.
Working with precision
It's important to remember that an estimate is never a single figure; it's a range of possibilities. If we use PERT, the range should be determined utilizing standard deviation. However, historical error data is required to calibrate this deviation, making the process more complex.
In 1981, Barry Boehm created the first version of what Steve McConnell dubbed the "cone of uncertainty." This tool will assist us in determining what range can estimate various project phases.
It's critical to note that the Cone of Uncertainty shows the maximum probable accuracy in software predictions at various project phases. The Cone is a graph that depicts the error in estimates produced by skilled estimators. It's easy to do much worse. There's no way to be more correct; all you can do is hope for better luck.
Another important thing to recognize is that if the project isn't well managed, or if the estimators aren't skilled, estimates may never improve. When the focus of a project isn't on reducing uncertainty — it's not a Cone but rather a Cloud that lingers until the end of the project.
What does it imply for an executive to review an estimate? Assume that an estimate is frequently provided before the requirements are finished. Because requirements are known to alter throughout the course of a project, there's no need to nail them down precisely. The cone states that the best achievable precision on this project stage is 0.67x to 1.5x. It means that if the upper bound isn't at least 2.24 times larger than the lower bound, the estimate isn't accurate enough or isn't fair.
How should we define requirements if we know they will change throughout the project? There is a fantastic agile development technique known as the user story. A user story is not even a formal requirement; rather, it's a promise to communicate later. For example: "A user can add a product to the cart" or "A user may pay for an order." The word "INVEST" is a popular acronym that signifies how user stories should be created.
Independent. It should be feasible to rearrange the order of user stories.
Valuable. It should be obvious why the user story is needed.
Estimable. It should be feasible to estimate each user story.
Small. It should be feasible to create many user stories in one iteration.
Testable. Whether the user story is complete or not should be evident by performing the user's action.
Here are some of the most common blunders that often result in under-estimates.
Loss of a project goal
Is it true that all of the project's goals have been satisfied by user stories? Before estimating, it's a good idea to write up a project goal statement.
Loss of user
A variety of users typically utilizes a system. It's wonderful that "Buyer can pay for the order" exists, but what about "Admin may fulfill the order," and "Executive may see a number of orders?"
Loss of non-functional requirements
There are a variety of non-functional criteria that are also known as quality characteristics. Karl Wiegers notes that improving one non-functional requirement may negatively influence another.
For example, we have a checkout store that needs additional security. At the same time, security harms performance. But it's evident that the checkout should be as quick as possible. Conflicts like these might result in an exponential rise in the effort, so discussing extra quality characteristics before estimation is a good idea.
Loss of transition requirements
Although a project is always moving from A to B, new sorts of requirements emerge along the way. For example, if we introduce a new system while keeping the customer and order data from an older system. In this situation, we must migrate the data to implement the project successfully. This is a transition requirement. Another example is instructing staff how to utilize the system. Unfortunately, these sorts of requirements are almost always under-estimated.
It is essential to understand that an estimate is not a commitment or quote; it's simply an analytical process. Estimates should only be used to assist with project decisions not as a tender bid. Here are some tips for making business estimates more useful:
Around 70% of projects do not fulfill expectations.
It's impossible to remove the factors that contribute to underestimating, such as scope changes, optimism, and unforeseen technological difficulties.
A good guess is honest, precise enough, and accurate enough to enable you to decide on whether or not to begin a project.
Counting and calculating is a lot more accurate than guessing.
If the upper bound of the anticipated range is not at least twice as big as the lower, it's a bad estimate.
Exact requirements will vary, so defining them in detail before starting a project does not aid in estimating. User stories are sufficient.
Common causes of significant underestimation include loss of goals, users, non-functional, and transition requirements.
I hope this post is more than a piece of content but also serves as a manual for executives who must assess them.
Answers to quiz:
Surface temperature of the Sun - 10,000°F/ 6,000°C
Latitude of Shanghai - 31 degrees North
The year of Alexander the Great's birth - 356 BC
Total value of U.S. currency in circulation in 2004 - $719.9 billion
Area of the Asian continent - 17,139,000 square miles 44,390,000 square kilometers
Total volume of the Great Lakes - 5,500 cubic miles 23,000 cubic kilometers 2.4 x 10^22 cubic feet 6.8 x 10^20 cubic meters 1.8 x 10^23 U.S. gallons 6.8 x 10^23 liters
Worldwide box office receipts for the movie Titanic - $1.835 billion
Number of book titles published in the U.S. since 1776 - 22 million
Total length of the coastline of the Pacific Ocean - 84,300 miles 135,663 kilometers
Heaviest blue whale ever recorded - 380,000 pounds 190 English tons 170,000 kilograms 170 metric tons
The Problematic Development of UI-Centric Software
- January 26, 2022