By Golo Roden, translated from German with the help of DeepL
Every developer knows the challenge of estimating effort for development code. Very few like to do it. Why is estimating so unpopular, why is it even necessary, and what should you look for?
The question of why it's necessary to estimate effort at all is easy to answer. Knowing how long a task is expected to take is essential for planning who in a team can do what and when. Even across team boundaries, a certain amount of time planning is essential; after all, teams have to be coordinated and resources have to be procured. In addition, other departments such as marketing also have an interest in being involved in planning at an early stage.
Ultimately, behind the task of estimating is simply the need to know approximately when one will be how far along, in order to be able to prepare accordingly so that everyone can work in the flow and no unnecessary waiting times occur. This is a perfectly legitimate approach, which is also necessary in order to work hand in hand. As a rule, everyone understands this and no one has anything against it. So why is the idea to estimate effort rejected so often?
A second aspect usually comes into play here, namely control. When asked how long something will take, developers know that this usually involves a certain amount of commitment, and that sooner or later this will lead to them having to justify why the plan is not being adhered to. And we weren't talking about a plan at all, just an estimate!
In practice, the buzzword "commitment" is often used, which is supposed to disguise the fact that it is a commitment expected from outside: Commitment sounds far too self-determined for that. As a rule, however, it is not.
The point here is the lack of trust. Outsiders are often unable to judge the cost of development, but are expected to manage it. In order to have a basis for this, any numbers are needed (with emphasis on "any"), as these are excellent for planning and managing. And this is exactly where the problem arises - an approximate and inaccurate estimate becomes a binding plan.
What people tend to forget is that an estimate per se must be fuzzy and inaccurate, because that is the definition of an estimate. If one knew exactly, one would not have to estimate. At this point a serious lack of understanding of the require effort becomes apparent, namely that it is not a repeatable process but a creative act, which is why it is called "development" and not "production".
Books like "The Art of Programming" by Donald E. Knuth show that software development can certainly be seen as an art and less as an engineering science. Of course, software development has some of that, namely the creation of structures, but there are also artistic-creative parts. And these are often forgotten.
The point is: it is legitimate to estimate effort in order to coordinate teams, and for that you also need approximate planning. But the question is, how do you do it?
In principle, there are two fundamentally different approaches to estimating: absolute or relative. Estimating in absolute terms means predicting the actual time required. Relative estimates, on the other hand, simply put the individual tasks in relation to one another. A relative estimate does not necessarily have to be related to time, but can also be based on other factors.
What are the arguments in favor of estimating absolute time? First of all, most developers have a reasonable sense of time, provided the task is clear. Conversely, if a task is unclear, it becomes extremely difficult to estimate time, therefore ambiguities become apparent very quickly.
However, the argument against estimating in terms of time is that the estimates often deviate drastically from the time actually required in the end. At first glance, this seems like a contradiction, but it is easy to explain: After all, all too often "code complete" is confused with "feature complete", i.e. when pure coding is complete, the entire feature still needs to be completed: documentation, testing, integration, etc.
In addition, in practice it is generally advisable to add extra time for unexpected tasks, such as debugging.
All of the above often leads to relative estimates. The idea is to choose a reference point and estimate tasks only in relation to it. This could be time, complexity or something else completely. Bullet points, which are often used in connection with Scrum, for example, suggest exactly that: Here, the complexity of tasks is estimated, instead of required time. The problem with this is that a high level of complexity is, however, associated with a high expenditure of time, although this does not necessarily have to be the case.
In practice, developers often secretly convert points into time, since it is not allowed to estimate time, but they can actually handle time much better than an abstract quantity like "8 points". So the estimate happens in absolute time after all, it is just disguised more or less cleverly.
To be able to estimate effectively in terms of absolute time, three important things have to be considered. First, tasks must be small enough to be kept track of. A good rule of thumb for this is 16 hours. Anything that is unlikely to be implemented in that time is too big and needs to be broken down further.
Second, it makes a huge difference who estimates. It highly depends on individual skills, knowledge, experience and the personal approach. In addition, there is the respective mental disposition of the day and of course unexpected external factors. Therefore it should be always those developers who estimate the effort required for a task who will eventually implement it. If estimates are made jointly in a team, agreement should be reached beforehand on how to deal with discrepancies, whether the mean value or the worst case should be used, etc.
Third, one should create an awareness that estimates will be wrong. This is the reason for that extra time mentioned above, which might be used, say, for debugging. However, this extra time should be calculated by multiplying instead of adding up extra time: it must be correspondingly longer for larger tasks. And there, too, if estimated in a team exercise, a common strategy must be considered beforehand how to reconcile different opinions.
If you take all this into account, you have a good starting point for coming up with at least a reasonably reliable estimate. But what happens if you take the estimate as a fixed plan with a rigid time frame?
This is where the magic triangle comes into play, which states that time, cost and scope are mutually dependent. If you change one of the quantities, it has an effect on the other two quantities. For example, less time must result in either an increase in cost or a decrease in scope. Since more cost usually means "a bigger team", but more people are not faster per se (but according to Frederick Brooks may even have the opposite effect), the only thing left to do is to reduce the scope.
Even if this sounds all too logical, it is all too often disregarded in practice. Although there is less time than originally planned, the cost and scope remain fixed - and at first everything seems to be working, because the end of the project is still a long way off. But delays add up and, in the end, time does run out: an unfortunately common "solution" is to increase the pressure on developers.
However, people don't think faster under pressure, which is why this doesn't work. As mentioned at the beginning, software development is development, not production, and the artistic-creative part in particular works much worse under pressure.
Nevertheless, the pressure is there, and it then seeks another outlet, namely quality. This is the only factor that can give when no other aspect can be changed. Of course, this is the worst outcome because then the customer is dissatisfied due to the poor quality, and the software developed in this manner is rife with subsequent maintenance challenges, not to mention the difficulties to develop it further.
This is exactly what is illustrated once again in the so-called "devil's quadrangle": the aspect of scope is divided into two here, into "content" and "quality".
Of course, this does not mean that time should never be fixed - especially when working towards trade fair or conference dates, for example, the time is fixed and cannot be negotiated. In this case, however, you must be aware that you will have to accept either a reduction in scope or a reduction in quality.
Especially if quality is cut, one should account for it with more than enough time to fix the existing deficiencies before using the resulting code as a starting point for further development: time should be allocated to "reduce technical debt".
Otherwise you run the risk of using ugly shortcuts in the code during development because you don't take the time to write the code properly. You deliberately build "workarounds" instead of solutions, adding "TODO" and "FIXME" comments. The initials of the three terms then result in the nice term "WTF" code ...
If such code is to be built on, no one dares to touch the code. Instead a layer put on top, for which again too little time is allocated, which leads to the next layer, lather, rinse, repeat. Each additional layer makes the code exponentially more complex and extensive. The effort for further development and maintenance increases enormously, as does the frustration and disappointment. Maintaining the code becomes increasingly tedious.
This makes development slower and more expensive until the point is reached where further development is no longer profitable. Then the only option is to stand still - or to rewrite everything from scratch, which then also means leaving the field to the competition for several years. Whether a company can survive this remains to be seen.
The only way to avoid this is to proclaim quality as the top priority from the outset and truly to live it. Of course, you're allowed to build prototypes and proof of concepts, but these should always considered throwaway code, in the literal sense.
Software development is, at least to some extent, an art that requires freedom to practice. Developers know this and therefore generally resist estimates that demand binding and exact planning from them. This can only be achieved if a number of important aspects are taken into account when estimating - which then makes estimating enormously time-consuming.
Ultimately, companies should realize that estimates are and must be imprecise and fuzzy, that they can only serve to coordinate teams, but that estimates must never become the plan against which success is measured. In all of this, above all, quality must not be allowed to fall behind, because this is what does the most damage in the long run: when quality is neglected.
This post offers a comprehensive and well-documented analysis of the challenges and considerations in software development. The author's thorough breakdown of different aspects, such as handling pull requests and maintaining stability, provides valuable insights for developers seeking to enhance their workflow efficiency and collaboration. Additionally, to further assist developers in their software development journey, I recommend reading the article "Software Development Time Estimation" (https://www.cleveroad.com/blog/software-development-time-estimation/). It offers practical tips and techniques for estimating project timelines accurately, aiding developers in better planning, resource allocation, and meeting deadlines.