Why software projects overrun, and how to stop it

Ask anybody who has commissioned software and they will tell you the same story. It was going to take three months. It took seven. Nobody could point at the moment it went wrong, and by the end neither side particularly wanted to talk about it.

The interesting thing is how rarely the code is the culprit. In our experience the overrun was decided long before anybody opened an editor, in a conversation nobody thought of as a decision at the time.

The real causes, in the order they bite

1. Nobody wrote down what was being built

This is the big one and it is not close. Two people leave a meeting genuinely believing they agreed the same thing. The client heard "a booking system". The developer heard "a booking system". Six weeks later it emerges that one of them assumed it would take payments, handle refunds, send reminders and sync to an existing calendar, and the other did not.

Neither party was being difficult. They simply never tested whether their two mental pictures matched, because nothing was written down in a form specific enough to disagree with. A scope document is not bureaucracy. It is the cheapest possible way to find out you were imagining different things, at the point when that discovery costs an afternoon rather than two months.

2. The estimate was a wish

Estimates given in a meeting, out loud, under mild social pressure, are consistently optimistic. That is not a character flaw, it is a well documented cognitive bias, and knowing about it does not make you immune to it.

The fix is unglamorous. Break the work down until each piece is small enough that you have genuinely done something like it before, estimate the pieces, and add the integration work that only exists because the pieces have to talk to each other. That last part is where most estimates quietly lose a fortnight.

3. Nobody owned the decisions

A developer hits an ambiguity on a Tuesday. Who decides? If the answer involves waiting for a fortnightly steering meeting, the developer will guess, because guessing keeps them moving and waiting does not. Guesses accumulate. Eventually one of them is wrong in a way that requires rework, and the rework is discovered late, because nobody was checking.

Every project needs one named person who can answer a question inside a day. Not a committee. A person.

4. Scope crept in through the side door

Scope creep is rarely a dramatic change of direction. It is a series of small, individually reasonable requests, each of which sounds like it takes ten minutes. "Could it also email the manager?" "Could we filter by date as well?" Any one of them is fine. Thirty of them is a second project running invisibly alongside the first, with no budget and no deadline attached.

The answer is not to say no. It is to make each request visible: in scope at no extra cost, or here is the price and here is the revised date. Once a change has a number next to it, everybody makes better decisions about it, including the person who asked.

5. Integration was treated as an afterthought

The demo works. Then it meets the client's actual data, their actual authentication, their actual mail server that rejects anything without a particular header, and the ten year old system that has to receive the output in a format documented nowhere.

Integration work is not the last five per cent. On a project that touches existing systems it is frequently a third of the effort, and it is the third most likely to be missing from the estimate entirely.

What actually prevents it

None of the fixes are clever. They are all things people already know and skip because they feel slow at the start of a project, when the pressure to be seen to be building is highest.

• Write the scope before you write the code. Goals, features, explicit exclusions, assumptions, dependencies. The exclusions matter as much as the inclusions, because they are the ones that surface disagreements early.
• Estimate in pieces, and include the joins. If a piece cannot be estimated, that is not a reason to guess. It is a signal that it needs breaking down further or investigating first.
• Name one decision maker on each side. Ambiguity resolved in a day costs nothing. The same ambiguity resolved in three weeks has already been built around.
• Price every change, including the free ones. "That is in scope, no charge" is still a decision worth recording. Six months later it is the record that settles the argument.
• Schedule the integration work explicitly. Give it real time in the plan, and do it early enough that a nasty surprise is still recoverable.
• Show working software early and often. Not a status report. The actual thing, in a browser, where somebody can click it and say "that is not what I meant" while that is still cheap.

Why we work the way we do

Every Rainbow Coders project starts with a systems analyst rather than a developer. Somebody sits with the client, works out what is actually needed as opposed to what was first asked for, and writes it down as a brief with a scope, a figure and a timeline. Only then does the work go to the people who will build it.

That stage is not free for us. It is the part clients most often expect to skip, and it is the part that removes most of the risk from everything after it. A project that starts with an agreed written scope can still go wrong. It just cannot go wrong quietly, for months, in a direction nobody noticed.

If you have a project that needs this kind of thinking, tell us about it. The first conversation and the written brief that follows cost nothing.