A take on injecting reality into software planning at any stage

A perennial source of angst for me is dealing with plans and deadlines in software. While there are many takes on this from #NoEstimates to good old blame, it’s a common experience that neither the people making the estimates, nor those receiving them are making any use of the information, instead allowing it to become a waste stream with no outlet that slowly toxifies relationships, becoming its own justification and justifying pressure that very often is strictly self imposed.

I’m going to expand here on a basic idea from here

https://twitter.com/prozacchiwawa/status/1302394289800933376

An experience most people who have worked for medium to larger companies in software have is timeline as powerplay; another manager says they can do it faster, our manager thinks they’ll look better with a better estimate, announcing this feature on a certain date works nicely with our media and PR schedule. In any of these cases and many more, extrinsic forces put pressure on a software team to deliver on a specific date for reasons that aren’t related to a need for the functionality itself. This is not as uncommon as one would hope.

That timelines and planning are a necessary aspect of software development are taken as assumed. Somewhere there’s a big ledger and we have to know what each line item is. How is that information used? It’s complicated; meetings are had and the data are discussed as a backdrop. Negotiations, golf games, offsites and late night ungrammatical texts are had. Ultimately, decisions about this data are taken in the moment, according to gut feelings almost 100% of the time.

Even worse, let’s say a company is working to produce a single software title, such as a company producing an MMO, the company lives and dies by this release, but what is actually the purpose of tracking, of planning? Is there any expense the company wouldn’t go to to ship? Is there any circumstance anyone can foresee early in the process that would cause some definite change? We can try to control scope as we go, but development is likely already moving in a strictly priority based order, so it’s unclear what decisions can be made based on macro-scoped information about the project’s timeline that isn’t already covered simply by prioritization. We can ask to cut more corners earlier at times or crunch harder earlier, but it’s unclear what else macro-scale time planning can give us from a dev-team perspective, and those are arguably the least healthy responses one might have.

What I’m going to propose does 2 things:

1. Justifies using an overall timeline for software development by providing responsibilities that flow both up and down the organization and makes this information actionable.
2. Balances the stakes of creating estimates up and down the organization, causing someone pushing to make the timeline shorter to take on greater risk, which is now not normally true.

How to impose responsibility on parties to a project plan without just making everything into an interminable crunch that winds up destroying everything

If it’s ok that pushing for a shorter timeline without changing scope or scaling back the project is completely without risk, then there’s no need for any of this; you just assign every project a symbolic amount of time, roughly in the same order of magnitude to the actual effort and you just keep doing it until it gets done. This is by far the most common way projects are budgeted. Decisions about changes to the project are usually taken in the moment, either near the end of or after the original timeline. I’d argue that in this environment, the project only needs progress reports at the 3/4 point, at the end of the original timeline, and at every 25% overrun thereafter.

If on the other hand, there’s a desire for the overhead of timekeeping, tracking, producing estimates and plans, and for there to be risk associated with pushing for both shorter and longer time, then this is actually a sincere proposal.

To start, before any work is started, we must set a trailing stop for the project. This is the percent overrun that the organization is willing to accept without complaint, and it’d actually be fine for it to be anything from 1000% to -99%. Literally any value in this range can be accommodated since the rest is all math.

Now we make an honest effort to estimate the project as normal, and come up with an estimated time as normal. We try to scale the point values in this estimate so that we have a reasonable belief that we can keep the burndown chart curve above the straight line between 0 time and 0 completion at the beginning, and 100% +allowed_overrun and 100% completion at the end. We can use techniques that are rarely used today, such as building in rampup times when the team needs to change focus that estimate for low point delivery over that time to control the shape of the curve.

The important part is that when (or if) we start the project, with management approval for the indicated timeline, we all agree that we will stop working on the project and accept the idea that the project may fail or be shelved if the lines cross. The one hard and fast rule in this is that work on the project halts and we come back to the table when the burndown chart indicates that we have gone over our margin, regardless of the circumstances, because that distributes risk equally among the stakeholders when speaking about the project’s timeline, and forces a healthy choice (a short break at least) when the project is floundering.

There are two price discovery mechanisms at work here; one is point scaling. If we work on a project for a while and it halts due to this mechanism, we actually know a lot more about our estimates. The dev team has good evidence that the estimates are not conservative enough and we should scale them better. Managers, PMs C-levels know that pushing for unrealistic goals will only land them back in front of the board to try to sell the project again (most likely with a rescaled remaining timeline, since the project is already going over), so smooth operation is made preferable to optimism. We can discover hidden assumptions and flexibilities among the participants when we come back together to discuss restarting the project. Perhaps requirements are more flexible than was originally stated? Perhaps the organization wants to look good on paper in cost terms but ultimately has no option but to do what it takes to complete the project as described. In either scenario, we repeat a simple process to tease this out.

This is a healthy way to start a project regardless of the relationship between players

I believe applying a process like this on top of the considerable estimation and tracking work that are basically universal in software projects can help everyone involved and remove some of the tensions that develop between participants. Restarting a project can be healthy in order to discover the best way to estimate.

Imagine that a project is started, everyone is optimistic and the timeline seems reasonable, but early on we notice that we’ve systematically biased our estimates toward optimism without intending to. A few weeks in, the tripwire goes off and we look at the plan again. Unlike the more traditional scenario where the dev team would face a lot of scrutiny for inconveniencing everyone by making a promise (the original timeline) and then suddenly trying to change it, it’ll be understood that what’s happening is based on data, and is a real event. It’s an agreement everyone made at the beginning, and it’s an easy time for the dev team’s management facing people to say “we’re re-scaling the estimate because it looks like it wasn’t as realistic as we hoped before, and we can be glad we caught this early, not 3 or 6 months from now. The new one is better for having had the opportunity to respond to real world information.”

Ultimately, my hope is that when we have the possibility to impose a burden on one party, we impose risk on another, that’s the framework at play here, and I believe it is not only morally and ethically sound, but also contributes to greater understanding and cooperation between people who may otherwise have very opposed incentives and goals within organizations.