The Cost of a Solution

By | May 10, 2021

Any solution to a problem involves trade-offs. One of my managers a long time ago used to be fond of saying some variation of

There are an infinite number of solutions to a problem. A subset of those will work. A subset of those will work well. A subset of those are actually worth doing.

Paraphrased from Rick Hoselton

When trying to find a solution to a problem, newer programmers spend all of their time trying to find an item in the first subset. More seasoned programmers can identify a number of solutions that will probably work, but mostly spend time working through trade-offs of those solutions. This really is not an criticism of newer programmers. Until you have a certain amount of experience, finding even one solution is a major accomplishment. As you gain experience, thinking of solutions is not as difficult because you have seen more examples of solutions.

Once you can think of more than one solution to a problem, you need to figure out a way to grade the solutions. One obvious criterion is whether or not a given solution will work. After you’ve eliminated the the non-working solutions, where do you go next?

In the various engineering disciplines, solutions are evaluated by their benefits and their costs. Every interesting solution, involves trade-offs between the costs and benefits of different solutions. You need to find a solution with all of the benefits you require, but with only costs you can afford to pay.

Costs

Some obvious costs involve buying software or hardware to solve the problem and spending time to write a solution in a particular programming language. But, those aren’t the only costs.

Other relatively obvious costs are computers to run any programs or store the data, electricity to power the systems, and air conditioning to keep the servers cool. Those people whose job is to keep these systems running could easily list another dozen costs that are associated with these. If you have access to systems with plenty of capacity, or you already pay for some sort of hosting, all of these costs get wrapped up into one bill and you may not think of it.

The next interesting cost that most developers don’t think of is the effort of keeping the software running, monitoring load, storage, and memory; heartbeat checks to confirm servers can talk to each other; network load; etc. These costs are largely invisible to the developer, but can spend time and resources from your operations group. Likewise, infrastructure changes like modifying the network topology or access permissions on certain services can have adverse effects on the development staff. Moving to a web API or micro-service architecture will add additional failure modes to these calls. (An in-process function call is not likely to ever time out, but ignoring time-outs on a network call is likely cause problems.)

When evaluating a solution, remember to consider all of the costs (that you can come up with). These extra costs may not be enough to prevent using a solution, but costs that you push off to another group or programmer are not gone. Someone else is paying those costs.

Consequences

I have seen quite a few people in business, development, and operations discount the cost to other teams when considering solutions.

Business people almost always discount the cost of loss of quality versus the perceived benefit of time to deliver. I’ve seen people make decisions that will make every change going forward 5% harder (just to pick a number out of the air) in order to get a project done a month sooner, even if that deadline is essentially arbitrary. Do that 4 times, and the developers can only implement 80% as much work. So, out of the next 5 things you want done, only 4 will be completed.

Developers may make decisions without regard for the hardware monitoring costs, or the manual time the ops group needs to maintain and monitor systems. Guess what, every time you do that, your ops team is more likely to be busy fighting fires to keep the system running when you need a new system spun up or an access change. A really painful change I’ve seen developers make in the past is changing programming languages, or frameworks. You may not be aware of the problems this will cause to your ops team. For instance, does your great new solution generate incidents because it does not have support for the monitoring system that your ops team uses?

When ops makes a permissions or networking change, one of the effects could be unexpected failures in the system that the developers will need to find time fixing. Or, the change could require code modifications that derail work needed for the business. This generates ill will and a stronger tendency to push back for changes later. It can also result in outages.

Developers often affect other developers by changing APIs or libraries without thinking about the cost. I’ve seen this from both internal developers and the makers of well-established frameworks. Every time a library or framework you depend on changes, you lose time modifying any code that depends on it, or you decide not to upgrade and hope that doesn’t hurt too much. Unfortunately, I’ve also seen this with programming languages. Some languages make a point of retaining backward compatibility as long as possible. Some break code each time they make a big release (every year or two). Most are somewhere in between.

Work with Those Who Will Pay the Cost

These costs are not necessarily a reason to discard a solution. It might be a good to discuss with any other group effected by your solution before deciding if the trade-off is worthwhile. Completely disregarding the cost another group has to pay for your wonderful idea, is a good way to harm your group and generate enemies where you don’t need to.

If you consider the costs to the people affected by your change, you stand a better chance of making the kind of progress you want without burning bridges you need later. There’s even the chance that those other people actually know what they are talking about and might help you avoid a misstep that could have consequences you did not foresee.

Leave a Reply

Your email address will not be published. Required fields are marked *