Managing complexity

A few weeks ago Joel warned you that there would be occasional guest posts - I am the first volunteer. The brief bio on beta.tech.lds.org should provide you with some understanding of my experience and biases. In this post, I leverage those experiences and biases to offer some observations about complexity....

One of the attributes I failed to develop during my academic training was a proper appreciation for the perils of complexity. I recall creating code that was fast, efficient, and utterly un-maintainable. In the academic context this seemed fine, since I was typically the only one maintaining the code, it was rarely used beyond the end of the class, and system failure affected only my grade. However, over the past 15 years my professional experience has changed my perspective and caused me to value simple, understandable, and maintainable solutions over those which lack the foregoing traits yet are fast, efficient, and theoretically “robust.” I offer the following observations relative to the impact of complexity on the reliability, maintainability, and scalability of the systems we create.

• Humans cause failures. In my experience, human failure is a more likely cause of downtime than the failure of a physical component. However, we often try to increase system availability by adding redundancy to mitigate for component failure. This redundancy has the collateral impact of adding complexity, which unfortunately increases the likelihood of human failure. I’m not sure we always end up net positive on the availability scale.


• Control planes, which manage highly redundant environments, are often themselves single points of failure. The likelihood of control plane failure generally increases as the component redundancy becomes more complex.


• Failure modes are difficult to predict and detect. As a result, sometimes secondary components go unused during primary component failure.


• With redundant systems, we assume that the joint probability of multiple independent failures is small. Unfortunately, the assumption of independence is often incorrect.


• Complex systems are difficult to scale. RFC 3439 quotes Mike O’Dell, the former Chief Architect of UUNET, as saying, “Complexity is the primary mechanism which impedes efficient scaling, and as a result is the primary driver of increases in both capital expenditures and operational expenditures.”


• And finally, an observation by Willinger and Doyle et al: “…we point out a very typical, but in the long term potentially quite dangerous engineering approach to dealing with network-internal and -external changes, namely responding to demands for improved performance, better throughput, or more robustness to unexpectedly emerging fragilities with increasingly complex designs or untested short-term solutions. While any increase in complexity has a natural tendency to create further and potentially more disastrous sensitivities, this observation is especially relevant in the Internet context, where the likelihood for ‘unforeseen feature interactions’ in the ensuing highly engineered large-scale structure drastically increases as the network continues to evolve. The result is a complexity/robustness spiral [i.e., robust design → complexity → new fragility → make design more robust → …] that—without reliance on a solid and visionary architecture—can easily and quickly get out of control.”

What can be done to help manage complexity or mitigate its impact? I doubt there is a silver bullet, but the following concepts have been helpful to me.

• Use a crutch to force yourself to remember what is important. My crutch was a note hanging on the side of my monitor to remind me that supportability, maintainability, and reliability were more important than performance and efficiency – not that the last two were not important – they were just not the most important.


• Document what you are doing as you are doing it. If your solution is simple it should be easy to describe. Consider the documentation process a litmus test for simplicity.


• Avoid tight coupling and interdependence. Focus on isolation, separation, and modularization.


• You are more likely to be successful tailoring your system to the capabilities of your operators than tailoring the capabilities of your operators to your system.


• Use automation, but continue to be vigilant about managing down the underlying complexity that the automation is abstracting.


• This one is going to be controversial: sometimes you have to make hard tradeoffs in which you abandon some amount of functionality (and possibly redundancy) to maintain simplicity. This involves understanding the difference between what you can do and what you should do.


• "Make everything as simple as possible, but not simpler."  --  Albert Einstein.

How has complexity manifest itself in the environments in which you work? What are you doing to manage it? Is it hypocritical for a complex post to extol the virtues of simplicity?