
Software complexity

Complexity is the primary enemy of software design and architecture. The primary job of a software architect is to remove, reduce, or manage complexity in that order. But what if the problem space itself is complex? How do you do that? What does it even mean? What is complexity, anyway?
What is complexity?
There are many definitions of complexity, and there is no single correct definition. It depends on the context (i.e., the scientific field). For the purpose of this article, however, let's define a complex space as one with the following attributes:
- Non-linear causality: You cannot predict the future.
- Interactions between parts of a system change the behavior of those parts in an unpredictable way.
- Any part of the system can start or stop interacting with another part at any time.
Before I examine the complex space, I would like to show you an example of a non-complex space: mechanics of the combustion engine. A combustion engine can be a very complicated piece of technology, but it is not complex; it follows the laws of physics exactly. Yes, people can make mistakes, and the engine can break down. There can be material fatigue, and the engine can break down. However, it is still based on the exact laws of physics, and it is predictable regardless of how complicated it is or whether we have noticed the issues. The interaction between the parts of the engine changes them, but in a predictable way.
Now let’s look at commercial software development:
Do you know if people will like your software? Do you know what changes you will need to make based on their feedback? Can you plan for that? What kind of architecture will best support the software in a year? How will people's demands and wants change? What will the competition's reaction be?
There are actually four types of complexity in software itself (not even accounting for anything else):
- essential complexity
- scale complexity
- support complexity
- accidental complexity
Essential complexity
Essential complexity refers to the inherent complexity of the problem itself. By nature, software that can add two numbers will be simpler than mature spreadsheet software. This has nothing to do with the solution; it's inherent to the problem.
One issue I often encounter relating to essential complexity is the assumption of a solution. "If we're doing this, we have to do it this way." This skips the essential complexity and goes straight to the solution (code). We should understand and address essential complexity before we start talking about code.
The size of essential complexity cannot be changed without altering the scope of the problem - that's what makes it essential. A complex system cannot be reduced in size to simplify things (complicated can). There’s a trade-off: you only move the complexity elsewhere.
So, what makes essential complexity problematic, and how should it be dealt with? Since it is essential and must exist to solve the problem, there are actually only two things to manage.
Our understandment of the problem
This is where practices like EventStorming might help. However, this is not about practices; it's about understanding the domain. It's essential for developers to communicate directly with domain experts, stakeholders, and end users without an interpreter.
There might be a place for a product manager (although there rarely is), however, the PM (or PO) cannot be the gatekeeper of information or the decider of what will be done and how it will be done.
There might be a place for pure coder (although there rarely is), however the ability to write a code is the least important skill of a software engineer.
But what is it for? Why is it important? A developer’s job is to solve the problem at hand, usually by writing code. To do this effectively and efficiently, developers must understand the problem in order to define an appropriate solution. It is (should be) the developer who understands whether it’s going to be simple or difficult, whether there are different solution to the problem, etc. It is through discussion where solution is created, code is just an execution.
Scope of the problem
But let's get technical for a moment. The issue here is the scope of the code. When you have a monolithic code base of 100,000 lines, it's impossible to reason about or define what each part does or know the blast radius of potential changes.
What do we want instead? System of well defined sub-systems with small responsibilities that communicate through API. The goal here is to be able to make changes:
- as long as the change of the sub-system does not alter the API of the sub-system, you do not need to be concerned with anything else
- if the change touches the API, blast radius is defined by consumers of this API
The solutions here are:
- high cohesion, low coupling
- modularization
- separation of concerns
and pretty much any principle or patter related to this.
Next, I will discuss the remaining three types of complexity.