Solving Complex Problems

Simple solutions to complex problems are possible if we meet the complexity head on, rather than trying to sidestep it or trusting that technology will magic it away for us.

Understanding Complexity

In my previous two posts on this site I wrote about how organisational and technological complexity gives rise to awkward customer experiences and negative outcomes. I've done this partly to illustrate the prevalence of the problem, but also to illustrate that it's a hard problem to solve.

The first obstacle to solving any complex problem is the time and effort required to fully understand it. We all know that simple solutions are the best, but the pressure to create a simple solution can lead to oversimplification of the problem.

We see this in IT all the time. Most readers will have experienced an IT solution failing to deliver its promised benefits. This frequently happens when we take a solution well suited to a specific problem and then stretch it to solve a larger, more complex problem. In doing so we make the solution bigger — and more expensive — so that it no longer represents value for money as a solution to the smaller problem, but it also fails to solve the larger one because the scope was never wide enough to capture the full complexity.

With organisational complexity the problem is exacerbated because we all view the problem through the lens of our own experiences. Some will see a complex, sprawling technology landscape. Others will experience complicated and ineffective processes. A few may struggle with duplicated or overlapping role responsibilities.

To truly solve the problem we must examine it through all of these lenses.

Mapping the Institution

In addition to the above examples of individual complexity, any large organisation will also have a degree of institutional complexity*. That is the complexity inherent in the nature of the organisation and what it does, such as the number of countries in which it operates and how many products or services it provides.

Institutional complexity is not necessarily a bad thing — a business selling a large number of products worldwide is usually happy to continue doing so — but when problems are identified it's important to be able to put them in context.

The most common (and valuable) tool for capturing the breadth of activity engaged in by an organisation is a Business Capability Model. This is a visual representation of all of the activities an organisation must perform — or be capable of performing — to operate successfully and fulfil its purpose.

Example of a simple capability model

Capability models can be simple (as in the example above) or they can go to multiple layers, breaking down each capability into its constituent parts.

There are many methods and approaches to creating a capability model, but the end result must be consistent. The capabilities documented should adhere to the MECE principle: that each capability is mutually exclusive — activities performed as part of one capability are not also performed as part of another — and collectively exhaustive — the model describes all of the activities the organisation carries out, with no exceptions.

Examining Perspectives

While the capability model provides context to the problem, we need a framework to help us uncover and examine its multiple aspects.

The Zachman Framework is one of the oldest and most controversial Enterprise Architecture frameworks in popular use. The controversy around it stems from its attempt to exhaustively detail every component of an organisation in a rigid ontology. However, its most valuable element (in my opinion) is its concept of Enterprise Perspectives.

The framework describes five perspectives of an organisation arranged in a hierarchy from the Executive Perspective at the top to the Technical Perspective at the bottom**. Each layer represents the information understood and required by a particular set of stakeholders.

Crucially, each layer is considered a to be realisation of the concepts and decisions made in the layer above it, so that the broad vision and strategy expressed at the executive level is translated into concrete action and deliverables at the technical level. This means that if a layer of information is missing, the implementor at the level below is forced to make assumptions about the intended outcome***.

Example applying Zachman perspectives to an API product

In the example above, the executives at a widget manufacturer have decided to adopt a strategy of creating an API ecosystem to interact with resellers. The table shows the sort of information that might be required at each layer to implement the strategy and turn it into working software.

If information was missing from the business management layer, the solution architect designing the API behaviours (in the architecture layer) would have to make assumptions about who the customer is, what behaviours are desirable, and what constitutes success. If information was missing from the engineering layer, the software developers would have to make guesses about the best structure of the system.

This is not just theoretical. It happens all the time. If you've never seen a product or project that failed to meet executive expectations you've either had a very short or very fortunate career.

Increasing Agility

None of this means that you need to have an army of analysts, architects and engineers for every single solution. Smaller products or projects will require less work to gather the information at each perspective and it's entirely possible for a single individual to produce the information required across multiple layers. The important thing is to keep it small by constraining scope at every layer, not by removing layers entirely.

It's also worth noting that artefacts produced at higher perspectives will usually be reusable across multiple solutions. An organisation only needs to invest in creating a capability model once. Similarly customer personas, value streams and success metrics at the business management layer will readily apply across multiple products and services.

In this way the effort expended to examine and understand the higher level perspectives actually works to increase organisational agility by reducing the load on discovery and design for each subsequent project.

Embracing Complexity

This is obviously only a brief summary, but I hope that it provides some indication of the benefits of the approach.

We can't afford to ignore complexity, or to blindly trust that new technology will somehow eliminate it.

Tools like automation and machine learning are incredibly powerful, but if we don't understand the processes we're automating — or the data we're collecting — we can find ourselves presented with a very large bill and very little to show for it.

Before embarking on a major new technology project, ask yourself whether you are in possession of all the information required to validate the promised benefits.

If we embrace complexity — if we seek to understand its components from all perspectives — we can find very good solutions to very complex problems.

Footnotes

* This distinction is drawn from a 2010 article by Julian Birkinshaw and Suzanne Heywood, Putting organisational complexity in its place.

** Actually, there are six perspectives. The bottom one (the Enterprise Perspective) represents the physical realisation of the organisation or solution, but it's not really relevant to this article. Also, the Technical Perspective is really called the Technician Perspective but I find that too much of a mouthful.

*** The likelihood of these assumptions being correct is arguable. John Zachman insists it's always zero, but I've argued that it's more likely to be inversely proportional to the size of the organisation. This is one reason startups can often outperform large enterprises at this sort of problem solving.