Thursday, September 01, 2011

Black Swan Blindness

In my post Black Swans and Complex System Failure, I talked about the architectural implications of some recent disasters, including the Gulf of Mexico oil spillage in 2010 and the partial melt-down in Japanese nuclear reactors following the tsunami in 2011. Both of these disasters involved something that isn't supposed to happen: the simultaneous failure of multiple fail-safe mechanisms.

A new study by Oxford University and McKinsey finds a similar phenomenon in technology investment, where large IT projects may experience spiralling costs as a result of multiple problems occurring simultaneously. According to the researchers, this is up to twenty times more frequent than traditional risk modelling techniques would expect, with one in six large IT projects going over budget by an average of over 200%. Researchers refer to the tendency to disregard rare but high-impact problems/risks as black swan blindness.

As an example, Professor Bent Flyvbjerg cites the collapse of Auto Windscreens, which went into administration in February following a disastrous attempt to implement a new IT system. "Black swans often start as purely software issues. But then several things can happen at the same time - economic downturn, financial difficulties - which compound the risk," he explained.

Professor Flyvbjerg has coined the term Black Swan Management, which currently merits its own Wikipedia page. Simon Moore (author of Strategic Project Portfolio Management) questions whether it is appropriate to use the term "black swan" for something that occurs with a one in six probability, but supports Flyvbjerg's conclusion that when projects go wrong they can go extremely wrong.

Flyvbjerg makes five fairly bland recommendations for avoiding IT project failure, including recruiting a "master builder". Some people may interpret this as an endorsement of the large IT service firms, but these firms have been responsible for some of the most extravagent failures. Is there any evidence that master builders are any more immune from "black swan blindness" than anyone else? Indeed, as a Scandinavian, Flyvbjerg will hardly need reminding of Ibsen's portrayal of madness in the play of the same name.


'Black swans' busting IT budgets (BBC News, 26 August 2011)

Bent Flyvbjerg and Alexander Budzier, Why Your IT Project May Be Riskier than You Think (Harvard Business Review, September 2011, pp. 601-603)

Natasha Lomas, Five ways to stop your IT projects spiralling out of control and overbudget (Silicon.com, 22 August 2011) (pdf)

Brenda Michelson, Complexity, Outliers and the Truth on IT Project Failure (HP Input-Output, 31 Aug 2011)

Simon Moore, Black Swans In Project Management (August 25, 2011)

7 comments:

Ironick said...

Richard,

What Prof. Flyvbjerg seems to be describing appears to be a "black elephant", not a "black swan." Project failures are quite common, even spectacular failures, so they can hardly be "black swans". A black elephant, on the other hand, is "an event which is extremely likely and widely predicted by experts, but people attempt to pass it off as a black swan when it finally happens."

For more on black elephants you can start here: http://bit.ly/n0gGyO .

-- Nick

Richard Veryard said...

Nick may be right about the black elephants, but we need to be careful to distinguish between different logical categories here.

1. There are individual setbacks on any project. Such setbacks are very common, and any competent project organization should be able to contain them.

2. But some combinations of these setbacks may introduce complications that even the most experienced project organization may be ill-equipped to deal with. Each specific combination of these setbacks is very rare, which helps to explain why people are slow in recognizing them and clumsy in dealing with them.

3. But there is an extremely large number of possible combinations of these setbacks, of which a significant proportion produce significantly bad outcomes. Therefore even though the probability of a specific bad combination may be very small (black swan), the probability of some bad combination or other may be uncomfortably high (black elephant).

Roger Sessions said...

The language of "black swan" and "black elephant" is leading us away from the real issue: project failure. We know exactly what causes failure: size. The evidence for this is overwhelming. Big projects fail. Small projects succeed. It is really as simple as that.

So the answer is equally simple. We shouldn't do big projects. We should do small projects.

This means we need to focus our energies on understanding better ways of breaking large projects up into small projects.

Once we understand how to split large projects up into small projects, we won't have either black swans or black elephants. We will only have plain old boring vanilla IT projects that provide value to the business.

Wenlock said...

With the advent of Web 2.0, crowd sourcing and other techniques, I would like to think we will start to see newer project management tools for managing risk.

I am a big fan of the works of Slevin & Pinto, who developed a methodology to help predict the outcome of projects and the ‘reasons why’ for project failure based on critical success factors. See: http://goo.gl/AFpmo

We must monitor a number of critical success factors to ensure a successful outcome and avoid potential black swans. Alas history repeats itself over and over again and we do not use the ideas and tools available to us.

Scale and complexity is also a very significant issue that contributes to time, cost and risk.

Legacy system migrations as an example are a class of project that can be very challenging, as they are often large and can take significant time to complete e, g, often having to migrate sub-systems or transactions one at a time.

The world in terms of business priorities can change very fast.

I like the ‘Black Elephant’ concept that has been mentioned as with hindsight the reasons for failure are often obvious and with a better upfront understanding and communication many could have been avoided and hence my hopes for Web 2.0 tools where anyone within the hierarchy can speak up and shout out.

Richard Veryard said...

Roger says that big projects fail. There are many reasons for this, including the belief that big projects need big and powerful IT companies.

This belief has three consequences. Firstly, the big boys use their influence to make sure that projects (especially in the public sector) are as big and complicated as possible. Secondly, such projects are always awarded to the big boys. And thirdly, when the big boys fail, those responsible can console themselves with the lie that nobody else could have done any better.

Roger Sessions said...

I think Richard is exactly right about the reasons we do big projects. We know they will fail, so why do we do them? The answer is easy. Follow the money.

For large consulting companies, it is far more profitable to do a big project that fails than a number of small projects that succeed. And for executives, it is far more prestigious to be in charge of a large project than a small project. Success doesn't matter, it is always easy to find somebody to blame later.

Things are not going to change until executives are rewarded not for the size of their projects, but for the success of their projects.

Peter Evans-Greenwood said...

Roger is on the money here: the business model behind SIs leads them to prefer larger projects that are prone to failure. Even worse, the people setting up the large project know that they won't be around in two years when it fails, so they have no incentive to set it up for success. Just as the people there at the end were not responsible for setting the project up so the failure is not their fault, and consequently they don't have much of a vested interest in the project's success or failure.

We can expect the same behaviour as long as we insist on tying pay and sales bonuses to project size. Sadly SaaS, which might have solved this problem, is heading the same direction as SaaS vendors try to be like Oracle and SAP, but in the cloud.