Monday, September 21, 2009

IT Complexity

According to @RSessions, "IT Complexity is a tax paid by everybody that benefits nobody."

Well not quite. @neilwd lists a few exceptions: vendors, integrators, analysts.

Roger's mission to eliminate complexity is fraught with two major difficulties. First, the number of powerful stakeholders who appear to benefit from complexity. Neil mentions a few obvious ones.

And second, the wide range of factors that cause complexity. Some forms of complexity may be justified in terms of requisite variety. Some forms of complexity emerge from a history of decisions that made sense at the time. And some forms of complexity may actually be caused by clumsy or one-sided attempts to eliminate complexity - for example, arrangements to protect one party in a procurement contract (especially in the absence of trust).

@patrickdlogan offers an analogy for IT complexity - the levees in New Orleans: no budget to fix them, but just enough budget to work most days, make more complex. Roger replies: Good analogy. But unlike most IT projects, the New Orleans' Levees worked at least for a while.

Actually, most IT projects work after a fashion. Where IT projects are regarded as failures, this is usually because something has gone wrong with such factors as cost and quality (value-for-money). One of the lessons for IT from Hurricane Katrina is about the ability of a large integrated system to withstand unexpected events. See my posts on Efficiency & Robustness 1, 2. See also my post on Processing New Orleans mail after Katrina.


Roger Sessions said...

Hi Richard,
I agree that the powerful interests like complexity. Complexity supports the status quo. I don't agree with you on the other factors.

I define the best architecture as the simplest possible architecture that meets the business needs. Any architecture that is more complex than the simplest possible architecture is overly complex.

I sometimes describe complexity as like heat. We need heat to survive, but the right amount of heat. Too little heat and we fail to solve the business problem (staying alive). Too much heat and we fry.

Richard Veryard said...

Thanks Roger. Your comment raises two questions.

(1) Powerful interests may benefit from complexity, but they generally don't admit to deliberately causing it for its own sake. What pretexts and mechanisms do they use to create the kind of complexity that will be useful to them? Do you agree that they sometimes blunder into creating complexity that damages their own interests as well?

(2) What other causes of complexity do you see? Do you agree that complexity sometimes results from well-intentioned but poorly coordinated actions? Do you agree that complexity sometimes results from unanticipated change?

Roger Sessions said...

How do the powerful interest justify complexity?
- We don't have the money to do the upfront work necessary to deal with complexity.
- We aren't really making a big change, so we don't need to worry about what will happen to complexity.
- We'll deal with it after the project is completed, if complexity turns out to be a problem.
- We know complexity is a problem, that is why we put our best people on the project.
- This project is too critical. We can't afford to try anything new. [even though their old approaches are known not to work!]
- This project is too complex. [yes, I see the irony]
- We're doing an SOA, so we don't have to worry about partitioning. [most SOAs are way overly complex]
- Our business people don't have the time to get involved.

The unspoken reason is often: I don't care if the project fails, as long as I am not blamed.

My experience is that those in power rarely hurt themselves with complexity. They are usually adapt at making sure somebody else gets blamed for failures. Large consulting firms bank on this fact.

And as far as intentions... I have never seen complexity caused by anything other than the best of intentions. The people designing and implementing the systems are doing their best. Little do they know they were doomed before they wrote their first line of code.

Paula Thornton said...

Mixing metaphors: complexity vs. complexity. There's the near-sighted version of complexity, that sees it as a mess and difficult to deal with and there's the far-sighted complexity that see's the underlying pattern of structure that makes it work.

IT is really bad at seeing what works. They do so because they hire no architects (read the job descriptions...they're all tool jockeys).

Classic case: MCI Friends & Family. We're talking 3MIL rows of data. New theory (circa mid 90's) said client-server beat mainframe for performance. The switch to client-server crashed and burned. The parts were never evaluated for the 'whole'. The 'facts' presented were not relevant to the unique implementation MCI already had in place that was 'far superior' to the 'norms' used in the industry assessments.

IT is driven by individual projects, driven by the budgeting mechanism. You can't change anything until you change the financial model.