IT analysis and trust

@mkrigsman asks "Trust is the currency that matters most. How many analysts / bloggers deserve it?"

@markhillary replies "surely in the same way as a journalist is trusted, by earning it"

@mkrigsman is particularly concerned about those who write about IT failure. (I'm not sure why he singles out that topic, but I note that the concern arose during a conversation with @benioff, boss of Salesforce.) "When someone writes on IT failures ask "What's their angle?". Usually sensationalism, currying favor, or threatening a vendor." When challenged about his own angle by @njames, @mkrigsman replies "I want to expose *why* projects fail, so we understand magnitude of the problem and can improve."

Trust is clearly a difficult issue for software industry analysts. Unfortunately, Michael's answer to Nigel's challenge cannot prove that he doesn't have a hidden agenda, because the untrustworthy are often just as able as the trustworthy to produce a plausible cover story. If we trust Michael it's not because he can answer the challenge but because of his track record.

We also need to ask - trusted by whom. Software companies might prefer industry analysts to be compliant and predictable, but intelligent software users might regard such analysts as being insufficiently independent. Who would you trust to tell you about Microsoft's new platform -  someone who is always pro-Microsoft, someone who is always anti-Microsoft, or someone who has a track record of making both positive and negative comments about Microsoft and its competitors?

Of course, this comment doesn't only apply to industry analysts. Robert Scoble, when he worked for Microsoft, made a point of distancing himself from the party line, and he therefore commanded a different kind of attention and respect than did Bill Gates or Steve Ballmer.

From a simplistic software industry perspective, an analyst who talks about IT success might be regarded as a friend, whereas an analyst who talks about IT failure is potentially an enemy. (This might explain Marc Benioff's wish to challenge the hidden agenda of the latter.) While many software and service companies might adopt the from-failure-to-success rhetoric - "the best way to avoid the risk of failure is to buy our software and hire our consultants" - this is not ideal from a sales perspective.

Mark Hillary appeals to a journalistic ethic, which would presumably include things like balance and transparency. But balance is not always appreciated by those with most at stake. In the past, I have written technology reports on new products, which I regarded as generally positive with a few small caveats. (I don't generally waste my time writing about products that are no good.) But the vendors concerned have often regarded my remarks as highly critical. (Fortunately, this over-sensitivity on the part of software companies is now changing, thanks in part to social media, and companies now understand that a robust debate can be just as beneficial as a highly controlled one-way marketing exercise.)

From a narrow software industry perspective, a trustworthy industry analyst is one who satisfies Simon Cameron's definition of an honest politician - "one who, when he is bought, will stay bought". But from a broader perspective, we should surely prefer to trust those industry analysts with independently critical mind, unafraid to ask awkward questions and publish the answers.

With the large industry analysis firms, the question of trust shifts from personal integrity to corporate integrity. The sales pitch for these firms depends not just on isolated flashes of insight from individual analysts, but on the collaborative intelligence of a community of analysts. Corporate integrity depends not just on transparency about the relationship between the work paid for by software vendors and the independent research consumed by CIOs, but also on a coherent and robust research methodology adopted consistently across the firm, typically supported by an apparatus of surveys and structured questionnaires and checklists and spreadsheets. However, there is a potential disconnect between the routine processing of supposedly objective raw data (this product with this market share in this geography in this time period) and the generation of useful interpretation and opinion, which is where the analytical magic and subjectivity comes in. One example of this magic, Gartner's Magic Quadrant, has been challenged in the courts; Gartner's defence has been that MQ represented opinion rather than fact. (See my post The Magic Sorting Hat is Innocent, Okay?) And the complicated relationship between fact and opinion, and the transparency of reasoning and evidence, is surely relevant to the level of trust that can be invested by different stakeholders in such analyses.

By the way, why am I writing about software industry analysis? Obviously, because I want to expose *why* analysis fails, so we understand magnitude of the problem and can improve. How can software industry analysis deliver greater levels of intelligence and value to the software industry as a whole?