Monday, June 22, 2009

Industry Analyst Coverage

@mcgoverntheory (James McGovern) complains about the completeness, balance and objectivity of industry analyst coverage. He believes that certain areas are neglected (security, open source), and attributes this to a commercial bias.
  • How important is it for industry analysts to include security analysis in their SaaS research?
  • Does non-commercial open source have a fighting chance to be mentioned by industry analysts to their customers? 
  • How can customers understand analyst transparency when it comes to coverage of non-commercial open source?

James has always been particularly exercised about the fact that OWASP lacks coverage. When he raised this issue with me last year, I responded by posting some questions on the OWASP wiki and the OWASP Linked-In group, as well as several posts on this blog. I'm still waiting for answers.

If there is something in the product offering from any of the large vendors that I don't understand, I can contact one of my analyst relations "minders" and get a reasonably quick answer. If it's a small vendor, I can usually get an answer straight from the CTO. In contrast, my questions to OWASP go into a black hole. One person even suggested that if I wanted to know something about OWASP I needed to start a project. No thanks. (And, to answer Jim's comment below, I don't want to join a mailing list either.)

Industry analysts simply cannot invest that amount of time in chasing non-existent information. If OWASP wishes to be taken seriously by industry analysts, then it needs to put some energy into briefing industry analysts properly, instead of expecting us to root around the OWASP website and complaining when we don't.

Large vendors may sometimes try to influence industry analysts by commissioning work, and many analysts declare this when they deem it relevant. (I think that's what James means by transparency.) But a much more subtle influence can be achieved simply by providing better quality information and making our lives easier.

Update February 2013. James has now returned to the subject Five Mistakes CIOs make in asking analyst firms to create vendor shortlists... (February 2013). See further comments below this post, plus discussion on Twitter OWASP and Industry Analysts (Storify, February 2013).


  1. I agree with your current assessment of OWASP and will take the following action items:

    1. If any analyst wants to publish detailed research (not just blog posts) on OWASP, please have them contact me directly and I will personally guarantee them a response to every single question they have within 48 hours.

    2. If they want to schedule a dialog, I will clear my work schedule and make time available.

    3. If I fail at either of the above, they can personally blog this fact. Transparency goes both ways...

  2. "In contrast, my questions to OWASP go into a black hole"

    Who are you emailing, seriously? The OWASP mailing lists are here are any of them being unresponsive?

  3. Anonymous4:47 am

    Richard, Good post.

    James, It is not enough to be willing to respond to requests or questions from analysts. You have to proactively reach out to the analysts and sell them on why they should be covering OWASP.

    This lack of outreach is not just problem with open source products but also commercial startups and even major vendors with new products outside their traditional markets. To see a little more on the subject check out:

    "Are the analysts laggards or have startups neglected to brief them? [Startup Saturday]"

  4. Great idea, OWASP would definitely benefit from a solid $100,000 USD donation so we can afford such an employee!

  5. 1. Should analysts be more transparent in declaring that they don't have time to actually perform proper research in their reports as a disclaimer and need to be spoonfed by a vendor briefing mechanism?

    2. What would an end buyer of technology learn if they were to understand how much/little time goes into producing a report vs the other activities analysts spend their time on?

  6. 3. Can any analyst guarantee that if OWASP spends time on briefing analysts that this will generate a positive ROI? As you are aware, OWASP is a volunteer organization. If analysts want to waste the time of analyst relations professionals, that is one thing. It is another to waste the time of people who are attempting goodness.

    4. Maybe you could identify an analyst or two in your network that would be willing to contribute time to a few open source projects. It may be beneficial to the analysts to understand what it is like to sit on the other side of the table with a compelling value proposition but zero money.

  7. I will take it one step further. If you know of any Gartner, Altimeter, Constellation, Ovum, Celent, Novarica or IDC analyst that wants a free conference pass to the upcoming OWASP conference in NYC, I will get them one.

  8. James, thanks for your questions. For answers see my Industry Analyst Coverage Update Feb 2013.