How to persuade me to participate in your survey

1. Please don't send me a laundry list of topics that you might want to pick my brains about.

Please indicate in which of the following 15 areas you have experience and/or will be able to provide examples, anecdotes, or case studies - dealing with best practices, failures, or successes, or in some way interesting. Please select all that apply.

2. Please don't send the same questionnaire to CIOs and consultants and industry analysts, and expect to be able to do anything useful with the answers.


3. Please don't address me as a "practitioner" without knowing what I am a practitioner of.

Researchers typically use the word "practitioner" to mean anyone without proper university affiliation. It sometimes carries a slightly patronizing tone.

4. Please read what I've written on the subject you are researching. Yes, even so-called Practitioners do publish stuff sometimes. A proper citation would be nice. Thank you.

---

Related post What's wrong with surveys (June 2010)

What's wrong with surveys?

From time to time, I receive requests to participate in various surveys, or see invitations posted on internet forums or via email lists. I have also received questionnaires from students asking my opinion on a range of topics. There seems to be no clear basis for selecting me as a member of some panel, no clear process (such as Delphi) for allowing the panel to converge on a collective expert opinion, and scant evidence that the student would be able to produce a meaningful result from the questionnaire returns.

On a recent occasion, the sender indicated that this was part of their doctoral research! (In discussion with colleagues at the time, we wondered whether the student’s supervisor really has no notion of what a ‘true sample’ might look like, or whether they were cynically promoting intellectually sloppy work in order to milk some funding institution that knows no better.)

On the other side of the fence, I have sometimes reviewed papers (both academic and commercial) whose main source of data is an opinion survey of some kind. For example, asking IT managers what they thought were the important technical issues in a given domain. Although such surveys could be useful for marketing purposes (enabling a vendor to align products and services to the perceptions of the customers), they rarely add anything to our technical knowledge or understanding of the domain.

I don't want to discourage anyone from participating in surveys, but I always wonder what this kind of survey might tell us. If the results of the survey were to contradict our current beliefs on some topic of importance, how many of us would be willing to alter our beliefs, and how many would merely dismiss the survey as obviously inaccurate? And if such a survey is only taken seriously if it confirms what we think we already know, what's the point of doing such surveys at all?

Even the most open-minded, flexible and inquisitive people don't just change their minds at the drop of a hat. It takes rather more than a casual Internet questionnaire to persuade any of us to alter our opinion or our actions; and if a survey doesn't have any effect on our actions then it is not worth a lot to practical people. I am not against surveys as such, but unfortunately most surveys fall short of providing actionable intelligence, and serve merely to reinforce received opinion.

Update

An additional problem with these surveys is that they tend to be circulated to a person's existing contacts, or within an existing filter bubble/echo chamber. Which makes the results additionally suspect.

Related posts

Received Opinion (January 2005), Industry Analysis by Survey (July 2009), How to persuade me to participate in your survey (April 2013)

Updated 22 March 2021

Are BPM professionals experts in collaboration?

@ReduxOnline posts a link to a student project How to measure the collaboration of knowledge workers in the enterprise.

1. To understand the factors supporting collaboration between knowledge workers, a questionnaire was sent to BPM professionals. The paper doesn't make clear whether they were being asked about their own personal collaboration, or about their opinions about collaboration in general. In any case, we might imagine that BPM professionals have a particular perspective on collaboration, which might distort the survey.

2. Nearly a quarter of the BPM professionals couldn't make sense of the collaboration model on which the survey was based, and were unable to answer all the questions. Instead of treating this as a sign that there might be a problem with the model, the researchers chose to exclude these from the analysis. They then argue that the remaining responses validate their model.

3. The survey doesn't measure collaboration, it measures opinions about collaboration, from a carefully selected group of knowledge workers. Perhaps not surprisingly, the opinions are pretty consistent with the kind of management literature that these knowledge workers might be expected to have read. Except that the answers about "purpose" were all over the place (which I can well imagine, given the uncertain intentions of the question), so this factor failed a statistical test (Cronbach's Alpha) and could be quietly dropped from the model.

4. The direct relation between collaboration and the performance of an enterprise is not tested, because the questionnaire did not consists of any financial questions. (It would seem that financial information is "sensitive"; collaboration itself presumably isn't.) Let's hope the students manage to extend their research to include questions about the financial situation of an enterprise, allowing them to demonstrate and explain how maturity of collaboration of knowledge workers (as perceived and understood by BPM professionals) might actually help to improve the performance of an enterprise.

5. I generally regard opinion surveys as low-grade research because they usually merely recycle received opinion. While I understand that this may be the easiest and cheapest form of research, especially for students and software industry analysts, I expect to see some acknowledgement of the potential distortion, rather than merely taking the collected opinions at face value.

Industry Analysis by Survey

@pgiblett @chrisdpotts @seabird20 @

---

Following a recent poll of 900 CIOs, Gartner concludes "IT Spending Outlook Still Uncertain". Peter B. Giblett adds "... not necessarily recession related".

Chris Potts comments: "CIOs need to actively resist being cast as primarily interested in IT spending" and adds a hint to CIOs: "If a research organisation asks you how much your company is spending on IT, ask 'why does that matter?'". Chris Bird suggests that a more interesting question would be "What value are you delivering?". Chris Potts then adds a further question: "Which corporate strategy are you leading?"

Excellent questions Chris-and-Chris, but we know that surveys like these generally restrict themselves to asking numerical and multiple-choice questions, because the answers can then be "analysed" using simple Excel. As ChrisB points out, most of the hard numbers (?!) analysis is around purchasing.

The question "Why does that matter?" prompts the question "To whom does that matter?" So who cares about IT spending? Why would a CIO care what percentage of her peers are spending what amount on a given buzzword? Safety in numbers?

Perhaps, as ChrisB suggests, the people most interested in quantifying IT spend are the vendors and investors. But they can't learn much from an unqualified total figure. What they really need is a detailed breakdown, based on a consistent method of cost accounting, and they are not going to get that from surveys like these.

The problem is not just lack of detail, but lack of reliability. As ChrisB points out, people don't always tell the truth. Some CIOs may deliberately distort their answers, while others may simply guess the numbers in order to get the researcher off their backs.

In my opinion, then, the correct retort to "how much do you spend?" is "what meaningful conclusions can you possibly draw from whatever number I give you?"

---

Related post: Received Opinion (January 2005)

Standards

Andy Hayler's post on business rules Through a Rule Darkly includes a comment about a readers' poll reported by David Stodder, editor of Intelligent Enterprise, in an article entitled Mission Intelligence: Hide Complexity, Expose the Rules.

"I was amused by the readership poll quoted that said that 61% of respondents say that they have 'no standard process or practice' for business rules management. This might imply that 39% actually did, a number I would treat with considerable caution. Personally I have yet to encounter any that does so on a global basis."

For my part, I always treat these kind of polls with caution. My observation is that people often don't have a clue about what is really going on in their own organizations. They attempt to answer these questions by combining beliefs of what ought to be going on (largely based on what they read in the trade press) with an impression of the general capability of their own organization (linked to emotive factors such as pride and motivation).

[See my post on Received Opinion]

In interpreting this particular survey, Andy adopts the law of the excluded middle. Yes plus No must sum to 100%. Andy uses this calculation - presumably tongue in cheek - to call the Yes vote into question. But even if we assume the don't-knows have been taken care of, there are further problems with this kind of survey.

One of my first mentors in data modelling, Raimo Rikkilae, taught me that there were always three answers to any question: "zero", "one" and "many". Thus for example it is often useful to reframe apparently binary (Yes/No) questions into more complex questions (e.g. Always/Never/Sometimes).

So if 61% of organizations have no standards, this doesn't necessarily mean that the remaining 39% have exactly one standard. It could mean that they have many overlapping standards. (Of course, you might argue that having lots of different standards is tantamount to having no standards at all - but many organizations fail to operate according to this logic.)

Meanwhile, the easiest way to implement a single standard is to make it so vague and abstract that anything you like can be interpreted as conforming to the standard. (Of course, you could argue that having a vague and ambiguous standard is useless, and doesn't really count as a standard at all, but how many organizations follow this logic?)

On this analysis, it's hard to take figures such as 61% or 39% seriously - except as rhetorical gestures. If we want to assess the level of standardization in an organization (whether business rules or anything else), we need to investigate the matter properly, with a clear understanding of (a) what counts as a meaningful standard and (b) how to make sense of the multiplicity of standards in play.

And there is a wider question here - which links to governance. What is the relationship between standardization and the existence (and enforcement) of multiple standards? Does the software industry need more standards - or fewer? Obviously there are some strong vendor interests at stake, but there are also some independent bodies with strong standardization agendas. But while there is a lot of debate about individual standards, or the choice between apparent alternatives, there is little rigorous debate about standardization as a whole.

---

Related post Received Wisdom (January 2005)

Received Opinion

Opinion is recursive. I have always been wary of technical research based on opinion surveys, because most technical opinions are already based on second-hand information. How are we to interpret the fact that 75% of IT managers believe that platform X is flexible, or that vendor Y is attacking vendor Z, or that the most significant success factor is P? How many of these opinions are based on first-hand experience, and how much on reading what other people think – or what other people want them to think.

(The internet amplifies this effect. Google typically finds you the information that other people have already found; blogs and wikis often circulate received opinion without critical engagement; vested interests are often imperceptible behind a complex information supply chain).

There has recently been a vigorous debate between Lotus Domino and its admirers on the one hand, and the Radicati Group. John Vaughan argues that the Radicati Group analysis is based on out-of-date opinion; Lotus is now under new leadership, and the opinions collected by Radicati don't reflect this.

I haven't looked at the rights and wrongs of this particular debate, but I want to comment on some general principles of software industry analysis.

Clearly there is often a delay between a real change and the perceived change. Outsiders don't change their opinions just because there is a new management team with some fine-sounding words. And even when the new management has started to achieve something, it may take a while before this disseminates through received opinion.

Given that there is always a huge legacy of out-of-date opinion, the challenge with opinion surveys is to detect and interpret shifts in opinion, rather than simply report the current distribution of opinion.

Obviously the vendor marketing departments care about the prevailing opinion, because this is what they are selling into. Purchasers care about prevailing opinion for a different reason: because they don't want to look stupid buying a technology that nobody else wants. But in some cases, the ideal scenario for a purchaser is to buy just before everyone else buys. You may be able to negotiate deep discounts and lots of free support from a vendor who is hungry for revenue and for customer success stories. Then if everyone follows your lead, you may appear very smart indeed.

Just like in the stock market, good decisions are based on a shrewd estimate of tomorrow's prevailing opinion. Momentum only works in the short term. And if we analysts (whether stock analysts or technology analysts) simply reflect momentum, we aren't doing our job.

---

Related posts: Industry Analysis by Survey (July 2009), On Practices and Pitfalls (September 2009), On the Transparency of Algorithms (October 2016)