What's wrong with surveys?

From time to time, I receive requests to participate in various surveys, or see invitations posted on internet forums or via email lists. I have also received questionnaires from students asking my opinion on a range of topics. There seems to be no clear basis for selecting me as a member of some panel, no clear process (such as Delphi) for allowing the panel to converge on a collective expert opinion, and scant evidence that the student would be able to produce a meaningful result from the questionnaire returns.

On a recent occasion, the sender indicated that this was part of their doctoral research! (In discussion with colleagues at the time, we wondered whether the student’s supervisor really has no notion of what a ‘true sample’ might look like, or whether they were cynically promoting intellectually sloppy work in order to milk some funding institution that knows no better.)

On the other side of the fence, I have sometimes reviewed papers (both academic and commercial) whose main source of data is an opinion survey of some kind. For example, asking IT managers what they thought were the important technical issues in a given domain. Although such surveys could be useful for marketing purposes (enabling a vendor to align products and services to the perceptions of the customers), they rarely add anything to our technical knowledge or understanding of the domain.

I don't want to discourage anyone from participating in surveys, but I always wonder what this kind of survey might tell us. If the results of the survey were to contradict our current beliefs on some topic of importance, how many of us would be willing to alter our beliefs, and how many would merely dismiss the survey as obviously inaccurate? And if such a survey is only taken seriously if it confirms what we think we already know, what's the point of doing such surveys at all?

Even the most open-minded, flexible and inquisitive people don't just change their minds at the drop of a hat. It takes rather more than a casual Internet questionnaire to persuade any of us to alter our opinion or our actions; and if a survey doesn't have any effect on our actions then it is not worth a lot to practical people. I am not against surveys as such, but unfortunately most surveys fall short of providing actionable intelligence, and serve merely to reinforce received opinion.

Update

An additional problem with these surveys is that they tend to be circulated to a person's existing contacts, or within an existing filter bubble/echo chamber. Which makes the results additionally suspect.

Related posts

Received Opinion (January 2005), Industry Analysis by Survey (July 2009), How to persuade me to participate in your survey (April 2013)

Updated 22 March 2021