To survey or not

Erlend’s comment on my earlier post on interviews prompted some thoughts, particularly along the lines that it would be worth adding a companion blog post discussing some similar issues around surveys, to go with my post a few days ago about interviews.

To reiterate, my fundamental view of the use of surveys, for student dissertations in particular, is quite similar to my view of interviews.  As with interview data, survey data can be fraught with bias: it can be influenced dramatically by the choice of questions, and can tell you what the subject wants you to hear, rather than what the subject is really thinking or doing.  Worse, there is a danger of survey data being completely false – for instance if people fill in the same survey several times, or if they fill it in with false information.  However in the context of an undergraduate dissertation particularly, a survey can be a quick and effective way of gathering some primary data, and if you acknowledge the problems with survey data and recognise the limitations, the survey results can be useful in giving your work a distinctive and original slant.

With surveys there is a particular paradox.  It is very easy to set up a simple survey and to send it out over the Internet.  Surveymonkey is perhaps the best known product but there are many other, including Qualtrics, which we have recently started using at Cass Business School.  It’s available to Cass students and I have had exceptionally good feedback from people who have used it in their dissertations.  However questionnaire design and sampling  are highly technical areas into which professionals, such as market researchers, put a lot of effort and it isn’t realistic for a student spending a few months on their dissertation to do the same.  So the sort of survey that you’re likely to use as a dissertation student will fall a long way short of the standards typically associated with the professionals.

However this isn’t a reason to adopt any really sloppy practice when you carry out surveys.  So word your questions carefully, and make sure they don’t obviously lead people in one direction of another.  Make it clear what is being done with the data that you collect, and offer to share any results (anonymously) with people who fill in your survey.  Define any obscure terms that you use.  Make sure that the survey wording is right for your audience – you would phrase questions differently according to whether you were surveying a group of professional web developers or surveying a group of people who were reluctant to use the Internet at all.  Think about what you want to find out from the survey, and make sure that your questions reflect these (if you have say 4 or 5 categories of information, make sure you have questions which cover each of these). 

As a sanity check I would also recommend you to look at chapter 9 of Business Research Methods by Bryman and Bell (Oxford, 2007) which discusses issues around the use of self-completed questionnaires.

And I would suggest that you can draw three completely different conclusions from the above.  You have to choose which one is right for your work.

Conclusion 1 would be that surveys are best avoided in a dissertation – that you won’t really get a useful sample and that if you want to get data from real people, you are best off using different methods such as interviews or focus groups.

Conclusion 2 would be that surveys are useful, but that the data should be treated more as though you’d been carrying out a large asynchronous electronic focus group than anything else – that you have a rather amorphous and unstructured group of people who might give you some interesting ideas, but you should be cautious in drawing any conclusions

Conclusion 3 would be that surveys are useful but that the results should still be treated with extreme caution.

Advertisements

2 Responses to “To survey or not”

  1. Jake Says:

    Great article, you make some very good points.
    Our teachers and students use a great web survey site at http://www.websurveymaster.com/ It is very easy to use, creates great professional looking surveys, and the results analysis tools are fantastic.
    Hope this helps!
    Jake

    • martinrich Says:

      Jake,

      Please could you clarify whether you are personally involved with this survey site or not? No problem if you are: it would just be nice to know

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: