Surveys: notes on methodology

According to this website, researchers at the Athens University of Business and Economics  released a study about the attitudes of the Greek electorate yesterday. In summary, the survey purports to show a massive disappointment with major parties, which have traditionally held power, and a concomitant rise in the popularity of smaller political groups. It is noteworthy document, because –in addition to its substantive findings– it brings into the foreground a number of issues relating to research methodology and ethics. I think I am not alone in wondering to what extent the results were influenced by the wording of questions such as the following:

If one of the major contending parties, which have been responsible for the decline of your standard of living, promises a better future, will you trust them and vote for them again? (p. 5).
Are you determined to punish the parties that brought the crisis upon us, by voting against them? (p. 15)

Leading questions are useful in politics, no doubt, but they make for bad science (if don’t know why, Sir Humphrey Appleby offers a concise reminder). In addition, in the course of discussing this study with other researchers and academically minded members of the public, it occurred to me that there is a danger of people lapsing into epistemological agnosticism, i.e. deciding that no survey can be trusted. This danger, I think, stems from the fact that there seems to be a lack of awareness of why some surveys generate surprising results. One often has an intuitive understanding that something must be wrong, but cannot pinpoint the features in the survey design that led to these results.

Prompted by these thoughts, I have put together a list of references that describe how survey research should be done. The resources I cite focus mostly on interviewing, although the advice on eliciting usable data can be transferred to other types of research, such as questionnaire surveys as well. The idea is that researchers might follow these methodological rules of thumb in order to avoid the more obvious pitfalls of data generation. In addition, such a resource might make it easier to scrutinize those studies which selectively apply the principles of social science in order to advance insidious agendas.

Bell, J. (2008) ‘Planning and conducting interviews’ in Doing your research project (4th  edn.), Maidenhead: Open University Press: pp. 156 – 172.

This introductory-level textbook contains practical information on topics such as the ethics of interviewing, advantages and limitations of interviews as a data-generation technique, question wording, constructing an interview schedule, tape-recording and transcribing.

Brown, J. D. (2001) Using surveys in language programs. Cambridge: Cambridge University Press: pp. 44-54.

This section contains one of the most exhaustive lists of advice for designing effective data elicitation questions, with excellent examples.

Cohen, L., Manion, L. & Morrison, K. (2007) Research methods in education (6th edn.). New York: Routledge.

Chapter 9 offers a comprehensive, if somewhat uninspiring, discussion of surveys, longitudinal, cross-sectional and trend studies. Additional information on how to design instruments for these surveys can be found in chapters 15 (questionnaires) and 16 (interviews). There is a discussion of common errors in question writing and sequencing (pp. 334-336), as well as sections of telephone interviewing (pp. 379 et seq.) and the ethics of interviewing (p. 382).

Crano, W. & Brewer, M. (2002) ‘Interviewing’ in Principles and methods of social research. Mahwah, N.J.: Lawrence Erlbaum Associates, Inc.: pp. 223-244.

Contains lots of practical advice on designing question content, interview format, sampling, interview conduct and data analysis. Also includes a discussion of telephone surveys.

DeMarrais, K. (2004) ‘Qualitative interview studies: Learning through experience’ in DeMarrais, K. and Lapan, S. (eds) Foundations for research: Methods of inquiry in education and social sciences. Mahwah, N.J.: Lawrence Erlbaum Associates, Inc.: pp. 51 – 68.

This chapter begins by defining interviews and linking them to their theoretical underpinnings. Following that, there’s advice on sampling, designing an interview guide, phrasing questions and conducting an interview. There is a discussion of the power relations between interviewer and interviewee, which seems quite pertinent to survey type interviews as well, even though this is not the topic of the chapter.

Dörnyei, Z. (2007). Research methods in applied linguistics : quantitative, qualitative, and mixed methodologies. Oxford: Oxford University Press.

Because of the focus of this monograph, there is no section focussing specifically on interview surveys, but the relevant information can be found scattered throughout this hightly readable book. Sound advice is offered on sampling (pp. 95-101), and the section on questionnaire surveys includes advice on question wording (pp. 102-109) and sequencing (pp.101-102), which can be transferred to interview surveys. There is also a section on internet surveys (pp. 121-123). Advice on designing and conducting interviews is also available (pp. 136-143).

Fielding, N. & Thomas, H. (2008) ‘Qualitative interviewing’ in Gilbert, N. (ed) Researching social life (3rd edn.). London: SAGE: pp. 245 – 265.

Discusses varieties of qualitative interviews and uses of data. There are sections on interviewer effects, impediments to the flow of candid information, as well as advice on designing and conducting an interview. There is also information on group interviews and telephone surveys.

Kvale, S., & Brinkmann, S. (2009) InterViews : learning the craft of qualitative research interviewing (2nd edn.). Thousand Oaks ; London: SAGE.

A comprehensive guide on all aspects of qualitative interviewing, thought by some to be the definitive work in the field. Perhaps less useful in survey-type interviews.

Nunan, D. (1992) ‘Elicitation techniques’ in Research methods in language learning. New York, NY: Cambridge University Press: pp. 137 – 158.

Technically this chapter contains advice on a variety of data generation methods, including surveys, interviews etc. It is, in my opinion, an excellent concise introduction, especially for ELT students working towards an M-level degree.

Punch, K. (2005) Introduction to social research (2nd edn.). London: SAGE: pp. 170-177.

This chapter contains information on the types and uses of interviews, including focus groups. There is advice on designing and managing interviews as well as a discussion on recording. The chapter also includes a very thorough discussion of possible challenges inherent in the textual nature of data and some theoretical responses.

Robson, C. (2002) ‘Interviews’ in Real world research (2nd edn.). Oxford: Blackwell: pp. 269-291.

Discusses different types of interviews, along with their uses, advantages and limitations. Also contains advice for designing effective interview questions and conducting interviews effectively. Includes sections on telephone surveys and focus groups.

Simons, R. (2008) ‘Questionnaires’ in Gilbert, N. (ed.) Researching social life (3rd edn.). London: SAGE: p. 187.

Brief discussion of advantages, disadvantages and uses of interviews. Brief advice on conducting an interview.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s