Back in 2012, I was still reading research methdology literature for my PhD studies. But as I was struggling to become more familiar with the proper use of research methods, the increasingly polarised political discourse in Greece was also providing me with examples of how research methods could be abused. It was a certain degree of indignation at such abuses, often originating from corrupt corners in Greek academia that triggered this post; that, and a belief that, while I was not able to challenge this abuse, I could at least show interested readers that there are better and more trustworthy ways of finding information. I no longer know how strongly I feel those feelings, but I have retained the post, which may still be of some use to anyone interested.
Abusing Research Methodology
According this website (update: link broken), researchers at the Athens University of Business and Economics released a study yesterday (10 April 2012) about the attitudes of the Greek electorate. In summary, the survey purports to show a massive disappointment with major parties, which have traditionally held power, and a concomitant rise in the popularity of smaller political groups.
It is noteworthy document, because –in addition to its substantive findings– it brings into the foreground a number of issues relating to poor understandings of research methodology and questionable ethics.
I think I am not alone in wondering to what extent the results were influenced by the wording of questions such as the following:
If one of the major contending parties, which have been responsible for the decline of your standard of living, promises a better future, will you trust them and vote for them again? (p. 5, emphasis added).
Are you determined to punish the parties that brought the crisis upon us, by voting against them? (p. 15, emphasis added)
Leading questions, such as the ones above, are no doubt useful in politics, but they make for bad science (if don’t know why, Sir Humphrey Appleby offers insights below).
Why abusing research methodology is a problem
In the course of discussing this study with other researchers and academically minded members of the public, it occurred to me that there is a danger of people lapsing into epistemological agnosticism, i.e. deciding that no survey can be trusted.
This danger, I think, stems from the fact that there seems to be a lack of awareness of why some surveys generate surprising results. One often has an intuitive understanding that something must be wrong, but cannot pinpoint the features in the survey design that led to these results.
A list of resources on research methodology
Prompted by these thoughts, I have put together a list of references that describe how survey research should be done. The resources I cite focus mostly on interviewing, although the advice on eliciting usable data can be transferred to other types of research, such as questionnaire surveys as well.
The idea is that researchers might follow these methodological rules of thumb in order to avoid the more obvious pitfalls of data generation. In addition, such a resource might make it easier to scrutinize those studies which selectively apply the principles of social science in order to advance insidious agendas.
- Bell, J. (2008). Planning and conducting interviews. In Doing your research project (4th edn., pp. 156-172). Open University Press.
This introductory-level textbook contains practical information on topics such as the ethics of interviewing, advantages and limitations of interviews as a data-generation technique, question wording, constructing an interview schedule, tape-recording and transcribing.
- Brown, J. D. (2001). Using surveys in language programs. Cambridge University Press.
The section between pp. 44-54 contains one of the most exhaustive lists of advice for designing effective data elicitation questions, with excellent examples.
- Cohen, L., Manion, L. & Morrison, K. (2007). Research methods in education (6th edn.). New York: Routledge.
Chapter 9 offers a comprehensive, if somewhat uninspiring, discussion of surveys, longitudinal, cross-sectional and trend studies. Additional information on how to design instruments for these surveys can be found in chapters 15 (questionnaires) and 16 (interviews). There is a discussion of common errors in question writing and sequencing (pp. 334-336), as well as sections of telephone interviewing (pp. 379 et seq.) and the ethics of interviewing (p. 382).
- Crano, W. & Brewer, M. (2002) ‘Interviewing’. In Principles and methods of social research (pp. 223-244). Lawrence Erlbaum Associates.
Contains lots of practical advice on designing question content, interview format, sampling, interview conduct and data analysis. Also includes a discussion of telephone surveys.
- DeMarrais, K. (2004). ‘Qualitative interview studies: Learning through experience’. In K. DeMarrais, & S. Lapan (Eds.), Foundations for research: Methods of inquiry in education and social sciences (pp. 51-68). Lawrence Erlbaum Associates.
This chapter begins by defining interviews and linking them to their theoretical underpinnings. Following that, there’s advice on sampling, designing an interview guide, phrasing questions and conducting an interview. There is a discussion of the power relations between interviewer and interviewee, which seems quite pertinent to survey type interviews as well, even though this is not the topic of the chapter.
- Dörnyei, Z. (2007). Research methods in applied linguistics: Quantitative, qualitative, and mixed methodologies. Oxford University Press.
Because of the focus of this monograph, there is no section focussing specifically on interview research, but the relevant information can be found scattered throughout this hightly readable book. Sound advice is offered on sampling (pp. 95-101), and the section on questionnaire surveys includes advice on question wording (pp. 102-109) and sequencing (pp.101-102), which can be transferred to interview surveys. There is also a section on internet surveys (pp. 121-123). Advice on designing and conducting interviews is also available (pp. 136-143).
- Fielding, N. & Thomas, H. (2008). ‘Qualitative interviewing’. In N. Gilbert (Ed), Researching social life (3rd edn., pp. 245-265). SAGE.
Discusses varieties of qualitative interviews and uses of data. There are sections on interviewer effects, impediments to the flow of candid information, as well as advice on designing and conducting an interview. There is also information on group interviews and telephone surveys.
- Kvale, S., & Brinkmann, S. (2009). InterViews: Learning the craft of qualitative research interviewing (2nd edn.). SAGE.
A comprehensive guide on all aspects of qualitative interviewing, thought by some to be the definitive work in the field. Perhaps less useful in survey-type interviews.
- Nunan, D. (1992). ‘Elicitation techniques’ In Research methods in language learning (pp. 137 -158). Cambridge University Press: .
Technically this chapter contains advice on a variety of data generation methods, including surveys, interviews etc. It is, in my opinion, an excellent concise introduction, especially for ELT students working towards an M-level degree.
- Punch, K. (2005). Introduction to social research (2nd edn., pp. 170-177). SAGE.
This chapter contains information on the types and uses of interviews, including focus groups. There is advice on designing and managing interviews as well as a discussion on recording. The chapter also includes a very thorough discussion of possible challenges inherent in the textual nature of data and some theoretical responses.
- Robson, C. (2002). ‘Interviews’. In Real world research (2nd edn., pp. 269-291). Blackwell.
Discusses different types of interviews, along with their uses, advantages and limitations. Also contains advice for designing effective interview questions and conducting interviews effectively. Includes sections on telephone surveys and focus groups.
- Simons, R. (2008). ‘Questionnaires’. In N. Gilbert (Ed.), Researching social life (3rd edn., pp. 187). SAGE.
Brief discussion of advantages, disadvantages and uses of interviews. Brief advice on conducting an interview.
In liew of a conclusion
I am not labouring under any misapprehansion regarding the motives of those who willingly fabricate research findings to suit a political agenda. However, it is my hope that the resources listed above may be of some use to readers who value methodological rigour and research integrity.
If you have found something in this list that is useful to you, that will make me very happy. I am also very keen on any ideas you have on how to expand this list and make it useful and relevant to more readers. Feel free to add more resources at the comment section or send me an email. Also feel free to share this with anyone who might find it interesting.
Note: This post was originally written in 2012, and the introductory text was revised in April 2023. You may find that the fundamentals of good research do not change fast, but the list of resources is nevertheless out of date. I will, perhaps, find the time to revise it in the future. But in the meanwhile, I hope it can still be useful as a starting point for your searches.
Leave a Reply