Writing effective questionnaire items in language education research (Part 1)

This article contains information and advice intended to help you write effective questionnaire items for your research project. It was mainly written for students working towards a degree in in language education, ELT or applied linguistics. However, you might still find this information relevant to your needs even if you’re doing research in other fields. You can skip directly to the topics that are covered in this post, namely:

• Keeping questions simple;
• Avoiding double-barrelled questions; and
• Avoiding negative constructions.

Before I do that, however, I’d like to say a few words about how this article came about.

---

Introducing the Writing More Effective Questionnaires series

A number of years ago, I was employed at an ‘experimental’ school affiliated to a university. Part of my job there involved liaising with university students who wanted to conduct research at the school, and vetting their research protocols. As one might expect from students who were keen but inexperienced, many of the research instruments they used were somewhat inexpertly designed. Even so, in most cases, it major improvements were possible with only minor changes. In this article, and the ones that will follow, I want to share a few pointers that will help you to avoid some of the most common mistakes I came across.

My assumptions about you

In writing this post, and the ones that will follow, I shall make some assumptions about you. For instance, I am assuming that you are a trainee researcher, such as an undergraduate student working on an assignment; or maybe you are a teacher doing an action research project to improve your professional practice. I am also assuming that you working in education or linguistics, because this is where my expertise lies. However, you may find that much of what I have to say can be applied equally well elsewhere.

---

Three ways to make questionnaire items more effective

We begin by looking into three tips that can make your questions (or, to use a slightly more technical term, questionnaire items) more effective.

Keep the questions simple

It is important that questionnaire items are short and straightforward. By making sure that your participants read and answer the questions easily, you can include more items in the questionnaire, and respondents are less likely to give up before completing the survey. Zoltán Dörnyei helpfully suggests 20 words as a maximum (2007: 108). However, you might need to adjust this rule of thumb depending on your respondents’ reading skills and the language in which the questionnaire is written.

It is often easy to lose sight of this rule: Sometimes, when we’re trying too hard to become clear, we tend write sentences that are rather longer than they need to be. Or maybe we’re too anxious to convince readers that we are credible researchers, so we go overboard with academic or technical language. I have been guilty of this myself: in a draft questionnaire addressed to 9-12 year old students, I once included questions like this one:

The number of students in your class is so large that it disrupts learning activities (i.e., it is difficult for the teacher to carry out the lesson, and it prevents you from focusing on your tasks).
YES    NO

After piloting the questionnaire, I sensibly decided to shorten this item to “There are too many children in your class”.

In addition to long, complex structures, you should try to avoid jargon, acronyms and abbreviations, and other technical terms with which readers will be unfamiliar. You should also bear in mind that your respondents’ reading skills may be less sophisticated than yours, so keep things simple.

---

Avoid double-barrelled questions

A double-barrelled, or dichotomous, item is an item that packs more than one questions. The following question, from a survey an MA student designed, is a good example of what I mean:

In science lessons, learners should spend more time on experiments and project work than on memorizing theory.
1=Strongly Agree, 2=Agree, 3=No Opinion, 4=Disagree, 5=Strongly Disagree

By parsing this item, we find that it consists of at least two propositions: (a) learners should do more experiments and (b) learners should do more project work. A third, implicit, proposition is that “learners should spend less time memorising theory”.

Double-barrelled questions are difficult to answer, and the responses are even harder to interpret. In the example above, ‘disagree’ could mean that the respondent believes that memorising rules is effective, or that they prefer experiments but not projects, or maybe they’d rather do more projects but not experiments. Essentially, there is no way of using this data.

To avoid double-barrelled questions, you should scrutinise your questionnaire and check every instance where words such ‘and’ or ‘or’ appear – these are good indicators of potential problems.

---

Avoid negative constructions

Negatively phrased questionnaire items are problematic for two reasons: First, many respondents fail to notice the negative words. Secondly, expressing disagreement with a negative proposition is too complicated. Consider how confusing the following questionnaire item is:

Test scores are not the only criterion you use for student evaluation.  
YES    NO

In the example above, the negative meaning is expressed syntactically (with the word no). Such cases are relatively easy to spot. However, it is also possible to express negative meaning semantically (e.g., “Teaching pronunciation is uninteresting”). The same caveates apply in this case as well.

You will want to go over your questionnaire carefully and check whether negative statements can be rephrased in a more straightforward way: “Students do not make noise”, for instance, can be changed to “Students are quiet”. If there is no way to avoid a negative construction, you will likely need to format the negative words with bold or underlining to make them more visible.

---

Before you go

If you arrived here while preparing for a student project, I wish you good luck with your work. If you have any other questions that I might be able to answer, I’d be happy to hear from you. You can ask me by posting a comment or using this form. You might want to use the social sharing buttons at the end of the post to forward this content to other students who might find it useful.