Writing effective questionnaire items in language education research (Part 1)

In my previous job assignment in an ‘experimental’ school affiliated to a university, part of my job involved liaising with university students who wanted to conduct research at the school, and vetting their research protocols. As one might expect from students who were keen but inexperienced, many of the research instruments they used were somewhat inexpertly designed. Even so, in most cases major improvements were possible with only minor changes.

This is the first in a series of five blog posts in which I want to share a few pointers that will help you to avoid some of the most common mistakes I came across. In writing this post, and the ones that will follow, I shall make some assumptions about you: I assume that you are a trainee researcher, such as an undergraduate working on an assignment, or maybe a professional in a field where small-scale research can be used to inform practice (e.g., a teacher doing an action research project). I also assume that you are working in the fields of education or linguistics, because this is where my expertise lies, although you may find that much of what I have to say can be applied equally well elsewhere. If the above applies to you, shall we read on?

Fragebogen zur Wikipedia - YOU Berlin 2008 (6566)

We shall begin by looking into three tips that can make your questions (or, to use a slightly more technical term, questionnaire items) more effective.

Keep the questions simple

It is important that questionnaire items are kept short and straightforward: By making sure that questions can be read and answered easily, you may include more items in the questionnaire, and respondents are less likely to give up before completing the survey. Zoltán Dörnyei helpfully suggests 20 words as a maximum (2007: 108) , but you may need to adjust this rule of thumb depending on your respondents’ reading skills and the language in which the questionnaire is written.

It is often easy to lose sight of this rule: Sometimes, we write too long sentences when we’re trying too hard to become clear; or maybe we might use academic or technical language to establish our credibility as researchers. I have been guilty of this myself: in a draft questionnaire addressed to 9-12 year old students, I included questions such as:

The number of students in your class is so large that it disrupts learning activities (i.e., it is difficult for the teacher to carry out the lesson, and it prevents you from focusing on your tasks).

On review, the item was shortened to “There are too many children in your class”.

In addition to long, complex structures, you should try to avoid jargon, acronyms and abbreviations, and other technical terms with which readers will be unfamiliar. You should also bear in mind that your respondents’ reading skills may be less sophisticated than yours, as was the case with the draft questionnaire I mentioned above.

Avoid double-barrelled questions

A double-barrelled, or dichotomous, question is an item which actually contains more than one questions. The following item, from a survey an MA student designed, is a good example of what I mean:

In science lessons, learners should spend more time on experiments and project work than on memorizing theory.
1=Strongly Agree, 2=Agree, 3=No Opinion, 4=Disagree, 5=Strongly Disagree

By parsing this item, we find that it consists of at least two propositions: (a) learners should do more experiments and (b) learners should do more project work (a third, implicit, proposition is “learners should spend less time memorising theory”).

Double-barrelled questions are difficult to answer, and the responses are even harder to interpret. In the example above, ‘disagree’ could mean that the respondent believes that memorising rules is effective, or that they prefer experiments but not projects, or maybe they’d rather do more projects but not experiments. Essentially, there is no way of using this data.

To avoid double-barrelled questions, you should scrutinise your questionnaire and check every instance where words such ‘and’ or ‘or’ appear – these are good indicators of potential problems.

Questionnaire extract
How might you re-write these questions?

Avoid negative constructions

Negatively phrased questionnaire items are problematic for two reasons: First, many respondents fail to notice the negative words. Secondly, expressing disagreement with a negative proposition is too complicated. Consider how confusing the following questionnaire item is:

Test scores are not the only criterion you use for student evaluation.  

In the example above, the negative meaning is expressed syntactically (with the word no). Such cases are relatively easy to spot. However, it is also possible to express negative meaning semantically (e.g., “Teaching pronunciation is uninteresting”). The same caveates apply in this case as well.

You will want to go over your questionnaire carefully and check whether negative statements can be re-phrased in a more straightforward way: “Students do not make noise”, for instance, can be changed to “Students are quiet”. If there is no way to avoid a negative construction, you will likely need to format the negative words with bold or underlining to make them more visible.

I hope that you found the advice in this post helpful. You might also want to take a look at the following posts on questionnaire design:

If you arrived here while preparing for a student project, I wish you good luck with your work. You may also want to use the social sharing buttons at the end of the post to share this content with other students who might find it useful.

If you have any other questions that I might be able to answer, feel free to ask by posting a comment or using this form.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.