Designing better questionnaires: Writing more effective questions (1)

In my previous job assignment in an ‘experimental’ school affiliated to a university, part of my job involved liaising with university students who wanted to conduct research at the school, and vetting their research protocols. As one might expect from students who were keen but inexperienced, many of the research instruments they used were somewhat inexpertly designed. Even so, in most cases major improvements were possible with only minor changes.

This is the first in a series of five blog posts in which I want to share a few pointers that may be of help in avoiding some of the most common mistakes I came across. In writing this post, and the ones that will follow, I shall make some assumptions about you: I assume that you are a trainee researcher, such as an undergraduate working on an assignment, or maybe a professional in a field where small-scale research can be used to inform practice (e.g., a teacher doing an action research project). I also assume that you are working in the fields of education or linguistics, because this is where my expertise lies, although you may find that much of what I have to say can be applied equally well elsewhere. If the above applies to you, shall we read on?

Fragebogen zur Wikipedia - YOU Berlin 2008 (6566)

We shall begin by looking into three tips that can make questions (or, to use a slightly more technical term, questionnaire items) more effective.

Keep the questions simple!

It is important that questionnaire items are kept short and straightforward: By making sure that questions can be read and answered easily, you may include more items in the questionnaire, and respondents are less likely to give up before completing the survey. Zoltán Dörnyei helpfully suggests 20 words as a maximum (2007: 108) , but you may need to adjust this rule of thumb depending on your respondents’ reading skills and the language in which the questionnaire is written.

It is often easy to lose sight of this rule: Sometimes, we write too long sentences when we’re trying too hard to become clear; or maybe we might use academic or technical language to establish our credibility as researchers. I have been guilty of this myself: in a draft questionnaire addressed to 9-12 year old students, I included questions such as:

  • The number of students in your class is so large that it disrupts learning activities (i.e., it is difficult for the teacher to carry out the lesson, and it prevents you from focusing on your tasks).      YES    NO

On review, the item was shortened to “There are too many children in your class”.

In addition to long, complex structures, you should try to avoid jargon, acronyms and abbreviations, and other technical terms with which readers will be unfamiliar. You should also bear in mind that your respondents’ reading skills may be less sophisticated than yours, as was the case with the draft questionnaire I mentioned above.

Avoid double-barrelled questions

A double-barrelled, or dichotomous, question is an item which actually contains more than one questions. The following question, from a survey an MA student designed, is a good example of what I mean:

  • In science lessons, learners should spend more time on experiments and project work than on memorizing theory.
    Strongly Agree, Agree, No Opinion, Disagree, Strongly Disagree

By parsing this item, we find that it consists of at least two propositions: (a) learners should do more experiments and (b) learners should do more project work (a third, implicit, proposition is “learners should spend less time memorising theory”). Double-barrelled questions are difficult to answer, and the responses are even harder to interpret. In the example above, ‘disagree’ could mean that the respondent believes that memorising rules is effective, or that they prefer experiments but not projects, or maybe they’d rather do more projects but not experiments. Essentially, there is no way of using this data. To avoid double-barrelled questions, you should scrutinise your questionnaire and check every instance where words such ‘and’ or ‘or’ appear – these are good indicators of potential problems.

Questionnaire extract
How might you re-write these questions?

Avoid negative constructions

Negatively phrased questionnaire items are problematic for two reasons: First, many respondents fail to notice the negative words; secondly, expressing disagreement with a negative proposition is too complicated. Consider how confusing the following questionnaire item is:

  • Test scores are not the only criterion you use for student evaluation.   YES    NO

Note that a negative meaning can be expressed syntactically (as was the case above) and semantically (e.g., “Teaching pronunciation is uninteresting”). You will want to go over your questionnaire carefully and check whether negative statements can be re-phrased in a more straightforward way: “Students do not make noise”, for instance, can be changed to “Students are quiet”. If there is no way to avoid a negative construction, you will likely need to format the negative words with bold or underlining to make them more visible.

~

In later posts, I shall discuss some more tips about writing questionnaire items; I’ll look into ways to improve the demographics section of questionnaires; I’ll explain how to use scale items effectively; and I shall talk about questionnaire layout. Till next time!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s