A voluntary organisation providing
support to homeless people finds it hard to provide the required level of
services during a time of economic recession. Two reasons put forward to explain
this are:
A. An increased number of people needed services
B. it is harder to raise money at this time
What do you think about the validity of these two reasons?
1. Neither A nor B are relevant to voluntary organisations.
2. Only A relevant to voluntary organisations.
3. Only B is relevant to voluntary organisations.
4. Both A and B are relevant to voluntary organisations.
Missing item
Give the learner an incomplete text or graphic-based
list and ask them to identify what is missing from the list.
Solution evaluation
Give the learner a question and an answer, and then provide
them with a list of possible ways of assessing the answer. For example, is
the answer given Incorrect, Partially correct, Acceptable or Excellent. You
will need to give the learner some criteria for categorising these descriptions.
Multiple true or false
Offer two statements A and B about the content. Then ask
the learner if A is true and B false, A is false and B true, A and B are both
true or A and B are both false. Four options from two statements.
Guidelines for effective multiple choice
questions
Always number or letter the list
Learners find it easier mentally to sort items that have
easy references, such as numbers (1, 2, 3, etc.) or letters (a, b, c, etc.)
Provide four or five possible
answers
A multiple choice question can offer, in theory, anything
from two possible answers upwards. Of course, with only two answers the learner
has a 50% chance of guessing the right answer, while with a large number of
answers it is increasingly difficult to guess correctly, but the question
becomes unwieldy both to design and read.
The best compromise is to offer no more than four or five
alternatives.
Make all the choices believable
The hardest part about writing multiple choice questions
is thinking of the distractors.
Also resist the temptation to put one in as a joke. It
merely increases the user’s chance of guessing the right answer from
those left.
For example, is option 3 in this question plausible?
Make the meaning of the question clear
to the user
Do not ask questions that mean different things to different
people. For example:
Clearly everyone will have different thoughts about this.
Keep the options as short as possible
You keep the options short by putting as much text as
possible in the stem of the question (the first part of the question). For
example:
This is preferable to writing December 5, December 17,
etc. in the list of answers.
Avoid giving the learner clues
It is surprisingly easy to give a learner clues about
right and wrong answers. We have already discussed implausible answers, but
you must also look out for such things as:
- presenting opposites: an obvious distractor is the direct
opposite of the key, so if a learner sees a pair in the list they will concentrate
on those two
- making the correct answer the longest: you may find yourself
doing this by adding information to make the correct answer unambiguously
correct
- 'a' and 'an': ending the stem with the word 'a' and including
distractors beginning with vowels (or vice versa)
- repetitive answer positions: repeatedly using the same
number in the list for the correct answer (keep a die in your desk and roll
it to decide the position of the key).
Avoid using 'not' in the stem
Avoid asking learners to say which is not a correct answer.
Such questions become tests of mental dexterity rather than of an understanding
of the subject. For example:
This type of questioning also makes people think about
the incorrect answer rather than the correct answer.
Avoid using ‘none’ or ‘all’
as an option
Look at this question:
The weakness with this type of question is that as soon
as the learner realises that you do not pay estate agent's fees, the ‘all’
option is not a possible correct answer. The question then becomes a simple
alternative response type, with an increased chance of guessing the right
answer.
Similarly, avoid using the words ‘never’
and ‘always’ in a question. Absolutes are always very hard to
find, and if a learner can think of an exception, however obscure, they have
a good reason to reject 'never' and 'always' options.
Put answers in alphabetical or numerical
order (where appropriate)
When answers are numbers or single words, it is good
practice to put them in numerical or alphabetical order.
Keep options the same length
Make all options look the same length. The correct answer
sometimes contains more detail than the distractorsto to make it precise and
correct: this draws attention to itself.
Consider multiple correct questions
It is perfectly acceptable to give learners questions
where there are two or more correct answers in a list. You should provide
a correspondingly higher number of distractors.
The main issue with using questions such as this in a
test is about awarding scores. You need to decide on a scoring scheme that
rewards partially correct answers as well.
It also can be difficult to provide error-contingent
feedback with multiple correct questions. It is possible to do it but
the question design time goes up considerably.
Avoid questions where all the options are correct. These
can irritate learners, particularly when they are used where it is obvious
that all are correct.
Also be careful about including multiple correct questions
within a series of single correct questions. Learners do not always read the
instructions and may assume that only one answer is correct!
Finding distractors
Following these guidelines will help you to design better
multiple choice questions. Do not underestimate the time taken to write these
questions: you could realistically spend an hour writing a well-designed multiple
choice question.
Here are some ideas.
Collect possible answers
Write your question down as an open-ended question and
ask typical learners to give you answers to the question. You will find that
you have right answers and wrong answers. The wrong answers make perfect distractors
because they are clearly plausible to the target group!
Think of different questions
Think of statements that would provide correct answers
to different but similar questions.
Classify the answer
Can you see what sort of ‘general class’
the correct answer is? Think of other items in that class and construct the
distracters from that. For example, consider this question:
The Ansoff growth model proposes four strategies that an
organisation can choose to follow. Three of them are market penetration,
market development and diversification. What is the fourth strategy?
The correct answer is ‘Product development’.
This is in the class of ‘doing things to products’, so alternatives
could be:
- Product enhancement
- Product protection
- Product distribution
All apparently meaningful, but quite wrong!
(C) Bryan Hopkins, 2005
A multiple choice question is any question
that asks the learner to select the correct answer(s) from a list of possible
answers. Some technical terms:
- the correct answer is the key
- the incorrect answers are the distractors
E-learning materials use a lot of multiple
choice questions as they are easy to program and easy for the computer to
judge. This does not mean, however, that they are easy to design! It can be
very difficult to design a set of multiple choice questions that really test
the material the learner is studying.
This page covers:
Ways of presenting a question
Types of question to ask
Good practice in multiple choice questions:
Number the list
Give four or
five possible answers
Make choices
believable
Make the question
clear
Keep options short
Avoid clues
Avoid 'not' questions
Avoid 'none' or 'all'
questions
Put answers in an
order
Make options the
same length
Consider multiple
correct
How to find distractors
Ways of presenting a question
Graphics and audio allow for a lot of variety
in the structure of a multiple choice question. Although the principle of
choosing a correct answer from a set of possibilities remains, there are various
ways in which the possibilities can be presented and the choice made. For
example, the learner can:
- select a graphic image
- click on the correct part of a graphic image
- drag a graphic image into an answer box
- select an audio item
Mechanisms for multiple choice questions are more limited
in paper-based materials, but the principles of good question design remain
the same.
Types of question to ask
The most common and obvious style of multiple choice question
is to ask a question and offer a number of possible answers. However, it can
sometimes be difficult to think of suitable questions in this style, so it
is useful to think about alternative styles.
Premise and consequence
Present the learner with a situation and provide them
with a choice of possible outcomes.
Case study
Give the learner a scenario based on the source material.
If your scenario is strong enough, you may be able to find several questions
within it.
An organisation wants to estimate how much it will cost to
offer existing services to a new group of clients. Which of the following
would be useful activities to carry out?
1. Marginal cost analysis.
2. Strategy-context evaluation
3. Programme evaluation review
4. Value for money analysis
Stamp duty is payable:
1. For a solicitor's services
2. To the local authority for carrying out searches
3. When you buy a stamp
4. On the purchase of a house
The best part about buying a house is:
1. Moving in
2. Beating the vendor down by £5000
3. Exchanging contracts
4. Moving out of your old house
Christmas Day is on December:
1. 5
2. 17
3. 25
4. 31
Which of the following do you not pay when buying a house?
1. Stamp duty
2. Capital Gains Tax
3. Solicitor's fees
4. Search fees
Which of the following do you pay when buying a house?
1. Estate agent's fees
2. Capital Gains Tax
3. Search fees
4. All of these
