This looks at various aspects of test design:
Different ways of scoring tests
What helps to make a good test?
Design the test first
If assessment is an important aspect of the learning, design the test first then design the content.
This is because the content of the materials should be based on objectives, so that by the time the person has completed everything they should be able to satisfy the criteria of the training objective. This implies that material that is not needed in order to help the learner to achieve the objective is not relevant and should therefore not be included. Writing the test question or questions first helps you to focus on what is actually needed and to only include relevant information.
A second reason is more pragmatic. Writing good test and quiz questions can be difficult and tiring, and after spending considerable time designing content it can be difficult to find the energy necessary to do this job properly! So design the questions first, while you are fresh and enthusiastic.
Base questions on objectives
Only put questions in a test if they are related to an objective identified. Do not include questions just because they are good questions.
Always give the learner some feedback
You must always give the learner feedback to their response. If there are particular reasons why they should not know whether their answer was right or wrong, give them the message "Thank you" or similar.
Give the learner control
Because tests are, by definition, important to the learner, you must take whatever steps are necessary to reduce the anxiety they will feel. One source of anxiety is in the level of control they have over the program. Things you can do include:
- give the learner the chance to change their answers
- allow the learner to review questions they have done
- build in barriers that will prevent them from leaving the test accidentally.
We can divide tests into two general types:
Performance tests
The learner has to do something that shows they can satisfy an objective. In an e-learning course you could do this by providing some form of simulation.
There would be no need for individual questions, and the success would depend on how well the learner performs in the simulation. You will need to be very clear about the success criteria and may also need to think about the level of fidelity that is needed in the simulation.
Criterion-referenced tests
The learner answers a series of questions with specific criteria for success based on the objectives.
Note that this contrasts with norm-referenced testing where success is relative to group performance, so that, say, 50% will always pass and 50% always fail regardless of absolute levels of performance.
Different ways of scoring tests
The simplest type of test comprises a fixed series of questions that the learner must answer. You decide how many questions to include and how many the learner must answer correctly in order to pass. This is probably the most common type of test in use, but it is often ineffective as a real test of mastery of a subject due to (amongst other reasons) the:
- difficulty in setting questions at the right level of difficulty
- somewhat arbitrary nature of the pass mark
There are a number of other ways in which tests can be presented that may be more effective.
Adaptive tests are programmed so that they can present different questions to different learners. The simplest form of adaptive test is illustrated here. If the learner answers a question correctly they move on to a question on the next subject. If they answer a question incorrectly, they are asked a second question on the same subject. This helps to see whether or not their first wrong answer was bad luck. |
 |
A more sophisticated form of adaptive testing is based around an area of research known as latent trait theory. Tests using this method have questions banded by level of difficulty. This diagram shows the principle of such a test. To start the test, the learner is asked a question drawn at random. If they answer it correctly they are asked a harder question. If they answer this question correctly, they are asked an even harder question, and so on. |
 |
If, however, they answer the question incorrectly they are asked an easier question. If they answer that correctly, they receive a harder one, and so on. In practice, this process is almost always convergent, and quite quickly the learner is being presented with questions almost continually at the same level of difficulty.
This type of adaptive testing has a number of significant advantages over conventional test methods. It provides a very clear idea of the level of mastery, and does not rely on any arbitrary pass mark. It also works very quickly; learners usually converge on their level of mastery within 10 to 15 questions.
The limitation of this type of testing is that all of the questions must be related to the same subject and this can be difficult to design.
Admissible probability measures testing
Another type of testing is that of Admissible Probability Measures testing. This is a form of multiple choice testing, but each question has three options, A, B and C. The learner marks their response on a template as shown here (although the red numbers are not shown). |
 |
Suppose that for this question the correct answer is A. If the learner is confident that A is correct, they mark the triangle at A. If they are confident that B is correct, they mark that corner. However, if they only think that A is correct but that it could be B, they can mark the triangle at a point along the line from A to B at a point representing their degree of confidence in A .
This method of testing overcomes the problem of guessing in a multiple choice test. The numbers on the triangle show the score given for each answer. If the correct answer is A but the student marks B, they will score –5. The sensible learner therefore hedges their bet if they are not sure and marks somewhere in the middle of a line. The system can therefore classify a learner as well informed, partially informed or misinformed.
|
|
(C) Bryan Hopkins, 2005