I have just been reading a new book by Will Thalheimer called "Smile Sheets", and an excellent read it is too.
If you don't know Will, he specialises in reading academic research about learning and thinking about how this can contribute to training. Training is, of course, an area where there are various strange practices based on mythical facts. One of my favourites is the cone of experience, the claim that we remember 10% of what we read, 20% of what we hear and so on. Claims such as this are presented in training, and because they seem to make some sense get repeated, and slowly becomes fact. This particular topic is one that Will has discussed in the past, and I can recommend a visit to his blog to learn more (www.willatworklearning.com/).
Anyway, he focuses in his latest book on the smile sheet, or to give it its polite name, the reaction questionnaire (a la Kirkpatrick). Although this is the bedrock of most training evaluation activities, the book discusses in some detail the lack of research data to prove that it is meaningful in any way. This is because of a number of different factors. One is that the types of questions often included in reaction questionnaires are often poorly constructed from a statistical point of view, and force the learner into giving a positive response. Another is that surveys conducted in the training environment while the training is still under way are heavily influenced by the fact of being there and because there is no time for reflection on what the training has been about. Finally, there is very little evidence to show that merely reacting positively to a training activity means that there will be learning, which is a fundamental principle in the Kirkpatrick framework, which, of course, underlies much thinking in training evaluation.
Will then goes on to talk about what learning actually means and provides a practical guide to how to design 'smile sheets' which can actually produce meaningful and useful data. It is a most entertaining and illuminating read, and I certainly wish that I had read it before sending my own manuscripts to my publisher!
If you do get involved in any way with training evaluation, buy yourself a copy. At $25, it's well worth it.