When assessment interferes with the measured

There was a time, not so long ago, when no scientific presentation could afford to omit at least one cartoon from The Far Side. One of my personal favourites (which can be seen here) depicts people of an apparently remote part of the world hiding their luxury Western goods as anthropologists arrive unannounced in the village.

I was reminded of this cartoon recently whilst washing my hands at work. This surprising mental leap was prompted by the temporary addition of a tool for monitoring water consumption in one of our buildings.

Can the method of assessment interfere with the thing it is supposed to be measuring?

Can the method of assessment interfere with the thing it is supposed to be measuring?

As can be seen in the photograph, the equipment being used scores few points for subtlety. I cannot believe that people use their usual amounts of water when confronted by this instrument. This raises the question of their value given that the method of monitoring is almost certainly interfering with the thing that is being measured. Continue reading

The perils on anonymity in educational research

The promise of anonymity can undermine the value of data

The promise of anonymity can undermine the value of education data

From time to time I am asked to comment on other people’s unpublished research. As part of the evidence offered in the manuscript, it is quite common to see analysis based on anonymous questionnaires conducted before and after a pedagogic intervention. In this post I want to raise some concerns about the significant limitations that arise from the unnecessary anonymisation of survey data.

Why offer anonymity?
Firstly, however, it is worth examining the allure of anonymity. From conversations I’ve held with colleagues, the main attraction of anonymisation is the perception that removal of identifiers will free participants to provide full and frank contributions, secure in the knowledge that there can be no personal come-back.

I want to argue here that there are important research benefits from *avoiding* complete anonymity, except in the vanishingly rare occasions where it is vital that contributors cannot be recognised.

1. Keeping identifiers allows for richer analysis. If you can match pre- and post-intervention data it is possible to report on changes relating to individuals which may have been masked by analysis of the cohort as a whole.

2. Keeping identifiers guards against inappropriate comparison of whole cohort data. There is a temptation to take all of the available pre-intervention data and compare it with the complete set of post-intervention data, thereby ensuring that a minimum of data is “wasted”. I believe that this is wrong-headed and to illustrate this point, consider the following scenario in education research. Continue reading

In praise of Psychology (as an A level)

Do you like Green Eggs and Ham?

Do you like Green Eggs and Ham?

I don’t think this warrant’s a spoiler alert, but if you don’t know the punchline of Green Eggs and Ham, you may want to skip to the next paragraph. In Dr Seuss’s classic book, the central protagonist is pestered by Sam-I-Am to try the eponymous delicacy. The man declines, insisting that he does not like green eggs and ham. When, however, he is finally persuaded to give it a try he find that, contrary to expectation, he is actually rather partial to this culinary concoction.

It seems to me that there are a good few people around who have a Green Eggs and Ham approach to A level Psychology. The Russell Group universities do not consider it in their list of “facilitating” (i.e. those it considers worthy-of-study) A level subjects*. Similarly, the snooty attitude of my elder son’s previous school in not offering Psychology was one of the main factors in the decision for him to move for his sixth form studies.

AS level psychology includes thorough evaluation of key studies

AS level psychology includes thorough evaluation of key studies

My suspicion, however, is that a significant proportion of those shunning Psychology have never actually looked into the content of the course. If they had done so, they might have been in for a pleasant surprise. Over the last few days, whilst helping the aforementioned sprog with his revision, I have been reminded of just how good the content of the AS level is (at least for the OCR specifications, I can’t speak for the course offered by the other boards).

The course is built around analysis of 15 classic studies. There are good descriptions of what has been done and why. However the feature for me that really makes the content valuable is the emphasis on evaluating the strengths and weaknesses of each study. The lessons about the importance of reliability and validity of data would be good grounding for students wanting to do a degree in any of the sciences. The discussion of ethical issues and, where applicable, what the investigators did to mitigate against them is also applicable for anyone intending to conduct research at any level.

On the basis of some of the manuscripts I’ve reviewed over the years, I couldn’t help feeling as well that there are a number of university-level teachers setting out for the first time to do pedagogic research who might usefully pick up on some of the do’s and dont’s of experimental design onto which this course sheds some light.

So, in short, I have had a Green Eggs and Ham conversion regarding A level Psychology and, with apologies to Dr Seuss, I say to  “the Russell Group” and others who dismiss it out of hand, “Try it, try it and you may. Try it and you may I say”. Oh, and if you really didn’t know the ending of GE&H… sorry.

*If you are interested the facilitating subjects are: Maths and further maths; Physics; Biology; Chemistry; History; Geography; Modern and classical languages; English Literature.