The perils on anonymity in educational research

The promise of anonymity can undermine the value of data

The promise of anonymity can undermine the value of education data

From time to time I am asked to comment on other people’s unpublished research. As part of the evidence offered in the manuscript, it is quite common to see analysis based on anonymous questionnaires conducted before and after a pedagogic intervention. In this post I want to raise some concerns about the significant limitations that arise from the unnecessary anonymisation of survey data.

Why offer anonymity?
Firstly, however, it is worth examining the allure of anonymity. From conversations I’ve held with colleagues, the main attraction of anonymisation is the perception that removal of identifiers will free participants to provide full and frank contributions, secure in the knowledge that there can be no personal come-back.

I want to argue here that there are important research benefits from *avoiding* complete anonymity, except in the vanishingly rare occasions where it is vital that contributors cannot be recognised.

1. Keeping identifiers allows for richer analysis. If you can match pre- and post-intervention data it is possible to report on changes relating to individuals which may have been masked by analysis of the cohort as a whole.

2. Keeping identifiers guards against inappropriate comparison of whole cohort data. There is a temptation to take all of the available pre-intervention data and compare it with the complete set of post-intervention data, thereby ensuring that a minimum of data is “wasted”. I believe that this is wrong-headed and to illustrate this point, consider the following scenario in education research. Continue reading

Questionnaire design: some tips on generating meaningful data

At the November 2011 meeting of our Bioscience Pedagogic Research group, attention was focused on Questionnaire Design. Emma Angell, from the University’s SAPPHIRE group (Social science APPlied to Healthcare Improvement REsearch) shared some tips she had picked up during a two-day course which she had attended in May 2011. The course took place at the London School of Economics and was led by Jon Krosnick of Stanford University, and Emma was keen to stress that credit for the insights was his not hers!

As the website advertising the original course points out: “Surveys and questionnaires are a common way of gathering data in the social sciences. The structuring, wording and ordering of questions has traditionally been viewed as an art, not a science, best guided by intuition. But in recent years, it has become clear that this is an antiquated and even dangerous view that does not reflect the accumulation of knowledge throughout the social sciences about effective question-asking. Intuition often leads us astray in the questionnaire design field, as becomes clear when putting intuitions to the test via scientific evaluation. A large body of relevant scientific studies has now accumulated, and when taken together, the findings point to a series of formal rules for how best to design questions.”

Emma talked us through a number of potential problems with questionnaires that can undermine the legitimacy of the data they generate. In gathering questionnaire-based data, we hope that the person surveyed is able to interpret the meaning of the question, searches for the most appropriate pre-set response (or offers a thorough and accurate open text response) and in so doing gives a true reflection of their views and/or experiences.  To do so will require them to search thoroughly for an appropriate memory and to convert that information into an  answer that correlates with the question asked. If they are doing this, then they are “optimising”.

Continue reading