To kick off the discussion I shared the following slides:

We had a good debate about the paper. For the most-part we thought it was an interesting and thought-provoking study, prompting us to consider greater use of oral examinations in the assessment repertoire at Leicester. A few questions were raised. It was felt to be a pity that the authors had not included an evaluation of the overall staff time involved in oral assessment versus written assessment (particularly for the first year cohort that had been randomly assigned one or other task). This would have been a valuable addition.

I don’t claim to be statistically-minded, but those with greater expertise in this field felt that the Mann-Whitney U-test might have been better than Student’s t-test for comparison of student scores in the oral and written assessments. The notion that a p-value of 0.079 was “not quite a significant difference” (p130) also ruffled some feathers.

Aside from these relatively minor issues, it was felt that the Napier study was a useful addition to the canon on assessment and readers of this short reflection are encouraged to seek out the original paper.

Thanks to Mark Huxham for some e-mail discussion prior to the meeting.

In September 2006 Banksy (briefly) included a painted "Elephant in the Room" in his LA show

In a thought-provoking article, available online ahead of publication in the February 2012 edition of Assessment and Evaluation in Higher Education, Teresa McConlogue looks into the pedagogical benefits of peer assessment. Her paper But is it fair? Developing students’ understanding of grading complex written work through peer assessment focuses on work conducted with engineering students at Queen Mary University of London.

Two distinct cohorts of students were required to peer assess a piece of coursework, leading to generation of a summative mark; a laboratory report (n=56, 10% of mark for module) and a literature review (n=26, 25%). Each piece of work was assessed by 4 or 5 peers who were required to provide both a mark and comments on the work. The students were then awarded the mean mark.

Thus far there is nothing exceptional about this process – peer assessment is an established practice in Higher Education (see, for example, Paul Orsmond’s excellent guide on Self- and Peer-Assessment). The controversial element of McConlogue’s activity comes with the fact that the authors of the peer-assessed work were provided with all of the comments made by their contemporaries AND a full record of the range of marks awarded. This “warts and all” approach exposed the students to the mechanics of marking – showing them both the reasoning that went into a mark (some of which seemed poorly aligned with the mark awarded or based on ‘trivialities’) and the fact that an individual “rogue” mark may have significantly influenced the mean. In some cases the individual marks awarded apparently spanned  several grade boundaries.

Qualitative evaluation (questionnaires and focus groups) showed that the students frequently found this to have been an unsettling experience. Exposure to a divergent range of scores ran contrary to an ingrained expectation that there ought to be a “correct” mark for their work. The author expresses surprise that many students also thought the process was “unfair” (especially since those with grievances were allowed to ask for a reassessment of the work by staff, a process that generally ended with a mark very close to the originally-awarded mean).

Some of the benefits of involvement in peer assessment can be accrued from formative rather than summative tasks (as, for example, in my own work ‘You have 45 minutes, starting from now’: helping students develop their exam essay skills, in which students offer formative feedback on essays prior to summative marking by tutors). The benefits for students include opportunities to understand more fully the standard of work expected at university (clarifying ‘the rules of the game’, Carless (2006)) and to benchmark their own work alongside those of their contemporaries (In my exercise they also get an appreciation of the difficulty markers have with the legibility of some students’ work).

Embracing subjectivity?

Providing students with only the mean mark rather than the individual scores would have avoided some of the flak received in the feedback. For McConlogue, however, the exposure of students to the inherent subjectivity of marking represents one of the potential benefits of the exercise. Engagement with the nuances of marking can enhance students’ critical thinking skills and provides opportunity for “assessment dialogue”. There is educational merit in realising that diverse influences can shape the mark awarded for a piece of work and that “application of assessment criteria in HE is a matter of professional judgement not a matter of fact” (Bloxham, 2009). Despite this, comments from participating students give the impression that they would have preferred the work to have been marked by staff since they are perceived to bring (a) greater subject knowledge and (b) greater marking experience to the process and are therefore more likely to get the mark “right”.

Whilst the students may expect their tutors to do a better job than their peers, McConlogue’s review of the literature on consistency in marking by academics suggests that this faith may be unwarranted. In many ways this was the most interesting feature of the paper for me since it puts into the spotlight the issue that is so often the “elephant in the room”, the suspicion that marking – even marking by experienced tutors – is not as reliable as we would like to believe.

Drawing on work by Sue Bloxham (2009), Suellen Shay (2005), John Schacter (1999) and  Susan Orr (2006), amongst other, McConlogue notes that marking of anything more than trivial/fact-regurgitation tasks can be prone to inconsistency. The differences may stem from all manner of factors, including the time of day, the number of scripts and the order of marking (i.e. a mediocre piece of work may look poor when marked directly after a first-class essay whereas it may stand out in a positive way if the preceding script was lousy).

To try an counter these effects, course directors organise training for markers and provide ever more proscriptive assessment guidelines, assessment criteria, and/or grade descriptors in advance of marking so as to standardise tutor responses. This is supplemented after the event by moderation across markers. Yet despite these interventions variability persists.

Should we worry? As someone who takes the moderation process seriously and spends many hours tweaking up or down the marks awarded by different tutors, I would like to think that students are usually getting as fair a mark as possible. It would, however, be naive to think that unwarranted differences never occur. On one level this is simply a manifestation of the broader injustices that exist in life. It could also be argued that a 5% or even 10% “error” in the marking of one piece of work that counts for 10% of the mark for a module that in turn represents 20 credits out of a total of 120 credits in a year than contributes 40% to an overall degree classification is not something about which we should be unduly concerned. Over the course of an entire degree programme, individual differences are going to even out.

Yet, at the same time, McConlogue’s paper is a timely reminder that we ought to be doing everything reasonable and practicable to ensure that fairness is achieved. Whether or not the benefits of peer assessment (of summative work) outweigh potential issues of consistency between student markers is an issue with which individual module convenors will have to grapple.

The CABS programme started in 2007, and the supporting website in 2009

The December 2011 edition of Bioscience Education included an account I wrote concerning our Careers After Biological Science (CABS) programme at the University of Leicester. The CABS series of careers talks was started in 2007. Since 2009 it has been supported and enhanced by the Bioscience careers blog which includes copies of the slides used in the presentations, as well as a variety of videos and/or audio recordings.

As the Abstract of the paper states:

Graduate employability is an important concern for contemporary universities. Alongside the development of employability skills, it is also crucial that students of bioscience, a ‘non-vocational’ subject, have awareness of the breadth of potential careers that can follow from their initial degree.

Over the past five years we have developed the Careers After Biological Science (CABS) programme. Former students are invited back to describe their current role and offer practical advice to undergraduates who may be considering moving into a similar discipline. The speakers’ career profiles and associated resources are then collated onto an open-access website for the benefit of the wider community.

This project is characterised by two principal innovations; the pivotal role of alumni in the delivery of careers education, and the integrated use of multiple social media (web2.0) technologies in both the organisation of careers events and development of an open access repository of careers profiles and associated resources.

To read the full article “Here’s one we prepared earlier”: involving former students in careers advice click here.

A&EinHE now has an impact factor

Assessment for Learning (AfL) has been a key notion in recent curriculum developments in both secondary and tertiary education (see this link for previous left-handed biochemist posts on AfL).

The December 2011 edition of Assessment and Evaluation in Higher Education featured a paper Does assessment for learning make a difference? The development of a questionnaire to explore the student response by Liz McDowell and colleagues from the recently-closed AfL CETL in Northumbria. Quoting AfL guru Paul Black, the authors point out that the definition of Assessment for Learning has become overly flexible, “a free brand name to attach to any practice,” before clarifying that for them AfL must encompass six dimensions:

• Formal feedback – e.g. from tutor comments or self-assessment
• Informal feedback – e.g. from peer interaction or dialogue with staff
• Practice – opportunity to try out skills and rehearse understanding
• Authenticity – assessment tasks must have real-life relevance
• Autonomy – activities must help students develop independence
• Summative/Formative balance – involves an appropriate mix of both tasks that are “for marks” and those that are not

The bulk of the paper describes the development and testing of a questionnaire used for evaluation of students’ experience of a module. The questionnaire, which can be downloaded from the AfL CETL website, could be used to provide evidence to justify curriculum change and/or to support the case for quality enhancement. Each of the questions maps to at least one of the six key dimensions.

In analysing the use of this research instrument to evaluate modules at their own institution, the authors highlighted three principal factors distinguishing AfL and non-AfL courses: staff support and module design; engagement with subject matter; and the role played by peer support. Overall they suggest that the student experience was more positive in modules where AfL approaches were employed.

Institutional repositories are playing an increasingly important role in academic publishing

Prompted by Alan and Melissa’s enthusiasm for using social media to promote awareness of published work, in mid-November I started to use Twitter to advertise the existence of some of the papers I have deposited in the Leicester Research Archive (LRA). Some of my tweets were retweeted by others in the community, especially Alan, who also shared some of these within his Google+ circles.

Partway through this process it occurred to me that I had stumbled into a little experiment. So in the end I selectively tweeted about 8 of the 27 documents I currently have in the LRA. Admittedly these were probably the 8 papers that I felt were of most interest to the broader community on Twitter, but this did not mean they had previously received the most hits in the archive. In fact, if you rank the 25 works that had been in the Leicester repository throughout the 6 months (May to October 2011) from most to least popular,  then these 8 were ranked: 4th, 5th, 10th, 12th, 13th, 18th, 23rd and 24th= (2 documents were not added to the archive until November).

So what happened? I’m still picking over the data, but here are some of the main take-home messages:

• All of the resources promoted via Twitter were accessed more times than any of the resources that were not advertised in this way.
• Apart from the resources that were newly listed in the archive in November (and de facto got 100% of their views within that month), the highest scores for “hits in November as percentage of lifetime hits” were also all for the promoted articles. The range was 36.7% of lifetime hits for An exercise to teach bioscience students about plagiarism, through to 77.6% for The increasing significance of ethics in the bioscience curriculum and 81.8% for Ethics and Plagiarism – helping undergraduates write right.
• Interestingly, these highest two resources (as a percentage of lifetime views) were for articles that had been listed in the archive for several months, but had only had an authors’ draft copy deposited as a PDF file within the last few weeks. I wouldn’t want to overplay this observation though, since the third highest percentage (76.6%) was for ‘You have 45 minutes, starting from now’: Helping Students Develop their Exam Essay Skills, which has been available via the archive for much longer.
• The highest three articles downloaded from the site (lifetime statistics, not specifically November 2011) are all for resources that are not freely available as open access resources via any other (official) route.
• My most accessed resource remain our chapter “Ain’t We All the Same? Underneath, Ain’t We All Kin?” Humans, Daleks and the Species Problem in Doctor Who, which was not actively promoted in November. Carus Publishing, the publishers of the book Doctor Who and Philosophy: Bigger on the Inside, from which this chapter is taken, were kind enough to let us post the final proofs of the chapter. This is a Popular Culture and Philosophy title and as such is probably being accessed by a wider audience than the other more academic papers.

Leicester’s Gareth Johnson, current chair of the UK Council of Research Repositories, suggests that after publishing any paper academics ought to provide copies to their local archive (subject to copyright restrictions) and then alert colleagues to the existence of the work. Gareth calls this the ”Publish – Deposit – Share” model of academic publishing.

There is no doubt that if you have gone to the trouble of carrying out research and/or developing a new resource, and then writing the relevant paper, then it seems logical to use whatever channels are at your disposal to you to alert interested parties to the existence of the work.  Wherever possible, you will want to provide a copy of the text and institutional repositories are a good way of doing this, even if it is not the final, formatted version. My experience over the last month clearly illustrates the value of using Twitter and Google+ to advertise your work.

As the website advertising the original course points out: “Surveys and questionnaires are a common way of gathering data in the social sciences. The structuring, wording and ordering of questions has traditionally been viewed as an art, not a science, best guided by intuition. But in recent years, it has become clear that this is an antiquated and even dangerous view that does not reflect the accumulation of knowledge throughout the social sciences about effective question-asking. Intuition often leads us astray in the questionnaire design field, as becomes clear when putting intuitions to the test via scientific evaluation. A large body of relevant scientific studies has now accumulated, and when taken together, the findings point to a series of formal rules for how best to design questions.”

Emma talked us through a number of potential problems with questionnaires that can undermine the legitimacy of the data they generate. In gathering questionnaire-based data, we hope that the person surveyed is able to interpret the meaning of the question, searches for the most appropriate pre-set response (or offers a thorough and accurate open text response) and in so doing gives a true reflection of their views and/or experiences.  To do so will require them to search thoroughly for an appropriate memory and to convert that information into an  answer that correlates with the question asked. If they are doing this, then they are “optimising”.

One key danger is “satisficing”, a term coined by Herbert Simon to cover the combination of “satisfying” and “sufficing” to describe behaviour in which respondents do not fully engaging with the exercise. It may be that they enter into the entire process in a superficial way, choosing the first answer that they think is acceptable, or picking an answer they think is expected. There may be a bias, for example, towards picking “agree” over “disagree”. Alternatively, they may take the easy way out, picking “don’t know” or omitting a question since they can subsequently claim that they did not properly understand what was being asked.

Factors influencing whether a respondent “optimises” or “satisfices” may include the difficulty of the task, their ability to perform the task and their motivation; if any of these are compromised then the respondent may be more prone to satisficing.

• Difficulty might include complexity of language, answer options that do not include their preferred response, the presence of distracting elements (either within the survey or in the environment where the survey is conducted), the overall length of the survey and – for oral questionnaires – the speed at which questions were asked.
• Ability might include prior experience of this type of questioning (it was noted, for example, that parents of children with long-term illness become more adept at answering surveys as they are familiar with the pattern and mental processes involved). If someone has answered similar surveys before, or has actually entered into profound thought on the topic independent of the survey, then they may be able to offer a well-formulated “preconsolidated” answer.
• Motivation includes the respondents personal views on the importance of the information being collected, their accountability for the answers they give, and appropriate encouragement from an interviewer during the course of the task.

Take-home messages from the session included:

• the need to phrase questions in a way that fitted, as far as possible, with conversational norms (e.g. “ladies and gentlemen” not “gentlemen and ladies”, avoiding double-negative where possible).
• If in doubt open questions are preferable to closed questions. It was noted in discussion that this presupposes the necessary resource to subsequently code and interpret the open responses. However, provided this is feasible, an open question ought to generate a truer answer. A set of stated answers plus “other…. please specify” was said to be an unsatisfactory model. If using questions with a pre-determined set of options then it is vital that adequate pilot testing has been undertaken to ensure that all necessary options have been included.
• Question order effects can be significant, with a satisficing  participant either picking the first response on the list (primacy effects) or the last option given, especially in oral surveys (recency effects).
• Where a scale of responses if offered, there ought to be an odd number of choices, ideally 7. The options should be offered in full, not as a series of numbers with only then ends and the middle options labelled.

The book was edited by David Adams, Director of the UK Centre of Biosciences until September 2011

Today I have received my copy of Effective Learning in the Life Sciences: how students can achieve their full potential. As the subtitle implies, the book is targeted first and foremost at students wanting to make the most of their time at university, and at academics helping them to reach that goal.

1. Creativity (David Adams and Kevin Byron)

2. Problem solving: developing critical, evaluative, and analytical thinking skills (Tina Overton)

3. In the laboratory (Pauline Millican and David Adams)

4. Fieldwork (Julie Peacock, Julian Park and Alice Mauchline)

5. In vivo work (David Lewis)

6. Research projects (Martin Luck)

7. Maths and stats for biologists (Dawn Hawkins)

8. E-learning for biologists (Jo Badge, Jon Scott and Terry McAndrew)

9. Bioethics (Chris Willmott)

10. Assessment, feedback and review (Steve Maw and Paul Orsmond)

11. Communication in the biosciences (Joanna Verran and Maureen Dawson)

12. Bioenterprise (Lee Beniston, David Adams and Carol Wakeford)

A survey of Google Scholar looking at the number of articles published over past thirty years with "graduate" and "employability" in the title

The increase in papers on graduate employability is striking, but probably not a surprise. Having done this research, however, I elected not to include the data in the paper. Why? My main concern was uncertainty about appropriate controls for the fact that there has been a general increase in information (specifically academic literature) during the same period. I was therefore uncomfortable about the dangers of over-interpretation.

Should I have worried? Is it a valid observation? What could serve as a legitimate control?  Any thoughts gratefully received.

The ERA allows for legal use of TV and radio programmes

At the same time, however, there is something about the OU that I have found increasingly frustrating. As a frequent user of multimedia clips in my teaching, I take advantage of the Educational Recording Agency (ERA) licensing scheme that permits educators to hold copies of TV and radio programmes specifically for teaching purposes and provided that they adhere to a number of straightforward rules regarding both the storage and use of the material (I have written about the merits of the ERA licensing scheme).

.

The OU run their own off-air scheme, distinct from the ERA

What brings this issue into sharp focus for me is the increasing number of programmes that are branded as OU/BBC co-productions. If these fell within the ERA scheme I would happily use clips of all of the following co-productions within my teaching:

.

Comparing 'before' and 'after' data needs some identification

When undertaking educational research you often want to know how an intervention has affected a cohort, and ideally to be able to drill down into the data to see the impact on individuals. In order to match pre-and post- activity surveys, some kind of identifier is required. You could ask the students to put their names on the forms, but they may have concerns that this will have ramifications for their coursework. What else you could do?

There are a range of semi-anonymised labels you could use. At various times in my own work I’ve used formal candidate number, email username and date of birth (the latter often throws up more than one student with the same date, but handwriting can then distinguish). In each of these cases, however, it remains a relatively trivial step for someone with access to the right databases to decode the label and convert it into a name. Of course there is generally no reason why a researcher would want to do this, and students trust that you are not going to waste your precious time doing so.

What else might you do? You could ask the students to pick a bogus name or their favourite superhero, but these run several risks – including having surveys completed multiple “lady gaga”s or “dr [insert your name here]“. The students might also forget the random name they picked between the first and the second test.

During a recent coffee-time conversation with a colleague from the social sciences, she recommended a system for developing an essentially unbreakable code for times when proper anonymity is necessary (or desirable) whilst retaining the potential to match ‘before’ and ‘after’ surveys.

The system generates a series of letters and numbers that is consistent and sufficiently detailed to be unique to an individual without allowing participant identification. The questions used to produce the code are an amalgamation of the sort of security questions banks and websites sometimes pose to authenticate ID. By combining components from a variety

1. The first letter of the town or city where you were born (e.g. if born in Lincoln, put L).

2. The last letter of your mother’s maiden name (e.g. if her name was Carter, put R)

3. The first digit of your house number (e.g. if you live at number 43, put 4). If the house only has a name, put zero.

4. The second digit of the day of the month in which you are born (e.g. if born on the 27th September, put a 7).

5. The second letter in the name of your oldest brother or sister (e.g. if your brother is called Austin, put U). If you are an only child put ‘Z’.

This string of five letters and numbers ought to be adequate to produce a unique code for each participant. For most pedagogic research this is over and above the necessary security, but for times when the data is particularly sensitive it might give a useful.