Oral versus written assessments

The January 2012 meeting of the Bioscience Pedagogic Research group at the University of Leicester included a “journal club” discussion of a paper Oral versus written assessments: a test of student performance and attitudes by Mark Huxham and colleagues from Napier University, Edinburgh. The paper had recently been published online in advance of a paper copy appearing in the February 2012 edition of Assessment and Evaluation in Higher Education.

To kick off the discussion I shared the following slides:

We had a good debate about the paper. For the most-part we thought it was an interesting and thought-provoking study, prompting us to consider greater use of oral examinations in the assessment repertoire at Leicester. A few questions were raised. It was felt to be a pity that the authors had not included an evaluation of the overall staff time involved in oral assessment versus written assessment (particularly for the first year cohort that had been randomly assigned one or other task). This would have been a valuable addition.

I don’t claim to be statistically-minded, but those with greater expertise in this field felt that the Mann-Whitney U-test might have been better than Student’s t-test for comparison of student scores in the oral and written assessments. The notion that a p-value of 0.079 was “not quite a significant difference” (p130) also ruffled some feathers.

Aside from these relatively minor issues, it was felt that the Napier study was a useful addition to the canon on assessment and readers of this short reflection are encouraged to seek out the original paper.

Thanks to Mark Huxham for some e-mail discussion prior to the meeting.

Marking (in)consistency – the elephant in the assessment room?

In September 2006 Banksy (briefly) included a painted "Elephant in the Room" in his LA show

In a thought-provoking article, available online ahead of publication in the February 2012 edition of Assessment and Evaluation in Higher Education, Teresa McConlogue looks into the pedagogical benefits of peer assessment. Her paper But is it fair? Developing students’ understanding of grading complex written work through peer assessment focuses on work conducted with engineering students at Queen Mary University of London.

Two distinct cohorts of students were required to peer assess a piece of coursework, leading to generation of a summative mark; a laboratory report (n=56, 10% of mark for module) and a literature review (n=26, 25%). Each piece of work was assessed by 4 or 5 peers who were required to provide both a mark and comments on the work. The students were then awarded the mean mark.

Thus far there is nothing exceptional about this process – peer assessment is an established practice in Higher Education (see, for example, Paul Orsmond’s excellent guide on Self- and Peer-Assessment). The controversial element of McConlogue’s activity comes with the fact that the authors of the peer-assessed work were provided with all of the comments made by their contemporaries AND a full record of the range of marks awarded. This “warts and all” approach exposed the students to the mechanics of marking – showing them both the reasoning that went into a mark (some of which seemed poorly aligned with the mark awarded or based on ‘trivialities’) and the fact that an individual “rogue” mark may have significantly influenced the mean. In some cases the individual marks awarded apparently spanned  several grade boundaries.

Involving alumni in careers education

The CABS programme started in 2007, and the supporting website in 2009

The December 2011 edition of Bioscience Education included an account I wrote concerning our Careers After Biological Science (CABS) programme at the University of Leicester. The CABS series of careers talks was started in 2007. Since 2009 it has been supported and enhanced by the Bioscience careers blog which includes copies of the slides used in the presentations, as well as a variety of videos and/or audio recordings.

As the Abstract of the paper states:

Graduate employability is an important concern for contemporary universities. Alongside the development of employability skills, it is also crucial that students of bioscience, a ‘non-vocational’ subject, have awareness of the breadth of potential careers that can follow from their initial degree.

Over the past five years we have developed the Careers After Biological Science (CABS) programme. Former students are invited back to describe their current role and offer practical advice to undergraduates who may be considering moving into a similar discipline. The speakers’ career profiles and associated resources are then collated onto an open-access website for the benefit of the wider community.

This project is characterised by two principal innovations; the pivotal role of alumni in the delivery of careers education, and the integrated use of multiple social media (web2.0) technologies in both the organisation of careers events and development of an open access repository of careers profiles and associated resources.

To read the full article “Here’s one we prepared earlier”: involving former students in careers advice click here.

An instrument to evaluate Assessment for Learning

A&EinHE now has an impact factor

Assessment for Learning (AfL) has been a key notion in recent curriculum developments in both secondary and tertiary education (see this link for previous left-handed biochemist posts on AfL).

The December 2011 edition of Assessment and Evaluation in Higher Education featured a paper Does assessment for learning make a difference? The development of a questionnaire to explore the student response by Liz McDowell and colleagues from the recently-closed AfL CETL in Northumbria. Quoting AfL guru Paul Black, the authors point out that the definition of Assessment for Learning has become overly flexible, “a free brand name to attach to any practice,” before clarifying that for them AfL must encompass six dimensions:

• Formal feedback – e.g. from tutor comments or self-assessment
• Informal feedback – e.g. from peer interaction or dialogue with staff
• Practice – opportunity to try out skills and rehearse understanding
• Authenticity – assessment tasks must have real-life relevance
• Autonomy – activities must help students develop independence
• Summative/Formative balance – involves an appropriate mix of both tasks that are “for marks” and those that are not

The bulk of the paper describes the development and testing of a questionnaire used for evaluation of students’ experience of a module. The questionnaire, which can be downloaded from the AfL CETL website, could be used to provide evidence to justify curriculum change and/or to support the case for quality enhancement. Each of the questions maps to at least one of the six key dimensions.

In analysing the use of this research instrument to evaluate modules at their own institution, the authors highlighted three principal factors distinguishing AfL and non-AfL courses: staff support and module design; engagement with subject matter; and the role played by peer support. Overall they suggest that the student experience was more positive in modules where AfL approaches were employed.

Institutional repositories, social media and academic publication: a simple experiment

Over at Science of the Invisible, my colleague Alan Cann has been reflecting on the contemporary landscape within academic publication. Specifically, he’s been thinking aloud about the role played by institutional repositories alongside (or, more radically, instead of) more formal journal publication (for example, see Wit’s End, which links in turn to Melissa Terras’ post What happens when you tweet an open access paper).

Institutional repositories are playing an increasingly important role in academic publishing

Prompted by Alan and Melissa’s enthusiasm for using social media to promote awareness of published work, in mid-November I started to use Twitter to advertise the existence of some of the papers I have deposited in the Leicester Research Archive (LRA). Some of my tweets were retweeted by others in the community, especially Alan, who also shared some of these within his Google+ circles.

Partway through this process it occurred to me that I had stumbled into a little experiment. So in the end I selectively tweeted about 8 of the 27 documents I currently have in the LRA. Admittedly these were probably the 8 papers that I felt were of most interest to the broader community on Twitter, but this did not mean they had previously received the most hits in the archive. In fact, if you rank the 25 works that had been in the Leicester repository throughout the 6 months (May to October 2011) from most to least popular,  then these 8 were ranked: 4th, 5th, 10th, 12th, 13th, 18th, 23rd and 24th= (2 documents were not added to the archive until November).

Questionnaire design: some tips on generating meaningful data

At the November 2011 meeting of our Bioscience Pedagogic Research group, attention was focused on Questionnaire Design. Emma Angell, from the University’s SAPPHIRE group (Social science APPlied to Healthcare Improvement REsearch) shared some tips she had picked up during a two-day course which she had attended in May 2011. The course took place at the London School of Economics and was led by Jon Krosnick of Stanford University, and Emma was keen to stress that credit for the insights was his not hers!

As the website advertising the original course points out: “Surveys and questionnaires are a common way of gathering data in the social sciences. The structuring, wording and ordering of questions has traditionally been viewed as an art, not a science, best guided by intuition. But in recent years, it has become clear that this is an antiquated and even dangerous view that does not reflect the accumulation of knowledge throughout the social sciences about effective question-asking. Intuition often leads us astray in the questionnaire design field, as becomes clear when putting intuitions to the test via scientific evaluation. A large body of relevant scientific studies has now accumulated, and when taken together, the findings point to a series of formal rules for how best to design questions.”

Emma talked us through a number of potential problems with questionnaires that can undermine the legitimacy of the data they generate. In gathering questionnaire-based data, we hope that the person surveyed is able to interpret the meaning of the question, searches for the most appropriate pre-set response (or offers a thorough and accurate open text response) and in so doing gives a true reflection of their views and/or experiences.  To do so will require them to search thoroughly for an appropriate memory and to convert that information into an  answer that correlates with the question asked. If they are doing this, then they are “optimising”.

Effective Learning in the Life Sciences

The book was edited by David Adams, Director of the UK Centre of Biosciences until September 2011

Today I have received my copy of Effective Learning in the Life Sciences: how students can achieve their full potential. As the subtitle implies, the book is targeted first and foremost at students wanting to make the most of their time at university, and at academics helping them to reach that goal.

1. Creativity (David Adams and Kevin Byron)

2. Problem solving: developing critical, evaluative, and analytical thinking skills (Tina Overton)

3. In the laboratory (Pauline Millican and David Adams)

4. Fieldwork (Julie Peacock, Julian Park and Alice Mauchline)

5. In vivo work (David Lewis)

6. Research projects (Martin Luck)

7. Maths and stats for biologists (Dawn Hawkins)

8. E-learning for biologists (Jo Badge, Jon Scott and Terry McAndrew)

9. Bioethics (Chris Willmott)

10. Assessment, feedback and review (Steve Maw and Paul Orsmond)

11. Communication in the biosciences (Joanna Verran and Maureen Dawson)

12. Bioenterprise (Lee Beniston, David Adams and Carol Wakeford)

Graduate employability: a growing concern

During the research for a recently-submitted paper, I decided to investigate the rising importance of graduate employability as a concern for universities (and the wider society). As an indicator of this trend I searched Google Scholar for articles with “graduate” and “employability” in the title – the results are shown in the chart below.

A survey of Google Scholar looking at the number of articles published over past thirty years with "graduate" and "employability" in the title

The increase in papers on graduate employability is striking, but probably not a surprise. Having done this research, however, I elected not to include the data in the paper. Why? My main concern was uncertainty about appropriate controls for the fact that there has been a general increase in information (specifically academic literature) during the same period. I was therefore uncomfortable about the dangers of over-interpretation.

Should I have worried? Is it a valid observation? What could serve as a legitimate control?  Any thoughts gratefully received.

The best and worst of the OU

I have been a long-time admirer of the Open University; my mother completed a degree with them when I was a child and another of my relatives was one of the first ever cohorts of OU students. At a recent conference a presentation on the OU’s new “Science Investigations” module* was truly inspiring – the notion of involving novice scientists dispersed across the planet in collaborative experiments shows real vision.

The ERA allows for legal use of TV and radio programmes

At the same time, however, there is something about the OU that I have found increasingly frustrating. As a frequent user of multimedia clips in my teaching, I take advantage of the Educational Recording Agency (ERA) licensing scheme that permits educators to hold copies of TV and radio programmes specifically for teaching purposes and provided that they adhere to a number of straightforward rules regarding both the storage and use of the material (I have written about the merits of the ERA licensing scheme).

.

For some reason, however, programmes that are produced by the OU fall outside the ERA scheme. To hold and to show OU broadcasts you need a different licence. I imagine the source of this anomaly may be historical – the OU may have set up their arrangement with other institutions before the ERA scheme was developed (I speculate, I don’t have any data on this). Unlike the ERA scheme, there is an annual fee (of about £30) to hold a copy of an OU programme and if you decide you no longer want to use the programme then you need to physically return your copy to them. For a number of reasons, which I will outline below, it strikes me that this arrangement is increasingly anachronistic.

The OU run their own off-air scheme, distinct from the ERA

What brings this issue into sharp focus for me is the increasing number of programmes that are branded as OU/BBC co-productions. If these fell within the ERA scheme I would happily use clips of all of the following co-productions within my teaching:

• Bang Goes the Theory

• The Cell (3-part series)

• The Gene Code (2-part series)

• Mental: A History of the Madhouse and Sectioned

• Justice: A Citizen’s Guide to the 21st Century (and associated Justice lectures)

• Can Gerry Robinson Fix the NHS?

• Charles Darwin and the Tree of Life

• Inside the Ethics Committee (Radio 4)

.

For the most part I would be looking to use a clip of perhaps 2 or 3 minutes in each case, rather than showing a full episode (though there may be an exception here for Adam Rutherford’s excellent series The Cell and The Gene Code).

.

The popular series Coast, and Jim Al-Khalili’s current series Shock and Awe: the story of electricity  are also OU/BBC co-productions but don’t really fit with my teaching which focuses on bioethics and/or molecular biology.

Anonymous data and educational research

Comparing 'before' and 'after' data needs some identification

When undertaking educational research you often want to know how an intervention has affected a cohort, and ideally to be able to drill down into the data to see the impact on individuals. In order to match pre-and post- activity surveys, some kind of identifier is required. You could ask the students to put their names on the forms, but they may have concerns that this will have ramifications for their coursework. What else you could do?

There are a range of semi-anonymised labels you could use. At various times in my own work I’ve used formal candidate number, email username and date of birth (the latter often throws up more than one student with the same date, but handwriting can then distinguish). In each of these cases, however, it remains a relatively trivial step for someone with access to the right databases to decode the label and convert it into a name. Of course there is generally no reason why a researcher would want to do this, and students trust that you are not going to waste your precious time doing so.

What else might you do? You could ask the students to pick a bogus name or their favourite superhero, but these run several risks – including having surveys completed multiple “lady gaga”s or “dr [insert your name here]“. The students might also forget the random name they picked between the first and the second test.