Finding research articles that are worth finding

An enormous amount of scientific literature is generated each month

The world is awash with scientific papers. Even if we restricted a survey to research within biological sciences, I guestimate that there are more papers published each month than your average academic could be expected to read in a lifetime. In these rich fields of information, how are students unfamiliar with the genre  to develop the ability to discern the wheat from the chaff (let alone the weeds)?

At the University of Leicester, we have a task for undergraduates, conducted towards the end of their second year, in which they produce a poster describing a particular research method (selected from a short list relevant to their chosen discipline). As part of the exercise, the students need to choose a primary research article which illustrates one application of that method.

It is fair to say that many of the students initially struggle to select an appropriate paper*. There are several reoccurring  problems:

• Failure to distinguish between a research article and a review
• Failure to recognise that all journals are not equal in terms of their academic quality and rigour of the work they publish
• Selection of papers that do not really utilise the technique that should be the focus of their poster.

In truth, the ability to select the right kind of paper is one of the important learning outcomes from this exercise; these students will commence their final year dissertations immediately after they return from summer vacation and need to avoid wasting hours reading papers that are not worthy of their attention. However, since I frequently find myself making the same points in email correspondence with individual students, I felt it was worth using this forum to share some of my overarching reflections on the fine art of finding appropriate research articles.

Research Article or Review?

Most searches these days are conducted online

When we’ve been involved in academia for many years, knowing whether a paper is a primary research paper or a review comes as second nature. This is not necessarily true for inexperienced students. In their defence, the variation of the naming of research articles in different journals does not help. Cell and EMBO Journal call them “Articles”. Nature also has “Articles” but the majority of original research is labelled as “Letters” and in Science primary literature is mostly “Reports” .

Although it ought to be possible to spot a research article by the sub-sections it contains (i.e. Abstract, Introduction, Materials and Methods, etc…) these are not necessarily given the same names in all journals, and the order in which they included can be different. Of course, if the journal has the word “Review”, “Trends in…” or “Current opinions in…” then it is likely to include, almost exclusively, articles summarising the research of other scientists rather than containing original reports of new experiments . But in other journals a review might actually be called “Perspectives”, “Commentary” or “News and Views” to name but three.

Review articles as a mean to an end

For the specific activity we set our students (finding a primary research paper that demonstrates the use of a particular technique), reviews are not going to be a suitable destination. This does not mean, however, that review articles are of no merit in the search for good quality experimental data. In some senses, the authors of a review article have done crucial legwork for you. Already experts in the field, they have read broadly about the topic and will then have selected what they consider to be the most significant recent experiments for their reflections. Looking at the reference list in a review can therefore be an efficient way to shine a light on the best primary literature.

Is it a good paper?

If you are not familiar with a research technique or particular discipline, how can you know whether a paper is a “good” paper? There are three useful clues. The first is the one we have just described, namely do authors of reviews rate the work as worthy of their attention?

Secondly, where has it been published? Like it or not, there is a hierarchy of journals; some titles are in the Premier League, some are in the Championship, and some are non-league. Experienced heads know that research published in Nature, Science, Cell, Proceedings of the National Academy of Science and EMBO Journal is considered (usually with some justification) to be more worthy than papers coming out in the Journal of Knitting and Spectroscopy, but recognising this is, once again, an aspect of the maturing of undergraduates into fully-fledged scientists.

Thirdly, how many times has the work been cited by other researchers? If work has been in circulation for fifteen years but has only been quoted twice during that period, and both times by the author of the original paper, then it is fair to conclude that it did not contain ground-breaking discoveries. One of the useful features of Web of Knowledge is the citation count. If you don’t have access to Web of Knowledge, Google Scholar has a similar (though slightly less rigorous) feature.

Finally, if you want to refine a search so that it is restricted to only those “Premier league” journals name above,  then why not go to a journal’s own search engine rather than, or in addition to, a more generic search tool? (e.g. for Nature, for Science).

* Note, if you are wondering how this squares with my recent description of a  source evaluation exercise for first years, I need to point out that this is NOT the same cohort.

Use of social media in careers education

At the Higher Education Academy STEM Conference in April 2012 I gave a presentation about our Careers After Biological Science project at the University of Leicester. The focus of the talk was the pivotal role played by social media in recruiting speakers for careers talks, archiving various resources associated with those careers, and advertising their existence to a broader audience.

Slides from the talk are available below (or, in the event that they haven’t loaded properly, via this link )

But is it any good? An information literacy tutorial

At the Higher Education Academy STEM conference in April 2012, I presented a poster offering an outline into a blended-learning tutorial we have produced in order to help undergraduates develop their abilities to evaluate the academic merit of different resources they might find on the internet. The tutorial involves the students working individually to critique eight specially chosen online sources presented as the results of a search on the topic of “mitochondria”. This is followed up by a group tutorial in which the quality and relevance of the materials are discussed more fully.

To see a pdf version of the poster, click on this image

Oral versus written assessments

The January 2012 meeting of the Bioscience Pedagogic Research group at the University of Leicester included a “journal club” discussion of a paper Oral versus written assessments: a test of student performance and attitudes by Mark Huxham and colleagues from Napier University, Edinburgh. The paper had recently been published online in advance of a paper copy appearing in the February 2012 edition of Assessment and Evaluation in Higher Education.

To kick off the discussion I shared the following slides:

We had a good debate about the paper. For the most-part we thought it was an interesting and thought-provoking study, prompting us to consider greater use of oral examinations in the assessment repertoire at Leicester. A few questions were raised. It was felt to be a pity that the authors had not included an evaluation of the overall staff time involved in oral assessment versus written assessment (particularly for the first year cohort that had been randomly assigned one or other task). This would have been a valuable addition.

I don’t claim to be statistically-minded, but those with greater expertise in this field felt that the Mann-Whitney U-test might have been better than Student’s t-test for comparison of student scores in the oral and written assessments. The notion that a p-value of 0.079 was “not quite a significant difference” (p130) also ruffled some feathers.

Aside from these relatively minor issues, it was felt that the Napier study was a useful addition to the canon on assessment and readers of this short reflection are encouraged to seek out the original paper.

Thanks to Mark Huxham for some e-mail discussion prior to the meeting.

Marking (in)consistency – the elephant in the assessment room?

In September 2006 Banksy (briefly) included a painted "Elephant in the Room" in his LA show

In a thought-provoking article, available online ahead of publication in the February 2012 edition of Assessment and Evaluation in Higher Education, Teresa McConlogue looks into the pedagogical benefits of peer assessment. Her paper But is it fair? Developing students’ understanding of grading complex written work through peer assessment focuses on work conducted with engineering students at Queen Mary University of London.

Two distinct cohorts of students were required to peer assess a piece of coursework, leading to generation of a summative mark; a laboratory report (n=56, 10% of mark for module) and a literature review (n=26, 25%). Each piece of work was assessed by 4 or 5 peers who were required to provide both a mark and comments on the work. The students were then awarded the mean mark.

Thus far there is nothing exceptional about this process – peer assessment is an established practice in Higher Education (see, for example, Paul Orsmond’s excellent guide on Self- and Peer-Assessment). The controversial element of McConlogue’s activity comes with the fact that the authors of the peer-assessed work were provided with all of the comments made by their contemporaries AND a full record of the range of marks awarded. This “warts and all” approach exposed the students to the mechanics of marking – showing them both the reasoning that went into a mark (some of which seemed poorly aligned with the mark awarded or based on ‘trivialities’) and the fact that an individual “rogue” mark may have significantly influenced the mean. In some cases the individual marks awarded apparently spanned  several grade boundaries.

Involving alumni in careers education

The CABS programme started in 2007, and the supporting website in 2009

The December 2011 edition of Bioscience Education included an account I wrote concerning our Careers After Biological Science (CABS) programme at the University of Leicester. The CABS series of careers talks was started in 2007. Since 2009 it has been supported and enhanced by the Bioscience careers blog which includes copies of the slides used in the presentations, as well as a variety of videos and/or audio recordings.

As the Abstract of the paper states:

Graduate employability is an important concern for contemporary universities. Alongside the development of employability skills, it is also crucial that students of bioscience, a ‘non-vocational’ subject, have awareness of the breadth of potential careers that can follow from their initial degree.

Over the past five years we have developed the Careers After Biological Science (CABS) programme. Former students are invited back to describe their current role and offer practical advice to undergraduates who may be considering moving into a similar discipline. The speakers’ career profiles and associated resources are then collated onto an open-access website for the benefit of the wider community.

This project is characterised by two principal innovations; the pivotal role of alumni in the delivery of careers education, and the integrated use of multiple social media (web2.0) technologies in both the organisation of careers events and development of an open access repository of careers profiles and associated resources.

To read the full article “Here’s one we prepared earlier”: involving former students in careers advice click here.

An instrument to evaluate Assessment for Learning

A&EinHE now has an impact factor

Assessment for Learning (AfL) has been a key notion in recent curriculum developments in both secondary and tertiary education (see this link for previous left-handed biochemist posts on AfL).

The December 2011 edition of Assessment and Evaluation in Higher Education featured a paper Does assessment for learning make a difference? The development of a questionnaire to explore the student response by Liz McDowell and colleagues from the recently-closed AfL CETL in Northumbria. Quoting AfL guru Paul Black, the authors point out that the definition of Assessment for Learning has become overly flexible, “a free brand name to attach to any practice,” before clarifying that for them AfL must encompass six dimensions:

• Formal feedback – e.g. from tutor comments or self-assessment
• Informal feedback – e.g. from peer interaction or dialogue with staff
• Practice – opportunity to try out skills and rehearse understanding
• Authenticity – assessment tasks must have real-life relevance
• Autonomy – activities must help students develop independence
• Summative/Formative balance – involves an appropriate mix of both tasks that are “for marks” and those that are not

The bulk of the paper describes the development and testing of a questionnaire used for evaluation of students’ experience of a module. The questionnaire, which can be downloaded from the AfL CETL website, could be used to provide evidence to justify curriculum change and/or to support the case for quality enhancement. Each of the questions maps to at least one of the six key dimensions.

In analysing the use of this research instrument to evaluate modules at their own institution, the authors highlighted three principal factors distinguishing AfL and non-AfL courses: staff support and module design; engagement with subject matter; and the role played by peer support. Overall they suggest that the student experience was more positive in modules where AfL approaches were employed.

Institutional repositories, social media and academic publication: a simple experiment

Over at Science of the Invisible, my colleague Alan Cann has been reflecting on the contemporary landscape within academic publication. Specifically, he’s been thinking aloud about the role played by institutional repositories alongside (or, more radically, instead of) more formal journal publication (for example, see Wit’s End, which links in turn to Melissa Terras’ post What happens when you tweet an open access paper).

Institutional repositories are playing an increasingly important role in academic publishing

Prompted by Alan and Melissa’s enthusiasm for using social media to promote awareness of published work, in mid-November I started to use Twitter to advertise the existence of some of the papers I have deposited in the Leicester Research Archive (LRA). Some of my tweets were retweeted by others in the community, especially Alan, who also shared some of these within his Google+ circles.

Partway through this process it occurred to me that I had stumbled into a little experiment. So in the end I selectively tweeted about 8 of the 27 documents I currently have in the LRA. Admittedly these were probably the 8 papers that I felt were of most interest to the broader community on Twitter, but this did not mean they had previously received the most hits in the archive. In fact, if you rank the 25 works that had been in the Leicester repository throughout the 6 months (May to October 2011) from most to least popular,  then these 8 were ranked: 4th, 5th, 10th, 12th, 13th, 18th, 23rd and 24th= (2 documents were not added to the archive until November).

Questionnaire design: some tips on generating meaningful data

At the November 2011 meeting of our Bioscience Pedagogic Research group, attention was focused on Questionnaire Design. Emma Angell, from the University’s SAPPHIRE group (Social science APPlied to Healthcare Improvement REsearch) shared some tips she had picked up during a two-day course which she had attended in May 2011. The course took place at the London School of Economics and was led by Jon Krosnick of Stanford University, and Emma was keen to stress that credit for the insights was his not hers!

As the website advertising the original course points out: “Surveys and questionnaires are a common way of gathering data in the social sciences. The structuring, wording and ordering of questions has traditionally been viewed as an art, not a science, best guided by intuition. But in recent years, it has become clear that this is an antiquated and even dangerous view that does not reflect the accumulation of knowledge throughout the social sciences about effective question-asking. Intuition often leads us astray in the questionnaire design field, as becomes clear when putting intuitions to the test via scientific evaluation. A large body of relevant scientific studies has now accumulated, and when taken together, the findings point to a series of formal rules for how best to design questions.”

Emma talked us through a number of potential problems with questionnaires that can undermine the legitimacy of the data they generate. In gathering questionnaire-based data, we hope that the person surveyed is able to interpret the meaning of the question, searches for the most appropriate pre-set response (or offers a thorough and accurate open text response) and in so doing gives a true reflection of their views and/or experiences.  To do so will require them to search thoroughly for an appropriate memory and to convert that information into an  answer that correlates with the question asked. If they are doing this, then they are “optimising”.

Effective Learning in the Life Sciences

The book was edited by David Adams, Director of the UK Centre of Biosciences until September 2011

Today I have received my copy of Effective Learning in the Life Sciences: how students can achieve their full potential. As the subtitle implies, the book is targeted first and foremost at students wanting to make the most of their time at university, and at academics helping them to reach that goal.

1. Creativity (David Adams and Kevin Byron)

2. Problem solving: developing critical, evaluative, and analytical thinking skills (Tina Overton)

3. In the laboratory (Pauline Millican and David Adams)

4. Fieldwork (Julie Peacock, Julian Park and Alice Mauchline)

5. In vivo work (David Lewis)

6. Research projects (Martin Luck)

7. Maths and stats for biologists (Dawn Hawkins)

8. E-learning for biologists (Jo Badge, Jon Scott and Terry McAndrew)

9. Bioethics (Chris Willmott)

10. Assessment, feedback and review (Steve Maw and Paul Orsmond)

11. Communication in the biosciences (Joanna Verran and Maureen Dawson)

12. Bioenterprise (Lee Beniston, David Adams and Carol Wakeford)