From time to time examples of scientific fraud come to light and raise questions about the integrity of scientific endeavour. The most well-known example of recent years must surely be South Korean stem cell biologist Hwang Woo-Suk, whose ground-breaking discoveries in the field of therapeutic cloning were exposed as bogus (In addition to his science reputation being in tatters, Hwang was convicted in October 2009 of embezzlement and violation of bioethical laws, although he escaped a custodial sentence).

In physics, the multiple re-use of the same graphs as data for entirely different experiments led to the downfall of a leading young nanoscientist (this was the subject of a 2004 episode of the BBC’s Horizon series The dark secret of Hendrik Schön). Are Hwang and Schön rare examples bringing unwarranted criticism to a body of otherwise exemplary scientists, or are their crimes indicative of much wider malpractice within the scientific community?

University of Edinburgh researcher Daniele Fanelli has shed some light on the the extend of scientific fraud in an article How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. Published in the open access journal PLoS ONE in May 2009, the research brought together data from a number of earlier smaller studies on scientific misconduct to generate “the first meta-analysis of these surveys” (p1).

Fanelli was interested in examining the rates of self-reporting of scientific misconduct and knowledge about the misconduct of colleagues. Recognising that “any boundary defining misconduct will be arbitrary” (p9), he limited discussion to incidents where there was clear “intention to deceive” (p1, p9) rather than generation of incorrect results as a consequence of shoddy experimental design and/or accidental misinterpretation of the data. For the purposes of this study, Fanelli also excluded plagiarism and other examples of “questionable research practices” (QRPs; such as failure to include a contributor amongst the list of authors for a paper) from his definition of scientific misconduct, which was instead limited to fabrication and falsification. The grounds for this decision seem valid; whereas fabrication (the invention of data) and falsification (the wilful distortion of results) change the actual body of scientific knowledge, these other unprofessional activities lead instead to changes in the distribution of credit for the work, the substance of which remains unaltered.

To identify the previous studies of misconduct, Fanelli conducted a search of citation databases, scientific journals, “grey literature” databases and internet search engines using the terms “research misconduct”, “research integrity”, “research malpractice”, “scientific fraud”, “fabrication, falsification” and “falsification, fabrication”. An initial search generated 3276 potentially relevant studies. The vast majority (3207) were easily excluded because they were not surveys of research misconduct.

The author then applied very strict criteria to limit the meta-analysis to genuinely appropriate studies. For example, papers were excluded if there was no quantitative data, if the data included no clear category of never/none/nobody (e.g. if only mean values were shown), if the sample had been generated in a non-random manner, or if undergraduate and/or other non-researchers were included in a manner that did not permit their removal from the dataset). Having done so, the initial pool of potential papers was whittled right down to 18 suitable studies.

Quantifying research malpractice

What were the conclusions of Fanelli’s analysis? The main issues addressed were the proportion of respondent admitting to misconduct or questionable practices of their own, or knowledge of similar behaviour committed by colleagues on at least one occasion.

In the various studies reviewed, between 0.3% and 4.9% of respondents confirmed that they had modified results to improve the outcomes. This led to an average of about 2% self-reporting of misconduct (although it was nearer 1% if the responses were limited to those that specifically mentioned ‘falsification’ or ‘fabrication’.

A rather larger number, 9.5%,  were willing to admit that they had carried out broader questionable practices. Again, however, the phrasing was important with more respondents willing to say they had “modified research results” than admitting that had reported results that they “knew to be untrue”. This may fit with an underlying assumption that it is okay to omit data that you “know” are outliers or otherwise “wrong”. As Fanelli puts it “many did not think that the data they “improved” were falsified” (p9).

When asked about the actions of others, a crude average of around 16.7% (range 5.2% to 33.3%, Fanelli elects to report this statistic as “up to 34%” (p10)) of scientists said they had personal knowledge that a colleague had fabricated or falsified data on at least one occasion. A much wider range (6.2% to 72%; crude mean 28.5%) said that they were aware of peers who had indulged in QRPs.

So, were Hwang and Schön isolated miscreants or does their identification mark the tip of an iceberg of scientific misconduct? The truth seems to lie somewhere in between. As Fanelli notes, usual rules of self-reporting bias – in which some people (typically older women) under-report criminal behaviour whereas others (typically younger males) over-report such activity – do not apply here. It is highly unlikely that anyone in a community where trust is taken seriously will over-report their own wrong actions. It is likely, therefore, that the calculated values of self-reported malpractice are underestimates.

The data regarding knowledge of other researchers’ actions are harder to validate. It is theoretically possible, for example, that more than one correspondent might be describing the wrongdoings of the same colleague. In contrast, the criteria was knowledge of malpractice on “at least one occasion” and therefore the data may not take into account serial offences.

Other approaches to measuring misconduct, as reviewed by Fanelli, have generated a range of figures which might be seen as very broadly equivalent. For example, about 0.02% of paper  are retracted from PubMed due to misconduct (Claxton, 2005). 1% of papers submitted to the Journal of Cell Biology were found to have been inappropriately manipulated (Steneck, 2006). 2% of clinical researchers were found guilty of serious scientific misconduct in routine US Food and Drug Administration audits (Glick, 1992).

Whatever the accuracy of these numbers, however, it remains true that the vast majority of science is carried out in a spirit of accuracy and integrity.

At the risk of sounding like a Carlsberg advert, “The Journal of the Left-handed Biochemist doesn’t do award ceremonies, but if we did…” – what would be the winner of “Best Science programme” during the last 12 months?

In truth, I think it has been a bumper year for science programmes. There has been a tangible return to form at Horizon – the first three episodes of the current series, for example,  have all included significant coverage of molecular biology. There were commendable features to the mini-series Prof Regan’s Diet Clinic/Medicine Cabinet/Nursery/Health Spa (I have posted a separate review of Prof Regan’s Medicine Cabinet here) although it did let itself down at times by commiting some of the same mistakes it was accusing others of making. Pitched at a slightly different audience, it is also good to welcome Bang Goes the Theory to finally fill a gap in popular science coverage left empty since the demise of Tomorrow’s World around 2003.

The BBC Four War Beneath the Skin season in the late summer was consistently strong. Breaking the Mould: The Story of Penicillin helped to amend the traditional account of the birth of antibiotics as therapeutics and Spanish Flu: The Forgotten Fallen recreated events surrounding the battle to protect Manchester from the 1918 influenza epidemic (it also gave them a chance to re-transmit Mutant Mouse and Superfly, documentaries on research using model organisms, from 2004).

My winner of the grand prize would be another programme from that season. Adam Rutherford’s Cell – a 3-part series The Hidden Kingdom, The Chemistry of Life and The Spark of Life – was outstanding. Of the three, it is The Chemistry of Life that is my outright favourite.

The programme tells the story of the identification of DNA as the molecule of inheritance. Beginning with Friedrich Miescher’s original isolation of DNA in 1868, and his amazing determination of its chemical composition, the episode reflects on the key experiments of Theodore Boveri, Thomas Hunt Morgan, Fred Griffith, Oswald Avery, Maurice Wilkins & Rosalind Franklin, James Watson & Francis Crick, culminating with Walter Gehring’s discovery and characterisation of homeobox genes in the 1980s and 1990s (more details can be seen in my notes on the programme, via this link).

The Chemistry of Life can serve as an excellent component of an introductory module on molecular biology for first year undergraduates. Important discoveries are laid out in an accurate yet engaging way. I particularly like the way that the episode naturally demonstrates the evolution of scientific ideas.

The programme has a certain ”yuk factor” that serves to keep it captivating for a youthful audience; description of Miescher’s isolation of pus from the sheets of wounded soldiers and footage of a transgenic Drosophila fly with eyes all over its body being two examples. The episode also has its lighter moments – Rutherford’s visit to the Bay of Naples includes him partaking of a local delicacy as he eats raw urchin gonad straight from the spine-encrusted shell. Later on he deliberately burns his arm with a hot spoon, but it is evident that it was rather hotter than he intended. Perhaps best of all, archive footage of Maurice Wilkins describing ‘MOWlecules’ with received pronunciation and his explanation that X-rays are “wavy” were reminiscent of the Mr Cholmondley-Warner sketches from the old Harry Enfield series.

Maurice Wilkins explains how X-ray can be used to study molecules

Having said that, the film would be a useful teaching resource for first year undergraduates, I think that you would want to include time to emphasise the main points and to put back some of the detail that has inevitably been skipped. For example, fuller description of the basis of Griffith’s experiments with smooth and rough strains of Streptococcus pneumoniae would be beneficial. It is a minor irritation that no on-screen captioning supports the introduction of dead scientists, so offering a class a sheet with the names of the individuals might help students to structure any notes they wished to take.

Do you think a different science programme is more worthy of the title “Best Science Programme of the year”? If so, how about using the comment facility to share your suggestions.

The following are notes written for a session I was asked to run with sixth form students about preparing for Med School interviews. I am quite sure there are lots of sensible suggestions that I have inadvertently omitted – please feel free to use the Comments facility to offer your additional advice.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Your personal statement: You’ve got an interview! Apart from anything else, that means you must have done something right in your personal statement. Even though it may be months since you wrote it, it is important that you re-read it thoroughly about a week before the interview to remind yourself what you said and then reflect on what questions this may lead onto. In particular, think about:

• What did you learn via the experiences you described? You’ve probably used time spent shadowing or participating in medical activities as part of your statement. You may also have mentioned a part-time job or being captain of a sports team. Your interviewers will not so much be interested in what you’ve done as to what you’ve learnt by the experience. For example, how have your attitudes been altered? What was it about the things you’ve been involved in that makes you more suitable to be a doctor than the next person waiting outside?
• Can you provide evidence for the claims you made? I am assuming that all the things you said on your UCAS form are true (if not, you are just setting yourself up for more problems – be warned, don’t make stuff up). There is, however, a difference between you yourself knowing something to be true and being able to demonstrate that to somebody else. Let’s take an example from my wife’s experience. She wrote on her UCAS form that she was a keen dress-maker. Somebody wisely pointed out that if she was going to put that in her statement then she ought to wear something she’d made when going for interview. It was good advice; at every interview she attended somebody asked if she’d made the clothes she was wearing. If you are not yet in the habit of collecting documentation to back up work experience, Saturday employment etc then start now – portfolios of evidence are a fact of life in lots of jobs but none more so than Medicine, where even Consultants need to log their Continuing Professional Development (CPD) activities.

Body language: It may seem unfair, but it is certainly true that non-verbal communication can have a huge bearing on people’s opinions about you. You may not be able to iron out all of your faults in this area in one go, but think about the following:

• Posture: the way that you come into the room and the way that you sit in your chair are important. Try neither to slouch nor to grip the top of your other arm so tightly that you look like a survivor in a disaster movie. Avoid looking defensive or disinterested. Ideally sit upright, with both feet planted on the floor and slightly forward in your seat so that your body, and your face, looks engaged in the whole process.
• Irritating habits: do you repeatedly make the same gestures in a way that will inadvertently wind up your interviewer? Do you flick or twizzle your hair every 15 seconds? Do you click the lid of a pen on and off? Do you jangle keys or coins in your pocket? I’ve seen all of the above. I’ve also been told about the person wearing a large gold cross on a chain around her neck who wafted it back and forth during the interview in a way that made it look like her interviewers were vampires she was trying to keep at bay. Seeing yourself on video can be a useful way to reveal habits of this kind, and also verbal ‘ticks’ you might have such as finishing each sentence with “you know” or “that’s what I think” (see Practice, below).

Current issues: New developments in biomedicine are constantly being reported in the media. Make sure that you are up to date on several recent issues. Keep an eye on the BBC health pages and maybe subscribe for the year to the student BMJ. The NHS choices website runs an excellent Behind the Headlines service that unpicks the reality from the spin in media accounts of new developments. You should try and think about whether there is more than one opinion on a topic, particularly if it has clear ethical dimensions. At the time of writing, changes to the law on assisted suicide would be a classic example. Be informed but not unduly dogmatic – do think carefully about what the majority view of doctors might be on the issue, but don’t assume that everyone conducting the interview holds that opinion. The Radio4 series Inside the Ethics Committee fleshes out different views on genuine controversial cases within the NHS. Listening to one or two episodes would give you a feel for some of the controversies, and how decisions about how to proceed are made.

Motivation: In the days when the Miss World beauty competition was deemed suitable prime time viewing, people used to joke that all the contestants claimed that they wanted to “help children and bring about world peace”.

Similar clichés can tarnish a Med School interview. When asked about your motivation for being a doctor, if you reply with a bland “I want to help people” then you are likely to be rebuffed with the suggestion you should become a nurse, or a social worker, or a bin man. They all help people. Try to think about some more sophisticated reason for your application, something that shows you’ve truly understood what it means to be a doctor.

Other typical questions: Each institution is likely to have a fairly standardised set of questions they ask at interview, but these will be different for each university. Two areas that may well crop up, however, are Changes in the NHS/Medical training and some element of Self-reflection.

• Changes in the NHS – as I am sure you are aware, graduation from Med School does not mean that your training is complete. You have many years of additional study and exams to come (although you will at least get paid whilst you are doing it!) You may well be asked about your knowledge of the further training required to become a GP, or perhaps a hospital anaesthetist. You might also be asked about the impact of the (European) Working Time Directive – recently implemented legislation limiting the number of hours anyone can work in a given week. There are mixed feelings about this; the WTD ought to have brought an end to dangerously long shift, but fewer hours at work may also mean less clinical experience.
• Reflection about yourself: demonstration of an ability to be self-reflexive is another big theme in Medicine as a career. Interview questions may start to tap into this, e.g. by asking “Give an example of a time you were under pressure and how you coped” or “What would you say were your weaknesses?“. It is very common to say “I have a tendency to overwork” in response. Try to think about something that is more creative (and honest). For example, “I’m bad with names, so I’ve had to come up with a system to help me remember them”.

Practice…: If you can do so, try to organise a practice interview with someone you do not already know well. Alternatively (or additionally) try to make a video of yourself in an interview setting; there is nothing like seeing yourself on film to highlight body posture, repetitive behaviours and verbal mannerisms that might be off-putting to an interviewer.

…but don’t be ‘rehearsed’: It is a weird thing to observe, but students in interviews, and also in talks, sometimes start to speak in a robotic and detached way. Subconsciously they slip into an automaton mode which says – irrespective of the words coming out of their mouth – “I thought you were going to ask me that question and this is the answer I prepared”. This phenomenon can be particularly embarrassing when the student loses their train of thought mid-sentence and either comes to an abrupt halt, or starts again from the beginning of their ‘script’.

You do need to think carefully about the sort of things you might be asked, but avoid becoming overly wedded to a fixed set of phrases. Perhaps memorise a few key words instead – and don’t beat yourself up if you fail to cover all of the points you hoped to make.

Relax: Above all else – and in spite of all the things above (!) – try to relax. For the most part the interview will not be like any police interrogation or court scene you may have seen on telly. The rottweilers employed to cross-examine the candidates in The Apprentice will not be conducting your interview. It is rare these days for interviewers to be deliberately trying to catch you out, and if they are I would seriously question whether their institution is an appropriate place to study. They are trying to see whether or not you are the sort of person they want to engage with and invest in during the next four or five years prior to your becoming a significant asset to the health service.

(First version 23rd September 2009; this version 24th September 2009. With thanks to Jon Scott, Christine Wells and Anne Willmott for off-line advice and suggestions.)

There are many reasons why I am grateful to have spent some of my summer reading Ben Goldacre’s excellent book Bad Science, including the fact that it brought to my attention a paper The Seductive Allure of Neuroscience Explanations. The article is an account of experiments conducted by Deena Weisberg and colleagues at Yale University, and was published in the Journal of Cognitive Neuroscience in 2008.

Recognising that neuroscience is an area of research that fascinates the public and where discoveries are frequently picked up by the general press, Weisberg et al generated four explanatory statements for each of 18 different psychological phenoma. In each case the four statements represented:

• a good explanation without specific mention of neuroscience
• the same good explanation with the addition of plausible, but logically irrelevant, neuroscientific details
• a bad explanation without specific mention of neuroscience
• the same bad explanation with the addition of the same plausible (but irrelevant) neuroscience as in the second example

Satisfaction (on a 7-point scale) engendered by the various statements was tested with three cohorts:

• naive adults (n=81), i.e. individuals with no formal neuroscience training
• students from an introductory cognitive neuroscience course (n=22)
• experts in neuroscience (n=48), although the definition of “expert” in this context was quite generous – 6 members of this group had completed an undergraduate course but were yet to start their advanced degrees, 29 were currently in graduate school and the other 13 were beyond grad school.

In developing their neuroscience explanations, the authors held to three important criteria: (1) the phrasing would indicate that this was a field in which knowledge was already established, (2) the same information would be added to both the good and the bad explanation, and (3) the neuroscience information should not alter the underlying logic of the explanation such as it was before addition of the extra ’science’.

With all three cohorts, the participants were informed that the study being reported was scientifically robust, but that the explanation offered may not be genuine. For both the ‘naive’ and ‘expert’ groups any individual always saw only statements that contained neuroscientific information (but might be a good or bad explanation), or they never had the additional details. This approach was adopted so that subjects were not alerted to the fact that some of the explanations were fuller than others. Unfortunately the relatively small size of the current undergraduate cohort meant that the method was different, any individual could be exposed to all 4 different types of explanation. This is a shame, since it does require some caution when interpreting side-by-side comparisons of the three cohorts.

So what were the findings? For all three groups, participants rated the good explanations as more satisfactory than the bad ones. Both the naive group and the students found explanations with added neuroscience to be better, and the effect was more striking for the bad explanations than for the good. With the experts, however, addition of the spurious scientific details to the good explanation actually led to a reduction in their satisfaction – an indication that with their fuller knowledge they saw through the vacuous additions.

The authors argue that the findings with novices and students may be manifestations of the “seductive details effect”. Previous studies have suggested interesting-but-irrelevant information can have a detrimental effect on cognitive tasks, e.g. memory tests.

They also point to a general observation that individuals respond more favourably when given a reason for a request, however obvious. They cite one example of such “placebic” information in which research subjects were more amenable to letting someone carry out some photocopying if they added the phrase “I have some copies to make” to a bland request “May I use your Xerox machine?”

In Bad Science, Ben Goldacre argues that ‘quacks’ exploit this phenomenon by dressing up their claims with sciencey-sounding explanations. This reminded me too of the BBC documentary Professor Regan’s Medicine Cabinet (first shown in April 2009) in which, amongst other things, Regan dresses up a test of pills for treating insomnia with all sorts of detailed instructions about what must, and what must not, be done in order for the therapy to be effective. The participants stuck to the rules and reported beneficial effects of the pills, despite the fact that they were – in fact – sugar cake decorations.

Weinberg D.S., Keil F.C., Goodstein J., Rawson E. and Gray J.R. (2008) The seductive allure of neuroscience explanations Journal of Cognitive Neuroscience 20:470-477

A tweet this morning from @jon_scott alerted me to the fact that sometime over the weekend, the University of Leicester has been visited by the PR machine for the Viper service. Paving slabs had been stencilled with the company’s logo and web address. Rather ingeniously, the marketeers have jet-washed the image rather than painting it on, which I presume guards them against accusations of vandalism because all they’ve actually done is remove dirty (thanks to @jobadge for pointing this out, she is obviously more ‘direct action’ savvy than me). The image on the bumpy pavement at the traffic lights makes the wash v paint strategy most clearly.

Viper is marketing itself as a way for students to check that their work is not guilty of plagiarism. Several institutions have already wrestled with the question of whether to let students pre-submit their work to Turnitin so that they can see for themselves if it is going to get pinged by that software with the same intentions. However laudible this seems, one of the difficulties is the fact that students will simply learn how to mask their tracks rather than developing bona fide study skills. The subversive nature of the current marketing strategy reinforces the view that this is a way to “beat the system”. I was interested also that one of the recommendations for the software on the Tucows site seems to come from a student who bought a ‘bespoke’ essay for her course and was now asking for a refund as the software showed it was not quite the original work she thought she’d paid for!

Harper Perennial edition (2009)

If you have not yet read Ben Goldacre’s book Bad Science, then I thoroughly recommend that you do. As readers of his regular Guardian column or his website will already know, Goldacre has embarked on a campaign to root out example of pseudoscience and shoddy science whereever they may be found.

All the usual villians are present – homeopaths, nutritionists, slack journalists, pharmaceutical companies and AIDS dissenters. Some are mentioned by name, but given their alleged predilection for litigation, and since I do not have the time, the money or the inclination to do battle with them in the courts, I shall not repeat their identities here!

It would be wrong, however, to give the impression that Goldacre is merely on a crusade against high profile exponents of “bad science”. True, the author does sometimes betray a little too much glee as he places a bomb under the throne of a media ”health expert” (in a way that I found disturbingly reminiscent of the Physiology lecturer, when I was a first year undergraduate, recalling his boyhood experiments on frogs). Nevertheless, Goldacre is keen to emphasise that his purpose is to “teach good science by examining the bad” (p165 in my copy), adding that ”the aim of this book is that you should be future-proofed against new variants of bullshit” (p87).

It seems to me that Goldacre is correct in his assertion that the public needs help in ‘bullshit-spotting’ and that this book is an extremely valuable tool in achieving that goal. Scientific colleagues will (hopefully!) be familiar with at least some of the pitfalls of poor study design, inappropriate use of statistics and outright spin that lead to dramatic-but-spurious headlines in the newspapers. I am, however, convinced that there is plenty here that will improve the scientific literacy of undergraduates in medicine and bioscience subjects, as well as a more general readership.

Trial design
For an experiment involving human subjects to have at least some hope of generating objective data, it is important that the research method includes:

• control groups – you need something against which to compare your  intervention, whether it be a placebo or sham treatment, or the best treatment currently in use;
• appropriate blinding - i.e. that neither researcher nor participants know during the trial which individual is receiving each intervention;
• randomisation - trial subjects need to be assigned to different regimes in a genuinely unbiased way (some randomisation protocols are actually open to significant abuse, albeit subconscious);
• documentation - when the work is published, the account needs to include suitably transparent and complete details of the methods and the results such that any reader will know how the study was conducted and can therefore have a sporting chance of spotting the glitches.

The case of the widely-reported Durham trial of fish oil tablets containing Omega-3 fats (Chapter 8, ‘Pill solves complex social problem‘) is a chastening tale of ways in which poor research methodology can effectively ruin a study before it has even started. Alarm bells ought to have been triggered as soon as the trial (I will call it that for simplicity, although those involved in the research have shyed away from this term) was trumpeted in advance as a test to prove the effectiveness of fish oils in boosting academic performance. The fact that the participants knew that they were in a trial has been shown in itself to ellicit improvements (the so-called ‘Hawthorne effect’), even without the media scrum that accompanied this particular trial. Add to this the influence of potential ‘confounding factors’ (see below) and this study was never going to give clear and unequivocal results.

Common mistakes involving science literature
Goldacre’s critique of ‘nutritionists’ highlights four frequent errors in the way that science literature is handled. These are:

• extrapolating and overinterpreting data – For example studies conducted on isolated cells in vitro, can provide useful pointers for future studies in humans, but it is wrong to naively take findings from cell-based work and assume the equivalent is true in vivo in a whole organism. To purloin one of Goldacre’s favourite phrases “I think you’ll find it’s a bit more complicated than that” (p100)
• extrapolating from observational studies to make claims that require an interventional study to be conducted – ‘confounding variables’, that is differences between individuals that may or may not be linked to the factor under investigation, are hard enough to control in a study where the researcher is deliberately intervening in the partcipants’ lives to measure any apparent effects. If the study is merely observing differences between people reported to have an important lifestyle or dietary factor, there may be a lot more going on. Superficial analyses are prone to come up with erroneous conclusions.
• Cherry-picking only results that fit the hypothesis – it is, as Goldacre points out, a facet of human nature both to see patterns in data and to be more receptive to results that fit your expectation than those that do not. We need therefore to guard against selectively quoting only experiments that give the results that we want, and ignoring data (possibly the majority of findings) that don’t fit our model. This is why ‘systematic review‘ of all of the data on a particular topic is an essential process.
• Referring to studies that are not published in peer-reviewed journals, and frequently not published at all – it is bad enough when conference papers and press releases are reported with the same gravitas and authority as experiments which have been scrutinised by experts in the same field as part of the peer-review process. It is even worse, however, when some interviewees are prone to make specific claims such as “a study published just last week in America has described the same effect we see here” whilst it later turns out that no such article exists. In written work, some authors have increasingly given their books a spurious air of authority by adopting the trappings of good citation practice, e.g. use of superscript numbers to direct readers to their sources. When you flick on to check the reference, however, it turns out to be a non-scholarly document or something that they themselves have said on a different occasion.

Lies, damned lies and statistics
Statistics are clearly vital in substantiating the findings of any kind of trial and Goldacre attacks abuse of statistics on two fronts. Firstly, there is the deliberate use of an inappropriate statistical test to generate a positive-sounding number. Pharmaceutical companies are said to be guilty of this sleight of hand, and it requires a certain amount of statistical nous in order to detect when this crime is being perpetrated.

Secondly, there is the way that the numbers are presented to the public. Newspapers are prone to report the ‘relative risk increase‘, i.e. the percentage increase in condition X when presented with risk Y because it generates the most attention-grabbing numbers. The shock statistic “reading science-related blogs increases the chance that you’ll have a heart attack by 50%” may alarm you (so just in case, let me say straight way that I made this up).  A very different impression is given if we consider the ‘absolute risk increase‘ which state that “reading science-related blogs increases the chance that you’ll have a heart attack by 0.2%”. Goldacre recommends that there ought to be a move towards quoting ‘natural frequencies‘, i.e. as intelligible numbers. In this case, therefore we might say “reading science-related blogs increases the chance that you’ll have a heart attack from 4 in every 1000 people if you don’t, to 6 in every 1000 people if you do”.

Putting Bad Science to use in formal education
Are there ways in which Bad Science might be employed as a teaching tool in either secondary or tertiary education? The specifications for GCSE Science in England and Wales were altered in 2006 to place greater emphasis on “How Science Works“, and A levels were similarly altered in 2008 when this cohort passed on to the higher qualification. The reading level required to appreciate Bad Science probably procludes recommending it for the majority of 16 year olds. I believe, however, that the text would make an excellent resource for students of A level biology and/or General Studies. I do not know if the publishers have considered producing a structured guide based on the book or inclusion of end of chapter  study questions in future editions, but there is certainly scope for this.

Similarly, the book would be valuable reading for first year undergraduates in Medicine, Bioscience or Journalism. I think there would be more merit in having this as prescribed reading for a Year One skills or introductory module than several of the more ‘academic’ alternatives.

As an admissions tutor, I receive several e-mails each summer from students starting the following term and asking which textbooks to buy. My consistent response this time around has been to recommended that they read Bad Science now and wait until the course has started before they part with money for a chunky Biochemistry text.

Gripes
This is not to say that Bad Science is without faults. I do have a number of minor quibbles with the book, but I would say for the most part the fault lies with the editorial process rather than with the author per se.

Haven’t I read that before? Understandably much of the content of the book has already seen the light of day in shorter pieces in the Guardian’s Bad Science column. Repetition and/or poor ordering (by which I mean a point is introduced at length after it has already been previously noted) betray the ‘cut and shut’ nature of some of the present material. As an example of the former, we are told twice in consecutive paragraphs on page 113 about the crusade led by cereal magnate John Harvey Kellogg against a particular personal vice. Similarly, the fact that Durham council altered a press release on their website sometime after its release in order to remove the word ‘trial’ is mentioned on pages 143 and 149.

Examples of the ‘introduction after being stated’ phenomenon include the mention on page 157 that Equazen had been acquired by Galenica, follwed on p160 by a fuller account of this transaction in a tone that gave the impression it was ‘new news’. Similarly we are told on page 313 that some researchers did “something called a ‘case-control’ study” despite the fact that case-control studies were amongst the variety of experimental models discussed on page 103 and pages 295-296.

Page numbering: The cover of my edition of the book (Harper Perennial, 2009) trumpets the addition of an extra chapter. This material has not been added at the end of the text, but rather inserted at the appropriate point in the unfolding ’story’. In consequence, page numbering downstream of the insertion is altered. Although this has been recognised in the index, there are several examples of in-text cross-references where the page numbers are now 17 out. (In case anyone with influence on the next version is reading this review the reference on page 106 to p240 should be p257; page 282 should cite p294 not p277; page 330 should point to p293 not p276).

Referencing: Bad Science is intended to be a popular book not an academic tome. As such, it would be completely inappropriate for the text to be peppered with citations in a way that would interrupt the flow. I think the solution chosen here works very well – the notes in the back use page numbers and a short quote from the text as the identifiers of the source. It is partly because I know Goldacre makes regular criticism of the lack of referencing in media reports of science that I was disappointed on a couple of occasions to turn eagerly to the back and not find a citation. These tended to be times when a broad statement had been made – for example on page 75 “A huge amount of research...” does not provide any corroborating references and on page 144 “there is a lot of history here… the field of essential fatty acid research has seen research fraud, secrecy, court cases, negative findings that have been hushed up, media misreporting on a massive scale…[the list continues]” but no notes are offered. If a new edition is produced, please can these be added.

Summary
As I have already said, these are minor (some would say picky) criticisms of an otherwise extremely valuable book. Overall, I believe Ben Goldacre has provided all of us with a toolbox for evaluating sciencey-sounding stories in the media and alerted future scientists to some of the pitfalls they should avoid in the design and reporting of their work. Bad Science would make an excellent resource for post-16 education and I hope to see it adopted as a course text on A level and undergraduate programmes.

Bad Science has a cover price of £8.99 At the time of writing it is available from Amazon for £3.60.

Amongst the major science research journals, Science magazine has consistently been the most prominent in flying the flag for science education. I was very interested, therefore, in an Editorial by Carl Wieman in the September 4th 2009 issue of the magazine. In his piece Galvanising Science Departments, Wieman describes some fairly radical innovations in Science Education currently underway at the University of Colorado and the University of Bristish Columbia. The aim is to adopt evidence-based teaching methodologies with emphasis on the development of scientific thinking and problem-solving skills rather than fact regurgitation.

I have no direct experience of teaching in the USA, either as provider or recipient. I know, for example, that much greater emphasis is placed on the recommended course text in the USA than in the UK, but beyond that I cannot speak with any authority. It does sound like some of the reported innovations are things that have taken place here for some while, such as the addition of specific (skill-centred) learning goals to modules. A cornerstone of the strategy has been appointment of science education specialists, individuals who not only have expertise in their subject discipline, but are also au fait with educational and cognitive psychology studies, a variety of effective teaching strategies and – I note with some mirth – possess diplomatic skills!  The programme is ongoing, the University of Colorado is in the 4th year of an initial six year project and so the full impact of the developments will not be known for some while.

What really struck me, however, was the extent of the commitment at an institutional level, including provision of serious money to fund these changes. Far too many pro-active educators, motivated by genuine desire to improve the learning experience for their students actual receive flak not gratitude. In many cases this is, I believe, because their approach to pedagogy is different to the cultural norm and, as such, moved students (and staff) outside their comfort zone.

So much of education, even at University level, is about recall of a prescribed body of information. This is relatively easy – for both the teacher and the student; it is the mindset that underlies that chirping questions “will this be in the test?” Development of thinking skills demands more from everyone. If innovations that require higher skills are associated with just one module, or even just one academic, then that individual may unfairly receive criticism in feedback from the class.

I believe this is why Wieman is absolutely right when he says “an entire department must be the unit of change”. Depending upon institutional structure, it may even require a larger body – School, Faculty, College – to move together with shared commitment to the new goals. So far at Colorado, 60% of academic staff in three Science departments have embraced the new teaching approaches, impacting 80% of their students’ credit hours. Faculty are reported to enthusiastically discussing teaching as a scholarly activity – that’s surely got to be a good thing. But – it needs time and it needs money.

Wieman C (2009) Galvanising Science Departments Science 325:1181

I have been a devotee of social bookmarking tool delicious since 2007 and now have nearly 4000 items tagged. Although the ‘before’ and ‘after’ photos (slide 17) in my July 2008 presentation Knowing where it’s at: find it? flag it? share it? (or how delicious saved my life) were staged for effect, the ability to accumulate links to resources online rather than generate piles on (unread) papers in my office has been a genuine revelation.

Alongside delicious, I also dabbled briefly with Connotea, the online reference management tool from the Nature stable. It has the same potential as delicious for user-generated tags, but at the time I couldn’t really see what additional value it was adding and I let my interest wither, electing instead to use delicious alone for all of my bookmarks, including journal articles.

More recently, I’ve been persuaded by a colleague to take a close look at rival social citation application citeulike. This time around I think I get it. One of the features that really appeals is the potential to import comprehensive bibliographic information armed only with the Digital Object Identifier (DOI). With journals making the DOI of articles increasingly obvious on their websites and in table of contents alerts, this becomes a very straightforward way to collate large quantities of metadata whilst retaining the capability to tag a paper with whatever keywords reflect its relevance to you.

In truth, I have not conducted a rigorous side-by-side comparison of citeulike v connotea (or any of the other similar tools). I am quite sure, for example, that they all have the potential to assimilate bibliographic details armed only with the DOI. For the foreseable future I will continue to tag journal articles using delicious. However, this feature of citeulike, couple with the capabililty to establish shared libraries of articles relevant to members of a particular list, has persuaded me to also give the latter a prolonged trial.

I was intrigued by a recent paper Cognitive control in media multitaskers in the highly-regarded journal Proceedings of the National Academy of Sciences. The study looked at the information processing styles of self-reported media multitaskers, defined as users of two or more content streams simultaneously, compared with those who do not multitask in this way. (I will skirt over the irony both that I was supposed to be doing something else at the time I spotted the BBC report of the research, and that my son has just switched on the TV as I type this review on my laptop.)

An electronic copy of the paper was posted online in August 2009, ahead of publication in the print journal (doi: 10.1073/pnas.0903620106)

In their research, Ophir et al asked 262 University students to complete an online self-assessment questionnaire regarding both the total number of hours spent using different media (they specified 12 formats including TV, online video use, music, print media, e-mail and text messaging) and how likely they might be to use some of these concurrently alongside a primary task. The authors then generated a numerical Media Multitasking Index (MMI) and ranked the students. Those with a score one or more standard deviations below the mean (light media multitaskers, LMMs) or one or more SDs above the mean (heavy media multitaskers, HMMs) were then invited to participate in a series of cognitive ability tests. In total there were between 30 and 41 students taking the various tests, evenly split between LMMs and HMMs.

Central to the whole study was the issue of working memory, a concept we have discussed previously on JLB. Some of the experiments were designed to look at the students abilities to filter out distractions and remain focused on the main task. In one test, for example, the students were asked to compare the orientation of two red rectangles in consecutive images shown to them 900 milliseconds apart. To investigate the influence of distractions, the image also included 0, 2, 4 or 6 blue rectangles. Each student was asked to do this 200 times in one sitting,  during which there were an equal number of slides in which the position changed and in which it did not (the order in which they saw each of the image pairs was randomised). This test revealed a difference between the LMMs and the HMMs. Contrary to the hopes of self-confessed media multitaskers everywhere, the HMM group seemed to be more easily distracted than the LMM, whose performance was largely unaffected by the presence of blue rectangles.

In a second test, the investigators looked into the ability of participants to hit the correct button “animal” v “nonanimal” when presented with 24 words. Each word was presented one at a time, and all 24 words were used twice in the sequence (ie 48 words total). The variation was then introduced whereby the students were presented with the same words again, but this time with the additional instruction that they should refrain from hitting the button if they heard a stop tone before they responded. The signal was transmitted 225 ms before the mean response time, and was used on 25% of occasions. In this test there was not a significant difference between the two groups.

There was also no significant difference between the HMMs and LMMs in the first part of a third test. Participants were shown a sequence of letter pairs 300 ms apart (the letters were red on a black background, the relevance of this will be reached in a moment). The students were to hit the “no” button on all occasions except when there was a cue “A” followed by a probe “X”, hence this is known as an AX Continuous Performance Task (AX-CPT).

A difference was reported, however, when distractors were introduced into the AX-CPT. Between the cue letter (red on black, remember) and the probe letter (also red on black), the participants were presented with three distractor letters which were white on black. The distractors were never A, X or indeed K or Y both of which were entirely omitted from the study to avoid shape recognition clashes of X v Y/K. The overall time between cue and probe was held constant (4900 ms) relative to original AX-CPT test.

In the AX-CPT test with distractors, the HMM students were found to have the same accuracy as the LMM cohort, but responded more slowly. Alongside the relative responses in the rectangle distraction test, it seems that HMMs are worse than LMMs at filtering out irrelevant environmental stimuli.

Given the inherent need to swap between activities implicit in multi-tasking, the findings from the task-switching test are probably the most illuminating. In this experiment, the students were presented with a cue that said either “letter” or “number” followed by a stimulus that always featured one letter and one number. If the “letter” cue had been shown, the participants were to ignore the number and press the left button for a vowel and the right button for a non-vowel (only 4 consonants (K, N, P, S)  and 4 vowels (A, E, I, U) were ever used to avoid bias to the non-vowel side; O was not included). conversely if the “number” cue was used then it was left button for “odd” and right button for “even” (again, only 4 of each sort of stimuli were shown; the numbers 1 and 0 were omitted. Each student saw 4 sets of 80 cue and stimulus pairs, 40% involved task switching and 60% non-switching. They were given “warm-up” activities either involving switching between numbers and letters or non-switching.  HMMs were slower to respond than LMMs in both non-switch (259 ms) and switch (426 ms) trials. This result is interpreted as showing that HMMs are less effective at suppressing irrelevant tasks.

In the final set of studies, the researchers looked at abilities to ignore distractions within memory. This was investigated by conducting so-called “two-back” and “three-back” tasks in which participants were asked to indicate whether a letter on the screen was a “target”, i.e. the same as the letter 2 or 3 previously (in respective study) or “non-target”, i.e. different. Each student has 3 x 30 goes, in which 10 were targets and 20 non-targets.

In the two/three-back tasks the HMMs again performed less well than the LMMs. Unsurprisingly, both groups were worse in the three-back tests than in the two-back tests and their relative declines between the two were similar. Interestingly, however, HMMs had a greater tendancy to identify false positives, i.e. to report a non-target as target, particularly in association with letters than had come earlier in the sequence and were thus “familiar” but not in the correct relative position. This effect became more apparent as the task progressed, presumably as there were more “familiar” letters that had been and gone.

So, the headline message was that self-reported multitaskers are less adept at multitasking than those who shun multiple media. As someone who will admit to being an HMM, this is a disappointment. We could argue that the tests used in the study were surrogates – the real worldquestion would be “can you get work done appropriately whilst Twitter and E-mail are open on your dashboard and you are listening to your iPod?” not “can you remember if the red rectangles are in the same orientation if there are lots of blue ones there too?” I would also defend my position by acknowledging that my Twitter network is now a major source of information with “bang on the money” relevance for my work. Nevertheless I do know for myself that when I’ve really got to get something written I hide myself away with minimum distractions, the only added media will be background music to block out other noise, and this must be lyric-free to be fit for purpose! So, I will reflect on my work practice and perhaps make accessing of e-mail and Twitter more punctuated and less continuous, but I don’t feel that cutting off these channels entirely would – on balance – be a benefit. As the authors themselves point out, it may be that HMMs are demonstrating “a bias towards exploratory, rather than exploitative, information processing”.

Psychology and cognitive science are not my normal field of study and I therefore hope that I haven’t misrepresented any of the methods or findings of the PNAS paper in the proceeding discussion. I thought this was a truly interesting investigation, and highly topical – it is no longer just adolescents that are habitual multitaskers. Finally, from an educational research point of view, it was curious to note that participants in this research received course credits for their involvement in the study, both for taking the questionnaire in the first place and then for their role in the tasks themselves. I’m not sure we’d get away with that in a UK context.

Back in 2004, Sir David King (at the time, the Government’s Chief Scientific Adviser) initiated a discussion about generating a Code of Conduct for Scientists. The consultation process led, in 2006, to the publication of Rigour, respect and responsibility: a universal ethical code for scientists. None of the contents was particularly surprising or radical but it brought together in one place a list of seven key principles that ought to be foundational for the ethical conduct and communication of science.

The Code of Conduct emphasises seven key points

The Code received a public launch at the BA Festival of Science in September 2007 and was reported in the general press at the time (see, for example, UK science head backs ethics code). During the intervening two years, conversations with scientist colleagues (across a range of institutions) have revealed almost universal ignorance about the existence of the Code, let alone its content.

The new guidelines from Research Councils UK will see greater emphasis on research ethics

All this is about to change. In July 2009 Research Councils UK (RCUK), the umbrella organisation for the seven publicly-funded Research Councils, produced Integrity, Clarity and Good Management, a Policy and Code of Conducton the Governance of Good Research Conduct. An accompanying letter from Prof Ian Diamond, Chair of the RCUK Exec Committee, makes it clear that from 1st October 2009 adherence to the guidelines and code will be a requirement for all those seeking funding via the Research Councils.

The RCUK document is quite broad in its definition of unacceptable conduct and includes poor and inadequate record keeping alongside deliberate dishonesty. The misconduct highlighted includes:

• Fabrication: producing false data or documents
• Falsification: manipulating data, including images
• Plagiarism: taking someone else’s work or ideas without credit
• Misrepresentation: knowingly, recklessly or negligently presenting a wrong interpretation of data; inappropriate exclusion (or inclusion) of authors in publications arising from the work; duplicate publication
• Mismanagement: poor record-keeping or inadequate retention of primary data
• Breach of duty of care: revealing identity or research participants; endangering people involved in research; failure to fulfil legal requirements regarding use of animals and/or tissue samples; improper involvement in the peer review process including failure to disclose conflicts of interest

In addition to the aspects of conduct themselves, the RCUK paper also includes guidelines on the reporting and investigation of unacceptable research conduct.  Alongside the 2008 Procedure for the Investigation of Misconduct in Research (UK Research Integrity Office) and Investigating Research Misconduct Allegations in International Collaborative Projects (OECD Global Science Forum), requirements for informal and formal investigation and, where necessary disciplining by the Research Organisation hosting the affected study in detailed.

I believe that the vast majority of research in the UK is conducted to high ethical standards. For most people, the recommendations in the code will not be new, but having them in one place, and the clear indication from RCUK that funding will be linked to demonstration of high standards will hopefully serve to curb abuses (deliberate or accidental) where they do exist.