Somewhat belatedly I have been catching up on a couple of reports about the future of Science teaching in Europe. Both were prompted by widespread concern that school science in its present form is not meeting the needs of society for the 21st Century. The decline in students’ attitudes towards science – apparently universal across Europe – is a particular worry.

Science Education Now: A renewed pedagogy for the future of Europe was published in 2007 and Science Education in Europe: Critical reflections in 2008

Published in 2007, Science Education Now: A renewed pedagogy for the future of Europe was written at the behest of the European Commission with the specific objective “to examine a cross-section of on-going initiatives and to draw from them elements of know-how and good practive that could bring about a radical change in young people’s interest in science” (p2).

The second paper, Science Education in Europe: Critical reflections follows on from two seminars held in 2006 at the Nuffield Foundation in London. The final report was published in January 2008.

It is not my intention to analyse the documents in detail; both are written in a clear and accessible style, and at only approximately 30 pages each they are eminently worthy of your direct consideration. Here instead I will reflect on a couple of the major issues that struck me as I read the reports.

.

What is the purpose of science education?

A fundamental problem relating to science education is apparent from the outset; namely the objective of the whole process. There has long been a tension between ’school science as foundational knowledge for future scientists’ and ’school science as a mean of equipping the general citizen for engagement with science as they will encounter it in everyday life’.

In most countries science is now compulsory and as such, it is argued, the emphasis must be on the second of the two aims – i.e. promoting scientific literacy for all. Osborne and Dillon, authors of the Nuffield report, see this as warranted on both moral and economic grounds – it is poor management of resources to flog the majority for a programme geared for the minority who will take science to a higher level.

This was the motivation of the major GCSE curriculum changes that took place in the UK from October 2006, with much more emphasis placed on the understanding the process of science  - “How Science Works” – than on fact regurgitation. Advocates of the changes point out that the new curriculum benefits both the general student and the future scientist, though the latter will need additional instruction to establish the core knowledge necessary for higher study.

Critics have suggested that the baby has been jettisoned with the bathwater, that the new approach is “more suitable to the pub than the classroom“. I suspect that many of my science colleagues would intuitively feel nearer this latter position. Of course no-one is yet able to measure the effect on future scientists; the first students that took the new GCSE specifications will not start undergraduate courses until September 2010 at the earliest. I have genuine expectations that we will encounter a new breed of more science-savvy students from the autumn.

Of course one of the perennial dilemmas in education in general,but particularly science education, is whether we will be afforded time to see the current developments through before some new government initiative will force additional changes to be made. Like incoming Students’ Union sabbatical officers, new political leaders have an overwhelming desire to do something differently and thereby leave their legacy, but it does make it difficult to tell which innovations were genuinely beneficial when nothing is left to run its course.

.

Induction v Deduction, Inquiry v Instruction

There are some nice metaphors in the reports; I particularly like the analogy given by Osborne and Dillon that much traditional science teaching is like “being on a train with blacked-out windows – you know you are going somewhere but only the train driver knows where” (p8). Both documents hold that this type of deductive, “top-down” transmission delivers fragments of knowledge whose relevance does not become clear until you have got to your destination; if you fall by the wayside before reaching that point then it will never make any sense. Apparently random snippets of information hold no appeal for all except quiz addicts, and thus the majority, especially girls, switch off.

Instead, inquiry-based science education (IBSE) is advocated, a model in which a structured problem, ideally with a hands-on or practical dimension, comes first and then students are encouraged to develop their critical thinking and reflection skills whilst formulating an understanding of what is going on.

This latter approach has much going for it, but there are significant difficulties in adopting pedagogy of this kind. One problem is the fact it is fundamentally different to traditional approaches – you cannot get to it by minor tweaks of current practice. Secondly, ‘correct’ scientific interpretations do not always flow intuitively from observation. Thirdly, if IBSE is to avoid descent into “the blind leading the blind” it requires the science teacher to feel confident in their own underlying knowledge of the subject and in their classroom management skills, such that they will allow free-flowing discussion to occur before pulling together the salient points at the end.

This is a major sticking point. Combined with research that shows attitudes towards science are frequently cast in stone before the age of 14, a pivotal role is played by the primary school teacher. We know that relatively few primary school teachers have science backgrounds and many feel uncomfortable with even straightforward scientific experiments. There is thus a vital role to be played in equipping primary school teachers for this task. Good work is being done – both papers put a spotlight on the POLLEN programme, an initiative in 12 European countries trialling continuing professional development for primary school teachers in a way that will be scaleable across the wider community.

.

Careers from Science > Careers in Science

The final point that struck a chord with me was the need to convey to students that studying science can lead on to more varied (and more interesting) careers than simply wearing a white coat and working in a lab. Osborne and Dillon talk about an emphasis on Careers from science not just Careers in science. This is one of the things I am trying to convey to undergraduate scientists through the Careers After Biological Science seminar programme where we try to include a blend of ‘obvious’ and less obvious careers that can follow advance study of science (see our sister site biosciencecareers for more details). A rung or two further down the educational ladder I take it that this is also the goal of the scienceandmaths.net programme currently being advertised in cinema trailers before the feature film.

Development of appropriate curricula, appropriate pedagogy, appropriate assessement tasks and appropriately-trained teachers are going to be crucial in producing scientifically-literate societies and socially-literate scientists in the 21st Century.

I have been doing some reading for a while now on the ethics of research involving model organisms, particularly the potential for studies on lower species to offer insights into human disease (and thereby contribute to the 3Rs). Some of my musings on the topic can be found here.

Aware of this interest, a colleague recommended that I read a 2004 paper published in the journal Cell. I am very grateful that he did, since the study really has the “wow” factor – demonstrating beautifully the potential of comparative genomics, experiments on model organisms and knowledge of human disease to work together to produce new insights that would have been much harder if any one component was missing. The paper is Comparative genomics identifies a flagellar and basal body proteome that includes the BBS5 human disease gene by Li JB et al. The following notes are my attempt to summarise the best bits.

The importance of cilia and basal bodies in disease
The role of cilia in respiration (and the detrimental effects of smoking on their function) were features of the school biology curriculum when I was a child. However, research over the last ten years or so has demonstrated that cilia have surprisingly diverse roles in development, from determination of left-right symmetry in the body, through to formation and function of specific organs such as the kidneys (for more detail see the Wikipedia entry on Ciliopathy or, if you have access permissions,  Badano et al (2006), The ciliopathies: an emerging class of human genetic disorders Annual Review of Genomics and Human Genetics 7:125-148). Bardet-Biedl syndrome (BBS) is one disorder associated with non-functional or malfunctional cilia. The clinical features can be varied, but include obesity, mental retardation, progressive-onset blindness and polydactylism (i.e. possession of extra digits).

Cilia, and flagella in organisms that possess these related structures, develop from, and remain attached to, modified centrioles called basal bodies. These complex structures involve upwards of 150 proteins.

Comparative genomics
The complete genome sequence for a growing collection of organisms is now known. The authors of the Cell paper – headed up by Susan Dutcher of Washington University, St Louis -  reasoned that they could identify novel genes involved in basal body development by judicious use of such genomic information.

The logic goes like this: basal bodies are needed for both cilia and flagella, therefore genes necessary for proteins in the basal body should be present in the genomes of an organism with cilia and an organism with flagella – even if otherwise morphologically dissimilar – but not in the genome of a species with neither cilia nor flagella. A Venn diagram can be helpful in visualising this subset of genes.

researchers were interested in finding genes expressed in humans (an organism with cilia) and Chlamydomonas (an organism with a flagellum) but not Arabidopsis (which has neither cilia nor flagella) - see text for details

When they looked at the genomes of humans (ciliated, approx 32,ooo genes) and Chlamydomonas (a unicellular flagellate with approximately 20,000 genes) they homed in on a set of about 4,350 genes that were common to both species despite their obvious diversity.

Next, they subtracted from that list all of the genes that were also present in the genome of the thale cress Arabidopsis (approximately 29,000 genes in total).  This plant species has neither cilia nor flagella and therefore ought not to have the genes necessary for their development.

It turned out that a total of 686 genes fit into the subset “in human and in Chlamydomonas but not in Arabidopsis“, termed the flagellar apparatus-basal body (FABB) group by the researchers, and corresponding to the yellow sector in the Venn Diagram above. This may still seem a large number, but when compared with the total of at least 20,000 genes it represents elimination of over 96% of the genome as not being involved in basal bodies.  In this way they hoped that they had shortened the odds on identifying the role(s) played by genes of previously unknown function.

.

Alzheimer’s disease (AD) remains the most common form of dementia, particularly the late-onset version which typically develops in patients aged over 65. Although there is believed to be a strong genetic basis to the disease, the only gene previously identified as a susceptibility factor in all version of the disease was APOE, coding for Apolipoprotein E. In addition, genes for Amyloid precursor protein (APP), Presenilin 1 (PSEN1) and Presenilin 2 (PSEN2) have been noted as factors in the less common early-onset form of AD, which has a strong pattern of familial inheritance. Other attempts to find genes influencing the more common late onset form of AD have been ‘under-powered’, i.e. have involved insufficient individuals (≤1,100) to reveal any further statistically-significant correlations.

In October 2009, however, two independent studies published “back-to-back” in the journal Nature Genetics identified a number of other genes in which Single Nucleotide Polymorphisms (SNPs) seem to be associated with development of late-onset AD.

Genome-wide association study identifies variants at CLU and PICALM associated with Alzheimer’s disease
Harold et al (2009) Nature Genetics 41:1088-1093

The authors of the first study (n = 86, headed by Julie Williams from Cardiff University) began by examining SNPs in about 19,000 research subjects (reduced to about 16,000 after quality control). These genotypes were drawn from seven different primary studies, hence the fairly stringent application of quality control leading to the exclusion of a significant number of genomes.

The investigation was conducted in two stages. In the first part, over 500,000 SNPs from nearly 12,000 individuals (4000 known AD cases, 7850 controls) were investigated. Analysis was limited to autosomal chromosomes (except for one SNP on the X-chromosome already known to be associated with AD). In addition to the established link to APOE, the study suggested involvement of two further genes CLU and PICALM. CLU (on chromosome 8) encodes clusterin, another significant brain apolipoprotein . The specific SNP identified lies within an intron.  PICALM (chromosome 11) encodes phosphatidylinositol-binding clathrin assembly protein (aka clathrin assembly lymphoid-myeloid leukaemia gene).  The specific SNP identified is 88.5 kb 5’ to the gene. The authors of this study had prior notice of the study of Lambert et al (see below) and note that CLU is also the novel gene demonstrating the strongest association with AD in their work.

During the second stage, the researchers looked at the two newly identified SNPs in five additional European cohorts, drawn from Belgium, Greece, Bonn, the Medical Research Council, UK and the Alzheimer’s Research Trust. These included just over 2,000 AD cases and about 2,300 age-matched, cognitively screened controls.

Armed with the data of Amouyel et al, these authors also re-examined their own results looking for evidence of associations identified in the other paper. They confirm “suggestive evidence” for association with CR1, the gene encoding complement receptor 1. They also report the genes for bridging integrator 1 (BIN1) and disabled homolog 1 (DAB1) as being “noteworthy”.

Finally, the authors consider the physiological implications of the various mutations.

• APOE acts as a chaperone for Ab, influencing the latter’s conformation, conversion to an insoluble form, aggregation and toxicity.
• Clusterin is a molecule that occurs in a fairly wide number of tissue types and with several likely functions. The gene product is a 449 AA protein which is later glycosylated.  It binds soluble Ab in a specific and reversible manner; the resulting complex has been demonstrated to cross the blood-brain barrier.
• PICALM is ubiquitously in all tissues, but particularly so in neurons. It is involved in clatharin-mediated endocytosis (as too is BIN1). The role of PICALM in trafficking of VAMP2, a protein involved in synaptic vesicle fusion to presynaptic membranes, may underlie its role in AD.

Genome-wide association study identifies variants at CLU and CR1 associated with Alzheimer’s disease
Lambert et al (2009) Nature Genetics 41:1094-1099

In the second paper, the authors (n = 50, plus unnamed members of the European AD Initiative Investigators, and headed by Philippe Amouyel in Lille) conducted an initial study of about 2000 French cases of AD and about 5,300 controls, also from France. As was also seen in the separate study by Harold et al, this analysis identified the CLU gene as having genome-wide significance in AD.

A number of further markers across several chromosomes had “suggestive evidence of association”. Therefore in a second stage, analogous to the methodology used by Harold et al, the researchers looked to confirm the observed association with CLU and probe more deeply the correlation with other markers in the major loci identified. The individuals in this stage (approx 4000 AD cases and 3,300 controls) came from Belgium, Finland, Italy and Spain. This second stage confirmed the significant association with CLU, and also identified a second gene CR1 (for complement receptor 1, the main protein interacting with complement protein C3b).

These authors review several previous experiments that support a suggestion that CR1 is involved in clearance of Ab. It is therefore their contention that whereas the gene errors implicated in early-onset (familial) AD result in Ab overproduction, those involved in standard late-onset AD may lead to inadequate Ab clearance.

None of the genes identified in the recent studies guarantee the development of AD – in this paper it is estimated the risk factors are 25.5% for APOE, 8.9% for CLU and 3.8% for CR1.

Two further points of note. One is the care taken to ensure results are genuine – when aware that a Belgian dataset had been examined in both this study and in the work by Harold et al the present authors recalculated their results excluding the shared data lest it introduced any unwarranted significance by being counted twice. Secondly, this paper labels the gene for phosphatidylinositol-binding clathrin assembly protein as PILCAM. Looking at the literature it seems that this is a typo, not an alternative name.

If you’ve read this far, you may also be interested in the assessment of this research published on the excellent NHS Choice Behind the headlines website.

From time to time examples of scientific fraud come to light and raise questions about the integrity of scientific endeavour. The most well-known example of recent years must surely be South Korean stem cell biologist Hwang Woo-Suk, whose ground-breaking discoveries in the field of therapeutic cloning were exposed as bogus (In addition to his science reputation being in tatters, Hwang was convicted in October 2009 of embezzlement and violation of bioethical laws, although he escaped a custodial sentence).

In physics, the multiple re-use of the same graphs as data for entirely different experiments led to the downfall of a leading young nanoscientist (this was the subject of a 2004 episode of the BBC’s Horizon series The dark secret of Hendrik Schön). Are Hwang and Schön rare examples bringing unwarranted criticism to a body of otherwise exemplary scientists, or are their crimes indicative of much wider malpractice within the scientific community?

University of Edinburgh researcher Daniele Fanelli has shed some light on the the extend of scientific fraud in an article How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. Published in the open access journal PLoS ONE in May 2009, the research brought together data from a number of earlier smaller studies on scientific misconduct to generate “the first meta-analysis of these surveys” (p1).

Fanelli was interested in examining the rates of self-reporting of scientific misconduct and knowledge about the misconduct of colleagues. Recognising that “any boundary defining misconduct will be arbitrary” (p9), he limited discussion to incidents where there was clear “intention to deceive” (p1, p9) rather than generation of incorrect results as a consequence of shoddy experimental design and/or accidental misinterpretation of the data. For the purposes of this study, Fanelli also excluded plagiarism and other examples of “questionable research practices” (QRPs; such as failure to include a contributor amongst the list of authors for a paper) from his definition of scientific misconduct, which was instead limited to fabrication and falsification. The grounds for this decision seem valid; whereas fabrication (the invention of data) and falsification (the wilful distortion of results) change the actual body of scientific knowledge, these other unprofessional activities lead instead to changes in the distribution of credit for the work, the substance of which remains unaltered.

To identify the previous studies of misconduct, Fanelli conducted a search of citation databases, scientific journals, “grey literature” databases and internet search engines using the terms “research misconduct”, “research integrity”, “research malpractice”, “scientific fraud”, “fabrication, falsification” and “falsification, fabrication”. An initial search generated 3276 potentially relevant studies. The vast majority (3207) were easily excluded because they were not surveys of research misconduct.

The author then applied very strict criteria to limit the meta-analysis to genuinely appropriate studies. For example, papers were excluded if there was no quantitative data, if the data included no clear category of never/none/nobody (e.g. if only mean values were shown), if the sample had been generated in a non-random manner, or if undergraduate and/or other non-researchers were included in a manner that did not permit their removal from the dataset). Having done so, the initial pool of potential papers was whittled right down to 18 suitable studies.

Quantifying research malpractice

What were the conclusions of Fanelli’s analysis? The main issues addressed were the proportion of respondent admitting to misconduct or questionable practices of their own, or knowledge of similar behaviour committed by colleagues on at least one occasion.

In the various studies reviewed, between 0.3% and 4.9% of respondents confirmed that they had modified results to improve the outcomes. This led to an average of about 2% self-reporting of misconduct (although it was nearer 1% if the responses were limited to those that specifically mentioned ‘falsification’ or ‘fabrication’.

A rather larger number, 9.5%,  were willing to admit that they had carried out broader questionable practices. Again, however, the phrasing was important with more respondents willing to say they had “modified research results” than admitting that had reported results that they “knew to be untrue”. This may fit with an underlying assumption that it is okay to omit data that you “know” are outliers or otherwise “wrong”. As Fanelli puts it “many did not think that the data they “improved” were falsified” (p9).

When asked about the actions of others, a crude average of around 16.7% (range 5.2% to 33.3%, Fanelli elects to report this statistic as “up to 34%” (p10)) of scientists said they had personal knowledge that a colleague had fabricated or falsified data on at least one occasion. A much wider range (6.2% to 72%; crude mean 28.5%) said that they were aware of peers who had indulged in QRPs.

So, were Hwang and Schön isolated miscreants or does their identification mark the tip of an iceberg of scientific misconduct? The truth seems to lie somewhere in between. As Fanelli notes, usual rules of self-reporting bias – in which some people (typically older women) under-report criminal behaviour whereas others (typically younger males) over-report such activity – do not apply here. It is highly unlikely that anyone in a community where trust is taken seriously will over-report their own wrong actions. It is likely, therefore, that the calculated values of self-reported malpractice are underestimates.

The data regarding knowledge of other researchers’ actions are harder to validate. It is theoretically possible, for example, that more than one correspondent might be describing the wrongdoings of the same colleague. In contrast, the criteria was knowledge of malpractice on “at least one occasion” and therefore the data may not take into account serial offences.

Other approaches to measuring misconduct, as reviewed by Fanelli, have generated a range of figures which might be seen as very broadly equivalent. For example, about 0.02% of paper  are retracted from PubMed due to misconduct (Claxton, 2005). 1% of papers submitted to the Journal of Cell Biology were found to have been inappropriately manipulated (Steneck, 2006). 2% of clinical researchers were found guilty of serious scientific misconduct in routine US Food and Drug Administration audits (Glick, 1992).

Whatever the accuracy of these numbers, however, it remains true that the vast majority of science is carried out in a spirit of accuracy and integrity.

At the risk of sounding like a Carlsberg advert, “The Journal of the Left-handed Biochemist doesn’t do award ceremonies, but if we did…” – what would be the winner of “Best Science programme” during the last 12 months?

In truth, I think it has been a bumper year for science programmes. There has been a tangible return to form at Horizon – the first three episodes of the current series, for example,  have all included significant coverage of molecular biology. There were commendable features to the mini-series Prof Regan’s Diet Clinic/Medicine Cabinet/Nursery/Health Spa (I have posted a separate review of Prof Regan’s Medicine Cabinet here) although it did let itself down at times by commiting some of the same mistakes it was accusing others of making. Pitched at a slightly different audience, it is also good to welcome Bang Goes the Theory to finally fill a gap in popular science coverage left empty since the demise of Tomorrow’s World around 2003.

The BBC Four War Beneath the Skin season in the late summer was consistently strong. Breaking the Mould: The Story of Penicillin helped to amend the traditional account of the birth of antibiotics as therapeutics and Spanish Flu: The Forgotten Fallen recreated events surrounding the battle to protect Manchester from the 1918 influenza epidemic (it also gave them a chance to re-transmit Mutant Mouse and Superfly, documentaries on research using model organisms, from 2004).

My winner of the grand prize would be another programme from that season. Adam Rutherford’s Cell – a 3-part series The Hidden Kingdom, The Chemistry of Life and The Spark of Life – was outstanding. Of the three, it is The Chemistry of Life that is my outright favourite.

The programme tells the story of the identification of DNA as the molecule of inheritance. Beginning with Friedrich Miescher’s original isolation of DNA in 1868, and his amazing determination of its chemical composition, the episode reflects on the key experiments of Theodore Boveri, Thomas Hunt Morgan, Fred Griffith, Oswald Avery, Maurice Wilkins & Rosalind Franklin, James Watson & Francis Crick, culminating with Walter Gehring’s discovery and characterisation of homeobox genes in the 1980s and 1990s (more details can be seen in my notes on the programme, via this link).

The Chemistry of Life can serve as an excellent component of an introductory module on molecular biology for first year undergraduates. Important discoveries are laid out in an accurate yet engaging way. I particularly like the way that the episode naturally demonstrates the evolution of scientific ideas.

The programme has a certain ”yuk factor” that serves to keep it captivating for a youthful audience; description of Miescher’s isolation of pus from the sheets of wounded soldiers and footage of a transgenic Drosophila fly with eyes all over its body being two examples. The episode also has its lighter moments – Rutherford’s visit to the Bay of Naples includes him partaking of a local delicacy as he eats raw urchin gonad straight from the spine-encrusted shell. Later on he deliberately burns his arm with a hot spoon, but it is evident that it was rather hotter than he intended. Perhaps best of all, archive footage of Maurice Wilkins describing ‘MOWlecules’ with received pronunciation and his explanation that X-rays are “wavy” were reminiscent of the Mr Cholmondley-Warner sketches from the old Harry Enfield series.

Maurice Wilkins explains how X-ray can be used to study molecules

Having said that, the film would be a useful teaching resource for first year undergraduates, I think that you would want to include time to emphasise the main points and to put back some of the detail that has inevitably been skipped. For example, fuller description of the basis of Griffith’s experiments with smooth and rough strains of Streptococcus pneumoniae would be beneficial. It is a minor irritation that no on-screen captioning supports the introduction of dead scientists, so offering a class a sheet with the names of the individuals might help students to structure any notes they wished to take.

Do you think a different science programme is more worthy of the title “Best Science Programme of the year”? If so, how about using the comment facility to share your suggestions.

There are many reasons why I am grateful to have spent some of my summer reading Ben Goldacre’s excellent book Bad Science, including the fact that it brought to my attention a paper The Seductive Allure of Neuroscience Explanations. The article is an account of experiments conducted by Deena Weisberg and colleagues at Yale University, and was published in the Journal of Cognitive Neuroscience in 2008.

Recognising that neuroscience is an area of research that fascinates the public and where discoveries are frequently picked up by the general press, Weisberg et al generated four explanatory statements for each of 18 different psychological phenoma. In each case the four statements represented:

• a good explanation without specific mention of neuroscience
• the same good explanation with the addition of plausible, but logically irrelevant, neuroscientific details
• a bad explanation without specific mention of neuroscience
• the same bad explanation with the addition of the same plausible (but irrelevant) neuroscience as in the second example

Satisfaction (on a 7-point scale) engendered by the various statements was tested with three cohorts:

• naive adults (n=81), i.e. individuals with no formal neuroscience training
• students from an introductory cognitive neuroscience course (n=22)
• experts in neuroscience (n=48), although the definition of “expert” in this context was quite generous – 6 members of this group had completed an undergraduate course but were yet to start their advanced degrees, 29 were currently in graduate school and the other 13 were beyond grad school.

In developing their neuroscience explanations, the authors held to three important criteria: (1) the phrasing would indicate that this was a field in which knowledge was already established, (2) the same information would be added to both the good and the bad explanation, and (3) the neuroscience information should not alter the underlying logic of the explanation such as it was before addition of the extra ’science’.

With all three cohorts, the participants were informed that the study being reported was scientifically robust, but that the explanation offered may not be genuine. For both the ‘naive’ and ‘expert’ groups any individual always saw only statements that contained neuroscientific information (but might be a good or bad explanation), or they never had the additional details. This approach was adopted so that subjects were not alerted to the fact that some of the explanations were fuller than others. Unfortunately the relatively small size of the current undergraduate cohort meant that the method was different, any individual could be exposed to all 4 different types of explanation. This is a shame, since it does require some caution when interpreting side-by-side comparisons of the three cohorts.

So what were the findings? For all three groups, participants rated the good explanations as more satisfactory than the bad ones. Both the naive group and the students found explanations with added neuroscience to be better, and the effect was more striking for the bad explanations than for the good. With the experts, however, addition of the spurious scientific details to the good explanation actually led to a reduction in their satisfaction – an indication that with their fuller knowledge they saw through the vacuous additions.

The authors argue that the findings with novices and students may be manifestations of the “seductive details effect”. Previous studies have suggested interesting-but-irrelevant information can have a detrimental effect on cognitive tasks, e.g. memory tests.

They also point to a general observation that individuals respond more favourably when given a reason for a request, however obvious. They cite one example of such “placebic” information in which research subjects were more amenable to letting someone carry out some photocopying if they added the phrase “I have some copies to make” to a bland request “May I use your Xerox machine?”

In Bad Science, Ben Goldacre argues that ‘quacks’ exploit this phenomenon by dressing up their claims with sciencey-sounding explanations. This reminded me too of the BBC documentary Professor Regan’s Medicine Cabinet (first shown in April 2009) in which, amongst other things, Regan dresses up a test of pills for treating insomnia with all sorts of detailed instructions about what must, and what must not, be done in order for the therapy to be effective. The participants stuck to the rules and reported beneficial effects of the pills, despite the fact that they were – in fact – sugar cake decorations.

Weinberg D.S., Keil F.C., Goodstein J., Rawson E. and Gray J.R. (2008) The seductive allure of neuroscience explanations Journal of Cognitive Neuroscience 20:470-477

Harper Perennial edition (2009)

If you have not yet read Ben Goldacre’s book Bad Science, then I thoroughly recommend that you do. As readers of his regular Guardian column or his website will already know, Goldacre has embarked on a campaign to root out example of pseudoscience and shoddy science whereever they may be found.

All the usual villians are present – homeopaths, nutritionists, slack journalists, pharmaceutical companies and AIDS dissenters. Some are mentioned by name, but given their alleged predilection for litigation, and since I do not have the time, the money or the inclination to do battle with them in the courts, I shall not repeat their identities here!

It would be wrong, however, to give the impression that Goldacre is merely on a crusade against high profile exponents of “bad science”. True, the author does sometimes betray a little too much glee as he places a bomb under the throne of a media ”health expert” (in a way that I found disturbingly reminiscent of the Physiology lecturer, when I was a first year undergraduate, recalling his boyhood experiments on frogs). Nevertheless, Goldacre is keen to emphasise that his purpose is to “teach good science by examining the bad” (p165 in my copy), adding that ”the aim of this book is that you should be future-proofed against new variants of bullshit” (p87).

It seems to me that Goldacre is correct in his assertion that the public needs help in ‘bullshit-spotting’ and that this book is an extremely valuable tool in achieving that goal. Scientific colleagues will (hopefully!) be familiar with at least some of the pitfalls of poor study design, inappropriate use of statistics and outright spin that lead to dramatic-but-spurious headlines in the newspapers. I am, however, convinced that there is plenty here that will improve the scientific literacy of undergraduates in medicine and bioscience subjects, as well as a more general readership.

Trial design
For an experiment involving human subjects to have at least some hope of generating objective data, it is important that the research method includes:

• control groups – you need something against which to compare your  intervention, whether it be a placebo or sham treatment, or the best treatment currently in use;
• appropriate blinding - i.e. that neither researcher nor participants know during the trial which individual is receiving each intervention;
• randomisation - trial subjects need to be assigned to different regimes in a genuinely unbiased way (some randomisation protocols are actually open to significant abuse, albeit subconscious);
• documentation - when the work is published, the account needs to include suitably transparent and complete details of the methods and the results such that any reader will know how the study was conducted and can therefore have a sporting chance of spotting the glitches.

The case of the widely-reported Durham trial of fish oil tablets containing Omega-3 fats (Chapter 8, ‘Pill solves complex social problem‘) is a chastening tale of ways in which poor research methodology can effectively ruin a study before it has even started. Alarm bells ought to have been triggered as soon as the trial (I will call it that for simplicity, although those involved in the research have shyed away from this term) was trumpeted in advance as a test to prove the effectiveness of fish oils in boosting academic performance. The fact that the participants knew that they were in a trial has been shown in itself to ellicit improvements (the so-called ‘Hawthorne effect’), even without the media scrum that accompanied this particular trial. Add to this the influence of potential ‘confounding factors’ (see below) and this study was never going to give clear and unequivocal results.

Common mistakes involving science literature
Goldacre’s critique of ‘nutritionists’ highlights four frequent errors in the way that science literature is handled. These are:

• extrapolating and overinterpreting data – For example studies conducted on isolated cells in vitro, can provide useful pointers for future studies in humans, but it is wrong to naively take findings from cell-based work and assume the equivalent is true in vivo in a whole organism. To purloin one of Goldacre’s favourite phrases “I think you’ll find it’s a bit more complicated than that” (p100)
• extrapolating from observational studies to make claims that require an interventional study to be conducted – ‘confounding variables’, that is differences between individuals that may or may not be linked to the factor under investigation, are hard enough to control in a study where the researcher is deliberately intervening in the partcipants’ lives to measure any apparent effects. If the study is merely observing differences between people reported to have an important lifestyle or dietary factor, there may be a lot more going on. Superficial analyses are prone to come up with erroneous conclusions.
• Cherry-picking only results that fit the hypothesis – it is, as Goldacre points out, a facet of human nature both to see patterns in data and to be more receptive to results that fit your expectation than those that do not. We need therefore to guard against selectively quoting only experiments that give the results that we want, and ignoring data (possibly the majority of findings) that don’t fit our model. This is why ‘systematic review‘ of all of the data on a particular topic is an essential process.
• Referring to studies that are not published in peer-reviewed journals, and frequently not published at all – it is bad enough when conference papers and press releases are reported with the same gravitas and authority as experiments which have been scrutinised by experts in the same field as part of the peer-review process. It is even worse, however, when some interviewees are prone to make specific claims such as “a study published just last week in America has described the same effect we see here” whilst it later turns out that no such article exists. In written work, some authors have increasingly given their books a spurious air of authority by adopting the trappings of good citation practice, e.g. use of superscript numbers to direct readers to their sources. When you flick on to check the reference, however, it turns out to be a non-scholarly document or something that they themselves have said on a different occasion.

Lies, damned lies and statistics
Statistics are clearly vital in substantiating the findings of any kind of trial and Goldacre attacks abuse of statistics on two fronts. Firstly, there is the deliberate use of an inappropriate statistical test to generate a positive-sounding number. Pharmaceutical companies are said to be guilty of this sleight of hand, and it requires a certain amount of statistical nous in order to detect when this crime is being perpetrated.

Secondly, there is the way that the numbers are presented to the public. Newspapers are prone to report the ‘relative risk increase‘, i.e. the percentage increase in condition X when presented with risk Y because it generates the most attention-grabbing numbers. The shock statistic “reading science-related blogs increases the chance that you’ll have a heart attack by 50%” may alarm you (so just in case, let me say straight way that I made this up).  A very different impression is given if we consider the ‘absolute risk increase‘ which state that “reading science-related blogs increases the chance that you’ll have a heart attack by 0.2%”. Goldacre recommends that there ought to be a move towards quoting ‘natural frequencies‘, i.e. as intelligible numbers. In this case, therefore we might say “reading science-related blogs increases the chance that you’ll have a heart attack from 4 in every 1000 people if you don’t, to 6 in every 1000 people if you do”.

Putting Bad Science to use in formal education
Are there ways in which Bad Science might be employed as a teaching tool in either secondary or tertiary education? The specifications for GCSE Science in England and Wales were altered in 2006 to place greater emphasis on “How Science Works“, and A levels were similarly altered in 2008 when this cohort passed on to the higher qualification. The reading level required to appreciate Bad Science probably procludes recommending it for the majority of 16 year olds. I believe, however, that the text would make an excellent resource for students of A level biology and/or General Studies. I do not know if the publishers have considered producing a structured guide based on the book or inclusion of end of chapter  study questions in future editions, but there is certainly scope for this.

Similarly, the book would be valuable reading for first year undergraduates in Medicine, Bioscience or Journalism. I think there would be more merit in having this as prescribed reading for a Year One skills or introductory module than several of the more ‘academic’ alternatives.

As an admissions tutor, I receive several e-mails each summer from students starting the following term and asking which textbooks to buy. My consistent response this time around has been to recommended that they read Bad Science now and wait until the course has started before they part with money for a chunky Biochemistry text.

Gripes
This is not to say that Bad Science is without faults. I do have a number of minor quibbles with the book, but I would say for the most part the fault lies with the editorial process rather than with the author per se.

Haven’t I read that before? Understandably much of the content of the book has already seen the light of day in shorter pieces in the Guardian’s Bad Science column. Repetition and/or poor ordering (by which I mean a point is introduced at length after it has already been previously noted) betray the ‘cut and shut’ nature of some of the present material. As an example of the former, we are told twice in consecutive paragraphs on page 113 about the crusade led by cereal magnate John Harvey Kellogg against a particular personal vice. Similarly, the fact that Durham council altered a press release on their website sometime after its release in order to remove the word ‘trial’ is mentioned on pages 143 and 149.

Examples of the ‘introduction after being stated’ phenomenon include the mention on page 157 that Equazen had been acquired by Galenica, follwed on p160 by a fuller account of this transaction in a tone that gave the impression it was ‘new news’. Similarly we are told on page 313 that some researchers did “something called a ‘case-control’ study” despite the fact that case-control studies were amongst the variety of experimental models discussed on page 103 and pages 295-296.

Page numbering: The cover of my edition of the book (Harper Perennial, 2009) trumpets the addition of an extra chapter. This material has not been added at the end of the text, but rather inserted at the appropriate point in the unfolding ’story’. In consequence, page numbering downstream of the insertion is altered. Although this has been recognised in the index, there are several examples of in-text cross-references where the page numbers are now 17 out. (In case anyone with influence on the next version is reading this review the reference on page 106 to p240 should be p257; page 282 should cite p294 not p277; page 330 should point to p293 not p276).

Referencing: Bad Science is intended to be a popular book not an academic tome. As such, it would be completely inappropriate for the text to be peppered with citations in a way that would interrupt the flow. I think the solution chosen here works very well – the notes in the back use page numbers and a short quote from the text as the identifiers of the source. It is partly because I know Goldacre makes regular criticism of the lack of referencing in media reports of science that I was disappointed on a couple of occasions to turn eagerly to the back and not find a citation. These tended to be times when a broad statement had been made – for example on page 75 “A huge amount of research...” does not provide any corroborating references and on page 144 “there is a lot of history here… the field of essential fatty acid research has seen research fraud, secrecy, court cases, negative findings that have been hushed up, media misreporting on a massive scale…[the list continues]” but no notes are offered. If a new edition is produced, please can these be added.

Summary
As I have already said, these are minor (some would say picky) criticisms of an otherwise extremely valuable book. Overall, I believe Ben Goldacre has provided all of us with a toolbox for evaluating sciencey-sounding stories in the media and alerted future scientists to some of the pitfalls they should avoid in the design and reporting of their work. Bad Science would make an excellent resource for post-16 education and I hope to see it adopted as a course text on A level and undergraduate programmes.

Bad Science has a cover price of £8.99 At the time of writing it is available from Amazon for £3.60.

Amongst the major science research journals, Science magazine has consistently been the most prominent in flying the flag for science education. I was very interested, therefore, in an Editorial by Carl Wieman in the September 4th 2009 issue of the magazine. In his piece Galvanising Science Departments, Wieman describes some fairly radical innovations in Science Education currently underway at the University of Colorado and the University of Bristish Columbia. The aim is to adopt evidence-based teaching methodologies with emphasis on the development of scientific thinking and problem-solving skills rather than fact regurgitation.

I have no direct experience of teaching in the USA, either as provider or recipient. I know, for example, that much greater emphasis is placed on the recommended course text in the USA than in the UK, but beyond that I cannot speak with any authority. It does sound like some of the reported innovations are things that have taken place here for some while, such as the addition of specific (skill-centred) learning goals to modules. A cornerstone of the strategy has been appointment of science education specialists, individuals who not only have expertise in their subject discipline, but are also au fait with educational and cognitive psychology studies, a variety of effective teaching strategies and – I note with some mirth – possess diplomatic skills!  The programme is ongoing, the University of Colorado is in the 4th year of an initial six year project and so the full impact of the developments will not be known for some while.

What really struck me, however, was the extent of the commitment at an institutional level, including provision of serious money to fund these changes. Far too many pro-active educators, motivated by genuine desire to improve the learning experience for their students actual receive flak not gratitude. In many cases this is, I believe, because their approach to pedagogy is different to the cultural norm and, as such, moved students (and staff) outside their comfort zone.

So much of education, even at University level, is about recall of a prescribed body of information. This is relatively easy – for both the teacher and the student; it is the mindset that underlies that chirping questions “will this be in the test?” Development of thinking skills demands more from everyone. If innovations that require higher skills are associated with just one module, or even just one academic, then that individual may unfairly receive criticism in feedback from the class.

I believe this is why Wieman is absolutely right when he says “an entire department must be the unit of change”. Depending upon institutional structure, it may even require a larger body – School, Faculty, College – to move together with shared commitment to the new goals. So far at Colorado, 60% of academic staff in three Science departments have embraced the new teaching approaches, impacting 80% of their students’ credit hours. Faculty are reported to enthusiastically discussing teaching as a scholarly activity – that’s surely got to be a good thing. But – it needs time and it needs money.

Wieman C (2009) Galvanising Science Departments Science 325:1181

I have been a devotee of social bookmarking tool delicious since 2007 and now have nearly 4000 items tagged. Although the ‘before’ and ‘after’ photos (slide 17) in my July 2008 presentation Knowing where it’s at: find it? flag it? share it? (or how delicious saved my life) were staged for effect, the ability to accumulate links to resources online rather than generate piles on (unread) papers in my office has been a genuine revelation.

Alongside delicious, I also dabbled briefly with Connotea, the online reference management tool from the Nature stable. It has the same potential as delicious for user-generated tags, but at the time I couldn’t really see what additional value it was adding and I let my interest wither, electing instead to use delicious alone for all of my bookmarks, including journal articles.

More recently, I’ve been persuaded by a colleague to take a close look at rival social citation application citeulike. This time around I think I get it. One of the features that really appeals is the potential to import comprehensive bibliographic information armed only with the Digital Object Identifier (DOI). With journals making the DOI of articles increasingly obvious on their websites and in table of contents alerts, this becomes a very straightforward way to collate large quantities of metadata whilst retaining the capability to tag a paper with whatever keywords reflect its relevance to you.

In truth, I have not conducted a rigorous side-by-side comparison of citeulike v connotea (or any of the other similar tools). I am quite sure, for example, that they all have the potential to assimilate bibliographic details armed only with the DOI. For the foreseable future I will continue to tag journal articles using delicious. However, this feature of citeulike, couple with the capabililty to establish shared libraries of articles relevant to members of a particular list, has persuaded me to also give the latter a prolonged trial.

I was intrigued by a recent paper Cognitive control in media multitaskers in the highly-regarded journal Proceedings of the National Academy of Sciences. The study looked at the information processing styles of self-reported media multitaskers, defined as users of two or more content streams simultaneously, compared with those who do not multitask in this way. (I will skirt over the irony both that I was supposed to be doing something else at the time I spotted the BBC report of the research, and that my son has just switched on the TV as I type this review on my laptop.)

An electronic copy of the paper was posted online in August 2009, ahead of publication in the print journal (doi: 10.1073/pnas.0903620106)

In their research, Ophir et al asked 262 University students to complete an online self-assessment questionnaire regarding both the total number of hours spent using different media (they specified 12 formats including TV, online video use, music, print media, e-mail and text messaging) and how likely they might be to use some of these concurrently alongside a primary task. The authors then generated a numerical Media Multitasking Index (MMI) and ranked the students. Those with a score one or more standard deviations below the mean (light media multitaskers, LMMs) or one or more SDs above the mean (heavy media multitaskers, HMMs) were then invited to participate in a series of cognitive ability tests. In total there were between 30 and 41 students taking the various tests, evenly split between LMMs and HMMs.

Central to the whole study was the issue of working memory, a concept we have discussed previously on JLB. Some of the experiments were designed to look at the students abilities to filter out distractions and remain focused on the main task. In one test, for example, the students were asked to compare the orientation of two red rectangles in consecutive images shown to them 900 milliseconds apart. To investigate the influence of distractions, the image also included 0, 2, 4 or 6 blue rectangles. Each student was asked to do this 200 times in one sitting,  during which there were an equal number of slides in which the position changed and in which it did not (the order in which they saw each of the image pairs was randomised). This test revealed a difference between the LMMs and the HMMs. Contrary to the hopes of self-confessed media multitaskers everywhere, the HMM group seemed to be more easily distracted than the LMM, whose performance was largely unaffected by the presence of blue rectangles.

In a second test, the investigators looked into the ability of participants to hit the correct button “animal” v “nonanimal” when presented with 24 words. Each word was presented one at a time, and all 24 words were used twice in the sequence (ie 48 words total). The variation was then introduced whereby the students were presented with the same words again, but this time with the additional instruction that they should refrain from hitting the button if they heard a stop tone before they responded. The signal was transmitted 225 ms before the mean response time, and was used on 25% of occasions. In this test there was not a significant difference between the two groups.

There was also no significant difference between the HMMs and LMMs in the first part of a third test. Participants were shown a sequence of letter pairs 300 ms apart (the letters were red on a black background, the relevance of this will be reached in a moment). The students were to hit the “no” button on all occasions except when there was a cue “A” followed by a probe “X”, hence this is known as an AX Continuous Performance Task (AX-CPT).

A difference was reported, however, when distractors were introduced into the AX-CPT. Between the cue letter (red on black, remember) and the probe letter (also red on black), the participants were presented with three distractor letters which were white on black. The distractors were never A, X or indeed K or Y both of which were entirely omitted from the study to avoid shape recognition clashes of X v Y/K. The overall time between cue and probe was held constant (4900 ms) relative to original AX-CPT test.

In the AX-CPT test with distractors, the HMM students were found to have the same accuracy as the LMM cohort, but responded more slowly. Alongside the relative responses in the rectangle distraction test, it seems that HMMs are worse than LMMs at filtering out irrelevant environmental stimuli.

Given the inherent need to swap between activities implicit in multi-tasking, the findings from the task-switching test are probably the most illuminating. In this experiment, the students were presented with a cue that said either “letter” or “number” followed by a stimulus that always featured one letter and one number. If the “letter” cue had been shown, the participants were to ignore the number and press the left button for a vowel and the right button for a non-vowel (only 4 consonants (K, N, P, S)  and 4 vowels (A, E, I, U) were ever used to avoid bias to the non-vowel side; O was not included). conversely if the “number” cue was used then it was left button for “odd” and right button for “even” (again, only 4 of each sort of stimuli were shown; the numbers 1 and 0 were omitted. Each student saw 4 sets of 80 cue and stimulus pairs, 40% involved task switching and 60% non-switching. They were given “warm-up” activities either involving switching between numbers and letters or non-switching.  HMMs were slower to respond than LMMs in both non-switch (259 ms) and switch (426 ms) trials. This result is interpreted as showing that HMMs are less effective at suppressing irrelevant tasks.

In the final set of studies, the researchers looked at abilities to ignore distractions within memory. This was investigated by conducting so-called “two-back” and “three-back” tasks in which participants were asked to indicate whether a letter on the screen was a “target”, i.e. the same as the letter 2 or 3 previously (in respective study) or “non-target”, i.e. different. Each student has 3 x 30 goes, in which 10 were targets and 20 non-targets.

In the two/three-back tasks the HMMs again performed less well than the LMMs. Unsurprisingly, both groups were worse in the three-back tests than in the two-back tests and their relative declines between the two were similar. Interestingly, however, HMMs had a greater tendancy to identify false positives, i.e. to report a non-target as target, particularly in association with letters than had come earlier in the sequence and were thus “familiar” but not in the correct relative position. This effect became more apparent as the task progressed, presumably as there were more “familiar” letters that had been and gone.

So, the headline message was that self-reported multitaskers are less adept at multitasking than those who shun multiple media. As someone who will admit to being an HMM, this is a disappointment. We could argue that the tests used in the study were surrogates – the real worldquestion would be “can you get work done appropriately whilst Twitter and E-mail are open on your dashboard and you are listening to your iPod?” not “can you remember if the red rectangles are in the same orientation if there are lots of blue ones there too?” I would also defend my position by acknowledging that my Twitter network is now a major source of information with “bang on the money” relevance for my work. Nevertheless I do know for myself that when I’ve really got to get something written I hide myself away with minimum distractions, the only added media will be background music to block out other noise, and this must be lyric-free to be fit for purpose! So, I will reflect on my work practice and perhaps make accessing of e-mail and Twitter more punctuated and less continuous, but I don’t feel that cutting off these channels entirely would – on balance – be a benefit. As the authors themselves point out, it may be that HMMs are demonstrating “a bias towards exploratory, rather than exploitative, information processing”.

Psychology and cognitive science are not my normal field of study and I therefore hope that I haven’t misrepresented any of the methods or findings of the PNAS paper in the proceeding discussion. I thought this was a truly interesting investigation, and highly topical – it is no longer just adolescents that are habitual multitaskers. Finally, from an educational research point of view, it was curious to note that participants in this research received course credits for their involvement in the study, both for taking the questionnaire in the first place and then for their role in the tasks themselves. I’m not sure we’d get away with that in a UK context.