I have been doing some reading for a while now on the ethics of research involving model organisms, particularly the potential for studies on lower species to offer insights into human disease (and thereby contribute to the 3Rs). Some of my musings on the topic can be found here.

Aware of this interest, a colleague recommended that I read a 2004 paper published in the journal Cell. I am very grateful that he did, since the study really has the “wow” factor – demonstrating beautifully the potential of comparative genomics, experiments on model organisms and knowledge of human disease to work together to produce new insights that would have been much harder if any one component was missing. The paper is Comparative genomics identifies a flagellar and basal body proteome that includes the BBS5 human disease gene by Li JB et al. The following notes are my attempt to summarise the best bits.

The importance of cilia and basal bodies in disease
The role of cilia in respiration (and the detrimental effects of smoking on their function) were features of the school biology curriculum when I was a child. However, research over the last ten years or so has demonstrated that cilia have surprisingly diverse roles in development, from determination of left-right symmetry in the body, through to formation and function of specific organs such as the kidneys (for more detail see the Wikipedia entry on Ciliopathy or, if you have access permissions,  Badano et al (2006), The ciliopathies: an emerging class of human genetic disorders Annual Review of Genomics and Human Genetics 7:125-148). Bardet-Biedl syndrome (BBS) is one disorder associated with non-functional or malfunctional cilia. The clinical features can be varied, but include obesity, mental retardation, progressive-onset blindness and polydactylism (i.e. possession of extra digits).

Cilia, and flagella in organisms that possess these related structures, develop from, and remain attached to, modified centrioles called basal bodies. These complex structures involve upwards of 150 proteins.

Comparative genomics
The complete genome sequence for a growing collection of organisms is now known. The authors of the Cell paper – headed up by Susan Dutcher of Washington University, St Louis -  reasoned that they could identify novel genes involved in basal body development by judicious use of such genomic information.

The logic goes like this: basal bodies are needed for both cilia and flagella, therefore genes necessary for proteins in the basal body should be present in the genomes of an organism with cilia and an organism with flagella – even if otherwise morphologically dissimilar – but not in the genome of a species with neither cilia nor flagella. A Venn diagram can be helpful in visualising this subset of genes.

researchers were interested in finding genes expressed in humans (an organism with cilia) and Chlamydomonas (an organism with a flagellum) but not Arabidopsis (which has neither cilia nor flagella) - see text for details

When they looked at the genomes of humans (ciliated, approx 32,ooo genes) and Chlamydomonas (a unicellular flagellate with approximately 20,000 genes) they homed in on a set of about 4,350 genes that were common to both species despite their obvious diversity.

Next, they subtracted from that list all of the genes that were also present in the genome of the thale cress Arabidopsis (approximately 29,000 genes in total).  This plant species has neither cilia nor flagella and therefore ought not to have the genes necessary for their development.

It turned out that a total of 686 genes fit into the subset “in human and in Chlamydomonas but not in Arabidopsis“, termed the flagellar apparatus-basal body (FABB) group by the researchers, and corresponding to the yellow sector in the Venn Diagram above. This may still seem a large number, but when compared with the total of at least 20,000 genes it represents elimination of over 96% of the genome as not being involved in basal bodies.  In this way they hoped that they had shortened the odds on identifying the role(s) played by genes of previously unknown function.

.

From time to time examples of scientific fraud come to light and raise questions about the integrity of scientific endeavour. The most well-known example of recent years must surely be South Korean stem cell biologist Hwang Woo-Suk, whose ground-breaking discoveries in the field of therapeutic cloning were exposed as bogus (In addition to his science reputation being in tatters, Hwang was convicted in October 2009 of embezzlement and violation of bioethical laws, although he escaped a custodial sentence).

In physics, the multiple re-use of the same graphs as data for entirely different experiments led to the downfall of a leading young nanoscientist (this was the subject of a 2004 episode of the BBC’s Horizon series The dark secret of Hendrik Schön). Are Hwang and Schön rare examples bringing unwarranted criticism to a body of otherwise exemplary scientists, or are their crimes indicative of much wider malpractice within the scientific community?

University of Edinburgh researcher Daniele Fanelli has shed some light on the the extend of scientific fraud in an article How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. Published in the open access journal PLoS ONE in May 2009, the research brought together data from a number of earlier smaller studies on scientific misconduct to generate “the first meta-analysis of these surveys” (p1).

Fanelli was interested in examining the rates of self-reporting of scientific misconduct and knowledge about the misconduct of colleagues. Recognising that “any boundary defining misconduct will be arbitrary” (p9), he limited discussion to incidents where there was clear “intention to deceive” (p1, p9) rather than generation of incorrect results as a consequence of shoddy experimental design and/or accidental misinterpretation of the data. For the purposes of this study, Fanelli also excluded plagiarism and other examples of “questionable research practices” (QRPs; such as failure to include a contributor amongst the list of authors for a paper) from his definition of scientific misconduct, which was instead limited to fabrication and falsification. The grounds for this decision seem valid; whereas fabrication (the invention of data) and falsification (the wilful distortion of results) change the actual body of scientific knowledge, these other unprofessional activities lead instead to changes in the distribution of credit for the work, the substance of which remains unaltered.

To identify the previous studies of misconduct, Fanelli conducted a search of citation databases, scientific journals, “grey literature” databases and internet search engines using the terms “research misconduct”, “research integrity”, “research malpractice”, “scientific fraud”, “fabrication, falsification” and “falsification, fabrication”. An initial search generated 3276 potentially relevant studies. The vast majority (3207) were easily excluded because they were not surveys of research misconduct.

The author then applied very strict criteria to limit the meta-analysis to genuinely appropriate studies. For example, papers were excluded if there was no quantitative data, if the data included no clear category of never/none/nobody (e.g. if only mean values were shown), if the sample had been generated in a non-random manner, or if undergraduate and/or other non-researchers were included in a manner that did not permit their removal from the dataset). Having done so, the initial pool of potential papers was whittled right down to 18 suitable studies.

Quantifying research malpractice

What were the conclusions of Fanelli’s analysis? The main issues addressed were the proportion of respondent admitting to misconduct or questionable practices of their own, or knowledge of similar behaviour committed by colleagues on at least one occasion.

In the various studies reviewed, between 0.3% and 4.9% of respondents confirmed that they had modified results to improve the outcomes. This led to an average of about 2% self-reporting of misconduct (although it was nearer 1% if the responses were limited to those that specifically mentioned ‘falsification’ or ‘fabrication’.

A rather larger number, 9.5%,  were willing to admit that they had carried out broader questionable practices. Again, however, the phrasing was important with more respondents willing to say they had “modified research results” than admitting that had reported results that they “knew to be untrue”. This may fit with an underlying assumption that it is okay to omit data that you “know” are outliers or otherwise “wrong”. As Fanelli puts it “many did not think that the data they “improved” were falsified” (p9).

When asked about the actions of others, a crude average of around 16.7% (range 5.2% to 33.3%, Fanelli elects to report this statistic as “up to 34%” (p10)) of scientists said they had personal knowledge that a colleague had fabricated or falsified data on at least one occasion. A much wider range (6.2% to 72%; crude mean 28.5%) said that they were aware of peers who had indulged in QRPs.

So, were Hwang and Schön isolated miscreants or does their identification mark the tip of an iceberg of scientific misconduct? The truth seems to lie somewhere in between. As Fanelli notes, usual rules of self-reporting bias – in which some people (typically older women) under-report criminal behaviour whereas others (typically younger males) over-report such activity – do not apply here. It is highly unlikely that anyone in a community where trust is taken seriously will over-report their own wrong actions. It is likely, therefore, that the calculated values of self-reported malpractice are underestimates.

The data regarding knowledge of other researchers’ actions are harder to validate. It is theoretically possible, for example, that more than one correspondent might be describing the wrongdoings of the same colleague. In contrast, the criteria was knowledge of malpractice on “at least one occasion” and therefore the data may not take into account serial offences.

Other approaches to measuring misconduct, as reviewed by Fanelli, have generated a range of figures which might be seen as very broadly equivalent. For example, about 0.02% of paper  are retracted from PubMed due to misconduct (Claxton, 2005). 1% of papers submitted to the Journal of Cell Biology were found to have been inappropriately manipulated (Steneck, 2006). 2% of clinical researchers were found guilty of serious scientific misconduct in routine US Food and Drug Administration audits (Glick, 1992).

Whatever the accuracy of these numbers, however, it remains true that the vast majority of science is carried out in a spirit of accuracy and integrity.

Harper Perennial edition (2009)

If you have not yet read Ben Goldacre’s book Bad Science, then I thoroughly recommend that you do. As readers of his regular Guardian column or his website will already know, Goldacre has embarked on a campaign to root out example of pseudoscience and shoddy science whereever they may be found.

All the usual villians are present – homeopaths, nutritionists, slack journalists, pharmaceutical companies and AIDS dissenters. Some are mentioned by name, but given their alleged predilection for litigation, and since I do not have the time, the money or the inclination to do battle with them in the courts, I shall not repeat their identities here!

It would be wrong, however, to give the impression that Goldacre is merely on a crusade against high profile exponents of “bad science”. True, the author does sometimes betray a little too much glee as he places a bomb under the throne of a media ”health expert” (in a way that I found disturbingly reminiscent of the Physiology lecturer, when I was a first year undergraduate, recalling his boyhood experiments on frogs). Nevertheless, Goldacre is keen to emphasise that his purpose is to “teach good science by examining the bad” (p165 in my copy), adding that ”the aim of this book is that you should be future-proofed against new variants of bullshit” (p87).

It seems to me that Goldacre is correct in his assertion that the public needs help in ‘bullshit-spotting’ and that this book is an extremely valuable tool in achieving that goal. Scientific colleagues will (hopefully!) be familiar with at least some of the pitfalls of poor study design, inappropriate use of statistics and outright spin that lead to dramatic-but-spurious headlines in the newspapers. I am, however, convinced that there is plenty here that will improve the scientific literacy of undergraduates in medicine and bioscience subjects, as well as a more general readership.

Trial design
For an experiment involving human subjects to have at least some hope of generating objective data, it is important that the research method includes:

• control groups – you need something against which to compare your  intervention, whether it be a placebo or sham treatment, or the best treatment currently in use;
• appropriate blinding - i.e. that neither researcher nor participants know during the trial which individual is receiving each intervention;
• randomisation - trial subjects need to be assigned to different regimes in a genuinely unbiased way (some randomisation protocols are actually open to significant abuse, albeit subconscious);
• documentation - when the work is published, the account needs to include suitably transparent and complete details of the methods and the results such that any reader will know how the study was conducted and can therefore have a sporting chance of spotting the glitches.

The case of the widely-reported Durham trial of fish oil tablets containing Omega-3 fats (Chapter 8, ‘Pill solves complex social problem‘) is a chastening tale of ways in which poor research methodology can effectively ruin a study before it has even started. Alarm bells ought to have been triggered as soon as the trial (I will call it that for simplicity, although those involved in the research have shyed away from this term) was trumpeted in advance as a test to prove the effectiveness of fish oils in boosting academic performance. The fact that the participants knew that they were in a trial has been shown in itself to ellicit improvements (the so-called ‘Hawthorne effect’), even without the media scrum that accompanied this particular trial. Add to this the influence of potential ‘confounding factors’ (see below) and this study was never going to give clear and unequivocal results.

Common mistakes involving science literature
Goldacre’s critique of ‘nutritionists’ highlights four frequent errors in the way that science literature is handled. These are:

• extrapolating and overinterpreting data – For example studies conducted on isolated cells in vitro, can provide useful pointers for future studies in humans, but it is wrong to naively take findings from cell-based work and assume the equivalent is true in vivo in a whole organism. To purloin one of Goldacre’s favourite phrases “I think you’ll find it’s a bit more complicated than that” (p100)
• extrapolating from observational studies to make claims that require an interventional study to be conducted – ‘confounding variables’, that is differences between individuals that may or may not be linked to the factor under investigation, are hard enough to control in a study where the researcher is deliberately intervening in the partcipants’ lives to measure any apparent effects. If the study is merely observing differences between people reported to have an important lifestyle or dietary factor, there may be a lot more going on. Superficial analyses are prone to come up with erroneous conclusions.
• Cherry-picking only results that fit the hypothesis – it is, as Goldacre points out, a facet of human nature both to see patterns in data and to be more receptive to results that fit your expectation than those that do not. We need therefore to guard against selectively quoting only experiments that give the results that we want, and ignoring data (possibly the majority of findings) that don’t fit our model. This is why ‘systematic review‘ of all of the data on a particular topic is an essential process.
• Referring to studies that are not published in peer-reviewed journals, and frequently not published at all – it is bad enough when conference papers and press releases are reported with the same gravitas and authority as experiments which have been scrutinised by experts in the same field as part of the peer-review process. It is even worse, however, when some interviewees are prone to make specific claims such as “a study published just last week in America has described the same effect we see here” whilst it later turns out that no such article exists. In written work, some authors have increasingly given their books a spurious air of authority by adopting the trappings of good citation practice, e.g. use of superscript numbers to direct readers to their sources. When you flick on to check the reference, however, it turns out to be a non-scholarly document or something that they themselves have said on a different occasion.

Lies, damned lies and statistics
Statistics are clearly vital in substantiating the findings of any kind of trial and Goldacre attacks abuse of statistics on two fronts. Firstly, there is the deliberate use of an inappropriate statistical test to generate a positive-sounding number. Pharmaceutical companies are said to be guilty of this sleight of hand, and it requires a certain amount of statistical nous in order to detect when this crime is being perpetrated.

Secondly, there is the way that the numbers are presented to the public. Newspapers are prone to report the ‘relative risk increase‘, i.e. the percentage increase in condition X when presented with risk Y because it generates the most attention-grabbing numbers. The shock statistic “reading science-related blogs increases the chance that you’ll have a heart attack by 50%” may alarm you (so just in case, let me say straight way that I made this up).  A very different impression is given if we consider the ‘absolute risk increase‘ which state that “reading science-related blogs increases the chance that you’ll have a heart attack by 0.2%”. Goldacre recommends that there ought to be a move towards quoting ‘natural frequencies‘, i.e. as intelligible numbers. In this case, therefore we might say “reading science-related blogs increases the chance that you’ll have a heart attack from 4 in every 1000 people if you don’t, to 6 in every 1000 people if you do”.

Putting Bad Science to use in formal education
Are there ways in which Bad Science might be employed as a teaching tool in either secondary or tertiary education? The specifications for GCSE Science in England and Wales were altered in 2006 to place greater emphasis on “How Science Works“, and A levels were similarly altered in 2008 when this cohort passed on to the higher qualification. The reading level required to appreciate Bad Science probably procludes recommending it for the majority of 16 year olds. I believe, however, that the text would make an excellent resource for students of A level biology and/or General Studies. I do not know if the publishers have considered producing a structured guide based on the book or inclusion of end of chapter  study questions in future editions, but there is certainly scope for this.

Similarly, the book would be valuable reading for first year undergraduates in Medicine, Bioscience or Journalism. I think there would be more merit in having this as prescribed reading for a Year One skills or introductory module than several of the more ‘academic’ alternatives.

As an admissions tutor, I receive several e-mails each summer from students starting the following term and asking which textbooks to buy. My consistent response this time around has been to recommended that they read Bad Science now and wait until the course has started before they part with money for a chunky Biochemistry text.

Gripes
This is not to say that Bad Science is without faults. I do have a number of minor quibbles with the book, but I would say for the most part the fault lies with the editorial process rather than with the author per se.

Haven’t I read that before? Understandably much of the content of the book has already seen the light of day in shorter pieces in the Guardian’s Bad Science column. Repetition and/or poor ordering (by which I mean a point is introduced at length after it has already been previously noted) betray the ‘cut and shut’ nature of some of the present material. As an example of the former, we are told twice in consecutive paragraphs on page 113 about the crusade led by cereal magnate John Harvey Kellogg against a particular personal vice. Similarly, the fact that Durham council altered a press release on their website sometime after its release in order to remove the word ‘trial’ is mentioned on pages 143 and 149.

Examples of the ‘introduction after being stated’ phenomenon include the mention on page 157 that Equazen had been acquired by Galenica, follwed on p160 by a fuller account of this transaction in a tone that gave the impression it was ‘new news’. Similarly we are told on page 313 that some researchers did “something called a ‘case-control’ study” despite the fact that case-control studies were amongst the variety of experimental models discussed on page 103 and pages 295-296.

Page numbering: The cover of my edition of the book (Harper Perennial, 2009) trumpets the addition of an extra chapter. This material has not been added at the end of the text, but rather inserted at the appropriate point in the unfolding ’story’. In consequence, page numbering downstream of the insertion is altered. Although this has been recognised in the index, there are several examples of in-text cross-references where the page numbers are now 17 out. (In case anyone with influence on the next version is reading this review the reference on page 106 to p240 should be p257; page 282 should cite p294 not p277; page 330 should point to p293 not p276).

Referencing: Bad Science is intended to be a popular book not an academic tome. As such, it would be completely inappropriate for the text to be peppered with citations in a way that would interrupt the flow. I think the solution chosen here works very well – the notes in the back use page numbers and a short quote from the text as the identifiers of the source. It is partly because I know Goldacre makes regular criticism of the lack of referencing in media reports of science that I was disappointed on a couple of occasions to turn eagerly to the back and not find a citation. These tended to be times when a broad statement had been made – for example on page 75 “A huge amount of research...” does not provide any corroborating references and on page 144 “there is a lot of history here… the field of essential fatty acid research has seen research fraud, secrecy, court cases, negative findings that have been hushed up, media misreporting on a massive scale…[the list continues]” but no notes are offered. If a new edition is produced, please can these be added.

Summary
As I have already said, these are minor (some would say picky) criticisms of an otherwise extremely valuable book. Overall, I believe Ben Goldacre has provided all of us with a toolbox for evaluating sciencey-sounding stories in the media and alerted future scientists to some of the pitfalls they should avoid in the design and reporting of their work. Bad Science would make an excellent resource for post-16 education and I hope to see it adopted as a course text on A level and undergraduate programmes.

Bad Science has a cover price of £8.99 At the time of writing it is available from Amazon for £3.60.

Back in 2004, Sir David King (at the time, the Government’s Chief Scientific Adviser) initiated a discussion about generating a Code of Conduct for Scientists. The consultation process led, in 2006, to the publication of Rigour, respect and responsibility: a universal ethical code for scientists. None of the contents was particularly surprising or radical but it brought together in one place a list of seven key principles that ought to be foundational for the ethical conduct and communication of science.

The Code of Conduct emphasises seven key points

The Code received a public launch at the BA Festival of Science in September 2007 and was reported in the general press at the time (see, for example, UK science head backs ethics code). During the intervening two years, conversations with scientist colleagues (across a range of institutions) have revealed almost universal ignorance about the existence of the Code, let alone its content.

The new guidelines from Research Councils UK will see greater emphasis on research ethics

All this is about to change. In July 2009 Research Councils UK (RCUK), the umbrella organisation for the seven publicly-funded Research Councils, produced Integrity, Clarity and Good Management, a Policy and Code of Conducton the Governance of Good Research Conduct. An accompanying letter from Prof Ian Diamond, Chair of the RCUK Exec Committee, makes it clear that from 1st October 2009 adherence to the guidelines and code will be a requirement for all those seeking funding via the Research Councils.

The RCUK document is quite broad in its definition of unacceptable conduct and includes poor and inadequate record keeping alongside deliberate dishonesty. The misconduct highlighted includes:

• Fabrication: producing false data or documents
• Falsification: manipulating data, including images
• Plagiarism: taking someone else’s work or ideas without credit
• Misrepresentation: knowingly, recklessly or negligently presenting a wrong interpretation of data; inappropriate exclusion (or inclusion) of authors in publications arising from the work; duplicate publication
• Mismanagement: poor record-keeping or inadequate retention of primary data
• Breach of duty of care: revealing identity or research participants; endangering people involved in research; failure to fulfil legal requirements regarding use of animals and/or tissue samples; improper involvement in the peer review process including failure to disclose conflicts of interest

In addition to the aspects of conduct themselves, the RCUK paper also includes guidelines on the reporting and investigation of unacceptable research conduct.  Alongside the 2008 Procedure for the Investigation of Misconduct in Research (UK Research Integrity Office) and Investigating Research Misconduct Allegations in International Collaborative Projects (OECD Global Science Forum), requirements for informal and formal investigation and, where necessary disciplining by the Research Organisation hosting the affected study in detailed.

I believe that the vast majority of research in the UK is conducted to high ethical standards. For most people, the recommendations in the code will not be new, but having them in one place, and the clear indication from RCUK that funding will be linked to demonstration of high standards will hopefully serve to curb abuses (deliberate or accidental) where they do exist.

The annual Learning and Teaching in the Sciences event at the University of Leicester was held on May 23^rd 2007.  Three invited speakers brought very different insights into the effective communication of science. This entry focuses specifically on the first of the presentations.  Other talks, by Melanie Cooper (Clemson University, USA) and Alan Cann (University of Leicester) will follow in subsequent posts.

Norman Reid (Professor of Science Education, University of Glasgow) addressed the subject of the ways we can maximise the impact of our teaching by taking into account scientific studies into the factors that influence learning.  I had heard Norman speak previously on the subject of pedagogic research methodology (he has written a very useful booklet on the subject on behalf of the Physical Sciences Centre, Higher Education Academy).  I had high expectations, and I wasn’t disappointed. 

Early on in his talk, Norman emphasised the importance of Working Memory Capacity (WMC), in other words how many ideas are we capable of holding in our short-term memory at any one time.  In an exercise reminiscent of the 1980s gameshow The Krypton Factor, we were asked to convert a date into single digits, and put them in numerical order (without writing them down).  So, for example, 7th April 96 would be 4-6-7-9.  As the number of digits involved increased, the capability to solve the puzzle diminished.  If, therefore, we are presenting students with more distinct pieces of information than they can cope with (in other words, if the information load of our teaching exceeds their working memory capacity, then this is going to have a detrimental impact on their learning. Rather than a linear decline in success as information load increases, there is a sudden collapse in performance.  For most people, the WMC seems to be about 7 items.  This number varies from person to person and, it seems, we can do little to change it. Norman mentioned grouping strategies and pattern recognition as ways in which we can carry more bits of information than our WMC, but this is making the best of what we’ve got, not stretching the underlying capacity.  He didn’t specifically discuss mnemonics, but I guess these are an example of a grouping strategy.

The place of WMC in an information processing model was then fleshed out.  In addition to Working Memory and Long-Term Memory, an important role is also played by a Perception Filter.  I took the latter to be a subconscious self-recognition of the number of bits of information you can cope with.  To draw an analogy (my own, apologies to Prof Reid if I’ve got this wrong!) – if you were the captain of a ship, you would know how much cargo you can carry on board.  You would decline extra items, even if they were on offer.  In similar vein, a perception filter allows you to ‘know your limits’ – there may be extra information on offer, but when you know you are in danger of overload you engage mechanisms that stop taking too much on board, lest the ’ship’ sinks.  I guess, by extension of my image, there is benefit in being able to distinguish valuable cargo from junk, which is probably one reason why our previous experience and our long-term memory influence the effective working of our perception filter.  Norman used the term field dependency for the ability to see what is important, to distinguish the ‘message’ from the ‘noise’.

Pushing my analogy to its conclusion, I suppose our role as educators would equate to the port authorities or harbour master.  We need to be aware of the number of fresh bits of cargo we are offering to our students, and ration their delivery so that we reduce the risk than anyone tries to set sail with too much on board (suspicion I pushed that too far – Ed). 

In the next phase of his talk, Prof Reid moved on to consider the idea of pre-learning. At its most simple, this might be starting a lesson or a lecture with a couple of minutes of reflection (“ok, who can remember what we discussed last time?”).  This is all about making connections between different nuggets of information.  Having a list of review questions up on the screen at the start of the lecture and asking students to work through them in pairs was a recommended model.  This might be extended to a formal short activity or exercise taking place before a major lecture or laboratory practical to draw attention to what are going to be the main points, thus equipping the students more effectively to distinguish message v noise.

Once again, these ideas rang true for me.  I know I’m not alone in seeing that one of the downsides of modularisation has been the compartmentation of knowledge.  Students do not necessarily see the connections between the different teaching within a module and less so between units.  It is one of the roles of the educator to make explicit the links to previous and future teaching, since they (hopefully!) have a better grasp of how the bits fit together.

Prof Reid emphasised that reducing the working memory load was emphatically not a call for ‘dumbing-down’.  The challenge is not to throw out the hard topics, but rather give conscious consideration to the order in which material is covered, to connections between material more overt and to break down complex items into more comprehensible sizes.

As the session moved towards questions, much of the discussion focussed on the research methodologies employed to produce the scientific data undergirding these views.  In particular, delegates and speaker alike expressed a frustration that the demands for ‘fairness’ meant that it was becoming very difficult to conduct proper comparisons between groups experiencing different teaching.  True, crossover studies (where group A is taught using method X and group B is taught using method Y, and then the two groups are swapped over for a second phase of teaching using the other method) can partially fulfil this need, but there are plenty of occasions when this is not truly feasible.  In consequence, many of the most informative studies have been performed outside of the UK.  Food for thought.