Avoiding Scientific Misconduct in Prague

I recently spent an excellent few days in Prague, attending the 43rd FEBS Congress, at which I gave a talk about the importance of bioethics teaching, and ran a workshop on developing case studies in ethics teaching. A session on the final morning Scientific (mis)conduct: how to detect (and avoid) bad science illustrated one reason why this is a crucial dimension in the education of scientists.

prague1

I live-tweeted the presentations and organised them at the time within five threads. The post below represents a first attempt to use Thread Reader (@threadreaderapp) which operates a very straightforward “unroll” tool. Following the sad demise of Storify, I was curious to see if this would be a suitable alternative for curation of tweeted content. I have elected to offer both links to the unrolled threads and screenshots of the resulting notes. I’m relatively pleased with the outcome.

Getting back to the content of the session, it proved a really insightful overview of several aspects of research misconduct, and publication ethics.

The first talk Research Integrity: threats and safeguards was given by László Fésüs (University of Debrecen, Hungary). Fésüs cited a 2013 report How Science Goes Wrong from The Economist (a PDF file is available here). He also noted a white paper How Can Scientists Enhance Rigor in Conducting Basic Research and Reporting Research Results?produced by the American Society for Cell Biology, which reported that 72% of survey respondents had been unable to replicate a published result.

Similarly, in a study reported in the journal Nature, company Amgen was only able to reproduce 6 out of 53 results in “landmark” cancer biology papers. Likewise, Bayer only managed to repeat a quarter of 67 published results.

Taking a slightly different tack, Vasilevky et alfound that more than half of 238 papers published across 84 journals included all the necessary details of materials and methods to facilitate attempts to reproduce the work. A survey in SABIO-RK also found missing and/or imprecise information.

Touching on the difficulties getting “negative results” published, Fésüs noted that the proportion of papers including refutations of other work, and/or reporting that something had NOT worked had actually fallen over the past quarter century. He suggested a number of interventions that should be adopted to help change this situation:

  • Journals must be more pro-active in enforcing standards
  • Young scientists must be trained to have innate scepticism about their own results and to have the statistical wherewithal to evaluate data
  • Researchers should be evaluated on the quality not the quantity of their work
  • Funding bodies should be more enthusiastic about replication studies
  • Negative results and failures to reproduce other studies ought to be given a greater hearing, and open access publication may facilitate this.

Fésüs moved on to discuss the Transparency and Openness Promotion initiative from the Center for Open Science. To quote from their website: “Published in Science in 2015 (OA), the Transparency and Openness Promotion guidelines include eight modular standards, each with three levels of increasing stringency. Journals select which of the eight transparency standards they wish to adopt for their journal, and select a level of implementation for each standard. These features provide flexibility for adoption depending on disciplinary variation, but simultaneously establish community standards.” He finished by quoting Bruce Alberts from the article Self-correction is science at work  “Instances in which scientists detect and address flaws in work constitute evidence of success, not failure”.

fesus1a

The second presenter was Alex Wlodawer from the NIH Center for Cancer Research, who spoke on the theme Crystallography and Cryo-EM: not always crystal clear. The longer I’ve been in Biochemistry, the more fascinated by the capabilities of structural biology I’ve become. Having said that, however, I’m not confident on interpretation of structural data, so I was rather taking Prof Wlodawer’s word for it as he steered us around some examples of sloppy work or intentional misrepresentation. As noted in the thread below, the latter included H.M. Krishna Murthy who was eventually brought to book for systematic fraud over more than a decade.

wlodawer1a

A sad refrain in Wlodawer’s talk was the difficulty getting journals and database owners to respond with appropriate seriousness to exposed examples of poor or false work in their repositories. The responses frequently seemed to be slow, non-existent or accompanied by weasel words exonerating the rest of the content of work containing dodgy structures. As he noted “if you remove the structure, and the rest of the paper is fine, then it begs the question why the structure was included in the paper in the first place”.

Finally, Wlodawer encourage delegates to be alert to unrealistic claims about resolution quality being made in the emerging field of cryo-EM. There was a habit, he suggested, for people to deposit refined data but talk of it as though it was the original record.

The third presentation was by Bernhard Rupp, author – we were told by the chair-of the leading textbook on Biomolecular Crystallography. Rupp’s talk was entitled The action is in the re(tr)action. I warmed to Rupp for two reasons. Firstly he was happy to wander around the stage, freed from the shackles of the lectern. Secondly, he was “out and proud2 about the ongoing value of the Comic Sans script long after I have been cowed into submission by those who consider it a frivolous font.

rupp1a

Whilst acknowledging that there are genuine problems with research fraud and data reproducibility, he did direct the audience to the recent PNAS opinion piece by Daniele Fanelli Is science really facing a reproducibility crisis, and do we need it to? Referring to Bayesian statistics, and acknowledging that scientific inference is inherently probabilistic, Rupp advocated wider use of the phrase “almost always” when reporting data. He went on to suggest that we should expect corrections and retractions as part of the normal process of science. Some fields of science – he picked out cell biology and psychology ARE inherently prone to variation. In contrast, structural biology ought to be a discipline in which the data were less varied.

Again mentioning Bayes, Rupp argued that the likelihood of a given model ought to be proportional to the quality of the evidence and the prior probability of that outcome. What this mean in practice was that a strong claim with little prior basis ought to require a large amount of corroborating evidence.

Rupp also directed the audience to a recent FEBS Journal commentary The Two Cultures: Where are we now?  Referring back to C.P. Snow’s famous observation of the “mutual incomprehension” between the Sciences and the Arts & Humanities, the article apparently discussed the impression that those on the Humanities side were likely, in an age of Critical Theory, to think that they DO understand Science and to disapprove of it.

He went on to suggest that fraud was actually quite easy to spot as the data was too well ordered; genuine data tends to encompass both random errors and systematic errors. Rupp observed that worries about the influence of the human will and emotions on the interpretation of data was not a new concern; Francis Bacon had raised concerns in this regard back in the 1620s. This was echoed by Richard Feynman in the 1970s, when he noted that you are the easiest person to fool by your results. The temptation to self-delusion embraced both expectation bias and confirmation bias.

Building on from what Wlodawer had stated about the training requirements for young scientists, Rupp suggested that alongside technical skills, we also needed to ensure that they had a thorough grasp of empirical reasoning. He reiterated that extraordinary claims ought to require extraordinary evidence. He used a term that I hadn’t previously encountered “Whataboutism” which I have subsequently also heard used in the context of Donald Trump.

Pointing out that you cannot prove the absence of something, only that there is a lack of evidence, Rupp noted that the burden of proof needs to be on the claimant not the critic. He finished his presentation with an intriguing “world map of crystallographers” which, he implied, demonstrated clusters of interconnected “rotten apples”. The take-home message was, however, that most of the others were doing much better!

The most personal presentation was the fourth talk, given by Stefan Franzen from NC State University. Under the title “How not to design misconduct policy: Lessons learned from the United States“, Franzen gave a graphic account of his own brush with fraudulent research. Having been invited to collaborate in a research project he very quickly had concerns about the validity of some of the key claims. By this stage the work involved intellectual property rights, patents and large budgets. Trying to challenge these false results was to draw him into years of hassle – legal actions against him, accusation that he had breached confidentiality agreements and an awareness that the financial landscape means that home institutions cannot be trusted to robustly defend the scientists trying to bring dishonesty into the light.

Universities have, he notes, a huge conflict of interest – supposedly charged with maintaining the integrity of research and calling out misconduct whilst being a financial beneficiary of that very work. Confidentiality regulations, intended to protect whistleblowers, are all too often abused to close down investigations and suppress robust scrutiny. The financial implications mean that all too often cases where misconduct is suspected or even proven are dealt with as financial cases rather than research misconduct – Universities are unlikely to drive for retractions of work done on their watch. A need for demonstration of intent means that demonstration of falsification is not always taken as seriously as it ought to be. Additionally you need good legal advice, cases that appear watertight can get thrown out on a technicality. This process is definitely not “scientific”.

Finally, Franzen reiterated just how much of your life challenging research falsehood can take. This whole episode has clearly been a huge burden for him; a lesser man would have folded at an earlier stage.

franzen1a

The last talk in the session was Research Integrity via Post-Publication Peer Review by the PubPeer Community, given by PubPeer President Brandon Stell.  Stell began by reiterating the point that flawed publications are more than an intellectual problem. Citing the Poldermans and Wakefield cases he pointed out that deaths can arise when bogus data influences public policy and/or public opinion. The “political” climate in which science is conducted, in which kudos comes from publishing an uncomplicated story in a “high impact” journal, can tempt researcher to downplay trickier aspects of their data so as to tell a more persuasive tale. Similarly, if it was later believed that work was incorrect there was little incentive for the authors, the journal and/or the institution where the work was conducted to publish a retractions since, by so doing, all were de facto admitting to previous error.

Stell then introduced us to PubPeer, a resource for bringing to light potential concerns about published works. Controversially, the critic can be anonymous. Stell argues that this is crucial (a similar service in which the questioner was identified floundered due to lack of engagement). He believes that the other rules they have in place serve as sufficient safeguards against malicious actions.

stell1a

Presenter Nick Ross used to sign off each episode of the BBC programme Crimewatch with the words “Don’t have nightmares, do sleep well“. In similar vein, someone (I think it was session chairman Mariusz Jaskollski) urged us to remember that the vast majority of scientists are doing great and ethical work. There is a need for all of us to be active in ensuring this remains the case, and that where misconduct is identified it gets flagged and rooted out.

After what was clearly a session that had struck a chord with delegates, there were pleas for a similar session to be run in a more prominent slot at the 44th FEBS Congress, coming up in Krakow (Poland) in July 2019. I would completely endorse that call, the live audience for this session was much smaller than it deserved.

Advertisements

Leave a comment

No comments yet.

Comments RSS TrackBack Identifier URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s