Some thoughts on eLife's New Model: One year on

 

I've just been sent an email from eLife, pointing me to links to a report called "eLife's New Model: One year on" and a report by the editors "Scientific Publishing: The first year of a new era". To remind readers who may have missed it, the big change introduced by eLife in 2023 was to drop the step where an editor decides on reject or accept of a manuscript after reviewer comments are received. Instead, the author submits a preprint, and the editors then decide whether it should be reviewed. If the answer is yes, then the paper will be published, with reviewer comments. 

Given the controversy surrounding this new publishing model, it seems timely to have a retrospective look at how it's gone, and these pieces by the journal are broadly encouraging in showing that the publishing world has not fallen apart as a consequence of the changes. We are told that the proportion of submissions published has gone down slightly from 31.4% to 27.7% and the demographic characteristics of authors and reviewers are largely unchanged. The ratings of quality of submissions are similar to those from the legacy model. The most striking change has been in processing time: median time from submission to publication of the first version with reviews is 91 days, which is much faster than previously. 

As someone who has been pushing for changes to the model of scientific publishing for years (see blogsposts below), I'm generally in favour of any attempt to disrupt the conventional model. I particularly like the fact that the peer reviews are available with the published articles in eLife - I hope that will become standard for other journals in future. However, there are two things that rather rankled about the latest communication from the journal. 

First, the report describes an 'author survey' which received 325 responses, but very little detail is given as to who was surveyed, what the response rate was, and what the overall outcome was. This reads more like a marketing report than a serious scientific apprasal. Two glowing endorsements were reported from authors who had good experiences. I wondered though about authors whose work had not been selected to go forward to peer review - were they just as enthusiastic? Quite a few tables of facts and figures about the impact of the new policy were presented with the report, but if eLife really does want to present itself as embracing open and transparent policies, I think they should bite the bullet and provide more information - including fuller details of their survey methods and results, and negative as well as positive appraisals. 

Second, I continue to think there is a fatal flaw in the new model, which is that it still relies on editors to decide which papers go forward to review, using a method that will do nothing to reduce the tendency to hype and the consequent publication bias that ensues. I blogged about this a year ago, and suggested a simple solution, which is for the editors to adopt 'results-blind' review when triaging papers. This is an idea that has been around at least since 1976 (Mahoney, 1976) which has had a resurgence in popularity in recent years, with growing awareness of the dangers of publication bias (Locasio, 2017). The idea is that editorial decisions should be made based on whether the authors had identified an interesting question and whether their methods were adequate to give a definitive answer to that question. The problem with the current system is that people get swayed by exciting results, and will typically overlook weak methods when there is a dramatic finding. If you don't know the results, then you are forced to focus on the methods. The eLife report states:

 "It is important to note that we don’t ascribe value to the decision to review. Our aim is to produce high-quality reviews that will be of significant value but we are not able to review everything that is submitted." 

That is hard to believe: if you really were just ignoring quality considerations, then you should decide on which papers to review by lottery. I think this claim is not only disingenuous but also wrong-headed. If you have a limited resource - reviewer capacity - then you should be focusing it on the highest quality work. But that judgement should be made on the basis of research question and design, and not on results. 

Bibliography 

Locascio, J. J. (2017). Results blind science publishing. Basic and Applied Social Psychology, 39(5), 239–246. https://doi.org/10.1080/01973533.2017.1336093 

Mahoney, M. J. (1976). Scientist as Subject: The Psychological Imperative. Ballinger Publishing Company. 

Previous blogposts

Academic publishing: why isn't psychology like physics? 

Time for academics to withdraw free labour.

High impact journals: where newsworthiness trumps methodology

Will traditional science journals disappear?

Publishing replication failures

A suggestion for eLife

According to a piece today in Nature, there’s uproar at eLife, where a new publishing model has been introduced by Editor-in-Chief, Michael Eisen. The idea is that authors first submit their paper as a preprint, and then a decision is made by editors as to whether it is sent out for review – at a cost of $2000 to the author.  Papers that are reviewed are published, with the reviews, and an editorial comment, regardless of any criticisms in the review.  Authors have an opportunity to update the article to take into account reviewer comments if they wish, but once reviewed, cannot be rejected.

 

Of course, this does not really remove a “quality control” filter by the journal – it just moves it to the stage where a decision is made on whether or not to send the paper out for review.

The guidance given to editors in making that judgement is “can you generate high-quality and broadly useful public reviews of this paper?”  Concerns have been expressed over whether this would disadvantage less well-known authors, if editors preferred to play it safe and only send papers for review if the authors had a strong track record.  But the main concern is that there will be a drop in quality of papers in eLife, which will lose its reputation as a high-prestige outlet.

 

I have a simple suggestion for how to counteract such a concern, and that is that the journal should adopt a different criterion for deciding which papers to review – this should be done solely on the basis of the introduction and methods, without any knowledge of the results. Editors could also be kept unaware of the identity of authors.

 

If eLife wants to achieve a distinctive reputation for quality, it could do so by only taking forward to review those articles that have identified an interesting question and tackled it with robust methodology. It’s well-known that editors and reviewers tend to be strongly swayed by novel and unexpected results, and will disregard methodological weaknesses if the findings look exciting. If authors had to submit a results-blind version of the manuscript in the first instance, then I predict that the initial triage by editors would look rather different.  The question for the editor would no longer be one about the kind of review the paper would generate, but would focus rather on whether this was a well-conducted study that made the editor curious to know what the results would look like.  The papers that subsequently appeared in eLife would look different to those in its high-profile competitors, such as Nature and Science, but in a good way.  Those ultra-exciting but ultimately implausible papers would get filtered out, leaving behind only those that could survive being triaged solely on rationale and methods.

 

 

 

So do we need editors?

It’s been an interesting week in world politics, and I’ve been distracting myself by pondering the role of academic editors. The week kicked off with a rejection of a preprint written with co-author Anna Abalkina, who is an expert sleuth who tracks down academic paper mills – organisations that will sell you a fake publication in an academic journal. Our paper describes a paper mill that had placed six papers in the Journal of Community Psychology, a journal which celebrated its 50th anniversary in 2021. We had expected rejection, as we submitted the paper to the Journal of Community Psychology, as a kind of stress test to see whether the editor, Michael B. Blank, actually reads papers that he accepts for the journal. I had started to wonder, because you can read his decision letters on Publons, and they are identical for every article he accepts. I suspected he may be an instance of Editoris Machina, or automaton, one who just delegates editorial work to an underling, waits until reviewer reports converge on a recommendation, and then accepts or rejects accordingly without actually reading the paper. I was wrong, though. He did read our paper, and rejected it with the comment that it was a superficial analysis of six papers. We immediately posted it as a preprint and plan to publish it elsewhere.

Although I was quite amused by all of this, it has a serious side. As we note in our preprint, when paper mills succeed in breaching the defences of a journal, this is not a victimless crime. First, it gives competitive advantage to the authors who paid the paper mill – they do this in order to have a respectable-looking publication that will help their career. I used to think this was a minor benefit, but when you consider that the paper mills can also ensure that the papers they place are heavily cited, you start to realise that authors can edge ahead on conventional indicators of academic prestige, while their more honest peers trail behind. The second set of victims are those who publish in the journal in good faith. Once its reputation is damaged by the evidence that there is no quality control, then all papers appearing in the journal are tainted by association. The third set of victims are busy academics who are trying to read and integrate the literature, who can get tangled up in the weeds as they try to navigate between useful and useless information. And finally, we need to be concerned about the cynicism induces in the general public when they realise that for some authors and editors, the whole business of academic publishing is a game, which is won not by doing good science, but by paying someone to pretend you have done so.

Earlier this week I shared my thoughts on the importance of ensuring that we have some kind of quality control over journal editors. They are, after all, the gatekeepers of science. When I wrote my taxonomy of journal editors back in 2010, I was already concerned at the times I had to deal with editors who were lazy or superficial in their responses to authors. I had not experienced ‘hands off’ editors in the early days of my research career, and I wondered how far this was a systematic change over time, or whether it was related to subject area. In the 1970s and 1980s, I mostly published in journals that dealt with psychology and/or language, and the editors were almost always heavily engaged with the paper, adding their own comments and suggesting how reviewer comments might be addressed. That’s how I understood the job when I myself was an editor. But when I moved to publishing work in journals that were more biological (genetics, neuroscience) things seemed different, and it was not uncommon to find editors who really did nothing more than collate peer reviews.

The next change I experienced was when, as a Wellcome-funded researcher, I started to publish in Wellcome Open Research (WOR), which adopts a very different publication model, based on that initiated by F1000. In this model, there is no academic editor. Instead, the journal employs staff who check that the paper complies with rigorous criteria: the proposed peer reviewers much have a track record of publishing and be clear from conflict of interest. Data and other materials must be openly available so that the work can be reproduced. And the peer review is published. The work is listed on PubMed if and when peer reviewers agree that it meets a quality threshold: otherwise the work remains visible but with status shown as not approved by peer review. 

The F1000/WOR model shows that editors are not needed, but I generally prefer to publish in journals that do have academic editors – provided the editor is engaged and does their job properly. My papers have benefitted from input from a wise and experienced editor on many occasions. In a specialist journal, such an editor will also know who are the best reviewers – those who have the expertise to give a detailed and fair appraisal of the work. However, in the absence of an engaged editor, I prefer the F1000/WOR model, where at least everything is transparent. The worst of all possible worlds is when you have an editor who doesn’t do more than collate peer reviews, but where everything is hidden: the outside world cannot know who the editor was, how decisions were made, who did the reviews, and what they said. Sadly, this latter situation seems to be pretty common, especially in the more biological realms of science. To test my intuitions, I ran a little Twitter poll for different disciplines, asking, for instance: 

 

 

 Results are below

% respondents stating Not Read, Read Superfially, or Read in Depth

 

Such polls of course have to be taken with a pinch of salt, as the respondents are self-selected, and the poll allows only very brief questions with no nuance. It is clear that within any one discipline, there is wide variability in editorial engagement. Nevertheless, I find it a matter of concern that in all areas, some respondents had experienced a journal editor who did not appear to have read the paper they had accepted, and in areas of biomedicine, neuroscience, and genetics, and also in mega journals, this was as high as around 25-33%

So my conclusion is that it is not surprising that we are seeing phenomena like paper mills, because the gatekeepers of the publication process are not doing their job. The solution would be either to change the culture for editors, or, where that is not feasible, to accept that we can do without editors. But if we go down that route, we should move to a model such as F1000 with much greater quality control over reviewers and COI, and much more openness and transparency.

 As usual comments are welcome: if you have trouble getting past comment moderation, please email me.

We need to talk about editors

Editoris spivia

The role of journal editor is powerful: you decide what is accepted or rejected for publication. Given that publications count as an academic currency – indeed in some institutions they are literally fungible – a key requirement for editors is that they are people of the utmost integrity. Unfortunately, there are few mechanisms in place to ensure editors are honest – and indeed there is mounting evidence that many are not. I argue here that we can no longer take editorial honesty for granted, and systems need to change to weed out dodgy editors if academic publishing is to survive as a useful way of advancing science. In particular, the phenomenon of paper mills has shone a spotlight on editorial malpractice.

Questionable editorial practices

Back in 2010, I described a taxonomy of journal editors based on my own experience as an author over the years. Some were negligent, others were lordly, and others were paragons – the kind of editor we all want, who is motivated solely by a desire for academic excellence, who uses fair criteria to select which papers are published, who aims to help an author improve their work, and provides feedback in a timely and considerate fashion. My categorisation omitted another variety of editor that I have sadly become acquainted with in the intervening years: the spiv. The spiv has limited interest in academic excellence: he or she sees the role of editor as an opportunity for self-advancement. This usually involves promoting the careers of self or friends by facilitating publication of their papers, often with minimal reviewing, and in some cases may go as far as working hand in glove with paper mills to receive financial rewards for placing fraudulent papers.

When I first discovered a publication ring that involved journal editors scratching one another’s backs, in the form of rapid publication of each other’s papers, I assumed this was a rare phenomenon. After I blogged about this, one of the central editors was replaced, but others remained in post. 

I subsequently found journals where the editor-in-chief authored an unusually high percentage of the articles published in the journal. I drew these to the attention of integrity advisors of the publishers that were involved, but did not get the impression that they regarded this as particularly problematic or were going to take any action about it. Interestingly, there was one editor, George Marcoulides, who featured twice in a list of editors who authored at least 15 articles in their own journal over a five year period. Further evidence that he equates his editorial role with omnipotence came when his name cropped up in connection with a scandal where a reviewer, Fiona Fidler, complained after she found her positive report on a paper had been modified by the editor to justify rejecting the paper: see this Twitter thread for details. It appears that the publishers regard this as acceptable: Marcoulides is still editor-in-chief at the Sage journal Educational and Psychological Measurement, and at Taylor and Francis’ Structural Equation Modeling, though his rate of publishing in both journals has declined since 2019; maybe someone had a word with him to explain that publishing most of your papers in a journal you edit is not a good look.

Scanff et al (2021) did a much bigger investigation of what they termed “self-promotion journals” - those that seemed to be treated as the personal fiefdom of editors, who would use the journal as an outlet for their own work. This followed on from a study by Locher et al (2021), which found editors who were ready to accept papers by a favoured group of colleagues with relatively little scrutiny. This had serious consequences when low-quality studies relating to the Covid-19 pandemic appeared in the literature and subsequently influenced clinical decisions. Editorial laxness appears in this case to have done real harm to public health.

So, it's doubtful that all editors are paragons. And this is hardly surprising: doing a good job as editor is hard and often thankless work. On the positive side, an editor may obtain kudos for being granted an influential academic role, but often there is little or no financial reimbursement for the many hours that must be dedicated to reading and evaluating papers, assigning reviewers, and dealing with fallout from authors who react poorly to having their papers rejected. Even if an editor starts off well, they may over time start to think “What’s in this for me?” and decide to exploit the opportunities for self-advancement offered by the position. The problem is that there seems little pressure to keep them on the straight and narrow; it's like when a police chief is corrupt. Nobody is there to hold them to account. 

Paper mills

Many people are shocked when they read about the phenomenon of academic paper mills – defined in a recent report by the Committee on Publication Ethics (COPE) and the Association of Scientific, Tehcnical and Medical Publishers (STM) as “the process by which manufactured manuscripts are submitted to a journal for a fee on behalf of researchers with the purpose of providing an easy publication for them, or to offer authorship for sale.” The report stated that “the submission of suspected fake research papers, also often associated with fake authorship, is growing and threatens to overwhelm the editorial processes of a significant number of journals.” It concluded with a raft of recommendations to tackle the problem from different fronts: changing the incentives adopted by institutions, investment in tools to detect paper mill publications, education of editors and reviewers to make them aware of paper mills, introduction of protocols to impede paper mills succeeding, and speeding up the process of retraction by publishers.

However, no attention was given to the possibility that journal editors may contribute to the problem: there is talk of “educating” them to be more aware of paper mills, but this is not going to be effective if the editor is complicit with the paper mill, or so disengaged from editing as to not care about them. 

It’s important to realise that not all paper mill papers are the same. Many generate outputs that look plausible. As Byrne and Labbé (2017) noted, in biomedical genetic studies, fake papers are generated from a template that is based on a legitimate paper, and just vary in terms of the specific genetic sequence and/or phenotype that is studied. There are so many genetic sequences and phenotypes, that the number of possible combinations of these is immense. In such cases, a diligent editor may get tricked into accepting a fake paper, because the signs of fakery are not obvious and aren’t detected by reviewers. But at the other extreme, some products of paper mills are clearly fabricated. The most striking examples are those that contain what Guillaume Cabanac and colleagues term “tortured phrases”. These appear to be generated by taking segments of genuine articles and running them through an AI app that will use a thesaurus to alter words, with the goal of evading plagiarism detection software. In other cases, the starting point appears to be text from an essay mill. The results are often bizarre and so incomprehensible that one only needs read a few sentences to know that something is very wrong. Here’s an example from Elsevier’s International Journal of Biological Macromolecules, which those without access can pay $31.50 for (see analysis on Pubpeer, here).

"Wound recuperating camwood a chance to be postponed due to the antibacterial reliance of microorganisms concerning illustration an outcome about the infection, wounds are unable to mend appropriately, furthermore, take off disfiguring scares [150]. Chitin and its derivatives go about as simulated skin matrixes that are skilled should push a fast dermal redesign after constantly utilized for blaze treatments, chitosan may be wanton toward endogenous enzymes this may be a fundamental preference as evacuating those wound dressing camwood foundation trauma of the wounds and harm [151]. Chitin and its derivatives would make a perfect gas dressing. Likewise, they dampen the wound interface, are penetrability will oxygen, furthermore, permit vaporous exchange, go about as a boundary with microorganisms, and are fit about eliminating abundance secretions"

And here’s the start of an Abstract from a Springer Nature collection called Modern Approaches in Machine Learning and Cognitive Science (see here for some of the tortured phrases that led to detection of this article). The article can be yours for £19.95:

“Asthma disease are the scatters, gives that influence the lungs, the organs that let us to inhale and it’s the principal visit disease overall particularly in India. During this work, the matter of lung maladies simply like the trouble experienced while arranging the sickness in radiography are frequently illuminated. There are various procedures found in writing for recognition of asthma infection identification. A few agents have contributed their realities for Asthma illness expectation. The need for distinguishing asthma illness at a beginning period is very fundamental and is an exuberant research territory inside the field of clinical picture preparing. For this, we’ve survey numerous relapse models, k-implies bunching, various leveled calculation, characterizations and profound learning methods to search out best classifier for lung illness identification. These papers generally settlement about winning carcinoma discovery methods that are reachable inside the writing.”

These examples are so peculiar that even a layperson could detect the problem. In more technical fields, the fake paper may look superficially normal, but is easy to spot by anyone who knows the area, and who recognises that the term “signal to noise” does not mean “flag to commotion”, or that while there is such a thing as a “Swiss albino mouse” there is no such thing as a “Swiss pale-skinned person mouse”. These errors are not explicable as failures of translation by someone who does not speak good English. They would be detected by any reviewer with expertise in the field. Another characteristic of paper mill outputs, featured in this recent blogpost, are fake papers that combine tables and figures from different publications in nonsensical contexts.

Sleuths who are interested in unmasking paper mills have developed automated methods for identifying such papers, and the number is both depressing and astounding. As we have seen, though some of these outputs appear in obscure sources, many crop up in journals or edited collections that are handled by the big scientific publishing houses, such as Springer Nature, Elsevier and Wiley. When sleuths find these cases, they report the problems on the website PubPeer, and this typically raises an incredulous response as to how on earth did this material get published. It’s a very good question, and the answer has to be that somehow an editor let this material through. As explained in the COPE&STM report, sometimes a nefarious individual from a paper mill persuades a journal to publish a “special issue” and the unwitting journal is then hijacked and turned into a vehicle for publishing fraudulent work. If the special issue editor poses as a reputable scientist, using a fake email address that looks similar to the real thing, this can be hard to spot.

But in other cases, we see clearcut instances of paper mill outputs that have apparently been approved by a regular journal editor. In a recent preprint, Anna Abalkina and I describe finding putative paper mill outputs in a well-established Wiley journal, the Journal of Community Psychology. Anna identified six papers in the journal in the course of a much larger investigation of papers that came from faked email addresses. For five of them the peer review and editorial correspondence was available on Publons. The papers,  from addresses in Russia or Kazakhstan, were of very low quality and frequently opaque. I had to read and re-read to work out what the paper was about, and still ended up uncertain. The reviewers, however, suggested only minor corrections. They used remarkably similar language to one another, giving the impression that the peer review process had been compromised. Yet the Editor-in-Chief, Michael B. Blank, accepted the papers after minor revisions, with a letter concluding: “Thank you for your fine contribution”. 

There are two hypotheses to consider when a journal publishes incomprehensible or trivial material: either the editor was not doing their job of scrutinising material in the journal, or they were in cahoots with a paper mill. I wondered whether the editor was what I have previously termed an automaton – one who just delegates all the work to a secretary. After all, authors are asked to recommend reviewers, so all that is needed is for someone to send out automated requests to review, and then keep going until there are sufficient recommendations to either accept or reject. If that were the case, then maybe the journal would accept a paper by us. Accordingly, we submitted our manuscript about paper mills to the Journal of Community Psychology. But it was desk rejected by the Editor in Chief with a terse comment: “This a weak paper based on a cursory review of six publications”. So we can reject hypothesis 1 – that the editor is an automaton. But that leaves hypothesis 2 – that the editor does read papers submitted to his journal, and had accepted the previous paper mill outputs in full knowledge of their content. This raises more questions than it answers. In particular, why would he risk his personal reputation and that of his journal by behaving that way? But perhaps rather than dwelling on that question, we should think positively about how journals might protect themselves in future from attacks by paper mills.

A call for action

My proposal is that, in addition to the useful suggestions from the COPE&STM report, we need additional steps to ensure that those with editorial responsibility are legitimate and are doing their job. Here are some preliminary suggestions:

1. Appointment to the post of editor should be made in open competition among academics who meet specified criteria.
2. It should be transparent who is responsible for final sign-off for each article that is published in the journal.
3. Journals where a single editor makes the bulk of editorial decisions should be discouraged. (N.B. I looked at the 20 most recent papers in Journal of Community Psychology that featured on Publons and all had been processed by Michael B. Blank).
   
4. There should be an editorial board consisting of reputable people from a wide range of institutional backgrounds, who share the editorial load, and meet regularly to consider how the journal is progressing and to discuss journal business.
5.  Editors should be warned about the dangers of special issues and should not delegate responsibility for signing off on any papers appearing in a special issue.
6. Editors should be required to follow COPE guidelines about publishing in their own journal, and publishers should scrutinise the journal annually to check whether the recommended procedures were followed.
7. Any editor who allows gibberish to be published in their journal should be relieved of their editorial position immediately.
   

Many journals run by academic societies already adopt procedures similar to these. Particular problems arise when publishers start up new journals to fill a perceived gap in the market, and there is no oversight by academics with expertise in the area. The COPE&STM report has illustrated how dangerous that can be – both for scientific progress and for the reputation of publishers.

Of course, when one looks at this list of requirements, one may start to wonder why anyone would want to be an editor. Typically there is little financial renumeration, and the work is often done in a person’s “spare time”. So maybe we need to rethink how that works, so that paragons with a genuine ability and interest in editing are rewarded more adequately for the important work they do.

P.S. Comment moderation is enabled for this blog to prevent it being overwhelmed by spam, but I welcome comments, and will check for these in the weeks following the post, and admit those that are on topic. 

 

Comment by Jennifer Byrne, 9th Sept 2022 

(this comment by email, as Blogger seems to eat comments by Jennifer for some reason, while letting through weird spammy things!).

This is a fantastic list of suggestions to improve the important contributions of journal editors. I would add that journal editors should be appointed for defined time periods, and their contributions regularly reviewed. If for any reason it becomes apparent that an editor is not in a position to actively contribute to the journal, they should be asked to step aside. In my experience, editorial boards can include numerous inactive editors. These can provide the appearance of a large, active and diverse editorial board, when in practice, the editorial work may be conducted by a much smaller group, or even one person. Journals cannot be run successfully without a strong editorial team, but such teams require time and resources to establish and maintain.

PEPIOPs – prolific editors who publish in their own publications

I recently reported on a method for identifying authors who contribute an unusually high proportion of papers to a specific journal. As I noted in my previous blogpost, one cannot assume any wrongdoing just on the grounds of a high publication rate, but when there is a close link between the author in question and the journal's editor, then this raises concerns about preferential treatment and integrity of the peer review process.

In running my analyses, I also found some cases where there wasn't just a close link between the most prolific author in a journal and the editor: they were one and the same person! At around the time I was unearthing these results, Elisabeth Bik tweeted to ask if anyone had examples of editors publishing in their own journals. I realised the analysis scripts I had developed for the 'percent by most prolific' analysis could be readily adapted to look at this question, and so I analysed journals in the broad domain of psychology and behavioural science from six publishers: Springer Nature, Wiley, Taylor and Francis, Sage, Elsevier and American Psychological Association (APA). I focused on those responsible for editorial decisions – typically termed Editor-in-Chief or Associate Editor. Sometimes this was hard to judge: in general, I included 'Deputy editors' with 'Editors-in-Chief' if they were very few in number. I ignored members of editorial boards.

Before reporting the findings, I thought I should consult more widely about whether people think it is appropriate for an editor to publish in their own journal.

As a starting point, I looked at guidelines that the Committee on Publication Ethics (COPE) has provided for editors  

Can editors publish in their own journal? 
While you should not be denied the ability to publish in your own journal, you must take extra precautions not to exploit your position or to create an impression of impropriety. Your journal must have a procedure for handling submissions from editors or members of the editorial board that will ensure that the peer review is handled independently of the author/editor. We also recommend that you describe the process in a commentary or similar note once the paper is published.

They link to a case report that notes how this issue can be particularly problematic if the journal is in a narrow field with few publication outlets, and the author is likely to be identifiable even if blinding is adopted.

I thought that it would be interesting to see what the broader academic community thought about this. In a Twitter poll I asked specifically about Editors-in-Chief:

The modal response was that it was OK for an Editor-in-Chief to publish in their own journal at a modest rate (once a year or less). There was general disapproval of editors publishing prolifically in their own journals – even though I emphasised that I was referring to situations where the editorial process was kept independent of the author, as recommended by COPE.

My poll question was not perfectly phrased - I did not specify the type of article: it is usual, for instance, for an editor to write editorials! But I later clarified that I meant regular articles and reviews, and I think most people interpreted it in that way.

The poll provoked some useful debate among those who approved and disapproved of editors publishing in their own journals.

Let's start with some of the reasons against this practice. I should lay my cards on the table and state that personally I am in agreement with Antonia Hamilton, who tweeted to say that as Editor in Chief of the Quarterly Journal of Experimental Psychology she would not be submitting papers from her lab to that journal during her term of office, citing concerns about Conflict of Interest. When I was Editor-in-Chief of Journal of Child Psychology and Psychiatry, I felt the same: I was concerned that my editorial colleagues would be put in a difficult position if they had to handle one of my papers, and if my papers were accepted, then it might be seen as involving a biased decision, even if it that was not the case. Chris Chambers, who was among the 34% who thought an Editor-in-Chief should never publish in their own journal, expressed concerns about breaches of confidentiality, given that the Editor-in-Chief would be able to access identities of reviewers. Others, though, questioned whether that was the case at all journals.

Turning to those who thought it was acceptable for an Editor-in-Chief to publish in their own journal, several people argued that it would be unfair, not just on the editor themselves, but also on members of their research group, if presence of an editorial co-author meant they could not submit to the journal. How much of an issue this is will depend, of course, on how big your research group is, and what other publication outlets are available.  Several people felt there was a big difference between cases where the editor was lead author, and those where they played a more minor role. The extreme case is when an editor is a member of a large research consortium led by someone else. It would seem unduly restrictive to debar a paper by a consortium of 100 researchers just because one of them was the Editor in Chief of the journal. It is worth remembering too that, while being a journal editor is a prestigious role, it is hard work, and publishers may be concerned that it becomes a seriously unattractive option if a major outlet for papers is suddenly off-limits.

Be this as it may, the impression from the Twitter poll was that three papers or more per annum starts to look excessive. In my analysis, shown here I found several journals where the Editor-in-Chief had coauthored 15 or more articles (excluding editorials/commentaries) in their own journal between 2015 and 2019.  I refer to these as PEPIOPs (prolific editors who publish in their own publications). For transparency, I've made my methods and results available, so that others can extend the analysis if they wish to do so: see https://github.com/oscci/Bibliometric_analyses. More legible versions of the tables are also available.

Table 1 shows the number of journals with a PEPIOP, according to publisher. 

Table 1: Number of journals with prolific authors. Columns AE and EIC indicate cases where Associate Editor or Editor-in-Chief published 15+ papers in the journal between 2015-2019

The table makes it clear that it is relatively rare for an Editor-in-Chief to be a PEPIOP: there were no cases for APA journals, and the highest number was 5.6% for Elsevier journals. Note that there are big differences between publishers in terms of how common it is for any author to publish 15+ papers in the journal over 5 years: this was true for around 25% of the Elsevier and Springer journals, and much less common for Sage, APA and Taylor & Francis. This probably  reflects the subject matter of the journals - prolific publication, often on multiauthored papers, is more common in biosciences than social sciences.

Individual PEPIOPs are shown in Table 2. An asterisk denotes and Editor-in-Chief who authored more articles in the journal than any other person between 2015-2019.

These numbers should be treated with caution – I did not check whether any of these editors had only recently adopted the editorial role - largely because this information is not easy to find from journal websites*. I looked at a small sample of papers from individual PEPIOPs to see if there was anything I had overlooked, but I haven't checked every article - as there a great many of them.  There was one case where this revealed misclassification: the editor of Journal of Paediatrics and Child Health wrote regular short commentaries of around 800 words summarising other papers: this seems an entirely unremarkable thing for an editor to do, but they were classified by Web of Science as 'articles' which led to him being categorised as a PEPIOP.  This illustrates how bibliometrics can be misleading.

Table 2: Editors-in-chief who published 15+ papers in own journal 2015-2019

In general I did not find any ready explanation for highly prolific outputs.  And I where I did spot checks I found only one case where there was an explanatory note of the kind COPE recommended (in the journal Maturitas) - on the contrary, in many cases there was a statement confirming no conflict of interest for any of the authors. Sometimes there were lists of conflicts relating to commercial aspects, but it was clear that authors did not regard their editorial role as posing a conflict. It was also worth mentioning there were cases where an Editor-in-Chief was senior author.

The more I have pondered this, the more I realise that the reason why I am concerned particularly by Editor-in-Chief PEPIOPs is because this is the person who is ultimately responsible for integrity of the publication process. Although publishers increasingly are alert to issues of research integrity, journal websites typically advise readers to contact the Editor-in-Chief if there is a problem. Complaints about editorial decisions, demands for retractions, or concerns about potential fraud or malpractice all come to the Editor-in-Chief. That's fine provided the Editor-in-Chief is a pillar of respectability. The problem is that not all editors are the paragons that we'd like them to be: one can adopt rather a jaundiced view after encountering editors who don't even bother to reply to expressions of concern, or adopt a secretive or dismissive attitude if plausible problems in their journal are flagged up. And there are also cases on record where an editor has abused the power of their position to enhance their own publication record. For this reason, I would strongly advise any Editor-in-Chief not to be a PEPIOP; it just looks bad, even if a robust, independent editorial process has been followed. We need to have confidence and trust in those who act as gatekeepers to journals.

My principal suggestion is that publishers could improve their record for integrity if they went beyond just endorsing COPE guidelines and started to actively check whether those guidelines are adhered to. 

*Note: 21st Aug 2020: A commenter on this blogpost has noted that Helai Huang fits this category - this editor has only been in post since 1 Jan 2020, so the period of prolific publication predated being an editor-in-chief.

Corrigendum: a word you may hope never to encounter

I have this week submitted a 'corrigendum' to a journal for an article published in the American Journal of Medical Genetics B (Bishop et al, 2006). It's just a fancy word for 'correction', and journals use it contrastively with 'erratum'. Basically, if the journal messes up and prints something wrong, it's an erratum. If the author is responsible for the mistake, it's a corrigendum.

 I'm trying to remember how many corrigenda I've written over the 40 odd years I've been publishing: there have been at least three previous cases that I can remember, but there could be more. I think this one was the worst; previous errors have tended to just affect numbers in a minor way. In this case, a whole table of numbers (table II) was thrown out, and although the main findings were upheld, there were some changes in the details.

I discovered the error when someone asked for the data for a meta-analysis. I was initially worried I would not be able to find the files, but fortunately, I had archived the dataset on a server, and eventually tracked it down. But it was not well-documented, and I then had the task of trawling through a number of cryptically-named files to try and work out which one was the basis for the data in the paper. My brain slowly reconstructed what the variable names meant and I got to the point of thinking I'd better check that this was the correct dataset by rerunning the analysis. Alas, although I could recreate most of what was published, I had the chilling realisation that there was a problem with Table II.

Table II was the one place in the analysis where, in trying to avoid one problem with the data (non-independence), I created a whole new problem (wrong numbers). I had data on siblings of children with autism, and in some cases there were two or three siblings in the family. These days I would have considered using a multilevel model to take family structure into account, but in 2005 I didn't know how to do that, and instead I decided to take a mean value for each family. So if there was one child, I used their score, but if there were 2 or 3, then I averaged them. The N was then the number of families, not the number of children.

And here, dear Reader, is where I made a fatal mistake. I thought the simplest way to do this would be by creating a new column in my Excel spreadsheet which had the mean for each family, computing this by manually entering a formula based on the row numbers for the siblings in that family. The number of families was small enough for this to be feasible, and all seemed well. However, I noticed when I opened the file that I had pasted a comment in red on the top row that said 'DO NOT SORT THIS FILE!'. Clearly, I had already run into problems with my method, which would be totally messed up if the rows were reordered. Despite my warning message to myself, somewhere along the line, it seems that a change was made to the numbering, and this meant that a few children had been assigned to the wrong family. And that's why table II had gremlins in it and needed correcting.

I now know that doing computations in Excel is almost always a bad idea, but in those days, I was innocent enough to be impressed with its computational possibilities. Now I use R, and life is transformed. The problem of computing a mean for each family can be scripted pretty easily, and then you have a lasting record of the analysis, which can be reproduced at any time. In my current projects, I aim to store data with a data dictionary and scripts on a repository such as Open Science Framework, with a link in the paper, so anyone can reconstruct the analysis, and I can find it easily if someone asks for the data. I wish I had learned about this years ago, but at least I can now use this approach with any new data – and I also aim to archive some old datasets as well.

For a journal, a corrigendum is a nuisance: they cost time and money in production costs, and are usually pretty hard to link up to the original article, so it may be seen as all a bit pointless. This is especially so given that a corrigendum is only appropriate if the error is not major. If an error would alter the conclusions that you'd draw from the data, then the paper will need to retracted. Nevertheless, it is important for the scientific record to be accurate, and I'm pleased to say that the American Journal of Medical Genetics took this seriously. They responded promptly to my email documenting the problem, suggesting I write a corrigendum, which I have now done.

I thought it worth blogging about this to show how much easier my life would have been if I had been using the practices of data management and analysis that I now am starting to adopt. I also felt it does no harm to write about making mistakes, which is usually a taboo subject. I've argued previously that we should be open about errors, to encourage others to report them, and to demonstrate how everyone makes mistakes, even when trying hard to be accurate (Bishop, 2018). So yes, mistakes happen, but you do learn from them.

References 
Bishop, D. V. M. (2018). Fallibility in science: Responding to errors in the work of oneself and others (Commentary). Advances in Methods and Practices in Psychological Science, 1(3), 432-438 doi:10.1177/2515245918776632. (For free preprint see: https://peerj.com/preprints/3486/)

Bishop, D. V. M., Maybery, M., Wong, D., Maley, A., & Hallmayer, J. (2006). Characteristics of the broader phenotype in autism: a study of siblings using the Children's Communication Checklist - 2. American Journal of Medical Genetics Part B (Neuropsychiatric Genetics), 141B, 117-122.

My collapse of confidence in Frontiers journals

Frontiers journals have become a conspicuous presence in academic publishing since they started in 2007 with the advent of Frontiers in Neuroscience. When they were first launched, I, like many people, was suspicious. This was an Open Access (OA) online journal where authors paid to publish, raising questions about the academic rigour of the process. However, it was clear that the publishers had a number of innovative ideas that were attractive to authors, with a nice online interface and a collaborative review process that made engagement with reviewers more of a discussion than a battle with anonymous critics. Like many other online OA journals, the editorial decision to publish was based purely on an objective appraisal of the soundness of the study, not on a subjective evaluation of importance, novelty or interest. As word got round that respectable scientists were acting as editors, reviewers and authors of paper in Frontiers, people started to view it as a good way of achieving fast and relatively painless publication, with all the benefits of having the work openly available and accessible to all.

The publishing model has been highly successful. In 2007, there were 45 papers published in Frontiers in Neuroscience, whereas in 2014 it was 3,012 (data from Scopus search for source title Frontiers in Neuroscience, which includes Frontiers journals in Human Neuroscience, Cellular Neuroscience, Molecular Neuroscience, Behavioral Neuroscience, Systems Neuroscience, Integrative Neuroscience, Synaptic Neuroscience, Aging Neuroscience, Evolutionary Neuroscience and Computational Neuroscience). If all papers attracted the author fee of US$1900 (£1243) for a regular article, this would bring in £3.7 million pounds in 2014: the actual income would be less than this because some articles are cheaper, but it's clear that the income is any in case substantial, especially since the journal is online and there are no print costs. But this is just the tip of the iceberg. Frontiers has expanded massively since 2007 to include a wide range of disciplines.  A Scopus search for articles with journal title that includes "Frontiers in" found over 54,000 articles since 2006, with 10,555 published in 2014.

With success, however, have come growing rumbles of discontent. Questions are being raised about the quality of editing and reviewing in Frontiers.  My first inkling of this was a colleague told me he would not review for Frontiers because his name was published with the article. This wasn't because he wanted confidentiality; rather he was concerned that it would appear he had given approval for the article, when in fact he had major reservations.

Then, there have been some very public criticisms of editorial practices at Frontiers. The first was associated with the retraction of a paper that claimed climate denialism was associated with a more general tendency to advocate conspiracy theories. Papers on this subject are always controversial and this one was no exception, attracting complaints to the editor. The overall impression from the account in Retraction Watch was that the editor caved in to legal threats, thereby letting critics of climate change muzzle academic freedom of speech. This led to the resignation of one Frontiers editor**.

Next, there was a case that posed the opposite problem: the scientific establishment were outraged that a paper on HIV denial had been published, and argued that it should be retracted. The journal editor decided that the paper should not be retracted, but instead rebranded it as Opinion – see Retraction Watch account here.

Most recently, in May 2015 there was a massive upset when editors of the journals Frontiers in Medicine and Frontiers in Cardiovascular Medicine mounted a protest at the way the publisher was bypassing their editorial oversight and allocating papers to associate editors who could accept them without the knowledge of the editor in chief. The editors protested and published a manifesto of editorial independence, leading to 31 of them being sacked by the publisher.   

All of these events have chipped away at my confidence in Frontiers journals, but it was finally exploded completely when someone on Twitter pointed me to this article entitled "First time description of dismantling phenomenon" by Laurence Barrer and Guy Giminez from Aix Marseille Université, France. I had not realised that Frontiers in Psychology had a subsection on Psychoanalysis and Neuropsychoanalysis, but indeed it does, and here was a paper proposing a psychoanalytic account of autism. The abstract states: "The authors of this paper want to demonstrate that dismantling is the main defense mechanism in autism, bringing about de-consensus of senses." Although the authors claim to be adopting a scientific method for testing a hypothesis, it is unclear what would constitute disproof. Their evidence consists of interpreting known autistic characteristics, such as fascination with light, in psychoanalytic terms. The source of dismantling is attributed to the death drive. This reads like the worst kind of pseudoscience, with fancy terminology and concepts being used to provide evidence for a point of view which is more like a religious belief than a testable idea. I wondered who was responsible for accepting this paper.  The Editor was Valeria Vianello Dri, Head of Child and Adolescent Neuropsychiatry Units in Trento, Italy. No information on her biography is provided on the Frontiers website. She lists four publications: these are all on autism genetics. All are multi-authored and she is not first or last author on any of these*. A Google search confirmed she has an interest in psychoanalysis but I could find no further information to indicate that she had any real experience of publishing scientific papers. There were three reviewers: the first two had no publications listed on their Frontiers profiles; the third had a private profile, but a Google search on his name turned up a CV but it did not include any peer-reviewed publications.

So it seems that Frontiers has opened the door to a branch of pseudoscience to set up its own little circle of editors, reviewers and authors, who can play at publishing peer-reviewed science. I'm not saying all people with an interest in psychoanalysis should be banished: if they do proper science, they can publish that in regular journals without needing this kind of specialist outlet. But this section of Frontiers is a disastrous development; there is no evidence of scientific rigour, yet the journal gives credibility to a pernicious movement that is particularly strong in France and Argentina, which regards psychoanalysis as the preferred treatment for autism. Many experts have pointed out that this approach is not evidence-based, but worse still, in some of its manifestations it amounts to maltreatment.  What next, one wonders? Frontiers in homeopathy?

Like the protesting editors of Frontiers in Medicine, I think the combined evidence is that Frontiers has allowed the profit motive to dominate. They should be warned, however, that once they lose a reputation for publishing decent science, they are doomed. I've already heard it said that someone on a grants review panel commented that a candidate's articles in Frontiers should be disregarded. Unless these journals can recover a reputation for solid science with proper editing and peer review, they will find themselves shunned.

*The Frontiers biography suggests she is last author on a paper in 2008, but the author list proved to be incomplete.
** Correction: Shortly after I posted this, Stephan Lewandowsky wrote to say that there were 3 editors who resigned over the RF retraction, plus another one voicing intense criticism