Preprints, Trust, and DORA

The journal article has long been the object that defines academic career trajectories. In this issue, we talk to one of the main disruptors of this status quo.

Rebecca Lawrence is Vice President, Knowledge Translation at Taylor & Francis. She is also the Founding Managing Director of the open research publisher F1000 (now part of Taylor & Francis), and currently serves as Vice-Chair of DORA (San Francisco Declaration on Research Assessment).

One of the researchers we have worked with has remarked “Scientia’s article has let our funder know of the broader appeal of our work”. This piqued our curiosity to explore how non-traditional publishing models have jolted power in acknowledging the critical role of funder support. We looked at F1000, and reached out to Rebecca Lawrence.

Rebecca holds a PhD in Pharmacology and has worked in STM publishing for over 25 years. She was responsible for the launch of F1000Research in 2013 and has subsequently led the initiative behind the launches of many funder- and institution-based publishing platforms. To provide a new trajectory in the way scientific findings and data are communicated, she established partnerships with the European Commission, Gates Foundation, Wellcome, and others. She was a member of the European Commission’s Open Science Policy Platform, chairing their work on next-generation indicators and being credited as an Editor in their final report (OSPP-REC).

Could you explain the difference between F1000 and F1000Research?

F1000 was originally founded by Vitek Tracz back in 2002, a visionary in scholarly communication for 30–40 years, and was something different to F1000Research. The name stood for Faculty of One Thousand. The original concept was to create a virtual faculty of 1000 of the leading researchers in biology. We asked them to identify papers they thought were particularly interesting or important, and write a short recommendation about that paper for our site. This could be a paper from their area, or it might be a paper from a completely different field that they found useful, or it might be a really interesting method; there were all sorts of reasons why they might think that something is a standout paper. So, we started out as basically a database of article recommendations. We had thought that there’s a lot of excellent research published all over the place, and not just in the top impact factor journals. Equally, there’s lots of stuff in the top impact journals that maybe isn’t so excellent. That proved to be the case, as most of what was recommended was not published in top impact factor venues.

In 2012, we launched F1000Research, with the aim to rethink scholarly publishing. We asked the question: if you look at technology today and you’ve got no legacy system and you just start again, what would you do? It came out of some initial work we had been doing through F1000Posters in trying to maximise the discoverability of new research presented in posters, slides, and grey literature. We asked: ‘how can we speed up scholarly publishing?’ The process of getting new discoveries in front of those that can start to build on them typically takes months or even years in the traditional process, so everything is very slow. Given that in the rest of our lives, we can just stick stuff on the internet, there’s just too much of a time differential between findings and dissemination.

Data preparation

Also, can we improve transparency and reduce bias in the peer review process, as well as improve reproducibility in terms of sharing the data and the code? Indeed, these are actually the core of the article; the text around it is just conjecture and the author’s narrative. How could you review a paper if you don’t have the data? So, we built the F1000Research publishing model, combining the benefits of a preprint server with those of a traditional journal. With us, the author remains in control of what they share, and when they share it, enabling the sharing of incremental or null findings. It’s not about being novel or interesting; as long as you meet basic integrity and ethics requirements, then you should be able to share findings and data, as and when you want. Then we tie this together with peer review, archiving, indexing, and all the other stuff that journals provide. And we make it all open, transparent and FAIR.

That’s basically the publish–review–curate model, but ahead of any of that terminology. We were the first to really support preprinting in the biomedical sciences, even before bioRxiv and medRxiv. And the first to have a mandatory FAIR data policy, and fully transparent peer review, where we disclose identities as well as the peer review reports, and publish the peer review reports even of those articles that do not pass peer review. The key distinction with a preprint is that with our model, you can’t go somewhere else, i.e. you can’t put your article on an F1000 venue and then say ‘Now, I’m going to send it to Nature.’ No, this is your publication. It’s just the process that differs, we put it out there and then we peer review it, but you can’t stop in the middle. Because of that, we included versioning, so you can also update and revise the publication—it also means that as your research evolves, you will be able to evolve your paper.

Article submission

One of the biggest challenges we had, and still have, is due to the fact that we very deliberately made sure that we didn’t have an impact factor. eLife are now experiencing this challenge too. Researchers will think ‘Will it count? I love the idea of the model, but will it count if I publish on here?’ That’s what motivated us to approach funders to partner on our platforms, which obviously at the time was not a standard thing to do. If we were able to partner with a funder, we could reassure researchers that their funder is approving this option to you for publication. Many of the funders we work with are keen to see all the outputs of their funding published, and so they are saying to their grantees, here is an option that will enable you to do this. The funder takes care of payment directly with the publisher, giving authors confidence and simplifying any admin. They can then apply for further funding from say Wellcome or the Gates Foundation, because they can have the confidence that the funder will recognise any work published in their venues. And we approached funders in the first instance as they have the greatest influence on researchers, as that is where they get their funding from.

This is why we started to partner with those funders that strived to support a shift in the system, to enable all those research outputs to be out there, quickly and efficiently. It also removes a lot of cost from the system, because it means that articles don’t go from journal to journal to journal, not passing peer review, or not being accepted by the editor, and starting again. Here, you do it once. Wellcome were the first partner to have a publishing platform for their grantees using the model we had developed, and we went from hardly any Wellcome-funded papers using the model to very quickly being the largest venue of publication of Wellcome-funded research.

Publication

Do you still find reviewers the way that other journals do, for instance relying on a pool of reviewers that you know? Or do you trust reviewer finding to services like Global Campus and Prophy?

Open peer review and user commenting

We started from the principle that the people who will know best who are the right reviewers are the authors themselves, because they’re in the same community. We still conduct a set of checks to make sure they are relevant names and there aren’t any obvious conflicts of interest. But the key thing was that our peer review is transparent, this acts as a sort of disinfectant.

We ran that approach until very recently, actually, because it puts the author in charge. However, over time we discovered it caused many problems, partly because authors just weren’t used to this task; beyond those authors that were very experienced, the rest of the community really struggled to know how to find reviewers and to identify relevant people. Despite sharing the reviewer criteria, the authors would struggle to find names that meet them and would eventually get fed up trying to suggest reviewers just to have us say ‘they don’t meet the criteria’. This just slowed everything down massively and annoyed everybody. So, we eventually decided that our internal teams of experts would identify reviewers for us to then invite. Those authors that wish to can still suggest reviewers, and we will of course run them through our usual checks, but it’s not a requirement anymore.

In terms of communities of reviewers, we publish across all fields, so we don’t have communities of reviewers as such, because our community is essentially the world.

What about the worst nightmare of traditional publishers, which are retractions and paper mills, and fraud at a large scale. Have you had problems with that?

We’re certainly not immune to getting those submissions but we’ve ended up with few retractions because we actually have really rigorous editorial checks. We have to because we don’t have peer review before we publish the research: the manuscript gets a DOI, but so do the data and the code and everything else. The DOIs of each version of the manuscript are linked together, and the peer view reports get DOIs as well.

Article revisions

In some parts of the community, the belief is that we’re an easy venue to get content through, but the reality is that when they approach us, they realise we’re actually far more stringent than many venues. From the occasional analysis that we’ve done looking at our retraction rates compared to standard journals, we typically find they are significantly lower.

For content that comes into the funder platforms, the first barrier is that you have to have been awarded a genuine grant from that funder; hence, the researchers have already been heavily screened by the funder in the first place. As you’d expect, there you don’t see these types of targeted content. But on our own platforms, we do get targeted just as everybody does. We have exactly the same policies around retractions as traditional journals and follow the same rules.

Tell me more about your model of approaching funders. I know that you have partnerships with Wellcome and with the Gates Foundation, but do you have partnerships with other funders?

We run the Open Research Europe platform for the European Commission, so that covers Horizon 2020 and Horizon Europe funding, as well as most EU funded initiatives like ERC and Marie Skłodowska-Curie Actions. We also partner with the Irish Health Research Board, and in the UK with the Association of Medical Research Charities, which gathers over 150 medical research charities, and with the NIHR. Those larger ones have their own platform. But there are various other funders that we also partner with where they have a space and/or collections on one of our own platforms such as  with WHO, NC3Rs or Science for Africa Foundation (SFA). We also partner with research institutions, with scholarly societies, and other organisations.

A lot of people, especially early career researchers, are not aware that there are resources that help to assess whether the venue that holds a paper is reputable or not. And that the place that they choose to submit their research to will influence their reputation. Is that why you also got involved with DORA?

The impact factor of journals is calculated broadly speaking by looking at the number of citations from the previous two years and dividing that by the number of publications, with some (largely opaque) adjustments to that. This metric has been used for a long time by most countries, institutions, and funders to make decisions on hiring, career progression, promotion, and funding decisions. Whilst some progress away from this has started in some regions, this approach still persists worldwide but probably especially strongly in parts of the Global South where even to graduate, you may need say at least one publication; if you want to get a PhD, it’s often two or three publications.

But the problem is, whilst this is a simple and straightforward approach in some ways, by using either volume of publications and/or journal-based metrics such as impact factors to support decisions on your career or your funding, means that everybody chases those numbers. Consequently, it is very competitive to get into those ‘top journals’, so there is an increasing amount of either poor practice or, in some cases, highly problematic practice. It can range from poor statistical techniques that ‘massage’ your findings, showing them as highly significant when maybe they’re not, through to selective reporting, when you report the experiments that work and just ignore the ones that didn’t. Of course, afterwards nobody can reproduce your research because that’s not actually the reality.

It can also lead to more extreme practices such as fabricated data and images, and even to the use of paper mills, where people go to buy papers because they just need a paper, so they end up as authors on research they had nothing to do with and/or is AI-generated.

There are also a whole set of predatory journals and publishers that pretend to do proper peer review. Some of these look like the journal you think you’re targeting, but it’s actually a fake journal, they just swapped the order of the journal name around or they’ve slightly misspelt it. Sometimes, they’ve even hijacked it and that’s even harder to spot. But there’s something called think, check, submit which is a really useful checklist. I would always check any call for papers request with that very carefully, to ensure you are really submitting to a reputable journal. Even if a website has got top names on it as Editors, that doesn’t necessarily mean those editors actually know they’re listed on it or have anything to do with it.

DORA (San Francisco Declaration on Research Assessment) was launched in 2012. It came about from an annual meeting of the American Society of Cell Biology, where they developed a declaration calling for a move away from using impact factors. Apart from the integrity issues already mentioned, impact factors actually do not provide a fair reflection of the article itself, because invariably the high impact factor of a journal is based on a very small number of papers that gather really high citation numbers while the rest may have very few citations.

Those journals obviously also want to publish articles that appear like they will be impactful in some way. There are a lot of outputs such as negative/null findings, that are really important to share, and those journals typically don’t really want to share these because they don’t bring many citations. DORA aims to drive a shift away from using journal and venue-based metrics towards assessment systems that reward a much broader range of outputs, activities and processes. For example, we are trying to incentivize open research practice that will lead to you to getting proper recognition for sharing your data, for sharing your methods, sharing your code, performing peer review, good mentoring, good teaching — critical skills for an active and well-functioning research ecosystem.

DORA is fully global, with in fact strong networks in Global South regions. It has now been signed by over 26,000 researchers and institutions. We’re now focussing on not just building awareness that there is a problem with traditional research assessment approaches, but on developing tools, frameworks and implementation guides to support organisations in reforming their policies and practices. This includes training materials and materials to help you take the topic to your senior leadership to raise awareness, so you can help start that conversation. Now is the time to build on the growing momentum in this space, to properly recognise researchers and their outputs and also support equity, diversity, and inclusion, as well as promote citizen science to genuinely solve society’s problems.

Do you believe that there is still value in the traditional publishing model? We usually hear that people that want to have an international career in science should not be stuck to one funder or one venue, but place their outputs in varied places.

As with most of these things, I think it’s a balance. It doesn’t usually look good to have only published in one venue, it’s better to have publications in a range of places. This also depends on what the output is and what you’re trying to achieve with it, so you have to determine where is best suited for the output and what is the best process. In some areas, there isn’t such a pressure to get new discoveries out quickly, so it varies according to the situation. We need to develop a more holistic way of looking at this. I don’t think we’re going to end up with one mechanism of publishing research that is going to work for everybody. Different communities, different fields, different regions are going to need different versions and nuances on it.

I do personally think that there’s a big problem with closed anonymous peer review, although I can understand why researchers and research communities place value on it. I’m really pleased to see the acceleration in the uptake towards open peer review, although this is typically just making the reviews open, and only when an article passes peer review and is published. I fully understand why people, particularly early career researchers, are nervous about open identities. Having said that, unless we also have open identities, I believe we’re still going to have a lot of problems as people can hide behind anonymity and say whatever they want, even if the review text is public.

In terms of the actual model, I believe it’s going to end up as a whole mixture of approaches depending on the field and its community, but we need to have an evaluation system that supports and enables that right now. The main way that researchers are evaluated presently disincentivizes uptake of new experiments, new venues, and new approaches.

Do you think that generative AI has a role to play in this? There are a lot of tools being made, and different platforms that now provide services to both researchers and publishers. These are mostly based on the claim to at least reduce the amount of time, sometimes even the amount of effort that you have to put in to get your output published. Do you think that this is applicable right now?

It’s still early days really with LLMs. They are progressing rapidly, and there’s a lot of work going on to try and improve them. As an aside, I did though see a paper recently talking about using these tools to summarise outputs. They showed that if you try and use these tools to create a plain language summary, they tend to over-generalise, and that’s a real problem as they end up making claims that essentially aren’t there. The concerning thing is that they then tested with the newer versions of these tools and found they were worse! Of course, it all depends on what they’re trained on, and any biases will likely get amplified over time.

Whether we like it or not, AI and LLMs are going to play a key role in publishing. People are going to use them, and if used right, they can be really helpful. For example, if English is not your first language, then they can really help you for translation and editing. They can also help from a publisher perspective, to assess how novel something is; if you’re an editor or a reviewer, you can only know research based on what is a comparatively small area of information, but an LLM can look genuinely at the whole world of information and give a far better answer to such questions.

There are definitely places, certainly part of workflows, where we can use gen AI to support efficiencies. But it’s critical to maintain what authorship stands for. An LLM cannot be an author because an author has to stand behind the content of the article. Whether it’s writing an article, whether it’s as peer reviewers, even if someone uses LLMs as part of the process, humans have to decide whether they agree with the output, and are willing to stand behind it. There’s a lot of work going on to improve these tools, so I’m sure they will get better. But we will always need humans to verify the outputs.

Is there something that you think can be improved in terms of communicating science, to the broader public that researchers themselves, or publishers, could be doing?

I think there are two areas. I recently wrote an article about trust markers. At the moment, we’re in the process of applying for a working group to NISO to develop cross industry standards around trust markers. As a member of the public and even as a researcher, it’s very hard to know what level of checks has each output been through, and therefore how much could to trust it. Every preprint server has different checks, usually quite light checks (and often it’s not very easy to find what those checks were), and a lot of preprint servers are facing some real challenges at the moment.  Equally, every publisher, and even journal programmes within publishers, often have different checks, and they’re sometimes not visible at all. We just rely on the fact that if something is peer reviewed, then it must be OK; this is highly problematic and not a valid assumption.

Information is being misused and even problematic publications are then being cited to support really problematic theories and policies. We need some way of having some kind of markers that signal ‘these are the checks that this output has gone through, these are the ones it has passed, these are the ones it hasn’t passed’. This would then enable people to better decide how much trust to place on some content they come across. That’s one area where we could certainly do something collectively.

The other area is within my new role, which is VP of Knowledge translation within Taylor & Francis. I’m tasked to better support the translation of new information to those who are actually going to build on it, which may be other researchers, but may well be policymakers, or politicians, or the public etc. Different types of communities need to have a better way of understanding what something really means, and we have to ‘de-hype’ the language, to get a sense of how innovative a new piece of research truly is. And then they need to know how much to trust that research.