Guidance for Readers

Every piece of work on this site is published with the reviews that experts wrote about it, and with their scores. This page explains what peer review is, what it can and cannot tell you, and how to read those reviews and scores. Peer Reviews is a new and experimental service, so it also explains how our approach differs from the way most journals work.

What is peer review?

Peer review means asking independent experts to examine a piece of work closely before other people rely on it. The experts, called reviewers, usually work in the same field. They ask questions such as: are the methods sound, do the results support the conclusions, and is anything important missing?

The idea is old. The Royal Society began publishing Philosophical Transactions in 1665, and from 1832 it required a written report from expert referees before accepting a paper. Routine review by outside experts is more recent: Nature only made it standard practice in 1973 (JSTOR Daily). Today almost every academic journal uses some form of it.

At a traditional journal, the reviews help an editor decide whether to accept or reject the work, and suggest improvements to the authors. Readers rarely see them. They see the final article and the label “peer reviewed”, and have to trust that the review was careful.

Benefits

Peer review is the best general check we have on specialist work that most people cannot assess for themselves. Done well, it:

• catches problems such as flawed methods, mistakes in analysis, or conclusions that go further than the evidence allows;
• gives an independent view, from someone with relevant expertise who has no stake in the work;
• helps readers focus, by pointing out what is genuinely new and what matters most;
• improves the work, where authors are given the chance to respond and make changes.

Reviewed work is not guaranteed to be right. But a close reading by capable, independent experts makes serious problems more likely to be found.

Limitations

Peer review is carried out by a few busy people, and it has well-documented weaknesses.

• Reviewers miss things. In one study, researchers put nine major errors into papers sent to reviewers at the BMJ. On average, reviewers found fewer than three of them (Schroter et al., 2008).
• Reviewers often disagree. An analysis combining dozens of studies found that agreement between reviewers of the same paper is low (Bornmann et al., 2010).
• Reviewers cannot check everything. They usually see the paper, not the raw data or the lab. Review is poor at catching deliberate fraud.
• It can be biased. Judgements can be swayed by an author’s reputation or institution, or by whether the findings fit the reviewer’s own views.
• It is often hidden. When reviews are not published, readers cannot see what the reviewers said, or how carefully they looked.

None of this makes peer review worthless. It means “peer reviewed” is a useful signal, not a guarantee.

Interpreting PR Scores

Peer review is run in several ways (Wiley). The biggest differences are whether readers can see the reviews, and whether review ends in an accept or reject decision. What this means for you as a reader:

Reviewers can choose to sign their review or remain anonymous.

There is no stamp of approval. A simple “accepted” hides a lot: a sound but modest piece of work and a bold but shaky one can both pass. Our scores show where work is strong and where it is weak.

You can read the reviews. You do not have to trust a journal’s name. You can see what the experts said and decide how much weight to give it.

The author decides whether to publish. Authors read their reviews first and choose whether to make them public. So work that reviewed well is more likely to appear here than work that reviewed badly.

The fee does not buy a good result. Authors pay for review, and most of the fee goes to the reviewers. The fee is the same whatever the reviewers conclude.

The scores describe the version you are reading. Work is published as it was reviewed, without a round of revisions.

Some tips for reading them:

• Read the reasons, not just the numbers. The written explanation tells you why a reviewer gave a score, and whether it matters for your purpose.
• Look at the pattern, not a total. The criteria measure different things, so we do not add them up. Careful work that repeats an earlier study may score low on novelty and high on validity. That may be exactly what you need.
• Treat a one-point difference as small. Scores are expert judgements, not measurements. A 3 from one reviewer and a 4 from another may say more about the reviewers than the work.
• Compare the reviewers. Where there is more than one reviewer and they broadly agree, that is reassuring. If they differ, read why: disagreement is common, and often reflects different expertise or priorities.
• Compare like with like. Scholarly and technical work are reviewed by different kinds of expert, with different expectations. A score on one route may not mean the same as on the other.

Scores cannot tell you whether a finding is true, or whether you should act on it. They tell you how a small number of qualified people judged the work at one point in time.

A new and experimental process

Peer Reviews is new, and this way of running peer review is an experiment. We believe published, scored reviews tell readers more than a journal’s stamp of approval. But we do not yet have evidence that our scores are consistent between reviewers, or that they predict which work holds up over time.

That means:

• treat the scores as one piece of evidence among several, not as a final verdict;
• the criteria, scale and process may change as we learn. If they do, we will say so, and earlier reviews will stay as they were written;
• we use automated tools, including AI, to check submissions, help find reviewers and flag problems in reviews. Every review is written by a human expert, and every decision is made by a person.

We welcome feedback from readers, authors and reviewers. If you think a review is wrong, unfair or has missed something, please contact us.

Publication bias: “work that reviewed well is more likely to appear here” follows from the author choosing whether to publish. It is the most important caveat for readers, but your call.

Fee to reviewers: “most of the fee goes to the reviewers” is taken from the current Process page.