Friday, February 3, 2012

Universal open review?

There has been discussion in recent years about opening up review of scientific articles by bringing down the artificial wall of anonymity behind which reviewers are able to hide. A common complaint of authors is that anonymity results in careless, even rude, and sometimes unethical statements by reviewers, and it is argued that removing anonymity would greatly reduce unscientific and careless criticism because reviewers would be more careful if they had to stand behind their statements with their name and reputation. A common concern raised about removing anonymity is that reviewers could be subject to unfair retaliation by authors who are not happy about negative reviews.

To the first point, I would like to describe my observations based on my experience as editor in chief of The Plant Cell (2003-2007). Yes, I will be explaining "how the sausage is made"... Before 2003, the practice at TPC had been that editors handling manuscripts were always anonymous, just as reviewers were. Instead of editors signing decision letters, the Managing Editor, a professional editor who was not a scientist, edited and signed letters drafted by the handling editor, who was a recognized expert in the field of the submitted manuscript. The reason stated in support of this practice was that editors, especially early career editors, needed this protection in order not to suffer retaliation by peers and senior colleagues, especially regarding grant proposals, but also regarding manuscript and promotion decisions.

However, simple inspection of a random sample of decision letters indicated to me that editors were too frequently careless and cavalier with their statements and did not often enough explain clearly the basis for their decision. It was easy to see that authors would be offended by such letters, and moreover that the reputation of the journal would necessarily suffer as a result.

Based on this, I felt it was imperative that editors sign their decision letters and stand by their decisions and statements with their names and their reputations. So, prior to assuming my role as EiC, I informed the editorial board that henceforth editors would have to sign each decision letter and that I expected the same high level of professionalism in decision letters as we expected of our authors in their manuscripts. Some editors resigned, and I happily appointed new editors who were willing to sign their decisions. The result was striking - the quality and professionalism of decision letters rose dramatically. In four and a half years at TPC I never received a single complaint from an editor that s/he felt retaliated against because of a publication decision. I believe that, in general, scientists appreciate and respect well-argued publication decisions, even if they disagree with them, and that removing anonymity, if done thoughtfully, will not necessarily cause rampant problems of retaliation, i.e, authors' greatest fears will not be realized.

I think it was also important that I told the board I was willing to work with them on their decisions and letters and that I would support them and stand behind their decisions as long as they were scientifically solid and were explained clearly and professionally to authors. When authors challenged a decision, I was always available to work with the editor to see if there was a scientific basis to the challenge that should cause us to reconsider, and I was available to work with the editor on writing a professional response to the author indicating that we had considered the author's challenge thoughtfully. It was always expected that in case of a 'decline' decision editors would explain to authors what would be necessary for publication in TPC, such as additional experiments or more rigorous interpretations at a level of standard that the plant biology community expected for articles published in TPC. I do believe that it requires effort on the part of editors (and reviewers) to prepare the quality of decision letters (and reviews) that authors deserve. Authors have generally worked very hard on their manuscripts prior to submission, and even if there may be flaws, editors and reviewers should also be willing to work hard to prepare a fair, substantive, scientific and professional decision letters and reviews. After all, this is only way the system is going to work well.

Although I personally wanted to remove the anonymity of reviewers, I was afraid the community was not yet ready for that in 2003. Because we allowed reviewers to retain anonymity, we continued to have problems with unprofessional reviews. We dealt with this in various ways. One was to reject a review that contained unprofessional statements and force the reviewer to rewrite the review, and if that request was refused, we disqualified the review. Of course, we generally chose not to invite recalcitrant reviewers in future. I believe that most good journals operate this way (but I am really only familiar with practices at TPC during my tenure there).

I am in favor of removing anonymity of reviewers, but because of the second concern mentioned at the outset - that of potential retaliation by angry authors - I think the change should be made at several levels and that as a community we should embrace the following principle: No academic scientific reviewer should have anonymity, regardless of whether the review addresses a manuscript, a grant proposal or a promotion or tenure decision. All academic scientists should be willing to stand behind their opinions regarding manuscripts, proposals and performance appraisals. Transparency should be the order of the day!

In my view, 'universal transparency' would greatly dampen the possibility of retaliation because all major opinions affecting a scientist's ability to be a scientist would be made without the possibility of hiding behind a veil of anonymity. I also believe based on my experience that scientists can generally be trusted to appreciate and respect well argued publication decisions even if they disagree with them, and that the likelihood of retaliation will be no greater, and in fact less, than is already the case.

In fact, as we all know, anonymity comes with an unintended consequence of increasing the disappointment and sometimes anger of the person being reviewed. Too often that anger is misdirected at someone the person guesses was responsible for a negative appraisal but actually was not. I saw this many times at TPC: authors usually guess wrongly and aim their wrath at the wrong person. In addition, people they think are their allies are often their strongest critics. For instance, when authors suggest a reviewer, it is not at all uncommon that the proposed reviewer is the most critical reviewer of the manuscript, even recommending outright rejection. The point here is simply that authors often guess wrongly and so anonymity causes grief for 'innocent bystanders'.

Would universal transparency solve all our problems? Of course not. This is not a perfect world, and human nature being what it is, we will still have problems. Less problems, however, in my opinion. And problems that mainly can be addressed directly and openly by those whose opinions differ. I would even go so far as to suggest that we should consider anonymity to be unethical, given that I believe the alternative, transparency, is more fair.

Anonymity is not a sacred cow. We should drive a stake through its heart. And once we do, we will be glad we did.


P.S. See this new post, suggesting "True-Peer Review":
http://ibiosphere.blogspot.com/2012/02/true-peer-review-would-it-change.html

40 comments:

  1. Totally agree. Was thinking about this a lot today after the tweeting last night. This model is the only future that works if one wants anonymity in any one.

    ReplyDelete
  2. So well argued Rich! This is something the scientific community should strive for, and make it happen.

    ReplyDelete
  3. I think you make some excellent points; consider the differences between web sites that require comments to use real names and those that are anonymous. The overall number of thoughtless or mean comments drops and the level of conversation dramatically improves. We've all experienced reviews that are unnecessarily rude, poorly thought out or simply spiteful, and I'm sure the quality of reviews would go way up if reviewers couldn't hide behind anonymity. Overall, from the point of view of someone who has papers and grants reviewed, disclosure would be a good thing. However, I think you are too easily brushing aside the concern that (especially) younger reviewers might have that they would suffer retaliation. I do a lot of reviewing, mostly out of a sense of obligation to the community. Like everyone else, I do it for free, often for a journal that is owned by company that makes a hefty profit. It's a great business model. The authors pay you to provide you with your product, and your quality control division works for free. Smart. At any rate, given that, and the sheer amount of time it takes to write a good review, I understand that it can often be difficult for editors to find reviewers. How hard will that be when the reviewers for journals and funding agencies have to worry about retaliation? Why on earth would I negatively review a grant or a paper from a major player in my field when I'm just trying to get started? So now, on top of taking a lot of time and paying me nothing, I'm supposed to risk damaging my career? I agree, most scientists do appreciate honest and rigorous reviews, even negative ones. Indeed, I depend on reviewers to catch things that I miss. However, scientists are human, and power in the scientific community is unevenly distributed. I am concerned that the proposal will lead to a system in which the powerful get more so, since no one wants to piss them off, and the powerless are silenced or simply censor themselves.

    ReplyDelete
  4. Rich, one concern that comes to mind is the potential for "buying favoritism" by writing soft reviews for those more senior in the field. Your suggestion would require significant adjustment of the current process, but I see genuine merit in moving in this direction. How do you propose creating such momentum for change? Sally Mackenzie

    ReplyDelete
  5. from Jean Greenberg:
    As usual, i appreciate your very thoughtful comments Rich. Some points for you to think about:
    1. you know well the endless hours editors take reading and responding to authors who want to argue about reviews and editorial decisions. As open reviewers, are we now also going to be further subject to authors contacting us seeking more dialoging? I dont know about you, but I am just about maxed out in the amount of time I spend critiquing other people's work and adjudicating editing decisions. If I also have to dialog with more authors, i wont get any of my own work done.
    2. I've had the converse also happen-reviewers who approach me for discussion of my grants. NIH proposal reviewers are only semi-anonymous since you can see the panel composition...The potential for these extra discussions can add an extra layer of inequity in future reviews.
    3. I feel your suggestion will lead to many contentious confrontations. I've already had plenty (from reviewers, authors...) For people who want to avoid confrontations, they will not give frank reviews or they will refuse to review.

    some alternative ideas that might be a little cumbersome but may may some things more fair:
    1. in the usa, we dont have the opportunity to respond to prejudicial grant reviews (at least for most programs) until we re-submit. If someone makes a mistake in what they say your grant can be unfairly killed. I would be in favor of a rebuttal process to deal with clearly prejudicial comments. Now there is a way to do this, but its also subject to the good will of the program officers, which can be quite variable.
    2. a compromise on tenure would be to show redacted letters to the candidate so decision process is more transparent.

    ReplyDelete
  6. This is an interesting idea, but I think that the situation may be even more complex than even this well-reasoned post presents. The most critical, and personally offensive, reviews that I have received have come from reviewers who signed their reviews. Requiring reviewers to sign their reviews may indeed alleviate some of the problems that Rick mentions, but it will not solve the problem of the foibles of fame. If you are famous enough you can say just about anything, get away with it, and be invited back to review again. If we could eliminate human nature, now that would definitely solve the problem.

    ReplyDelete
  7. Bruce,
    Fair point. That's why we need to publish all reviews, along with the publication, a point that I neglected to make above.

    ReplyDelete
  8. Sally, I agree the situation is complex. But if the current system required openness, and we proposed a change to anonymity, that too would be complex. Instituting change is rarely simple.

    ReplyDelete
  9. 'Unknown' makes the point that anonymity demonstrably leads to nasty exchanges in comments sections on the web, supporting the point that anonymity has a negative unintended consequence. S/he also raises the concern that young scientists will suffer retaliation, as has a 'tweeter' who responded to this blog post.

    The reason I proposed removing anonymity not only from manuscripts for publication but also for grant proposals and promotion and tenure review is precisely to force more senior scientists to stand behind their words and think about what they are saying before they write them. Also, I think 'arbiters' (editors, administrators...) have an ethical responsibility to reject or force revision of unfair reviews. Publishing the reviews with the publication would also help. It would not be possible to publish grant proposal reviews, of course, because proposals are meant to be confidential (I'm not advocating with change that). But perhaps we should consider publishing promotion and tenure reviews? Thoughts on this, anyone?

    ReplyDelete
    Replies
    1. from Jean Greenberg:
      Others have advocating publication of critiques after publication (ie anyone can critique). I favor this more than publishing the reviews. For one thing the reviews are made on the version that is often not final and there can be multiple rounds of review. Are we going to publish all the versions of the manuscript with the reviews? I think its too unwieldy. I also dont see the point of publishing promotion and tenure reviews. what is the goal? If someone doesnt make tenure at one place, this could hurt their chances at getting employment elsewhere. do we really want that?

      Delete
    2. I agree that review should be continuous, not end with publication. Jonathan Eisen makes this point nicely today in his blog post titled "Stop deifying 'peer review' of journal publications" http://phylogenomics.blogspot.com

      Delete
    3. Reviews take a lot of time- so why do we ever do them? My reward is that I (often) get to learn something new and important, prepublication. I have plenty of interesting things to review- I don't have to do it out of a sense of obligation. Editing's another matter, unfortunately...

      Delete
  10. Jean, I agree workload is problem for all of us, but my response would be that we have to self-regulate more - we have to decline to review some papers so we have time to write thoughtful comprehensive reviews and for dialog with authors, as is the current practice at all Frontiers journals, for instance. (In that case, reviewers remain anonymous until publication.) I don't think workload is a sufficient argument against creating a more fair process.

    I agree with several commenters that fear of confrontation is a real problem. Fear is a natural emotion, the function of which in nature is to paralyze. It works well when there's a tiger roaming around. Reviewers are not tigers. Sure, we feel threatened, but in civilized settings perhaps the following aphorism is helpful? "The only thing we have to fear is fear itself." Let's not let fear paralyze us into accepting and maintaining a primitive, unfair system. Let's create a civilized system based on mutual respect, trust, and openness. This is about evolving the culture of science toward a more fair, more respectful state.

    ReplyDelete
    Replies
    1. from Jean Greenberg
      i understand that we should self-regulate, and believe me I turn down plenty of requests and as an editor I try spread review requests around! But we all want high quality reviewers to judge our work and as an editor I want to get the best reviewers -I dont want to unduly burden reviewers by having authors contacting them.

      Anyway, do we need a one size fits all system? there are some journals where it is the culture to sign reviews. In cases where I have signed my reviews authors have contacted me. Have other people had that experience? do you find it a burden on your time?

      Delete
  11. I agree there is no need for 'one size fits all'. Journals have differing cultures and it's up to editorial boards to drive the evolution of science publishing culture and for administrators to drive the evolution of proposal reviewing and promotion assessment cultures.

    I have signed nearly all reviews of manuscripts for years. Occasionally authors contact me but usually through the review process, i.e., the editor. I've received emails today saying they have had a similar experience. NSF doesn't want us to sign reviews, so that's another situation. And universities redact letters when they provide them to candidates (except when they screw up, which I've seen them do...). Everyone institution has its own policy. Reviewers and authors can try to influence those policies and should. Ultimately those who wield the levers of power have the primary responsibility to effect positive change.

    ReplyDelete
  12. Science flourishes ONLY in the light. We trust scientists to attach their names to honestly produced data, in spite of powerful temptations to pick and choose results. Scientists should also provide honest reviews, to which they are happy to attach their names.
    If a young scientist feels unable to review something openly because of a perceived conflict of interest, they can refuse to review. If the same young scientist looks on a review as an opportunity to curry favor, attaching their name- as reviewer- to a favorable review of a piece of junk will provide the rest of the community with valuable information.
    All honest reviews are helpful- they improve the quality of the paper. The Frontiers model- in which there is facile back and forth communication between reviewer and author- is ideal.

    I also agree that mean-spirited reviews are often ascribed to the wrong reviewer! It happens to me all the time! I also get blamed for satiric anonymous plays, anonymous protest songs, etc- it's flattering but I'm just not that clever!

    ReplyDelete
  13. Why you make some good points, I fear your view is too utopian.

    As you know, NIH does publish the list of individuals on study sections. So you can look up and see who was on the committee, though you don't know who got your proposal. However, as there are 3 reviewers for every proposal, if there are only two plant biologists on a study section, it seems rather obvious who reviewed a plant proposal. I can't go into further detail due to the required confidentiality, but one time that I was on such a study section, a plant proposal received uniformly very bad reviews. I also can't explain how I know, but the PI has definitely remembered this slight and it has definitely caused me some problems. It has given me--and other people whose specialty is not well represented (for example, yeast biologists on a mammalian study section)--pause as to whether they wish to serve or not unless there are sufficient others to result in true anonymity.

    Maybe no one has told you that they have received retaliation for their Plant Cell editorships, but they may not actually know, as when they receive bad reviews on proposals or manuscripts, or failed to get an award that they don’t even know they were nominated for, they have no way of knowing who wrote the reviews or killed their nomination (unless some unauthorized leak from committee occurs….)

    Open reviews in publishing might work because the publishing rate is higher than the 10% funding rate now common at many granting agencies. Many perfectly good grant proposals go unfunded and decisions necessarily must therefore be arbitrary. I believe every good paper will be published--though sometimes not in as prestigious a journal as the authors might hope, or not without lengthy, sometimes unnecessary, extra experiments.

    I do agree with one responder that it is unfortunate that we can’t respond to absurd and incorrect grant reviews as we can to manuscript reviews. Good NSF program officers, by the way, do sometimes ask PIs for clarification about some issue that a reviewer has raised—this has sometimes allowed people to do indirect rebuttals to completely incorrect reviews. But the workload is so high at most granting agencies, this useful reform would be difficult to make. I also suspect that since there are many more worthwhile proposals than funding available for them, the agencies may not think it is important when proposals are unfairly knocked out of competition—there are still more than enough good ones left.

    I actually do believe that associate editors should not be anonymous. Editors are at least somewhat protected when they are the bearers of bad news—because they can point to the negative reviews that have been received. I do think this can improve the quality of editorial decisions. Unfortunately, what has been happening is that associate editors often do NOT do their job—they merely parrot what the reviews say, rather than reading them to see whether a reviewer is being unreasonable. Part of this is due to the press of time, all of us are overwhelmed. However, that can be ameliorated by reducing editors’ loads. Too many editors don’t examine the manuscripts at all and just pass along reviewers’ demands for far more additional experiments than are necessary, or reject manuscripts due to unduly harsh criticisms (perhaps due to imagined slights to the reviewer by one of the authors?) I do agree that something needs to be done about the increased nastiness in reviews to which the community is subjected. It should be made easier for Associate Editors to edit comments if they are inappropriate instead of just sending the unexpurgated versions to the authors. But with human nature what it is, and present state of severe competition, I don’t think we can abandon anonymity.

    Signed, Anonymous

    ReplyDelete
    Replies
    1. I think fear is once of the central problems. If we fear retaliation, we have lost the game before it has started. We should not fear retaliation we should take action to prevent it. That's the whole point of my post - remove anonymity not only from manuscript reviews, but also proposal and promotion reviews. Let the light shine on the whole process of evaluation. Allow rebuttals of proposal and promotion reviews. Set a high ethical standard that reviewers of all sorts are expected to meet - evidence-based, non-biased, professional assessments. And editors, administrators and program directors should reject reviews that do not meet standards. Doing so means more effort in obtaining high quality, professional, ethical reviews, but that is the price of fairness. Fairness should be our objective and unfair treatment should be prohibited and punished. Those who are too fearful to participate in a transparent, fair system that strives to protect all from retribution, well, I'm afraid I have to say, we don't want or need your reviews. When you're ready to let go of the fear, you will be welcomed back into the community with open arms, smiles and applause!

      Delete
    2. Rich,

      Thanks for opening this discussion with your blog. One of the difficulties I see with the current thread is that it tends to cast the issues in black and white when instead they tend toward grey-scale. You propose universal open review (pubs, grants etc). Others are concerned with retaliation (particularly for young investigators). You respond that universal open review would be self-correcting - someone retaliating for a bad review by unjustly savaging their prior reviewer would be recognized and punished by the community. The problem I see here is that the retaliation doesn't have to be obvious to the community (or even detectable by it). The last federal grant panel I was on had a funding rate in the single digits. You could torpedo a grant in such a situation not just by being critical of it but also by merely praising it faintly. In such situations retaliation is essentially invisible to the community.

      I also have a another concern. The peer review process has a secondary benefit - it teaches scientists to think critically. It takes practice to be able to read a colleague's ideas and identify their strengths and weaknesses. We get some training in this during our education but at a professional level the peer review process provides our practical training. I'm concerned that any system that intimidates younger scientists from participating (including the ones you say you don't "want or need") will weaken the scientific enterprise as a whole. As with most things I don't think either option (closed or open reviewer) is perfect - I think we're stuck trying to decide which is the lesser evil.

      Gregory P. Copenhaver
      (in the spirit of full disclosure I should mention that I'm the Deputy EiC of PloS Genetics)

      Delete
    3. Excellent points, Greg. The middle ground is certainly a place to look for solutions. Frontiers has one approach, summarized in the comment by Henry Markham, but I'm sure there are others, and that it would be good for journals to experiment with them.

      Another concern, raised by 'mikethemadbiologist' http://mikethemadbiologist.com/2012/02/06/why-h-r-3433-is-a-bad-bill-because-we-can-count-on-republicans-to-abuse-it/
      who appears to be anonymous, is that of politically motivated harassment. I think this is a valid concern, and one can easily imagine reviewers being harassed by a wide range of parties, from climate skeptics to animal rights activivists to anti-GMO activists (the list is potentially very long, covering large swathes of the political spectrum, left, right and center.

      I don't mean to minimize the concerns associated with disclosure of reviewers names. I acknowledge the risks. But I think we need to openly discuss how to resolve the real problems that already exist of retaliation by anonymous reviewers. The Frontiers approach is one good one, which is why I am involved with one of their journals. Another approach I put out for debate is what I've labeled 'true-peer' review (http://ibiosphere.blogspot.com/2012/02/true-peer-review-would-it-change.html) It is not a panacea either - there is almost certainly none available. I'm just suggesting possibilities as they arise and hoping to stimulate some thought and debate, and maybe better approaches will begin to be tried by some journals. At which point, authors will decide which models they prefer and publish in those journals.

      Delete
  14. Rich,

    Please consider the situation with letters of recommendation for persons applying for academic positions. When is the last time you saw a negative letter of recommendation for an academic job? A potential referee typically declines to write or writes a letter stretching the truth to explain how many centimeters (or meters) above the water a candidate typically walks. Why should we not expect the same pattern to emerge in reviews of academic papers?

    We considered these issues in some depth several times when I was Editor-in-Chief of Ecology and Ecological Monographs. We always came down on the side of wanting the reviewers to feel maximally comfortable giving an honest and in-depth appraisal of a paper, so each time we decided to allow a potential reviewer a choice between an anonymous or a signed review. It continues to become more difficult to obtain the desired number of quality reviewers, so we certainly do not wish to scare away potential objective reviewers.

    Robert Peet

    ReplyDelete
    Replies
    1. Robert, I don't know how to fix the recommendation problem. In point of fact, however, a number of journals now do publish reviewers' names (see Henry Markham comment below), and some even publish reviews and reviewers do write honest, in-depth reviews. Why wouldn't a reviewer be willing to sign an honest, in-depth review? If the ethical standard for the community of scientists becomes one that expects transparency of all, reviewers will be willing to sign their reviews. Those who aren't willing, simply won't be members of the community in good standing. I think we DO want to scare away those potential reviewers who feel they have to hide their identities. There are plenty of good reviewers available, it just takes a little more effort to find them, but even that is changing. Look at what is happening today with social media. People speak their minds publicly, with their identities known. That is the direction we are going and I doubt there is any turning back.

      Delete
  15. This is a great discussion - well done Rich in starting it off. I agree we need to phase out anonymous reviewing. I paraphrase the key comment above: OK, now what?! How can we move this forward? As a member of several editorial boards, I am finding it increasingly difficult to recruit referees. I fear that, for the journals that implemented this policy, they would find it even more difficult to recruit (unpaid) referees. As such, a significant level of co-ordinated change within the community might be necessary. Or do people think my conservative fears (as an editor) are unfounded?!

    Best wishes, Mark

    ReplyDelete
    Replies
    1. Mark,
      I apologize in advance for mostly repeating my comments above in reply to Anonymous, but I think they also apply here:
      I believe we need to start by setting a high ethical standard that reviewers of all sorts are expected to meet, and that is, evidence-based, non-biased, professional assessments of manuscripts, proposals, and promotions. Editors, administrators and program directors should reject reviews that do not meet standards. Doing so certainly means more effort in obtaining high quality, professional, ethical reviews - that has certainly been my experience - but that is simply the price of fairness. Fairness should be our objective and our standard, and unfair treatment should be prohibited and punished. Those who are too fearful to participate in a transparent, fair system that strives to protect all from retribution, well, I'm afraid I have to say, we don't want or need their reviews. We say to them: when you're ready to let go of the fear, you will be welcomed back into the community with open arms, smiles and applause!

      Delete
  16. The Frontiers journal series has thought about this issue for many years now and has actually done something about it. Most important for a comprehensive solution is that the rights of authors, reviewers and editors must all be balanced fairly.

    Firstly, the reviewer mandate is changed - focus on valid science and not subjective issues. This already solves many issues.

    Secondly, the reviewer is anonymous only for the first independent review phase. This is for the protection of the reviewer in case he/she wants to withdraw from the review. They also have that right and should be able to do so without any retribution. They can withdraw without a paper being rejected.

    Thirdly, if the reviewer does decide to stick it through into the second phase of the review (an interactive phase) then he/she must agree to to have their names published on the paper as the reviewers once the paper has been accepted. This serves the purpose of making referees responsible and constructive since they do not want to be associated with a bad paper. It also serves the purpose of acknowledging the reviewers for their hard work.

    Fourthly, there is no review voting in Frontiers - all reviewers and the editor in charge must agree to publish or reject. Unanimously. Most people thought this would never work, but in fact it works perfectly because it forces the authors, reviewers, and editors to reconcile differences. Frontiers provides a real-time interactive discussion forum to help all parties converge.

    The result of all this is that peer-review in Frontiers is becoming what it should always have been - a constructive and valuable process of enhancing the quality of science.

    Henry Markram
    www.frontiersin.org

    ReplyDelete
    Replies
    1. For full disclosure, Henry Markham above is head the Frontiers organization, and I am Chief Editor of one of its journals, Frontiers in Plant Genetics and Genomics.

      The diversity of views expressed above is great, and interesting. Robert, for instance, comes to a different conclusion than Henry, as do several others. Clearly the debate has not been settled yet and there is need for discussion by the community and experimentation by journals. Some journals not only remove anonymity, they publish reviews, which I would advocate. Frontiers believes it has found a useful middle ground, allowing anonymity to a point - if a paper is rejected the reviewers and editors remain anonymous to the authors. Perhaps that is a necessary step to get reviewers comfortable with their names being published before we take the next step, an important step in my view, of removing anonymity completely (and at the same time publishing the reviews). The more important point I was trying to make above is that we need to address the problem of anonymity not only for publications but also for grant proposals and promotions. That is the area where people fear they will be retaliated against by authors if they are not anonymous in writing reviews, and this valid concern is why we should consider removing anonymity also from grant reviews and performance reviews, which was the central point of my post.

      Delete
  17. Hereafter, commenters on this blog will have to identify themselves in order for their comments to appear here.
    Removing the veil of anonymity, one step at at time....

    ReplyDelete
  18. I like what you're doing here. I agree with the spirit of your piece and the idea that open, evidence-based, critiques serve science best. However, I am also aware of the problems, and don't think that we can or should ever completely abolish anonymity.

    As you know, I have authored non-anonymous reviews for Frontiers. I have also received reviews under the traditional system that were nevertheless signed (Jim Birchler likes to do that).

    I agree that a move towards greater transparency makes sense. I have heard many stories over the years that involve resentment towards specific people who are thought to have sabotaged a paper, a grant submission, a tenure package or even a career. "If it weren't for X, I would still be at Harvard, but now I'm waiting tables." or "If it weren't for Y, our paper would have been published first and I would have gotten the Nobel prize instead of the other guy." I'm also quite sure that most of these resentments are misdirected. People are overly sure of themselves when it comes to guessing the identity of reviewers. There are indeed many cases where transparency would clear the air.

    However, I cannot imagine an open system that truly protects young scientists from powerful people in their field. The powerful have feelings too, and I likewise know of real resentment on their part ("Who does he think HE is, rejecting OUR paper"). This poses a real danger. Transparency will never be universal and nothing can completely eliminate back-channel communication or private revenge. Driving the real discussion "offline" will not serve science either. This is the real point regarding what is referred to above as the "recommendation problem."

    So, I welcome the experiment, but think the optimal outcome would be a well-crafted hybrid rather than universal sunshine.

    ReplyDelete
  19. Steve,
    Glad you welcome the experiment, and many thanks for your contributions to Frontiers in Plant Genet Genom. I agree that the sticking point is protecting junior scientists from retribution by senior scientists. That seems to be the central concern of many. We need to find a middle ground, as you say. Frontiers' middle ground is to unveil anonymity only upon publication. That's one way.
    Here's an idea - instead or in addition. Let's call it 'true peer review'. It seems clear that many do not feel that a senior scientist is truly a peer of a junior scientist because there is a tremendous imbalance of power. What if reviewers of junior scientists were ONLY junior scientists? True peers. Just as a 'jury of one's peers' was originally intended to be. (Not that it is now... but we're returning to fundamental principle here.) Editors are generally more senior scientists (mid to late career) and should have no problem signing decision letters. What if such scientists were excluded from reviewing more junior scientists? What if all reviewers were true peers? Same fears, same career stage as the author/grant proposal/promotion candidate. That would seem to turn the tables and force more senior scientists to make their decisions based on true peer reviews ONLY.
    In that case, would more scientists support open review - non-anonymous review?

    ReplyDelete
  20. Thanks for starting this interesting discussion. I'm a more junior scientist myself, but I agree with the thrust of your argument. Still, I think that others may need a lot of convincing before signing on for complete anonymity. I like the sound of the Frontiers model as outlined by Henry Markram, but it still has the effect of removing anonymity outright. I also read your "true-peer" model, but I don't think that is really where the problem lies. Would I be correct in assuming that the lion's share of reviewing falls on junior scientists anyway (at least secondarily, after senior scientists pass it on to their students and postdocs)?

    Allow me to suggest another middle ground solution that would provide transparency:
    Assign each reviewer a unique (anonymous) identifier to be used across all journals, and make all of their reviews publicly accessible via a central website. That way, editors and authors can look up the reviewer to assess their reviewing record, including aggregate statistics on:
    - How much reviewing experience they have.
    - How positive/negative they have been in their reviews.
    - How well their scores agree with other reviewers.
    - The quality of their reviews in the opinions of the editors.
    - Any biases they have demonstrated over time (e.g. requiring that certain work be cited, perhaps their own).
    - The subsequent fate of the articles they have reviewed (accept/reject, citations?).

    I think that such a resource would be very useful for editors, in that it would allow them to calibrate each reviewer's recommendations. It would also preserve anonymity for those who wish to maintain it. After enough reviews, perhaps it would become obvious who a given reviewer is, but perhaps at that stage anonymity won't matter to the reviewer. Another nice feature of this model is that it would give reviewers the chance to point to their body of reviewing work if they so wish to. Perhaps if a reviewer could point to a large body of reviews with positive stats, that could become a positive consideration for hiring, tenure, recruitment to editorial boards, etc.

    Shaun Mahony, CSAIL, MIT

    ReplyDelete
    Replies
    1. I'm not sure that the lion's share of reviewing falls on junior scientists, but if it does, then 'true-peer' review wouldn't increase workload much. As far as senior (and junior) scientists passing manuscripts on to students and postdocs, ethically that has to be disclosed (including name(s) disclosed to the editor) and generally permission should be obtained from the editor, at least for most journals I am familiar with. This is important relative to conflict of interest policies, for instance.

      To your main point, I agree that unique identifiers are an interesting idea, assuming journals will adopt the system and share information about reviews, which however they are loathe to do. Many journals treat their reviewer lists as proprietary. Others publish the names of all reviewers at the end of the year, but don't disclose any information about quality of reviewer performance. It would be interesting if some journals experimented with your proposal. It would indeed be a nice way for reviewers' to document their service.

      Delete
  21. Great idea! I totally agree with that! But I have a question, what would happen to the reviewers in case of retaliation? Would the journal stay behind the reviewer? Would it "protect" him?
    Many thanks!
    André (IPK)

    ReplyDelete
    Replies
    1. Good question. It seems to me that sanctions should be as severe as for plagiarism. If a journal's policy is non-anonymity, it would have to protect reviewers to extent it can. Actually, it is employers who have the greatest ability to punish offenses like plagiarism. A journal can prohibit an offender from publishing in their journal, but not much else, except informing the institution the person works at. The institution does indeed take action, sometimes severe action in such cases. It could mean losing a tenured position. So, institutions need to have clear policies on this. Probably they already have rules that could be used to severely punish someone taking retribution on another scientist, but perhaps these need to be more explicit, I don't know. Proving the offense would be more difficult than in the case of plagiarism, perhaps more like proving that data are fraudulent, but I have no doubt the our institutions can develop the expertise to prosecute serious violations. A rhetorical question: Will some people 'get away with it'? Sure, people get away with murder. That doesn't mean we should allow murder just because we can't always catch the murderer.

      Delete
  22. These are good suggestions. One caution from the corporate world: it is possible that universal open review could lead to increased lawsuits when, for example, someone is denied tenure or promotion because of a negative review. This sort of thing has had a chilling effect on the content and honesty of personal recommendations in the private sector. It would be good to have a plan for dealing with this possibility.

    ReplyDelete
    Replies
    1. Hi Paul,
      I agree it is a problem. One that already exists even with 'confidentiality' because a court can subpoena confidential letters. It could become more of a problem with transparency. It's a good question as to how we might protect reviewers and also get honest critical professional reviews. It seems to me that if a review is written professionally and is based on the science there should not be a problem. In theory, authors already could challenge decision letters in the legal realm, but they don't, as far as I know. And I doubt that a court would tell a journal they have to publish a paper it doesn't want to publish. Should be the same for reviewers, it seems to me. Promotion is subject to employment law so that is indeed a different environment that may preclude the possibility of non-anonymous promotion and tenure letters. It should not be a problem for grant proposals any more than it is for publications, I would think.

      Delete
  23. At the very least this is an experiment that needs to be tried. I really don't think the system at it stands is working. Reviews range from critical, helpful analyses of the work to brief summaries that make you question wether the reviewer even bothered to read the paper.

    It seems the strongest argument in favour of anonymity is to protect reviewers. But without doing the experiment we really don't know if an honest yet critical review of a paper would actually harm anyone's career. It's an argument based on fear of an event that may never occur. While I accept that there are people who may abuse the knowledge of their reviewers identity, my experience is that most in the scientific community actually appreciate a really good solid review of their work - even if it is negative.

    Suggesting that some might use that information to abuse or harm a reviewer is to ascribe a behaviour to the author that I don't see much evidence of in our various communities. And yes I know we've all ranted about reviewers, but there is a difference between ranting in the lab and actually taking the step of harming someone's career.

    So as a negative we have the possibility of poor behaviour by authors. But as a positive we have the effect of having the reviewer knowing their name will be attached to the review for all to see. I believe that will have the effect of encouraging people to do the best job they can when they review. I think you are right Rich, making the names of reviewers (and their reviews) public will improve the quality of the reviews. That has to be a good thing.

    ReplyDelete
    Replies
    1. You make an important point that this would first be done as an experiment, as some journals already are in various different ways. It never has to be imposed. Authors can choose which journals to submit to and reviewers can choose which to review for. Choice is important, and in time it may be become apparent what the community's preferences are and what works best.

      Delete
  24. From Jean Greenberg:

    I mentioned this blog to a colleague today and she wondered about whether authors should be anonymous to reviewers. I know some journals do this...In practice I think you can often figure out whose group the article is from, but I wonder how it fits with the points being discussed here.

    ReplyDelete
  25. I've also had several emails making this point about 'double-blinding' so that authors would also be anonymous, and I've heard it proposed a number of times over the years, but I always run up against the problem that it is usually pretty obvious who the lead author is - they cite their own work, especially in methods, they have a certain point of view, etc. Much easier than guessing the reviewer. We know authors guess reviewer IDs incorrectly most of the time, and reviewers can actively try to disguise their identity by saying things that would throw an author off the track (perhaps that's why authors guess wrong so often?).

    The real problem it seems to me is fear. Authors are afraid of reviewers and reviewers are afraid of authors. Right now there is an imbalance - reviewers have the upper hand. It shouldn't be this way. We need to fix it. My feeling remains that no system will be ideal but openness is the most ethical choice available.

    ReplyDelete
  26. Peer review shouldn't be the only system in place. While it is certainly effective for people within the system it is as restrictive as any guild system for people outside said system.

    Dissertation Peer Review

    ReplyDelete