Comments to my earlier post, "Universal Open Review?", indicate that a major sticking point is protecting junior scientists from retribution by senior scientists. This seems to be the central concern of many. So, we need to find a middle ground. Here's an idea. Let's call it 'TRUE-PEER review'. Clearly, many scientists do not feel that a senior scientist is truly a peer of a junior scientist given that there is potentially a tremendous imbalance of power. So, what if reviewers of junior scientists were ONLY junior scientists? True peers. Just as a 'jury of one's peers' was originally intended to be. (Not that it is now... but we're returning to fundamental principle here...)
Editors are generally more-senior scientists (mid to late career) and typically have no problem signing decision letters. Such scientists always say they already have enough work to do, and we could also tell them they have enough power already, so why not simply exclude them from reviewing more junior scientists' manuscripts, proposals, and promotions?
What if all reviewers were true peers? Reviewers with the same concerns, at same career stage, as the author/grant proposal/promotion candidate is. That would seem to turn the tables and force more senior ('non-peer') scientists to act only as editors, program managers, and administrators on manuscripts, proposals and promotions of more junior scientists, and to force their decisions to be based ONLY on true-peer reviews. In that case, would more scientists support open review, i.e., non-anonymous review?
I can imagine some of the objections to excluding reviews from senior non-peers. But are such objections sufficient to override the inherent fairness of 'True-Peer' review? In the interest of fairness might it not more ethical to base career-making and career-breaking decisions on true-peer reviews?
Sunday, February 5, 2012
Friday, February 3, 2012
Universal open review?
There has been discussion in recent years about opening up review of scientific articles by bringing down the artificial wall of anonymity behind which reviewers are able to hide. A common complaint of authors is that anonymity results in careless, even rude, and sometimes unethical statements by reviewers, and it is argued that removing anonymity would greatly reduce unscientific and careless criticism because reviewers would be more careful if they had to stand behind their statements with their name and reputation. A common concern raised about removing anonymity is that reviewers could be subject to unfair retaliation by authors who are not happy about negative reviews.
To the first point, I would like to describe my observations based on my experience as editor in chief of The Plant Cell (2003-2007). Yes, I will be explaining "how the sausage is made"... Before 2003, the practice at TPC had been that editors handling manuscripts were always anonymous, just as reviewers were. Instead of editors signing decision letters, the Managing Editor, a professional editor who was not a scientist, edited and signed letters drafted by the handling editor, who was a recognized expert in the field of the submitted manuscript. The reason stated in support of this practice was that editors, especially early career editors, needed this protection in order not to suffer retaliation by peers and senior colleagues, especially regarding grant proposals, but also regarding manuscript and promotion decisions.
However, simple inspection of a random sample of decision letters indicated to me that editors were too frequently careless and cavalier with their statements and did not often enough explain clearly the basis for their decision. It was easy to see that authors would be offended by such letters, and moreover that the reputation of the journal would necessarily suffer as a result.
Based on this, I felt it was imperative that editors sign their decision letters and stand by their decisions and statements with their names and their reputations. So, prior to assuming my role as EiC, I informed the editorial board that henceforth editors would have to sign each decision letter and that I expected the same high level of professionalism in decision letters as we expected of our authors in their manuscripts. Some editors resigned, and I happily appointed new editors who were willing to sign their decisions. The result was striking - the quality and professionalism of decision letters rose dramatically. In four and a half years at TPC I never received a single complaint from an editor that s/he felt retaliated against because of a publication decision. I believe that, in general, scientists appreciate and respect well-argued publication decisions, even if they disagree with them, and that removing anonymity, if done thoughtfully, will not necessarily cause rampant problems of retaliation, i.e, authors' greatest fears will not be realized.
I think it was also important that I told the board I was willing to work with them on their decisions and letters and that I would support them and stand behind their decisions as long as they were scientifically solid and were explained clearly and professionally to authors. When authors challenged a decision, I was always available to work with the editor to see if there was a scientific basis to the challenge that should cause us to reconsider, and I was available to work with the editor on writing a professional response to the author indicating that we had considered the author's challenge thoughtfully. It was always expected that in case of a 'decline' decision editors would explain to authors what would be necessary for publication in TPC, such as additional experiments or more rigorous interpretations at a level of standard that the plant biology community expected for articles published in TPC. I do believe that it requires effort on the part of editors (and reviewers) to prepare the quality of decision letters (and reviews) that authors deserve. Authors have generally worked very hard on their manuscripts prior to submission, and even if there may be flaws, editors and reviewers should also be willing to work hard to prepare a fair, substantive, scientific and professional decision letters and reviews. After all, this is only way the system is going to work well.
Although I personally wanted to remove the anonymity of reviewers, I was afraid the community was not yet ready for that in 2003. Because we allowed reviewers to retain anonymity, we continued to have problems with unprofessional reviews. We dealt with this in various ways. One was to reject a review that contained unprofessional statements and force the reviewer to rewrite the review, and if that request was refused, we disqualified the review. Of course, we generally chose not to invite recalcitrant reviewers in future. I believe that most good journals operate this way (but I am really only familiar with practices at TPC during my tenure there).
I am in favor of removing anonymity of reviewers, but because of the second concern mentioned at the outset - that of potential retaliation by angry authors - I think the change should be made at several levels and that as a community we should embrace the following principle: No academic scientific reviewer should have anonymity, regardless of whether the review addresses a manuscript, a grant proposal or a promotion or tenure decision. All academic scientists should be willing to stand behind their opinions regarding manuscripts, proposals and performance appraisals. Transparency should be the order of the day!
In my view, 'universal transparency' would greatly dampen the possibility of retaliation because all major opinions affecting a scientist's ability to be a scientist would be made without the possibility of hiding behind a veil of anonymity. I also believe based on my experience that scientists can generally be trusted to appreciate and respect well argued publication decisions even if they disagree with them, and that the likelihood of retaliation will be no greater, and in fact less, than is already the case.
In fact, as we all know, anonymity comes with an unintended consequence of increasing the disappointment and sometimes anger of the person being reviewed. Too often that anger is misdirected at someone the person guesses was responsible for a negative appraisal but actually was not. I saw this many times at TPC: authors usually guess wrongly and aim their wrath at the wrong person. In addition, people they think are their allies are often their strongest critics. For instance, when authors suggest a reviewer, it is not at all uncommon that the proposed reviewer is the most critical reviewer of the manuscript, even recommending outright rejection. The point here is simply that authors often guess wrongly and so anonymity causes grief for 'innocent bystanders'.
Would universal transparency solve all our problems? Of course not. This is not a perfect world, and human nature being what it is, we will still have problems. Less problems, however, in my opinion. And problems that mainly can be addressed directly and openly by those whose opinions differ. I would even go so far as to suggest that we should consider anonymity to be unethical, given that I believe the alternative, transparency, is more fair.
Anonymity is not a sacred cow. We should drive a stake through its heart. And once we do, we will be glad we did.
P.S. See this new post, suggesting "True-Peer Review":
http://ibiosphere.blogspot.com/2012/02/true-peer-review-would-it-change.html
To the first point, I would like to describe my observations based on my experience as editor in chief of The Plant Cell (2003-2007). Yes, I will be explaining "how the sausage is made"... Before 2003, the practice at TPC had been that editors handling manuscripts were always anonymous, just as reviewers were. Instead of editors signing decision letters, the Managing Editor, a professional editor who was not a scientist, edited and signed letters drafted by the handling editor, who was a recognized expert in the field of the submitted manuscript. The reason stated in support of this practice was that editors, especially early career editors, needed this protection in order not to suffer retaliation by peers and senior colleagues, especially regarding grant proposals, but also regarding manuscript and promotion decisions.
However, simple inspection of a random sample of decision letters indicated to me that editors were too frequently careless and cavalier with their statements and did not often enough explain clearly the basis for their decision. It was easy to see that authors would be offended by such letters, and moreover that the reputation of the journal would necessarily suffer as a result.
Based on this, I felt it was imperative that editors sign their decision letters and stand by their decisions and statements with their names and their reputations. So, prior to assuming my role as EiC, I informed the editorial board that henceforth editors would have to sign each decision letter and that I expected the same high level of professionalism in decision letters as we expected of our authors in their manuscripts. Some editors resigned, and I happily appointed new editors who were willing to sign their decisions. The result was striking - the quality and professionalism of decision letters rose dramatically. In four and a half years at TPC I never received a single complaint from an editor that s/he felt retaliated against because of a publication decision. I believe that, in general, scientists appreciate and respect well-argued publication decisions, even if they disagree with them, and that removing anonymity, if done thoughtfully, will not necessarily cause rampant problems of retaliation, i.e, authors' greatest fears will not be realized.
I think it was also important that I told the board I was willing to work with them on their decisions and letters and that I would support them and stand behind their decisions as long as they were scientifically solid and were explained clearly and professionally to authors. When authors challenged a decision, I was always available to work with the editor to see if there was a scientific basis to the challenge that should cause us to reconsider, and I was available to work with the editor on writing a professional response to the author indicating that we had considered the author's challenge thoughtfully. It was always expected that in case of a 'decline' decision editors would explain to authors what would be necessary for publication in TPC, such as additional experiments or more rigorous interpretations at a level of standard that the plant biology community expected for articles published in TPC. I do believe that it requires effort on the part of editors (and reviewers) to prepare the quality of decision letters (and reviews) that authors deserve. Authors have generally worked very hard on their manuscripts prior to submission, and even if there may be flaws, editors and reviewers should also be willing to work hard to prepare a fair, substantive, scientific and professional decision letters and reviews. After all, this is only way the system is going to work well.
Although I personally wanted to remove the anonymity of reviewers, I was afraid the community was not yet ready for that in 2003. Because we allowed reviewers to retain anonymity, we continued to have problems with unprofessional reviews. We dealt with this in various ways. One was to reject a review that contained unprofessional statements and force the reviewer to rewrite the review, and if that request was refused, we disqualified the review. Of course, we generally chose not to invite recalcitrant reviewers in future. I believe that most good journals operate this way (but I am really only familiar with practices at TPC during my tenure there).
I am in favor of removing anonymity of reviewers, but because of the second concern mentioned at the outset - that of potential retaliation by angry authors - I think the change should be made at several levels and that as a community we should embrace the following principle: No academic scientific reviewer should have anonymity, regardless of whether the review addresses a manuscript, a grant proposal or a promotion or tenure decision. All academic scientists should be willing to stand behind their opinions regarding manuscripts, proposals and performance appraisals. Transparency should be the order of the day!
In my view, 'universal transparency' would greatly dampen the possibility of retaliation because all major opinions affecting a scientist's ability to be a scientist would be made without the possibility of hiding behind a veil of anonymity. I also believe based on my experience that scientists can generally be trusted to appreciate and respect well argued publication decisions even if they disagree with them, and that the likelihood of retaliation will be no greater, and in fact less, than is already the case.
In fact, as we all know, anonymity comes with an unintended consequence of increasing the disappointment and sometimes anger of the person being reviewed. Too often that anger is misdirected at someone the person guesses was responsible for a negative appraisal but actually was not. I saw this many times at TPC: authors usually guess wrongly and aim their wrath at the wrong person. In addition, people they think are their allies are often their strongest critics. For instance, when authors suggest a reviewer, it is not at all uncommon that the proposed reviewer is the most critical reviewer of the manuscript, even recommending outright rejection. The point here is simply that authors often guess wrongly and so anonymity causes grief for 'innocent bystanders'.
Would universal transparency solve all our problems? Of course not. This is not a perfect world, and human nature being what it is, we will still have problems. Less problems, however, in my opinion. And problems that mainly can be addressed directly and openly by those whose opinions differ. I would even go so far as to suggest that we should consider anonymity to be unethical, given that I believe the alternative, transparency, is more fair.
Anonymity is not a sacred cow. We should drive a stake through its heart. And once we do, we will be glad we did.
P.S. See this new post, suggesting "True-Peer Review":
http://ibiosphere.blogspot.com/2012/02/true-peer-review-would-it-change.html
Subscribe to:
Posts (Atom)