tag:blogger.com,1999:blog-7285260803381805722.comments2023-06-01T03:47:50.198-07:00iBiosphere ScienceRich Jorgensenhttp://www.blogger.com/profile/08713562874646987566noreply@blogger.comBlogger55125tag:blogger.com,1999:blog-7285260803381805722.post-21143190873856959062015-06-30T00:13:20.080-07:002015-06-30T00:13:20.080-07:00Thanks for the support. We'll let you know whe...Thanks for the support. We'll let you know when this system is applied (hopefully very soon)<br />Thomas Guillemaud<br />Inra, Sophia Antipolis, FranceAnonymoushttps://www.blogger.com/profile/02973643668063556878noreply@blogger.comtag:blogger.com,1999:blog-7285260803381805722.post-36961684868941597772015-06-26T09:39:28.297-07:002015-06-26T09:39:28.297-07:00Excellent points. Certainly a big step in the righ...Excellent points. Certainly a big step in the right direction and I would support that. Rich Jorgensenhttps://www.blogger.com/profile/08713562874646987566noreply@blogger.comtag:blogger.com,1999:blog-7285260803381805722.post-74470801680244685892015-06-26T09:14:32.661-07:002015-06-26T09:14:32.661-07:00There are several problems with the current review...There are several problems with the current review process. Some of them apply to <br />1-the Editors who expect reviews of better quality to ensure a good quality of the Editoral process and fair decisions.<br />2-the authors who can be hurt by unfair reviewers or think that the reviews are not of acceptable standards to properly evaluate their work<br />3-the readers who have no tangible elements to evaluate the quality of the review process because it is hidden<br /><br />Anonymity of reviewers and the resulting bad quality of reviews is a problem linked to the 2 first points listed above. It can be solved by the Editors at the cost of more Editor's work. After all, it is the duty of the Editor in Chief to ensure a fair and good quality review process. As the author of this blog wrote 3 years ago, as an editor of The Plant Cell he "dealt with [unprofessional reviews] in various ways. One was to reject a review that contained unprofessional statements and force the reviewer to rewrite the review, and if that request was refused, we disqualified the review."<br />This additional work of the editor may thus solve problems of unfair or bad quality reviews due to anonymity, simply by asking additional reviews to other reviewers. However it does not solve the 3rd point listed above: the lack of confidence of readers in the review process because this process lacks transparency and is totally hidden for readers.<br /><br />I would therefore propose the transparency of the review process by publishing the reviews, the editors' decisions, the authors' replies and the corrected versions of the articles (for accepted papers). This may be positive in various ways: readers would understand why a paper has been accepted for publication, reviewers would write reviews of good quality because they know that they will be published (even anonymously), and Editors would base their decision on good quality reviews only because their decision and the reviews will be readable by everyone.<br /><br />And this does not requires signed reviews. <br /><br />Abandoning anonymity of reviewers may generate other problems than the problem of retaliation discussed by the author of this blog. Signing reviews can also lead to the following unwanted effects: 1) increasing the number of undue flattering reviews; 2) biased sampling of scientists accepting to review papers towards those extroverts who enjoy showing themselves, and 3) towards the old scientists who already have important position. <br /><br />Thomas Guillemaud<br />Inra, Sophia Antipolis, France<br /><br />Anonymoushttps://www.blogger.com/profile/02973643668063556878noreply@blogger.comtag:blogger.com,1999:blog-7285260803381805722.post-55371136631232330222015-04-13T15:03:39.084-07:002015-04-13T15:03:39.084-07:00Thanks, Steen. I agree legislation is suboptimal. ...Thanks, Steen. I agree legislation is suboptimal. But it will happen if we aren't more proactive. What do we do when research institutions refuse to provide an acceptable summary of an investigation? Can we ban the institution and all its scientists from publishing in journals? That would have almost the force of law. Other ideas?Rich Jorgensenhttps://www.blogger.com/profile/08713562874646987566noreply@blogger.comtag:blogger.com,1999:blog-7285260803381805722.post-62196811660830571622015-04-13T14:57:30.594-07:002015-04-13T14:57:30.594-07:00I'm in favor of public reports, but not legal ...I'm in favor of public reports, but not legal obligations. It seems like publishing retractions and expressions of concern (as in this <a href="http://publicationethics.org/files/u7140/Fabricated%20data%20B_0.pdf" rel="nofollow">COPE flowchart</a>) will remain the best way of enforcing any new convention.Anonymoushttps://www.blogger.com/profile/07189759690825645003noreply@blogger.comtag:blogger.com,1999:blog-7285260803381805722.post-58053394971484775462015-04-13T10:32:20.555-07:002015-04-13T10:32:20.555-07:00I am also a big fan of post-publication review. Au...I am also a big fan of post-publication review. Authors should be able to publish and let qualified (validated) reviewers comment, or choose to submit for pre-publication review if that is there preference. Your proposal is also an option. I think we need lots of meaningful discussion and serious efforts at reform. The traditional system is demonstrably broken, and out of sync with the real world of the 21st century. Physics and math are far ahead of biology, of course. Rich Jorgensenhttps://www.blogger.com/profile/08713562874646987566noreply@blogger.comtag:blogger.com,1999:blog-7285260803381805722.post-67702637331193935002015-04-13T10:20:54.818-07:002015-04-13T10:20:54.818-07:00Rich,
as an editor what do you think of a system ...Rich,<br /><br />as an editor what do you think of a system where:<br /><br />1- Papers are submitted to pre-print server like biorxiv, arxiv etc.<br /><br />2- paper is open peer reviewed with reviewers signing reviews and getting public recognition for their time and work.<br /><br />3- changes and/or more experiments are done in response to reviewer comments/suggestion.<br /><br />4- paper is finally submitted to journal of choice alongside reviews and response to reviews and then editor can make choice of publishing without any further review.<br /><br />5- alternatively to # 4, editors might decide to choose an article that they like and invite authors to submit to their journal.<br /><br />Rubén.Rubenhttps://www.blogger.com/profile/17608071542951164575noreply@blogger.comtag:blogger.com,1999:blog-7285260803381805722.post-4299338274335327702015-04-11T17:20:25.761-07:002015-04-11T17:20:25.761-07:00I agree there are risks, but the magnitude of thos...I agree there are risks, but the magnitude of those is probably no greater than the risks of allowing anonymity. I think what we need to do is develop solutions to mitigate the risks of open review. One possibility is the 'true-peer review' suggestion I made in a post here in 2012. I'd love to hear more ideas. We are a creative community and I'm sure if we try we can come up with good ideas. Rich Jorgensenhttps://www.blogger.com/profile/08713562874646987566noreply@blogger.comtag:blogger.com,1999:blog-7285260803381805722.post-33783354011563516742015-04-11T17:14:15.431-07:002015-04-11T17:14:15.431-07:00I think the abuses of anonymity are, overall, far ...I think the abuses of anonymity are, overall, far worse than the negative aspects and risks of full openness in review. However, I accept that there are still risks and negative aspects of full openness and thus am not ready to say every review should be fully open ... I would love that ... but I still think some of the risks still may need to be reduced ..Jonathan Eisenhttps://www.blogger.com/profile/07953790938128734305noreply@blogger.comtag:blogger.com,1999:blog-7285260803381805722.post-38511198074469804502013-01-21T01:50:35.567-07:002013-01-21T01:50:35.567-07:00Peer review shouldn't be the only system in pl...Peer review shouldn't be the only system in place. While it is certainly effective for people within the system it is as restrictive as any guild system for people outside said system.<br /><br /><a href="http://dissertationpeerreview.com/" rel="nofollow">Dissertation Peer Review</a>Rohit Kaushikhttps://www.blogger.com/profile/14544435258118872290noreply@blogger.comtag:blogger.com,1999:blog-7285260803381805722.post-39080131322342325632012-02-07T08:03:10.809-07:002012-02-07T08:03:10.809-07:00I've also had several emails making this point...I've also had several emails making this point about 'double-blinding' so that authors would also be anonymous, and I've heard it proposed a number of times over the years, but I always run up against the problem that it is usually pretty obvious who the lead author is - they cite their own work, especially in methods, they have a certain point of view, etc. Much easier than guessing the reviewer. We know authors guess reviewer IDs incorrectly most of the time, and reviewers can actively try to disguise their identity by saying things that would throw an author off the track (perhaps that's why authors guess wrong so often?). <br /><br />The real problem it seems to me is fear. Authors are afraid of reviewers and reviewers are afraid of authors. Right now there is an imbalance - reviewers have the upper hand. It shouldn't be this way. We need to fix it. My feeling remains that no system will be ideal but openness is the most ethical choice available.Rich Jorgensenhttps://www.blogger.com/profile/08713562874646987566noreply@blogger.comtag:blogger.com,1999:blog-7285260803381805722.post-8791710216961697122012-02-06T21:47:16.833-07:002012-02-06T21:47:16.833-07:00You make an important point that this would first ...You make an important point that this would first be done as an experiment, as some journals already are in various different ways. It never has to be imposed. Authors can choose which journals to submit to and reviewers can choose which to review for. Choice is important, and in time it may be become apparent what the community's preferences are and what works best.Rich Jorgensenhttps://www.blogger.com/profile/08713562874646987566noreply@blogger.comtag:blogger.com,1999:blog-7285260803381805722.post-25780238648508845732012-02-06T20:54:57.204-07:002012-02-06T20:54:57.204-07:00From Jean Greenberg:
I mentioned this blog to a c...From Jean Greenberg:<br /><br />I mentioned this blog to a colleague today and she wondered about whether authors should be anonymous to reviewers. I know some journals do this...In practice I think you can often figure out whose group the article is from, but I wonder how it fits with the points being discussed here.thebathttps://www.blogger.com/profile/08736987553097520234noreply@blogger.comtag:blogger.com,1999:blog-7285260803381805722.post-47754613110822539622012-02-06T20:54:54.875-07:002012-02-06T20:54:54.875-07:00At the very least this is an experiment that needs...At the very least this is an experiment that needs to be tried. I really don't think the system at it stands is working. Reviews range from critical, helpful analyses of the work to brief summaries that make you question wether the reviewer even bothered to read the paper.<br /><br />It seems the strongest argument in favour of anonymity is to protect reviewers. But without doing the experiment we really don't know if an honest yet critical review of a paper would actually harm anyone's career. It's an argument based on fear of an event that may never occur. While I accept that there are people who may abuse the knowledge of their reviewers identity, my experience is that most in the scientific community actually appreciate a really good solid review of their work - even if it is negative.<br /><br />Suggesting that some might use that information to abuse or harm a reviewer is to ascribe a behaviour to the author that I don't see much evidence of in our various communities. And yes I know we've all ranted about reviewers, but there is a difference between ranting in the lab and actually taking the step of harming someone's career.<br /><br />So as a negative we have the possibility of poor behaviour by authors. But as a positive we have the effect of having the reviewer knowing their name will be attached to the review for all to see. I believe that will have the effect of encouraging people to do the best job they can when they review. I think you are right Rich, making the names of reviewers (and their reviews) public will improve the quality of the reviews. That has to be a good thing.Bart Janssenhttps://www.blogger.com/profile/17427129555177716757noreply@blogger.comtag:blogger.com,1999:blog-7285260803381805722.post-90185718749892783432012-02-06T13:03:01.200-07:002012-02-06T13:03:01.200-07:00Excellent points, Greg. The middle ground is certa...Excellent points, Greg. The middle ground is certainly a place to look for solutions. Frontiers has one approach, summarized in the comment by Henry Markham, but I'm sure there are others, and that it would be good for journals to experiment with them. <br /><br />Another concern, raised by 'mikethemadbiologist' http://mikethemadbiologist.com/2012/02/06/why-h-r-3433-is-a-bad-bill-because-we-can-count-on-republicans-to-abuse-it/<br />who appears to be anonymous, is that of politically motivated harassment. I think this is a valid concern, and one can easily imagine reviewers being harassed by a wide range of parties, from climate skeptics to animal rights activivists to anti-GMO activists (the list is potentially very long, covering large swathes of the political spectrum, left, right and center. <br /><br />I don't mean to minimize the concerns associated with disclosure of reviewers names. I acknowledge the risks. But I think we need to openly discuss how to resolve the real problems that already exist of retaliation by anonymous reviewers. The Frontiers approach is one good one, which is why I am involved with one of their journals. Another approach I put out for debate is what I've labeled 'true-peer' review (http://ibiosphere.blogspot.com/2012/02/true-peer-review-would-it-change.html) It is not a panacea either - there is almost certainly none available. I'm just suggesting possibilities as they arise and hoping to stimulate some thought and debate, and maybe better approaches will begin to be tried by some journals. At which point, authors will decide which models they prefer and publish in those journals.Rich Jorgensenhttps://www.blogger.com/profile/08713562874646987566noreply@blogger.comtag:blogger.com,1999:blog-7285260803381805722.post-25950368500340582602012-02-06T12:19:54.819-07:002012-02-06T12:19:54.819-07:00Rich,
Thanks for opening this discussion with you...Rich,<br /><br />Thanks for opening this discussion with your blog. One of the difficulties I see with the current thread is that it tends to cast the issues in black and white when instead they tend toward grey-scale. You propose universal open review (pubs, grants etc). Others are concerned with retaliation (particularly for young investigators). You respond that universal open review would be self-correcting - someone retaliating for a bad review by unjustly savaging their prior reviewer would be recognized and punished by the community. The problem I see here is that the retaliation doesn't have to be obvious to the community (or even detectable by it). The last federal grant panel I was on had a funding rate in the single digits. You could torpedo a grant in such a situation not just by being critical of it but also by merely praising it faintly. In such situations retaliation is essentially invisible to the community.<br /><br />I also have a another concern. The peer review process has a secondary benefit - it teaches scientists to think critically. It takes practice to be able to read a colleague's ideas and identify their strengths and weaknesses. We get some training in this during our education but at a professional level the peer review process provides our practical training. I'm concerned that any system that intimidates younger scientists from participating (including the ones you say you don't "want or need") will weaken the scientific enterprise as a whole. As with most things I don't think either option (closed or open reviewer) is perfect - I think we're stuck trying to decide which is the lesser evil.<br /><br />Gregory P. Copenhaver<br />(in the spirit of full disclosure I should mention that I'm the Deputy EiC of PloS Genetics)Gregory Copenhaverhttps://www.blogger.com/profile/16690831328373253854noreply@blogger.comtag:blogger.com,1999:blog-7285260803381805722.post-83808044863183245452012-02-06T12:09:35.037-07:002012-02-06T12:09:35.037-07:00I'm not sure that the lion's share of revi...I'm not sure that the lion's share of reviewing falls on junior scientists, but if it does, then 'true-peer' review wouldn't increase workload much. As far as senior (and junior) scientists passing manuscripts on to students and postdocs, ethically that has to be disclosed (including name(s) disclosed to the editor) and generally permission should be obtained from the editor, at least for most journals I am familiar with. This is important relative to conflict of interest policies, for instance. <br /><br />To your main point, I agree that unique identifiers are an interesting idea, assuming journals will adopt the system and share information about reviews, which however they are loathe to do. Many journals treat their reviewer lists as proprietary. Others publish the names of all reviewers at the end of the year, but don't disclose any information about quality of reviewer performance. It would be interesting if some journals experimented with your proposal. It would indeed be a nice way for reviewers' to document their service.Rich Jorgensenhttps://www.blogger.com/profile/08713562874646987566noreply@blogger.comtag:blogger.com,1999:blog-7285260803381805722.post-22958678046061803742012-02-06T12:00:43.050-07:002012-02-06T12:00:43.050-07:00Hi Paul,
I agree it is a problem. One that already...Hi Paul,<br />I agree it is a problem. One that already exists even with 'confidentiality' because a court can subpoena confidential letters. It could become more of a problem with transparency. It's a good question as to how we might protect reviewers and also get honest critical professional reviews. It seems to me that if a review is written professionally and is based on the science there should not be a problem. In theory, authors already could challenge decision letters in the legal realm, but they don't, as far as I know. And I doubt that a court would tell a journal they have to publish a paper it doesn't want to publish. Should be the same for reviewers, it seems to me. Promotion is subject to employment law so that is indeed a different environment that may preclude the possibility of non-anonymous promotion and tenure letters. It should not be a problem for grant proposals any more than it is for publications, I would think.Rich Jorgensenhttps://www.blogger.com/profile/08713562874646987566noreply@blogger.comtag:blogger.com,1999:blog-7285260803381805722.post-83668529268066758312012-02-06T11:18:12.139-07:002012-02-06T11:18:12.139-07:00These are good suggestions. One caution from the ...These are good suggestions. One caution from the corporate world: it is possible that universal open review could lead to increased lawsuits when, for example, someone is denied tenure or promotion because of a negative review. This sort of thing has had a chilling effect on the content and honesty of personal recommendations in the private sector. It would be good to have a plan for dealing with this possibility.Paul K. Wolberhttps://www.blogger.com/profile/02097714280768618308noreply@blogger.comtag:blogger.com,1999:blog-7285260803381805722.post-25326982659364123652012-02-06T08:07:16.992-07:002012-02-06T08:07:16.992-07:00Good question. It seems to me that sanctions shoul...Good question. It seems to me that sanctions should be as severe as for plagiarism. If a journal's policy is non-anonymity, it would have to protect reviewers to extent it can. Actually, it is employers who have the greatest ability to punish offenses like plagiarism. A journal can prohibit an offender from publishing in their journal, but not much else, except informing the institution the person works at. The institution does indeed take action, sometimes severe action in such cases. It could mean losing a tenured position. So, institutions need to have clear policies on this. Probably they already have rules that could be used to severely punish someone taking retribution on another scientist, but perhaps these need to be more explicit, I don't know. Proving the offense would be more difficult than in the case of plagiarism, perhaps more like proving that data are fraudulent, but I have no doubt the our institutions can develop the expertise to prosecute serious violations. A rhetorical question: Will some people 'get away with it'? Sure, people get away with murder. That doesn't mean we should allow murder just because we can't always catch the murderer.Rich Jorgensenhttps://www.blogger.com/profile/08713562874646987566noreply@blogger.comtag:blogger.com,1999:blog-7285260803381805722.post-55544022280272846972012-02-06T06:43:56.261-07:002012-02-06T06:43:56.261-07:00I agree, Rich. I also think that when we become f...I agree, Rich. I also think that when we become fixated on the big trendy result that would get into Science or Cell, it's easy to forget what we actually think is an important result, rather than what some editor at a journal thinks. And that sucks the life out of good science.Unknownhttps://www.blogger.com/profile/18316888023629755098noreply@blogger.comtag:blogger.com,1999:blog-7285260803381805722.post-35415900217329162832012-02-06T01:19:11.089-07:002012-02-06T01:19:11.089-07:00Great idea! I totally agree with that! But I have ...Great idea! I totally agree with that! But I have a question, what would happen to the reviewers in case of retaliation? Would the journal stay behind the reviewer? Would it "protect" him?<br />Many thanks!<br />André (IPK)Dabitz66https://www.blogger.com/profile/10517591919618956614noreply@blogger.comtag:blogger.com,1999:blog-7285260803381805722.post-47913245488114974122012-02-05T21:43:31.750-07:002012-02-05T21:43:31.750-07:00Thanks for starting this interesting discussion. I...Thanks for starting this interesting discussion. I'm a more junior scientist myself, but I agree with the thrust of your argument. Still, I think that others may need a lot of convincing before signing on for complete anonymity. I like the sound of the Frontiers model as outlined by Henry Markram, but it still has the effect of removing anonymity outright. I also read your "true-peer" model, but I don't think that is really where the problem lies. Would I be correct in assuming that the lion's share of reviewing falls on junior scientists anyway (at least secondarily, after senior scientists pass it on to their students and postdocs)?<br /><br />Allow me to suggest another middle ground solution that would provide transparency: <br />Assign each reviewer a unique (anonymous) identifier to be used across all journals, and make all of their reviews publicly accessible via a central website. That way, editors and authors can look up the reviewer to assess their reviewing record, including aggregate statistics on:<br /> - How much reviewing experience they have. <br /> - How positive/negative they have been in their reviews.<br /> - How well their scores agree with other reviewers.<br /> - The quality of their reviews in the opinions of the editors. <br /> - Any biases they have demonstrated over time (e.g. requiring that certain work be cited, perhaps their own).<br /> - The subsequent fate of the articles they have reviewed (accept/reject, citations?).<br /><br />I think that such a resource would be very useful for editors, in that it would allow them to calibrate each reviewer's recommendations. It would also preserve anonymity for those who wish to maintain it. After enough reviews, perhaps it would become obvious who a given reviewer is, but perhaps at that stage anonymity won't matter to the reviewer. Another nice feature of this model is that it would give reviewers the chance to point to their body of reviewing work if they so wish to. Perhaps if a reviewer could point to a large body of reviews with positive stats, that could become a positive consideration for hiring, tenure, recruitment to editorial boards, etc. <br /><br />Shaun Mahony, CSAIL, MITShaunhttps://www.blogger.com/profile/16550461306175388443noreply@blogger.comtag:blogger.com,1999:blog-7285260803381805722.post-6595662875805188032012-02-05T19:08:40.223-07:002012-02-05T19:08:40.223-07:00Steve,
Glad you welcome the experiment, and many ...Steve, <br />Glad you welcome the experiment, and many thanks for your contributions to Frontiers in Plant Genet Genom. I agree that the sticking point is protecting junior scientists from retribution by senior scientists. That seems to be the central concern of many. We need to find a middle ground, as you say. Frontiers' middle ground is to unveil anonymity only upon publication. That's one way. <br />Here's an idea - instead or in addition. Let's call it 'true peer review'. It seems clear that many do not feel that a senior scientist is truly a peer of a junior scientist because there is a tremendous imbalance of power. What if reviewers of junior scientists were ONLY junior scientists? True peers. Just as a 'jury of one's peers' was originally intended to be. (Not that it is now... but we're returning to fundamental principle here.) Editors are generally more senior scientists (mid to late career) and should have no problem signing decision letters. What if such scientists were excluded from reviewing more junior scientists? What if all reviewers were true peers? Same fears, same career stage as the author/grant proposal/promotion candidate. That would seem to turn the tables and force more senior scientists to make their decisions based on true peer reviews ONLY. <br />In that case, would more scientists support open review - non-anonymous review?Rich Jorgensenhttps://www.blogger.com/profile/08713562874646987566noreply@blogger.comtag:blogger.com,1999:blog-7285260803381805722.post-82551725279961236572012-02-05T18:53:47.497-07:002012-02-05T18:53:47.497-07:00I like what you're doing here. I agree with t...I like what you're doing here. I agree with the spirit of your piece and the idea that open, evidence-based, critiques serve science best. However, I am also aware of the problems, and don't think that we can or should ever completely abolish anonymity. <br /><br />As you know, I have authored non-anonymous reviews for Frontiers. I have also received reviews under the traditional system that were nevertheless signed (Jim Birchler likes to do that). <br /><br />I agree that a move towards greater transparency makes sense. I have heard many stories over the years that involve resentment towards specific people who are thought to have sabotaged a paper, a grant submission, a tenure package or even a career. "If it weren't for X, I would still be at Harvard, but now I'm waiting tables." or "If it weren't for Y, our paper would have been published first and I would have gotten the Nobel prize instead of the other guy." I'm also quite sure that most of these resentments are misdirected. People are overly sure of themselves when it comes to guessing the identity of reviewers. There are indeed many cases where transparency would clear the air. <br /><br />However, I cannot imagine an open system that truly protects young scientists from powerful people in their field. The powerful have feelings too, and I likewise know of real resentment on their part ("Who does he think HE is, rejecting OUR paper"). This poses a real danger. Transparency will never be universal and nothing can completely eliminate back-channel communication or private revenge. Driving the real discussion "offline" will not serve science either. This is the real point regarding what is referred to above as the "recommendation problem."<br /><br />So, I welcome the experiment, but think the optimal outcome would be a well-crafted hybrid rather than universal sunshine.Stevehttps://www.blogger.com/profile/15264977010144529019noreply@blogger.com