As far as I know the peer reviewers are in most cases now selected by the editor, they self-select to respond, are not paid for their work, and the process for alarmingly many journals is not even blind. I always thought that this makes the process vulnerable to network effects in the field, since people are obliged to a certain etiquette when commenting on established figures in their own field. So yes, I get where you are coming from, but similar to the scientific method, peer review is also great to describe in theory, in practice it would require much more precise protocols, like Web protocols I might say. I really don’t want to be a pessimist about science in the current political climate, but if we want these great ideals (Scientific method, Peer Reviewed evidence) we will have to abandon the existing situation as soon as possible.
agreed the existing system is deeply flawed and currently on a trajectory to critical failure.
regarding peer review itself, this is another point. people regard peer review as this binary thing which takes place prior to publication and is like a box which is ticked after publication.
which is ofc ridiculous, peer review is an ongoing process, meaning many of the important parts take place after publication. fortunately this does happen in a variety of fields and situations, however not being the norm leads to a number of the issues in discussion. further it creates an erroneous mindset that simply because something has been published that its now fully vetted, which is ofc absurd.
also agreed, the process should be blind. i believe it often already means the reviewer’s identities are hidden, but i also agree the authours should be hidden during the process too.
don’t see the role as unpaid being a problem though, introducing money would complicate things alot and create even more conflicts of interest and undermine what little integrity the process still has.
i really love your idea of standardising the process in a network-like protocol. this would actually make an excellent RFC and i’d totally support that.
on a similar vein, this is why i’ve been advocating for a complete restructuring of support given to reproduction. as you mentioned, the current process is vulnerable to a variety of human network effects. and among other issues with that problem, i also see the broken reproduction system playing a role here.
as it currently stands, reviewers can request more explanation or data, introduction of changes/additional caveats etc or reject the paper entirely. what this means is a reviewer can only really gauge whether something sounds right, or plausible. and as you correctly identify, certain personalities or flavours of prevailing culture will play a role in the reviewer’s assessment of what merely seems like it’s plausible or correct etc. this has shown to make major breakthroughs more difficult to communicate and face unfair resistance, which has frankly held back society at large.
whereas if there was an organised system of reproduction it’s no longer left to just a matter of opinion in how something sounds. this is ofc how its supposed to work already, and sometimes does, but all too often does not. imo it would be a great detail to include in your idea for a protocol-based review process.
i don’t envision this as always being something which must take place prior to publication, it can and should be an ongoing process. where papers could have their classification formerly upgraded over time. currently the only ‘upgrade’ a paper really receives is publicity or number of citations. the flaws of which are yet another discussion again.
Yes, that would be great. People put so much stock in peer review because there is the myth that every statement undergoes under a rigorous process of verification in multiple laboratories. The reality is, as you said, there is a culture of active discouragement of reproduction and the pushing of novel results.
Not to mention that to foster reproductions, researchers should be trained into a culture of replication and collective metanalyses. As it is now, reproductions are less than an afterthought for the vast majority of researchers, and virtually none knows how to handle multiple replicatory studies instead of p-hacking.
As far as I know the peer reviewers are in most cases now selected by the editor, they self-select to respond, are not paid for their work, and the process for alarmingly many journals is not even blind. I always thought that this makes the process vulnerable to network effects in the field, since people are obliged to a certain etiquette when commenting on established figures in their own field. So yes, I get where you are coming from, but similar to the scientific method, peer review is also great to describe in theory, in practice it would require much more precise protocols, like Web protocols I might say. I really don’t want to be a pessimist about science in the current political climate, but if we want these great ideals (Scientific method, Peer Reviewed evidence) we will have to abandon the existing situation as soon as possible.
agreed the existing system is deeply flawed and currently on a trajectory to critical failure.
regarding peer review itself, this is another point. people regard peer review as this binary thing which takes place prior to publication and is like a box which is ticked after publication.
which is ofc ridiculous, peer review is an ongoing process, meaning many of the important parts take place after publication. fortunately this does happen in a variety of fields and situations, however not being the norm leads to a number of the issues in discussion. further it creates an erroneous mindset that simply because something has been published that its now fully vetted, which is ofc absurd.
also agreed, the process should be blind. i believe it often already means the reviewer’s identities are hidden, but i also agree the authours should be hidden during the process too.
don’t see the role as unpaid being a problem though, introducing money would complicate things alot and create even more conflicts of interest and undermine what little integrity the process still has.
i really love your idea of standardising the process in a network-like protocol. this would actually make an excellent RFC and i’d totally support that.
on a similar vein, this is why i’ve been advocating for a complete restructuring of support given to reproduction. as you mentioned, the current process is vulnerable to a variety of human network effects. and among other issues with that problem, i also see the broken reproduction system playing a role here.
as it currently stands, reviewers can request more explanation or data, introduction of changes/additional caveats etc or reject the paper entirely. what this means is a reviewer can only really gauge whether something sounds right, or plausible. and as you correctly identify, certain personalities or flavours of prevailing culture will play a role in the reviewer’s assessment of what merely seems like it’s plausible or correct etc. this has shown to make major breakthroughs more difficult to communicate and face unfair resistance, which has frankly held back society at large.
whereas if there was an organised system of reproduction it’s no longer left to just a matter of opinion in how something sounds. this is ofc how its supposed to work already, and sometimes does, but all too often does not. imo it would be a great detail to include in your idea for a protocol-based review process.
i don’t envision this as always being something which must take place prior to publication, it can and should be an ongoing process. where papers could have their classification formerly upgraded over time. currently the only ‘upgrade’ a paper really receives is publicity or number of citations. the flaws of which are yet another discussion again.
Yes, that would be great. People put so much stock in peer review because there is the myth that every statement undergoes under a rigorous process of verification in multiple laboratories. The reality is, as you said, there is a culture of active discouragement of reproduction and the pushing of novel results.
Not to mention that to foster reproductions, researchers should be trained into a culture of replication and collective metanalyses. As it is now, reproductions are less than an afterthought for the vast majority of researchers, and virtually none knows how to handle multiple replicatory studies instead of p-hacking.