Peer
review is far from perfect, but the media presenting material that has
not been through review as accepted science is misleading
In my experience quite a lot of the misunderstandings over research, and the criticisms leveled at science and scientists seems to stem from an ignorance of some relatively basic aspects of scientific work. The Lost Worlds has not really been going long enough for me to have a great handle on the audience I have here, but I do hope that it has a broad appeal and that a good proportion of the readers are those without much of a science background. Even those that do may not have a great concept on how many things are done – it wasn't until my PhD that I really got a proper understanding on how scientific papers are published and the differences between say reviewed papers and book chapters, and I don't think it was even mentioned while I was working on my BSc. Naturally I can't say much about the vast majority of science degrees in the U.K., but I wouldn't be surprised if there's a good number of graduates who, like me, were never really introduced to aspects of science like getting papers published and applying for grants. These issues and others are things I want to cover on here as I think that people having an idea about them and how they work will give people a better appreciation of science and importantly, provide a basis for avoiding bad science.
Let's take peer review. In the last few
months there have been a couple of incidents where the popular media
have leaped on ideas about extinct reptiles and promoted them as, well
if not quite true, then certainly highly credible. Huge numbers of
people would have seen IO9s coverage of 'vampire pterosaurs' and others
would have heard the BBC's radio coverage of the 'aquatic dinosaurs'
idea (which was also covered in plenty of other places too). In both
cases, palaeontologists responded quickly and pointed out the huge holes
in the proposals and lack of scientific data supporting them (like here, here and here).
However, quite how many people saw those rebuttals is another question
and I'm sure there's a good number of people out there who saw these
things presented as credible science and still labour under the
misapprehension that they are mainstream views and quite correct. The
question is raised therefore as to why were news organisations promoting
such nonsense in the first place, and could they have known that they
were not credible?
The short answer is yes. While a quick call to an expert would have revealed the problem, the fact that neither had been through peer review should have had alarm bells ringing.
Now I'm not pretending that peer review is brilliant, it has plenty of problems. Enough bad papers get through and even the best researchers occasionally make major mistakes or miss those of others. However, the fact is that it does at least provide a minimum amount of screening of scientific research and hypotheses, and ideas that have not passed it should be treated with greater suspicion than those that have. With the examples listed above, one idea was published in a popular science magazine (i.e. one full of articles aimed at the public, rather than technical, scientific publications) and the other was a formal conference abstract (though again, not reviewed) and backed by a personal website. I suspect it might be obvious to even non-scientists that neither sound like the right kind of venue to present (allegedly) important new research to the scientific community.
So, what is peer review? Quite simply, when a researcher submits a manuscript to a technical scientific journal, the editors will check the paper for basic competence and then send it onto a number of expert referees (generally two, sometimes many more) to check the actual science. These people are generally experts in the field in question (i.e. a paper on tyrannosaur teeth might be sent to a tyrannosaur specialist and a dinosaur tooth specialist) and they submit reports to the journal about the quality of the work. Is all the necessary information in there? Were the right analyses done in the right way? Is the science repeatable? Is it backed by the right evidence? Are the conclusions supported by the data? Is this original research with proper credit to previous studies? and so on. Assuming the paper is considered suitable, these reports are passed back to the author to make changes suggested by the referees or indeed to challenge them if he disagrees. The paper may go through several rounds of review and ultimately be rejected if it is not considered acceptable and might have to be binned or started again. Those who have tried to get a paper published will know that a great many scientific referees are both thorough and unforgiving and a manuscript is regularly improved by the feedback provided by referees.
This means that a paper that has got into a peer-reviewed journal has at least got past an expert editor and a couple of referees. It might still be rubbish, but has gained the blessing of at least a couple of people in the field who do know the subject well. Contrast this with an article submitted to a popular magazine where the limit to the input of an editor might be to suggest rewriting a few sentences. One can see why giving media coverage to the latter as likely being accurate science on the par with reviewed research might be, at best, unwise.
I would hope that readers can also appreciate how things can be seen by the researchers when their work is covered in the media. It can take months to get reviews back from journals and there may be pages and pages of comments from each of two referees and the editor. Arguments and modifications may go back and forth and additional rounds of review undergone or additional data collected and analyses revised to get things up to scratch. Years can literally pass, you go through debates and discussions with colleagues, refine your arguments and augment your evidence and see your paper eventually published. A major news outlet picks it up and the first thing you see at the bottom is a comment from some smart-assed troll asking how this nonsense ever got published or suggesting some 'obvious' flaw in the analysis. One gets the strong impression that they think researchers can simply write whatever they want in a paper, send it to the journal and a few weeks later it appears in print, and thus one could write almost anything and see it published.
That is of course exactly what can happen with something that is not peer-reviewed, hence it is most important that journalists covering science recognize the difference between the two, as indeed should the audience. But it's this lack of recognition for the process that means the mainstream media can seriously push the idea that all dinosaurs were aquatic and stuck in the water based on a scientific paper that was nothing of the kind.
The short answer is yes. While a quick call to an expert would have revealed the problem, the fact that neither had been through peer review should have had alarm bells ringing.
Now I'm not pretending that peer review is brilliant, it has plenty of problems. Enough bad papers get through and even the best researchers occasionally make major mistakes or miss those of others. However, the fact is that it does at least provide a minimum amount of screening of scientific research and hypotheses, and ideas that have not passed it should be treated with greater suspicion than those that have. With the examples listed above, one idea was published in a popular science magazine (i.e. one full of articles aimed at the public, rather than technical, scientific publications) and the other was a formal conference abstract (though again, not reviewed) and backed by a personal website. I suspect it might be obvious to even non-scientists that neither sound like the right kind of venue to present (allegedly) important new research to the scientific community.
So, what is peer review? Quite simply, when a researcher submits a manuscript to a technical scientific journal, the editors will check the paper for basic competence and then send it onto a number of expert referees (generally two, sometimes many more) to check the actual science. These people are generally experts in the field in question (i.e. a paper on tyrannosaur teeth might be sent to a tyrannosaur specialist and a dinosaur tooth specialist) and they submit reports to the journal about the quality of the work. Is all the necessary information in there? Were the right analyses done in the right way? Is the science repeatable? Is it backed by the right evidence? Are the conclusions supported by the data? Is this original research with proper credit to previous studies? and so on. Assuming the paper is considered suitable, these reports are passed back to the author to make changes suggested by the referees or indeed to challenge them if he disagrees. The paper may go through several rounds of review and ultimately be rejected if it is not considered acceptable and might have to be binned or started again. Those who have tried to get a paper published will know that a great many scientific referees are both thorough and unforgiving and a manuscript is regularly improved by the feedback provided by referees.
This means that a paper that has got into a peer-reviewed journal has at least got past an expert editor and a couple of referees. It might still be rubbish, but has gained the blessing of at least a couple of people in the field who do know the subject well. Contrast this with an article submitted to a popular magazine where the limit to the input of an editor might be to suggest rewriting a few sentences. One can see why giving media coverage to the latter as likely being accurate science on the par with reviewed research might be, at best, unwise.
I would hope that readers can also appreciate how things can be seen by the researchers when their work is covered in the media. It can take months to get reviews back from journals and there may be pages and pages of comments from each of two referees and the editor. Arguments and modifications may go back and forth and additional rounds of review undergone or additional data collected and analyses revised to get things up to scratch. Years can literally pass, you go through debates and discussions with colleagues, refine your arguments and augment your evidence and see your paper eventually published. A major news outlet picks it up and the first thing you see at the bottom is a comment from some smart-assed troll asking how this nonsense ever got published or suggesting some 'obvious' flaw in the analysis. One gets the strong impression that they think researchers can simply write whatever they want in a paper, send it to the journal and a few weeks later it appears in print, and thus one could write almost anything and see it published.
That is of course exactly what can happen with something that is not peer-reviewed, hence it is most important that journalists covering science recognize the difference between the two, as indeed should the audience. But it's this lack of recognition for the process that means the mainstream media can seriously push the idea that all dinosaurs were aquatic and stuck in the water based on a scientific paper that was nothing of the kind.
No comments:
Post a Comment