Rating articles in PLoS ONE
As you may remember, last summer we released the data on ratings, notes, comments and trackbacks on PLoS ONE to the community for analysis and Euan Adie, Deepak Singh and Cameron Neylon wrote their takes on the statistics as a whole, not particularly separating the ratings from the rest of the analysis. Last week, I also wrote a general post about the reasons to post ratings, notes and comments, but today I’d like to focus specifically on Ratings.
It has been almost two years since Ratings were introduced to Ambra (the publishing system that first debuted with the launch of PLoS ONE and will soon be home to all the PLoS journals). The more computer minded among you might be interested to know that Ambra is built on top of Topaz (the application that stores data in a combination of a semantic database – which holds all the metadata and a digital repository – which stores images, and xml files etc).
If you go to articles in PLoS ONE you will often see that readers have rated an article, sometimes leaving a short commentary along with it. The articles are rated on three criteria: Insight, Reliability and Style. Check out the Rating guidelines for more details about the criteria.
The ratings are prominently displayed on the right side-bar of each article and, since the most recent upgrade, posting the Ratings is easier than ever. Also, if you are one of the authors of the paper or a part of the same group, or Academic Editor or reviewer of the article, you should declare the Conflict of Interest when you rate that particular article. Adding a brief comment explaining your Rating is a nice additional touch.
As I check all the new notes, comments and ratings on PLoS ONE articles daily, I have noticed a pattern about ratings that does not readily apply to notes or comments. It is quite common to see one of our readers post ratings on a series of articles in the same general area of science within a very brief time span of just a few minutes or hours, often rating articles that are quite old (as much as any article in such a young journal can be called “old”). I have been trying to figure out what kind of behavior is consistent with this rating pattern. What I think is happening is this:
Many people download the PDFs of articles and print them out at the time they get published (or later, discovered via searches). The print-outs are then deposited in a pile of papers on one’s desk until they are really needed – the time when the researcher is writing his/her own manuscript and is reading in great depth and detail all the articles that can serve as potential references for their paper. This is the time when they are fully engrossed in the literature and also the time when they are most likely to come back to our site and provide quick ratings to all the articles they have just read.
Why should you do the same – go back and rate the articles, old and new, that you have just carefully read? Think of it as an integral part of being a scientist – informing and educating your colleagues and students (as well as journalists, bloggers and lay audience) about the quality of the papers. As you know, manuscripts submitted to PLoS ONE are peer-reviewed for the quality of work and writing, not for unpredictable and subjective criteria like novelty that can be only ascertained by the community after some time has passed. So, once some time has passed and you are a member of the community, think of it as your role to participate in post-publication discussion.
As Liz wrote last year, you should rate articles:
1. To help out your colleagues and yourself – if you’ve spent a considerable amount of time reading an article, it makes sense to spend two minutes rating it at the end. That way next time you are trying to figure out whether to spend time reading an article, hopefully someone you trust will have rated it and shared their opinion helping you to make a faster decision. It’s the “what goes around comes around” philosophy.
2. To build a critical mass of opinion – the next time you see an article with one rating and you think that this is not enough information on which to decide its importance and you read it, add your rating to it.
3. To change the way science is done – if you believe in a fairer, more open system where researchers decide the importance of what they read then please contribute to making this a success.
4. To build yourself a solid online reputation – in the interests of transparency, ratings are not anonymous (although it’s worth knowing that it is only the ‘user name’, not the user’s real name or their email, that is publicly disclosed). and that’s to the good for thoughtful contributors who should have the confidence to share their opinions (even if others publicly disagree there are rules of engagement on the site and a moderator to enforce them). In the future of scientific and medical publishing, where we hope open access is the gold standard, different ways of evaluating a person’s contribution to science will emerge and this could become one of them.
As the scientific (and academic in general) world is slowly but surely moving away from flawed metrics like the Impact Factor towards new article-level measures, it is of great importance for all the members of the scientific community to become active participants in the evaluation of individual articles and our Ratings are one of those potentially important measures.
If an article has only one or two (or zero) ratings right now, you should add your own. This is a long-term process. A scientific article is not like a blog post where most of the commentary appears during the first day or two after publication. Scientific literature is not ephemeral like that – as more and more research goes on over the years, there is more and more information that can be used to evaluate past articles. And then, consider it your duty to do so. Over the years, as the article accumulates more and more ratings, their combined scores will become more and more meaningful as a metric to evaluate its importance and quality.
So, as Chris Surridge wrote on the day we introduced Ratings:
Never read a paper on PLoS ONE without leaving a rating