How to Review a Paper: A guide for newcomers and a refresher for the experienced

Davis, Hugh

English

EdShare

10/11/2011 12:17Reviewing a PaperPage 1 of 5http://users.ecs.soton.ac.uk/hcd/reviewing.htmlHow to Review a Paper:A guide for newcomers and a refresher for theexperienced.V2.0 16th Jan 2007Hugh Davis www.ecs.soton.ac.uk/~hcd(Please feel free to reproduce this document with acknowledgement, and sendsuggestions for improvement to the author)IntroductionReviewing a paper is a difficult skill that takes practice to acquire. The purpose is:to inform the programme committee or editors whether the paper is suitablefor publication in their venueto identify any faults with the current paper (technical and presentational)for you to confirm that in your opinion this is original work (not plagiarised,and with results that have not been previously published) to give constructive feedback to the authors so they can improve their paperAnswering the question of whether work is suitable for publication in a particularvenue requires you to understand something about the standards and aims of thejournal, conference or workshop concerned. Clearly the expectations for significanceof results, degree of originality of results, and completion of evaluation will be verydifferent for a workshop or small conference, compared to a major internationaljournal.  However no venue would expect to publish work that is flawed orplagiarised.Please also note the requirement for *constructive* criticism. Let us start with theassumption that most work has been conducted by other hard working and honestpeople like ourselves who are proud of the work they have done. If you don't thinkthe work is ready for publication, then explain why, and tell them how useful it willbe when suitably improved; don't 'diss' them! The main reason why papers areturned down is not because the work is flawed, but because it's value has beenbadly communicated or it has been submitted to an inappropriate venue.�Desirable traits in a referee include objectivity, fairness, speed, professionalism,confidentiality, honesty, and courtesy� (Parberry, 1989)What goes into a reviewTo start with, you should assume that everything you say in a review will be shown10/11/2011 12:17Reviewing a PaperPage 2 of 5http://users.ecs.soton.ac.uk/hcd/reviewing.htmlto the authors - and it is good practice to assume that the authors will find out whodid the review - so make sure you would be happy to defend your review to theauthors! There may be some things that should not go to the authors, and we willcover this later, under comments to the committee/editors1. Summarise the paper in your own words.In affect you should write the abstract you would have written if you had writtenthe paper. Clearly this abstract should say what the authors have done, what theircontribution is, and what  the significance of that contribution is (*however smallyou think the contribution is you should still do this*). The point here is for you toshow you have read and understood the point of the paper.There are many sorts of papers including technical papers, review papers, thoughtexperiments, experiences papers and evaluations. It is important that youunderstand what sort of paper this claims to be, and that you show that you havereviewed it with that understanding. Note that many review forms tend to assumethat all papers are straight technical papers, which may well not be the case inter-disciplinary areas such as Technology Enhanced Learning. You may have to fightagainst the constraints of the review form, and comment on this to thecommittee/editors.2. "Technical Content"Remember to keep it constructive! If you think they have not explained somethingfully, then ask the question, or explain that you could not understand. Don't just tellthem they have done it wrong.  Here you might examine:the extent to which the work is relevant to this workshop/conference/journalthe extent to which the work answers a valid research questionthe research methodologythe quality of results and argumentation (any suspected flaws)the evaluationawareness of related work (including the  correct number of appropriatereferences)the degree of significance of the resultsdo the conclusions follow from the work described?3. Writing and PresentationThis section is concerned with the quality of the writing.Can you understand the development of the argument?Does the abstract describe what you read (and did you know what you weregoing to read about when you had read the abstract)Do the intro and conclusions tell a story on their own?Have the authors kept to the format and length requirements of the10/11/2011 12:17Reviewing a PaperPage 3 of 5http://users.ecs.soton.ac.uk/hcd/reviewing.htmlpublishers?Are the diagrams and figures readable?Are the references and citations formatted properly?Are the references "fully formed"? (Are all the authors listed? Is all theinformation about the venue that would be needed to retrieve the paper listed?The list should not include references to commercial web-sites. Academic Webreferences should preferably include author, title and date published, and/ordate last accessed)4. TyposIt is good manners to produce a table of any grammatical errors, typos and otherminor textual problems that you have spotted. But it's not the reviewer's job tocopy edit the paper, so don't go out of your way to look for typos. As a rule ofthumb I would not expect to list more than ten such errors. If there are many moreI would say something like "There are multiple typos that need serious attentionbefore publication". If the English throughout the paper is poor, then you must sayso, but do be charitable. Presumably the problem is that English is not the authors'native language; they need to be advised to get help before submitting the finalversion.5. ConclusionsIn the conclusions you should:answer the questions "Will it interest people who attend this conference/ readthis publication? Why? Who?"justify your overall recommendation.6. Overall RecommendationMost review forms will have some tick list or Lickert Scale for you to use e.g.Accept in its present form with no revisionsAccept after minor revisions (re-review unnecessary)Accept after major revisions (after re-review)Reject but encourage re-submission in another form (e.g short paper, poster)RejectOR1. Excellent - This paper is amongst the best papers I have ever read (short-listfor best paper award)2. Very good paper (Consider short listing for best paper award)3. Sound paper - I recommend acceptance4. Borderline - This paper could be accepted if there is room5. Poor - This paper has limited contribution, or the work is not yet ready for10/11/2011 12:17Reviewing a PaperPage 4 of 5http://users.ecs.soton.ac.uk/hcd/reviewing.htmlpublication. I do not believe it should be accepted, but if other reviewers differ,I would not oppose strongly6. Unacceptable - The work makes no contribution or, worse, it is flawed orscurrilous. I believe that publication of this paper would reflect badly on ourcommunity. I would strongly oppose any other outcome.These are both good sets of words to consider if there is no particular responsemode prescribed.To the Editors/Committeeyou can assume that the authors will see all the responses above. Most reviewforms also provide a space for "Comments to the committee". If they do not, thenthese comments will typically go in an email to the editors/committee.This is NOT the place for "sneak attacks"! If you have anything to say about thecontent of the paper, you should be happy to say it to the authors.This place is where you can tell the committee/editors about your own reflection onthe quality of your review.What is your expertise in the subject area?How confident you are of your views of the paper?How much time did you put into this review?Did you actually check any equations and data presented? If not, say so.Are there any potential conflicts of interest you may have?Acknowledge anyone who helped you with the review.This is also the place where you might explain any difficulty you had in conformingto the review form template they have presented. For example you may onoccasions decide to recommend acceptance of a paper that is generally quite poorbecause you believe it would spark important debate in the community; or you maythink that it is an excellent review of the state of the art that you would like to seepublished in spite of the fact that it has no original technical content.Finally, this is the place to present any evidence you have of unprofessionalism onthe part of the authors. If you have evidence that they have plagiarised, fabricatedresults or otherwise mis-represented their work - or maliciously mis-representedthe work of others, then you should say so here. You will need to prove yourassertions.Some Useful ReferencesDale J. Benos, Kevin L. Kirk and John E. Hall. How to Review a Paper. Advan.Physiol. Edu. 27: 47-52, 2003;http://advan.physiology.org/cgi/content/full/27/2/47 (Last accessed 16th Jan 2007)Ralph E. Johnson, Kent Beck, Grady Booch, William Cook, Richard Gabriel, Rebecca10/11/2011 12:17Reviewing a PaperPage 5 of 5http://users.ecs.soton.ac.uk/hcd/reviewing.htmlWirfs-Brock. 1993 How to Get a Paper Accepted at OOPSLA.http://www.acm.org/sigplan/oopsla/oopsla96/how93.html (last accessed 16th Jan2007)Ian Parberry A Guide for New Referees in Theoretical Computer Science. ACMSIGACT News (Nov 1989)  Available at http://portal.acm.org/citation.cfm?id=74090Barak A. Pearlmutter. How to Review a Scientific Paper.http://www.cs.unm.edu/~bap/how-to-review.html (last accessed 16th Jan 2007)   

How to Review a Paper

Abstract

Similar works

Full text

Available Versions

EdShare