Thank you for volunteering your time to review for ICCV 2019! To maintain a high-quality technical program, we rely very much on the time and expertise of our reviewers. This document explains what is expected of all members of the Reviewing Committee for ICCV 2019.
In addition to the guidelines below, you should read this ICCV 2019 Reviewer Tutorial for a summary of the decision process, annotated examples of good/bad reviews, and tips. You may also be interested in the ICCV 2019 Area Chair Tutorial to give you an overview of the process from the Area Chairs’ point of view.
|Paper Submission Deadline||March 22, 2019|
|Papers Assigned to Reviewers||April 19, 2019|
|Reviews Due||May 31, 2019|
|Start of Post-Rebuttal Discussion Period||June 26, 2019|
|Final Recommendations Due||July 3, 2019|
|Decisions Released to Authors||July 22, 2019|
Author Guidelines have instructed authors to make reasonable efforts to hide their identities, including omitting their names, affiliations, and acknowledgments. This information will of course be included in the published version. Likewise, reviewers should make all efforts to keep their identity invisible to the authors.
With the increase in popularity of arXiv preprints, sometimes the authors of a paper may be known to the reviewer. Posting to arXiv is NOT considered a violation of anonymity on the part of the authors, and in most cases, reviewers who happen to know (or suspect) the authors’ identity can still review the paper as long as they feel that they can do an impartial job. An important general principle is to make every effort to treat papers fairly whether or not you know (or suspect) who wrote them. If you do not know the identity of the authors at the start of the process, DO NOT attempt to discover them by searching the Web for preprints.
Please read the FAQ at the end of this document for further guidelines on how arXiv prior work should be handled.
Check your papers
As soon as you get your reviewing assignment, please go through all the papers to make sure that (a) there is no obvious conflict with you (e.g., a paper authored by your recent collaborator from a different institution) and (b) you feel comfortable to review the paper assigned. If either of these issues arise, please let us know right away by emailing the Program Chairs (firstname.lastname@example.org).
Please read the Author Guidelines carefully to familiarize yourself with all official policies (such as double submission and plagiarism). If you think a paper may be in violation of one of these policies, please contact the Program Chairs. In the meantime, proceed to review the paper assuming no violation has taken place.
What to Look For
Each paper that is accepted should be technically sound and make a contribution to the field. Look for what's good or stimulating in the paper. We recommend that you embrace novel, brave concepts, even if they have not been tested on many datasets. For example, the fact that a proposed method does not exceed the state-of-the-art accuracy on an existing benchmark dataset is not grounds for rejection by itself. Rather, it is important to weigh both the novelty and potential impact of the work alongside the reported performance. Minor flaws that can be easily corrected should not be a reason to reject a paper.
Please be specific and detailed in your reviews. Your main critique of the paper should be written in terms of a list of strengths and weaknesses. You can use bullet points here, but also explain your arguments. Your discussion, more than your score, will help the authors, fellow reviewers, and Area Chairs understand the basis for your recommendation, so please be thorough. You should include specific feedback on ways the authors can improve their papers.
In the discussion of related work and references, simply saying "this is well known" or "this has been common practice in the industry for years" is not sufficient: cite specific publications, including books or public disclosures of techniques.
Please read the FAQ at the end of this document for further details on how to treat related work on arXiv, supplementary material, and rebuttals.
1. Protect Ideas
As a reviewer for ICCV, you have the responsibility to protect the confidentiality of the ideas represented in the papers you review. ICCV submissions are not published documents. The work is considered new or proprietary by the authors; otherwise they would not have submitted it. Of course, their intent is to ultimately publish to the world, but most of the submitted papers will not appear in the ICCV proceedings. Thus, it is likely that the paper you have in your hands will be refined further and submitted to some other journal or conference. Sometimes the work is still considered confidential by the authors' employers. These organizations do not consider sending a paper to ICCV for review to constitute a public disclosure. Protection of the ideas in the papers you receive means:
2. Avoid Conflict of Interest
As a reviewer of a ICCV paper, it is important for you to avoid any conflict of interest. There should be absolutely no question about the impartiality of any review. Thus, if you are assigned a paper where your review would create a possible conflict of interest, you should return the paper and not submit a review. Conflicts of interest include (but are not limited to) situations in which:
While the organizers make every effort to avoid such conflicts in the review assignments, they may nonetheless occasionally arise. If you recognize the work or the author and feel it could present a conflict of interest, email the Program Chairs (email@example.com) as soon as possible so they can find someone else to review it.
3. Be Professional
Belittling or sarcastic comments have no place in the reviewing process. The most valuable comments in a review are those that help the authors understand the shortcomings of their work and how they might improve it. Write a courteous, informative, incisive, and helpful review that you would be proud to sign with your name (were it not anonymous).
Finally, keep in mind that a thoughtful review not only benefits the authors, but also yourself. Your reviews are read by other reviewers and especially the Area Chairs, in addition to the authors. Being a helpful reviewer will generate good will towards you in the research community -- and may even help you to win an Outstanding Reviewer award.
Once you've been notified by email that papers have been assigned to you, please log into the CMT site (https://cmt3.research.microsoft.com/ICCV2019), choose the “Reviewer” role on top, and follow the steps below.
1. Download your papers.
To download individual papers, you can click the links underneath individual paper titles. Or, you can click the “Actions” button in the top right corner and then choose “Download Files”. This allows you to download a zip file containing all the papers plus supplementary files (if available).
2. Check for possible conflict or submission rule violations.
Contact the Program Chairs (firstname.lastname@example.org) immediately if:
3. Review papers and assign them a preliminary (pre-rebuttal) rating.
For a paper, under the review column, click "Edit Review" to get to the review form. You can hover the mouse over the “?” symbol next to each question for a more detailed explanation, in particular, for the interpretations of paper ratings in questions 4 and 10. Before you start writing your reviews, make sure you have read the Reviewer Guidelines above and the ICCV 2019 Reviewer Tutorial.
4. (Optional) Review papers offline.
To enable offline reviewing, go to “Actions - > Import Reviews”. You can select papers and click “Download” to obtain XML review stubs and the update files as needed. Once you are done updating, you can upload the file from the same page. We suggest that you use an XML editor to edit the file. You should always verify the review after uploading by inspecting it online.
5. Participate in discussions with Area Chairs and other reviewers.
After the rebuttal period, reviewers will work with Area Chairs to clear up any confusions and attempt to reach consensus on papers. The CMT site has an electronic bulletin board feature that allows Area Chairs to contact reviewers anonymously. Once the Area Chair posts a note, reviewers will be notified and asked to log in to see the post and respond. The identities of the reviewers will be hidden from each other (but not from the Area Chair).
6. Enter your final (post-rebuttal) recommendation.
After the rebuttal period you will enter your final recommendation on CMT. This may differ from your preliminary rating, and should reflect your judgment taking into account all the other reviews, the authors' rebuttal, and the discussion about the paper (if any).
Q. Is there a minimum number of papers I should accept or reject?
A. No. Each paper should be evaluated in its own right. If you feel that most of the papers assigned to you have value, you should accept them. It is unlikely that most papers are bad enough to justify rejecting them all. However, if that is the case, provide clear and very specific comments in each review. Do NOT assume that your stack of papers necessarily should have the same acceptance rate as the entire conference ultimately will.
Q. Can I review a paper I already saw on arXiv and hence know who the authors are?
A. In general, yes, unless you are conflicted with one of the authors. See next question below for guidelines.
Q. How should I treat papers for which I know the authors?
A. Reviewers should make every effort to treat each paper impartially, whether or not they know who wrote the paper. For example: It is not OK for a reviewer to read a paper, think "I know who wrote this; it's on arXiv; they are usually quite good" and accept the paper based on that reasoning. Conversely, it is also not OK for a reviewer to read a paper, think "I know who wrote this; it's on arXiv; they are no good" and reject paper based on that reasoning.
Q. Should authors be expected to cite related arXiv papers or compare to their results?
A. Consistent with good academic practice, the authors should cite all sources that inspired and informed their work. This said, asking authors to thoroughly compare their work with arXiv reports that appeared shortly before the submission deadline imposes an unreasonable burden. We also do not wish to discourage the publication of similar ideas that have been developed independently and concurrently. Reviewers should keep the following guidelines in mind:
Q. How should I treat the supplementary material?
A. The supplementary material is intended to provide details of derivations and results that do not fit within the paper format or space limit. Ideally, the paper should indicate when to refer to the supplementary material, and you need to consult the supplementary material only if you think it is helpful in understanding the paper and its contribution. According to the Author Guidelines, the supplementary material MAY NOT include results on additional datasets, results obtained with an improved version of the method (e.g., following additional parameter tuning or training), or an updated or corrected version of the submission PDF. If you find that the supplementary material violates these guidelines, please contact the Program Chairs.
Q. Can I request additional experiments in the author rebuttal? How should I treat additional experiments reported by authors in the rebuttal?
A. In your review, you may request clarifications or additional illustrations in the rebuttal. Per a passed 2018 PAMI-TC motion, reviewers should not request substantial additional experiments for the rebuttal, or penalize for lack of additional experiments. Authors should not include new experimental results in the rebuttal, and reviewers should discount any such results when making their final recommendation.