About PoPETs Reviews

This year PoPETs is running an experiment on consistency in the review process. Please read about it!

Who reviews papers for PoPETs?

PoPETs has a Program Committee (PC) / Editorial Board (EB) composed by internationally recognised experts, who are invited to the PC/EB by the current PC Chairs. The PoPETs PC members provide most of the reviews, with each paper being reviewed by at least 3 PC members. The review load is limited, typically to 2-4 papers per PC member per issue. PC members are expected to provide timely, high-quality reviews, actively participate in the discussion, contribute to reaching a consensus decision, and in some cases volunteer to shepherd a paper.

We ask PC members to not delegate their reviews.

If as PC member you are aware of an expert reviewer that would be perfect for a particular paper, please let the Chairs know and we can add them as an additional external reviewer (after checking for conflicts).

External reviewers only have access to papers assigned to them for review, and are bound to confidentiality with respect to the paper, the reviews and the discussion, in the same way as PC members are bound. PC members have access to all non-conflicted submissions and they are welcome to review or discuss papers that have not been assigned to them.

At PoPETs we would also like to increase the opportunities for junior researchers to participate in our peer review process, contribute and learn from it, benefiting both as authors and as future PC members.

We aim to have one junior researcher acting as external reviewer for each paper.

By junior researchers we refer to recent graduates (postdocs) and senior PhD students (e.g., who have at least 2 years of research experience and one or more publications in their topic), qualified to review papers in their specific topic of expertise, and who we may wish to consider for invitations to the PC in the future.

While junior researchers are often involved in peer-review through sub-reviewing or as part of "shadow student PCs", we believe that personal invitation, first-hand experience, and direct participation in the real reviews and discussion with senior PC members, is the most effective way to train and engage the younger generation.

As external reviewers, junior researchers have access to all the reviews for their assigned paper, to the author's rebuttal, and to the full discussion and comments. Besides providing a review, they are expected to participate in the discussion, argue their points, and contribute to reaching a decision, just like any other PC member reviewing the paper.

If as junior researcher you are interested in reviewing for PETS, or if you know any junior researchers that you would like to recommend as external reviewers, the information can be provided through this online form (https://bit.ly/2gxVDd8). It is important to be specific in the keywords of the research areas as these are used to find matches between reviewers and papers.

We hope that participating in the PoPETs review process will be a positive learning experience for these junior researchers, as well as an additional source of quality reviews for authors submitting their work to PoPETs.

Review Guidelines

Timeliness:
The review process for an issue consists of: bidding, individual reviews (typically four per paper), author rebuttals, discussion, consensus decision, and writing of meta-reviews. Given that reviews are released to authors for the rebuttal phase, it is critical that they are not delayed. We also ask reviewers to please participate actively with comments during the discussion phase.

Thorough and substantive reviews:
We expect reviewers to write thorough reviews and not just a couple of short paragraphs. By limiting the review load, we hope that PC members can focus on writing more substantive reviews based on a deeper reading of the paper.

Constructiveness:
While many papers will no doubt be rejected, authors should know that their papers were turned down for reasonable issues. The authors may disagree with the end decision, but they should feel like the reviewer provided reasonable and constructive feedback. We thus ask reviewers to focus on the positives of the paper, and try to be as constructive as possible so that authors can improve their papers, regardless of whether they eventually receive an acceptance or rejection decision.

Topics:
We are (continually) trying to broaden the scope of PETS and we are encouraging submissions from diverse areas. We have a diverse PC so that we have relevant reviewers for these areas. Still, the paper should relate to privacy (see the topics listed on the CFP for examples). We've included a review field related to relevance to PETS. Reviewers should comment on the relevance score in the review, especially if they gave it a low relevance score.

Novelty claims:
While it is reasonable to discuss how exciting and interesting are (or not) the results of a submission, if a reviewer states that the novelty of a paper is limited because the idea is not new (or has been known), we ask them to provide citations to back up this claim to show that the work has been published before.

Authors' Rebuttal:
The authors will have a chance to address/rebut reviewer concerns/questions through a short rebuttal phase. If a reviewer recommendation hinges on answers to certain questions, these should be specified in the review so that the authors may answer them in their rebuttal. Reviewers are asked to take the rebuttals into consideration during the discussion, and revise their review accordingly.

Discussion Lead (aka Lead Reviewer):
During the discussion phase, one of the reviewers for each paper will be assigned the Discussion Lead (DL) role to steer the discussion. The DL is also responsible for drafting a meta-review that summarizes the main points revision brought up in the reviews and discussion. This meta review should provide clear constructive suggestions for the authors on how to improve the paper depending on the decision: for minor or major revisions, clarify what revisions are expected; for rejected papers, clarify the main reasons why the paper is being rejected and what the authors might want to improve for a better shot for a resubmission here or elsewhere. The DL should also make sure that the discussion and reviews take the rebuttal into account.

Resubmissions:
Articles that have been previously submitted to PoPETs are accompanied by a summary of changes, regardless of whether they were rejected, or invited to resubmit with major revisions. Major revisions resubmitted to one of the next two issues are available for review two weeks after the submission deadline and assigned to the same reviewers.

Publicly available pre-prints:
It is recognized that, at times, information regarding the identities of authors may become public outside the submission process (e.g., if a pre-print is published as a technical report); reviewers are asked to ignore this external information. If the paper was not published in formal proceedings, please do not look for additional contributions beyond those informally published reports. In these cases we have allowed authors to say they have such a report with an anonymized reference (which is really a NOP reference). Additionally in the discussion a reviewer can note, e.g., "accept only if same authors" if they are concerned about plagiarism issues.

Prior published work:
A paper submitted to PoPETs must present original work not described in any prior publication that is more than 4 double-column ACM conference-style pages in length. A prior publication is a paper that has been accepted for presentation at a refereed conference or workshop with proceedings, or an article that has been accepted for publication in a refereed journal. If a PoPETs submission has overlap with a prior publication, the submission must cite the prior publication (in third person regardless of whether the work is done by the same authors), along with all other relevant published work, even if this prior publication is at or below the 4-page length threshold.

Ethical concerns:
Reviewers are asked to discuss any ethical concerns they may have about the study methodology, regardless of whether the study received approval from an ethics board. Papers describing experiments with users or user data (e.g., network traffic, passwords, social network information), should follow the basic principles of ethical research, e.g., beneficence (maximizing the benefits to an individual or to society while minimizing harm to the individual), minimal risk (appropriateness of the risk versus benefit ratio), voluntary consent, respect for privacy, and limited deception. Authors are encouraged to include a subsection on Ethical Principles if human subjects research is conducted, and such a discussion may be required if deemed necessary during the review process.

Systematization of Knowledge (SoK) papers:
We accept Systematization of Knowledge papers. These are papers that critically review, evaluate, and contextualize work in areas for which a body of prior literature exists, and whose contribution lies in systematizing the existing knowledge in that area. To be suitable for publication, SoK articles must provide an added value beyond a literature review, such as novel insights, identification of research gaps, or challenges to commonly held assumptions.

Security Proofs and Lengthy Appendices:
In most cases, reviewers are not required to read or review content in the appendix. Some papers require lengthy security proofs to support the technical validity of the contribution. These papers should indicate this in the body of the paper and include the proof in the appendix. Reviewers are not required to read the appendices at this phase (though they are welcome to do so if they wish). If the paper would receive an "Accept" decision pending a valid proof, one or two experts would be asked to review the proof. If the proof is lengthy this may result in delayed notification and publication for such papers (as the authors have been warned).

Review decisions

Papers receive one out of five possible decisions.

Accept

This decision is given to papers that can be accepted as they are, or that require very minor revisions that don’t need to be verified by a shepherd.

Shepherd with Minor Revisions

These are conditionally accepted papers that require minor revisions to be approved by a shepherd. Revisions within the scope of shepherding typically include: editorial improvements or restructuring of some sections in the text, adding missing relevant related work, improving the motivation, toning down claims, adding disclaimers on issues left out of scope, including more details on methodology or experiments, providing missing clarifications, etc.

Resubmit with Major Revisions

These are promising papers that we would like to eventually publish at PoPETs, but that have some important, yet fixable, flaws that need to be addressed before publication, and that are too major to be addressed with shepherding. Typical issues within the scope of a major revision include: performing additional experiments, demonstrating robustness against certain attacks, improving the security analysis, etc.

Papers with issues of novelty, insufficient contribution, or requiring changes so substantial that it is difficult to predict that a revised version will likely be acceptable, should not be given a major revision decision, but instead be rejected. We would like to minimise the number of cases in which resubmitted major revisions that have addressed the reviewer comments are still below the acceptance bar.

Articles that receive a major revision decision, if submitted to one of the following two issues, are assigned to the same reviewers and judged against the list of revisions requested in the meta-review. It is thus crucial that the meta-reviews of major revisions provide a concrete list of improvements that, if met, would make the paper acceptable. Those improvements should be feasible within at most four months.

Authors receiving a "Resubmit with Major Revisions" decision are warmly invited to address the issues pointed in the reviews (and in particular those included in the meta-review) and to resubmit a revised version to one of the next two issues.

Reject and Resubmit

Papers receiving this decision are deemed interesting by the reviewers and potentially acceptable to a future issue if they are revised and improved. The needed revisions are however too substantial or their outcome too unpredictable for the reviewers to provide a concrete, feasible list of changes that if implemented would bring the paper over the accept bar. Papers that have some interesting insights or contributions that are however not quite sufficient by themselves for a PoPETs publication also fall within this category.

Authors receiving a "Reject and Resubmit" decision are encouraged to revise their paper according to the feedback received in the reviews and resubmit an improved version of their work to a later issue.

Reject

This decision is given to papers that the reviewers deem inadequate for PETS, and that would almost need to be turned into a different paper for the work to have chances of publication at a later issue. This decision may be given on the basis of the paper being out of scope for the topics of the venue (e.g., general security papers with no specific connection to privacy), papers suffering from issues of novelty or lack of contribution, or papers with fatal technical flaws that are non-trivial to address.

Authors receiving a "Reject" decision should carefully consider if they can overcome in a revision the issues raised by the reviewers to justify their decision.