About PoPETs Reviews

Review Process

Note: This information applies to PoPETs 2024 and earlier.

The review process begins after the submission deadline and contains the following steps:

1. Initial review: Submissions are quickly reviewed by PC chairs for possible “desk rejection”. Such rejections will be made only because of length violations, author anonymity violations, and submission on topics that are out of scope.
2. Early rejection: Reviews are completed in two rounds. After the first round, papers with at least 2 reviews and for which all reviews recommend “reject” or “reject and resubmit” may be rejected without further review and without the opportunity for rebuttal. Authors will be notified of this decision by the beginning of the rebuttal period.
3. Reviews: PC members write detailed and constructive reviews of their assigned submissions and provide an initial acceptance recommendation. PoPETs aims to provide 4 reviews for papers that are not rejected early.
4. Rebuttal: Authors are sent the reviews and then may submit a short (e.g. 500 words) rebuttal to answer the reviewers’ questions and concerns. The rebuttal period is short (e.g. 3 days), but authors are given the chance to provide additional responses to reviews that come in late.
5. Discussion: Reviewers participate in an online discussion to come to a decision. Discussion leads are appointed to solicit opinions, obtain agreement, and write a meta-review summarizing the discussion and reasons for the decision.
6. Decisions: Decisions are finalized by the PC Chairs, and final reviews are sent to authors. Reviews may be updated based on the discussion and will include a meta-review.
7. Next step: The next step depends on the decision:
• Accept: Authors should prepare a camera-ready version of the paper, taking into consideration the comments of reviewers. No further review is done of the camera-ready version.
• Accept with Minor Revisions: Authors prepare a minor revision responding to reviewer requests, and a shepherd decides if the revision is sufficient.
• Resubmit with Major Revisions: Authors may submit a major revision to either of the next two issues, in which case they will receive the same reviewers and be judged under the criteria set out in the meta-review. A summary of changes is required.
• Reject and Resubmit: Authors are encouraged to consider addressing the reviewer concerns and submitting a revision to a future issue. The same reviewers may be assigned but otherwise continuity is not assured. A summary of changes is required.
• Reject: Author may submit a revision to any future issue. A summary of changes is required.

Review Decisions

Note: This information applies to PoPETs 2024 and earlier.

Decisions are made on the basis of reviews, author rebuttals, and online discussion. Each paper is judged independently, and in particular PoPETs does not consider the total number of accepted papers when making decisions (e.g. no “accept if room” decisions). Papers receive one out of five possible decisions.

Accept

This decision is given to papers that can be accepted as they are, or that require very minor revisions that don’t need to be verified by a shepherd.

Accept with Minor Revisions

These are conditionally accepted papers that require minor revisions to be approved by a shepherd. Revisions within the scope of shepherding typically include: editorial improvements or restructuring of some sections in the text, adding missing relevant related work, improving the motivation, toning down claims, adding disclaimers on issues left out of scope, including more details on methodology or experiments, providing missing clarifications, etc.

Resubmit with Major Revisions

These are promising papers that we would like to eventually publish at PoPETs, but that have some important flaws that need to be addressed before publication, and that are too major to be addressed with shepherding. Typical issues within the scope of a major revision include: performing additional experiments, demonstrating robustness against certain attacks, improving the security analysis, etc.

Papers with issues of novelty, insufficient contribution, or requiring changes so substantial that it is difficult to predict that a revised version will likely be acceptable, should not be given a major revision decision, but instead be rejected. We would like to minimise the number of cases in which resubmitted major revisions that have addressed the reviewer comments are still below the acceptance bar.

Articles that receive a major revision decision, if submitted to one of the following two issues, are assigned to the same reviewers and judged against the list of revisions requested in the meta-review. It is thus crucial that the meta-reviews of major revisions provide a concrete list of improvements that, if met, would make the paper acceptable. Those improvements should be feasible within at most four months.

Authors receiving a "Resubmit with Major Revisions" decision are invited to address the issues pointed out in the reviews (and in particular those included in the meta-review) and to resubmit a revised version to one of the next two issues. The acceptance rate for major revisions is typically very high, but please bear in mind that their acceptance is not guaranteed. The paper will not be accepted if the reviewers believe that it fails to adequately address the meta-review requests. Moreover, we avoid giving consecutive major revisions decisions, and so a revision that doesn’t satisfy the requests may be rejected.

Reject and Resubmit

Papers receiving this decision are deemed interesting with potential to be accepted in a future issue if they are revised and improved. The needed revisions are, however, too substantial or their outcome too unpredictable for the reviewers to provide a concrete, feasible list of changes that, if implemented, would bring the paper over the accept bar.

Authors receiving a "Reject and Resubmit" decision are encouraged to consider addressing the concerns raised in the reviews and then resubmitting the paper to a later issue. Note, however, that a reject and resubmit decision typically signifies that substantial revisions are necessary, and so it is strongly recommended that authors take the time to make changes, likely delaying resubmission by more than one issue.

We will try to assign the revised paper to some of the same reviewers, but new reviewers will likely be assigned as well. All reviewers will take into account how the previous concerns have been addressed, but they are free to bring up new concerns as well.

Reject

This decision is given to papers that the reviewers deem inadequate for PoPETs. While the structure of PoPETs/PETS does not forbid submitting any paper to a future issue, the reviewers believe that acceptance would be unlikely unless the paper is substantially changed—to the point of almost being turned into a different paper.

This decision may be given on the basis of the paper being out of scope for the topics of the venue (e.g., general security papers with no specific connection to privacy), papers suffering from issues of novelty or lack of contribution, or papers with fatal technical flaws that are non-trivial to address.

Who reviews papers for PoPETs?

Note: This information applies to PoPETs 2024 and earlier.

PoPETs has a Program Committee (PC) / Editorial Board (EB) composed of internationally-recognised experts, who are invited to the PC/EB by the current PC Chairs. The PoPETs PC members provide most of the reviews, with each paper being reviewed by at least 2 PC members. The review load is limited, typically to 2–4 papers per PC member per issue. PC members have access to all non-conflicted submissions, and they are welcome to review or discuss papers that have not been assigned to them. PC members are expected to provide timely, high-quality reviews, actively participate in the discussion, contribute to reaching a consensus decision, and in some cases volunteer to shepherd a paper.

We ask PC members to not delegate their reviews.

PoPETs often uses external reviewers to provide needed expertise. If, as a PC member, you are aware of an expert reviewer who would be perfect for a particular paper, please let the Chairs know and we can add them as an additional external reviewer (after checking for conflicts). External reviewers only have access to papers assigned to them for review, and are bound to confidentiality with respect to the paper, the reviews and the discussion, in the same way as PC members. We aim to assign an external reviewer to most papers.

PoPETs provides an opportunity for junior researchers in particular to contribute as external reviewers. By “junior researchers” we mean recent graduates (postdocs) and senior PhD students who have at least 2 years of research experience, one or more publications in their topic, and the ability to review papers in their area of expertise. While junior researchers are often involved in peer review at other venues through sub-reviewing or "shadow PCs", we believe that direct participation in the real reviews and discussion with senior PC members is the most effective way to train and engage newer researchers.

Please nominate excellent junior researchers to be external PoPETs reviewers through the online form. It is important to be specific in the keywords of the research areas as these are used to find matches between reviewers and papers. Self-nominations are welcome. Accepted nominees will be normal external reviewers, expected to provide reviews, participate in the discussion, and contribute to reaching a decision.

Review Guidelines

Note: This information applies to PoPETs 2024 and earlier.

Timeliness:
The review process for an issue consists of: bidding, individual reviews (typically four per paper), author rebuttals, discussion, consensus decision, and writing of meta-reviews. Given that reviews are released to authors for the rebuttal phase, it is critical that they are not delayed. We also ask reviewers to please participate actively with comments during the discussion phase.

Thorough and substantive reviews:
We expect reviewers to write thorough reviews and not just a couple of short paragraphs. By limiting the review load, we hope that PC members can focus on writing more substantive reviews based on a deeper reading of the paper.

Constructiveness:
While many papers will no doubt be rejected, authors should know that their papers were turned down for reasonable issues. The authors may disagree with the end decision, but they should feel like the reviewer provided reasonable and constructive feedback. We thus ask reviewers to focus on the positives of the paper, and try to be as constructive as possible so that authors can improve their papers, regardless of whether they eventually receive an acceptance or rejection decision.

Topics:
We are (continually) trying to broaden the scope of PETS and we are encouraging submissions from diverse areas. We have a diverse PC so that we have relevant reviewers for these areas. Still, the paper should relate to privacy (see the topics listed in the CFP for examples). We've included a review field related to relevance to PETS. Reviewers should comment on the relevance score in the review, especially if they gave it a low relevance score.

Novelty claims:
While it is reasonable to discuss how exciting and interesting are (or not) the results of a submission, if a reviewer states that the novelty of a paper is limited because the idea is not new (or has been known), we ask them to provide citations to back up this claim to show that the work has been published before.

Authors' Rebuttal:
The authors will have a chance to rebut/answer reviewer concerns/questions through a short rebuttal phase. If a reviewer recommendation hinges on answers to certain questions, these should be specified in the review so that the authors may answer them in their rebuttal. Reviewers are asked to take the rebuttals into consideration during the discussion, and revise their review accordingly.

Discussion:
Reviewers are expected to participate in the online discussions for their papers. They should consider the opinions of other reviewers with the goal of coming to a consensus on a decision for a paper. Reviews should be updated to include any substantial changes in the reviewers’ understanding or opinions so that authors can understand their reviews to reflect the views of the reviewers after rebuttal and discussion. The PC Chairs are responsible for finalizing the decisions.

Discussion Lead (aka Lead Reviewer):
During the discussion phase, one of the reviewers for each paper will be assigned the Discussion Lead (DL) role to steer the discussion. The DL is also responsible for drafting a meta-review that summarizes the main suggestions for revision brought up in the reviews and discussion. This meta review should provide clear constructive suggestions for the authors on how to improve the paper depending on the decision: for minor or major revisions, clarify what revisions are expected; for rejected papers, clarify the main reasons why the paper is being rejected and what the authors might want to improve for a better shot for a resubmission at PoPETs or elsewhere. The DL should also encourage authors to add a rebuttal response, to consider the rebuttal during the discussion, and to update their reviews to reflect significant changes in their opinions.

Resubmissions:
Articles that have been previously submitted to PoPETs are accompanied by a summary of changes, regardless of whether they were rejected, or invited to resubmit with major revisions. Major revisions resubmitted to one of the next two issues are available for review two weeks after the submission deadline and assigned to the same reviewers.

Publicly available pre-prints:
It is recognized that, at times, information regarding the identities of authors may become public outside the submission process (e.g., if a pre-print is published as a technical report); reviewers are asked to ignore this external information. If the paper was not published in formal proceedings, please do not look for additional contributions beyond those informally published reports. In these cases we have allowed authors to say they have such a report with an anonymized reference (which is really a NOP reference). Additionally in the discussion a reviewer can note, e.g., "accept only if same authors" if they are concerned about plagiarism issues.

Prior published work:
A paper submitted to PoPETs must present original work not described in any prior publication that is more than 4 double-column ACM conference-style pages in length. A prior publication is a paper that has been accepted for presentation at a refereed conference or workshop with proceedings, or an article that has been accepted for publication in a refereed journal. If a PoPETs submission has overlap with a prior publication, the submission must cite the prior publication (in third person regardless of whether the work is done by the same authors), along with all other relevant published work, even if this prior publication is at or below the 4-page length threshold.

Ethical concerns:
Reviewers are asked to bring up any ethical concerns they may have about the study methodology, regardless of whether the study received approval from an ethics board. Examples of research for which ethical issues may arise include papers describing experiments with users or user data (e.g., network traffic, passwords, social network information) and papers reporting privacy or security vulnerabilities in real-world systems. PoPETs reviewers should consider the basic principles of ethical research, e.g., beneficence (maximizing the benefits to an individual or to society while minimizing harm to the individual), minimal risk (appropriateness of the risk versus benefit ratio), informed consent, respect for privacy, and limited deception. Authors are encouraged to include a subsection on Ethical Principles if human subjects research is conducted, and such a discussion may be required if deemed necessary during the review process.

Claims of Benefits to Particular Populations
Like human subjects research and supporting statements from IRBs and similar, failure to support claims about usefulness for a target demographic should not be grounds for immediate rejection. If this isn't in the paper, reviewers should ask for it to be added/clarified before assuming the appropriateness of the proposed system/technology for that group. Authors should be expected to address this in rebuttal. And if reviewers don't feel competent to judge whether the claims about a system's benefit to a particular user population are appropriate, it might be a place to suggest seeking an external reviewer (if the paper is otherwise competitive enough).

However, progress and understanding sometimes does result from simplifying assumptions. Basic research work that makes clear that it is merely speculating usefulness for a particular user population and indicates potential problems and questions, and the need for validation of applicability, should still be accepted if the theoretical contribution is solid.

Systematization of Knowledge (SoK) papers:
We accept Systematization of Knowledge papers. These are papers that critically review, evaluate, and contextualize work in areas for which a body of prior literature exists, and whose contribution lies in systematizing the existing knowledge in that area. To be suitable for publication, SoK articles must provide an added value beyond a literature review, such as novel insights, identification of research gaps, or challenges to commonly held assumptions.

Security Proofs and Lengthy Appendices:
In most cases, reviewers are not required to read or review content in the appendix. Some papers require lengthy security proofs to support the technical validity of the contribution. These papers should indicate this in the body of the paper and include the proof in the appendix. Reviewers are not required to read the appendices at this phase (though they are welcome to do so if they wish). If the paper would receive an "Accept" decision pending a valid proof, one or two experts would be asked to review the proof. If the proof is lengthy this may result in delayed notification and publication for such papers (as the authors have been warned).