Comparing Large-Scale Privacy and Security Notifications

Over the last decade, web security research has used notification campaigns as a tool to help web operators fix security problems or stop infrastructure abuse. First attempts at applying this approach to privacy issues focused on single services or vendors. Hence, little is known if notifications can also raise awareness and encourage remediation of more complex, vendor-independent violations of privacy legislation at scale, such as informed consent to cookie usage under the EU’s ePrivacy Directive or the General Data Protection Regulation’s requirement for a privacy policy. It is also unclear how privacy notifications perform and are perceived compared to those about security vulnerabilities. To fill this research gap, we conduct a large-scale, automated email notification study with more than 115K websites we notify about lack of a privacy policy, use of third-party cookies without or before informed consent, and input forms for personal data that do not use HTTPS. We investigate the impact of warnings about fines and compare the results with security notifications to more than 40K domains about openly accessible Git repositories. Based on our measurements and interactions with operators through email and a survey, we find that notifications about privacy issues are not as well received as security notifications. They result in lower fix rates, less incentive to take immediate action, and more negative feedback. Specific reasons include a lack of awareness and knowledge of privacy laws’ applicability, difficulties to pinpoint the problem, and limited intrinsic motivation.


INTRODUCTION
Websites' availability and security depend on operators following best practices, update their systems, and stay alert of new threats. Over the last few years, compliance with privacy regulations has become another important task to ensure that services operate within the legal boundaries and protect user privacy. Recent years have seen the creation of new privacy laws across the globe to better tackle the 21st century's reality of ubiquitous processing of personal data by digital services. New privacy regulations including the European Union's General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA) have introduced extensive requirements for the processing of personal data and granted people individual rights to learn about and control the use of their personal information. As research has shown, the adoption of new regulations and guidelines in practice is often slow [10,58]. While the last few years have seen increasing numbers of GDPR-induced fines [8], a lack of monetary and human resources continues to pose a problem in large-scale enforcement of privacy laws [25,51]. Given that web privacy research has been identifying privacy issues on websites at scale for years, this raises the question if and how the scientific community could aid regulators in identifying and remediating website behavior that violates privacy law.
One promising means for privacy and security research to help boost GDPR compliance on the Web are large-scale email notification campaigns. Informing the operators of affected websites could help raise awareness of practices that do not comply with privacy law and encourage operators to fix the issue before they are subject to GDPR-mandated fines. Such notifications have been repeatedly used by security research to raise awareness and motivate fixes of diverse issues including Heartbleed [12], XSS [49,50], DDoS amplifiers [34,61] and potential information leaks [35,36]. If this approach turns out to also be viable for notifications about privacy violations, this could take the burden off data protection authorities (DPAs) in enforcing laws and help website owners to better protect user privacy. While both researchers and NGOs have conducted notification campaigns about privacy issues before, they have focused on selected Consent Management Platforms (CMPs) [44] or restricted their scope to a single vendor and locale [38], typically because of the manual verification involved. It is also unknown how notifications about privacy problems compare to those about security vulnerabilities in terms of remediation rates and timing.
In this work we explore the feasibility of large-scale, automated email notification campaigns for vendor-independent violations of privacy laws, namely the GDPR's transparency requirement, its mandate to use state-of-the-art data protection mechanisms, and consent to the use of not strictly necessary cookies under the ePrivacy Directive. To identify how notifications about privacy issues perform compared to those about security vulnerabilities, we also notify websites about publicly accessible Git repositories that may leak sensitive information. We compare fix rates between issues, investigate the impact of mentioning potential fines, and conduct qualitative analyses of feedback from emails and a survey to learn how privacy notifications are perceived by recipients and what could be done to help them address privacy issues in the future. More concretely, we make the following contributions: • We conduct the first large-scale, automated email notification study with 115K websites that investigates the feasibility of this approach for complex, vendor-independent privacy issues. Our notifications have significant impact on remediation for lack of a privacy policy or consent notice. We find no significant impact of warnings about potential fines. • We compare the effect of notifications about privacy issues with those about a security vulnerability. For privacy, fix rates are lower than for the security vulnerability, which is also addressed more quickly. Recipients also perceive emails about a privacy compliance issue more negatively, partially due to a lack of intrinsic motivation to fix it or (incorrect) assumption of inapplicability of the relevant laws. • As for the persisting reachability problem, we investigate if email addresses extracted from websites are an efficient and scalable alternative to prior approaches, manually collected addresses or email generics. Our results confirm this, with 87.8 % and 33.8 % of successful handovers to recipients' mail servers for extracted addresses and generics, respectively.

RELATED WORK
Security Notifications. In Web security research, large-scale notification campaigns were first used to alert server operators about abuse of their infrastructure for unintended purposes, including distribution of malicious downloads [6,59]. This approach was subsequently also applied to raise awareness and motivate fixes of security vulnerabilities including Heartbleed [12], DNS zone poisoning [7], XSS [50], HTTPS misconfigurations [61], DDoS amplifiers [34,35], misconfigured IPv6 firewalls [35], and leaks of information whose public accessibility could pose a security risk, including industrial control systems [35], Git or SVN repositories [36,49], cryptographic keys, database backups, server status information, and phpinfo files [36]. One core problem with large-scale Web security notifications is to reliably reach the people responsible for fixing the issue. While previous work found that more individual communication channels such as telephone [49], physical mail [36,38,49], websites' contact forms, associated social media accounts [49], or manually identified contact email addresses [36,38,49] can lead to higher delivery rates, the involved overhead in terms of human resources and monetary cost makes these infeasible for notifying websites at scale. Hence, most Web security notification campaigns have used generic approaches to contact websites via email, either directly via generic email addresses [6] such as info@DOMAIN or webmaster@DOMAIN (RFC 2142 [9]), WHOIS contact information [6,7,12,35,49,50,61] or through trusted third parties including CERTs [35,50], DNS nameserver operators [7], or hosting providers [6], depending on the investigated issue(s). Drawbacks of this approach include high bounce rates due to missing or outdated information in WHOIS records or non-use of RFC 2142 mailbox names [7,50]. Use of intermediaries carries the risk of them not forwarding notifications [35].
Even if a notification email is correctly targeted, it still risks being considered spam or otherwise malicious by both mail servers and human recipients. Prior work has studied how to increase perceived message authenticity in notification campaigns by evaluating the effect of sender reputation [6,38,61], email format such as plaintext or HTML [49], text localization [35,61], and use of S/MIME [49], but no clear "recipe" has emerged. Finally, findings also differ for the content of the notification message itself: While some studies did not identify a significant influence of message text on remediation rates [7,61], others found that more detailed explanations had a significant positive influence [35,59]. Message tone was found to not affect fix rates [49,61]. Maass et al. compared existing work and outlined practical recommendations for future notification studies [37].
Privacy Notifications. Compared to abuse and security notifications, previous work that notified website owners about privacy issues at scale is scarce. Challenges lie in such a study requiring 1) determining if the examined website is subject to the privacy legislation of interest, 2) certainty that a given issue is regarded a violation of this privacy legislation, and 3) a high-accuracy detection mechanism to keep the number of false positives low and not cause unnecessary anxiety and costly investigations with people whose websites do not have a privacy problem. The third prerequisite is challenging, as privacy issues are hard to detect automatically [46] unless focused on specific services or vendors, where a common implementation can allow for simple yes/no checks of a value, parameter, or URL. Otherwise, complex heuristics are necessary that may require, for example, contextual analysis and natural language processing to determine if a privacy notice contains the required disclosures.
Consequently, prior privacy notification campaigns focused on specific vendors or consent frameworks. Maass et al. [38] notified the owners of 4,754 German websites about the lack of IP anonymization in their Google Analytics integration, which the German DPA deemed necessary for GDPR compliance [33] and can be remotely detected with certainty through URL parameters passed in the HTTP request to Google. While the study found that framing notifications as legal compliance issues led to increased fix rates, the analysis is limited to German sites and those with an imprint, thereby limiting insights to this selected group. Our work uses a much larger domain set without this bias. In May 2021, Austrian privacy NGO noyb notified more than 500 companies about consent notices on their websites that used techniques considered to be non-GDPR-compliant by various national DPAs [44]. While 42 % of individual violations were fixed within 30 days, 82 % of companies still exhibited some type of GDPR compliance issue. Since the goal of the notification campaign was to file complaints to national DPAs, the automated detection mechanism was tailored to a single CMP, OneTrust, and supported by manual review by legal experts, limiting this approach in coverage and scalability.
Automated Detection of Web Privacy Issues. Automatically detecting vendor-agnostic privacy problems on websites is challenging due to a lack of standardization and concrete guidelines by lawmakers, DPAs, and court rulings on how to implement key legal requirements on a technical level, including transparency mechanisms mandated by privacy law such as privacy policies or consent notices. This is partly a deliberate decision to remain flexible towards future technological developments [46]. But even for concrete requirements, such as the wording of the "Do Not Sell My Personal Information" link mandated by the CCPA [48], actual implementations on websites widely vary [45,58].
Still, there is previous work that worked towards automatic detection of the privacy issues at the heart of our study. A growing body of literature has tackled the problem to automatically find privacy policies on websites and download them for further analysis. Hosseini et al. [28] discuss and evaluate different approaches and identify best practices. Recent research has also shown growing interest in (cookie) consent notices. They originate in Article 15 (3) of the EU's ePrivacy Directive, which requires website visitors' consent to store information on their devices unless the site cannot technically function otherwise, but are also used to obtain informed consent to data processing under Article 6(1)(a) GDPR. As with privacy policies, differences in implementation make automatic detection difficult [46]. Thus, past work has focused on specific consent frameworks or individual CMPs [4,26,39,43,44] or included manual analysis [10,56]. Despite EU law requirements for free, prior and informed consent to data processing [3,46], many consent notices were found not to offer sufficient choice [10,56], use dark patterns to nudge people into giving consent [43,56], or do not have a backend that ensures the visitors' selection is honored by the website [4,39]. Bollinger et al. [4] trained a machine learning classifier on cookie-purpose mappings from CMP classifications and manual categorization by web developers. Examining the 30K websites from the Tranco 1M list that featured one of the investigated CMPs with cookie-purpose mappings, they found that 94.7 % exhibited at least one violation of a consent requirement. Like prior approaches to determine the purpose for which specific cookies are set [29], this approach suffers from a lack of reliable ground truth.
Our work builds upon some of these techniques to automatically detect privacy issues at scale and independent of software or vendor, focusing on keeping the number of false positives low to avoid unnecessary notification of compliant websites.

MEASUREMENT & NOTIFICATION SETUP
Our study setup first required us to identify the security and privacy issues we wanted to notify operators about and how to check their presence at scale. Further, we describe how we created a set of domains to monitor, notification messages, and report infrastructure. We also explain ethical aspects of our large-scale measurements and notifications and discuss limitations of this approach.

Investigated Issues & Implemented Checks
In Section 2 we already identified three core requirements for privacy issues to investigate in a large-scale notification study. Of particular importance is a low number of false positive cases to not erroneously alert recipients and potentially trigger costly and stressful investigations. Thus, for privacy notifications, we required a clear violation of unambiguous regulatory data protection requirements that can be automatically detected. This ruled out issues requiring human judgment, such as dark patterns. To estimate the prevalence of false positives for our checks, we manually verified, for each issue, whether it was indeed present on 250 websites randomly drawn from the set of all domains we found to have the respective problem. We ended up selecting four privacy issues which fit our requirements and implemented them as custom functions in an established measurement framework, OpenWPM [14]. To compare the effect of privacy notifications to those about a security vulnerability, we also selected one security issue already used in prior notification studies, publicly accessible Git repositories [36,49]. For performance reasons, checks for the Git issue were not conducted with OpenWPM but with standard HTTP requests. All checks were performed once a day, launched shortly after midnight CET from CISPA servers on the premises of Saarland University in Germany. Performing the checks with an IP address in the EU is important because some websites, particularly with .com domains, show consent notices only to EU-based visitors [57].
3.1.1 No Privacy Policy. The GDPR requires that data subjects are informed about all processing of their personal data, which also comprises communications data such as users' IP addresses in web server logs [15], even if stored only temporarily. Thus, any website collecting such information must have a privacy policy that explains the use of visitors' personal data. To determine if a website had a privacy policy, we followed best practices identified by Hosseini et al. [28] and searched for privacy policy specific words in and around HTML link tags. For this, we extended a list of common words for privacy policy links, terms-of-service, and contact pages from a recent study [55] to cover all official EU languages. After a website had been fully loaded, we used the list of words identifying privacy policies to find links that likely lead to a privacy policy. If no such link was found, we also visited less privacy-specific subpages like terms-of-service and contact pages and searched them for words from the privacy list. If neither of these searches led to a match, the site was marked as violating the privacy policy requirement. For the manually checked sample of 250 sites drawn from those with this violation, 0.4 % were false positives.
3.1.2 Use of Third-Party Cookies Without Consent Notice ( No Consent) or Before Interaction With Consent Notice ( Before Consent). The setting of cookies is regulated by the EU's ePrivacy Directive (2009/136/EC) [54] and its implementations into national laws. Under its Article 5(3) the storing of information in a user's terminal equipment, including HTTP cookies that are not strictly necessary for the functioning of the website, is only allowed if the user has given prior, active consent. Mere continued use of the website does not constitute informed consent [16,17]. For the No Consent and Before Consent issues we focused on cookies set by third-party providers for advertising, analytics, and social media, because EU DPAs have universally deemed these non-essential for the website's functioning [1,32]. We used the WhoTracks.me database [23] to categorize a checked website's third-party requests by purpose and flagged those that included a Set-Cookie HTTP header and requested a third-party domain classified as "audio video player," "ad/pornvertising, " "site analytics, " or "social media". The presence of a cookie consent notice was determined based on two rule sets, a list of common HTML elements from the EasyList Cookie List [53] and the list of consent management providers vetted by IAB Europe's CMP Compliance Programme [30]. If one of the EasyList rules matched or a script referring to one of the CMPs was found on the front page, we assumed that the site had a consent notice. If a website issued a third-party request that required prior consent but a consent notice was not detected, we considered the site a case of No Consent. If a consent notice was detected but the flagged request was issued despite our script not interacting with the website, the website was labeled as having the Before Consent issue. The prevalence of false positive cases for No Consent was 2 % in our manual check and 6.8 % for Before Consent. The latter involved a tradeoff between false negatives for the presence of a consent notice and false positives for a notice without a working consent mechanism.
3.1.3 Input Fields for Personal Information Without HTTPS ( No HTTPS). The GDPR's requirements for "security of processing" (Article 32) and "data protection by design and by default" (Article 25) mandate the use of appropriate state-of-the-art technology for the collection and processing of personal data. One such mechanism is transport encryption of HTTP traffic via TLS, i. e., HTTPS. Better availability of certificates and browsers flagging HTTP-only connections as insecure have led to increased adoption, so that the majority of today's Web traffic uses HTTPS [10,18]. Thus, it is considered a state-of-the-art technology to protect personal information [31,52]. To detect if a website requested users' personal data without securing it with HTTPS, we created a list of terms for personal information (e. g., firstname or password) likely to be used as names for input fields that request the corresponding piece of information. In an iterative process we checked our list against the actual names of form fields used by websites, removed terms that led to many false positives, and added newly found, more specific terms (e. g., login_email). We ended up with a final list of 24 phrases (see Appendix A) that is not comprehensive but designed to reduce false positives. We flagged a site as violating the HTTPS requirement if one of the terms on the list was used in the name or id attribute of HTML input fields and the site did not use HTTPS. Manual validation yielded a prevalence of false positive cases of 3.6 % on the 250 sampled sites with this issue.
3.1.4 Publicly Accessible Git Repository ( Git). If repositories for software version control systems such as Git or SVN are accidentally publicly accessible, they could potentially leak confidential information to outsiders, such as hardcoded encryption keys or credentials [49]. Considered a security vulnerability, this issue was already the subject of previous security notification campaigns [36,49]. We selected it as a security issue to compare against our privacy notifications because it is still a common problem, can be accurately detected, and, like the privacy issues, concerns a specific domain rather than a specific server (that could host multiple domains). Most importantly, this issue can be tested in a non-intrusive manner, which is a core aspect of ethical security vulnerability checks [49,50] and excludes any vulnerability for which even a proof-of-concept would require some server-side code execution, which could be considered illegal in some countries. To check websites for publicly accessible Git repositories, we used standard HTTP requests, as they were faster and less resource intensive than OpenWPM. We tried to access the file domain.tld/.git/config; if it contained the line [core], we requested .git/HEAD. This either directly provided the hash of the currently checked out commit or pointed to a branch, so we could retrieve that branch's commit hash from .git/refs/heads/<branch>. If the commit hash could also be found on GitHub, we did not consider the domain problematic, assuming that a repository also published elsewhere does not increase the attack surface [49]. Our check did not further investigate if the repository indeed posed a security risk, because once the presence of a publicly accessible repository has been confirmed, it would be unethical to search its content for sensitive information.

Initial Domain Set
In order to obtain an a large and diverse initial set of websites to analyze, we leveraged a public domain list provided by the TheIn-ternetBackup project [5], whose goal was to compile a list of every domain on the Internet. Our starting point was a domain list with 252 million domains from February 2020. To reduce the number of sites subject to resource-intensive checks, we defined additional criteria for a website to be a candidate for a detailed check: • EU-based: To ensure that EU privacy laws applied to the monitored websites, we first resolved the domain names and checked that all requested IP addresses were within the EU, based on Maxmind's GeoIP database [40]. • Not parked: Next, we excluded domains for which the resolving nameserver was a known domain parking service. We identified these by manually extending the list by Vissers et al. [60]. These DNS-based checks reduced the number of candidate sites to 51 million. • Active web server: We issued HTTP requests to all remaining domains to check whether they provide a website. If the HTTP response status was below 400, we kept the domain in our data set, leading to around 30 million candidate sites. • No previous opt-out: We excluded 1,513 websites that had opted out of our previous notification studies [49,50]. • Public audience: We excluded sites that only offered limited content or did not seem to be targeted at a public audience (e. g., "under construction" sites). As a metric we required at least five same-site links on the front page. The check for internal links was part of a pre-study in which we visited all 30 million sites with our OpenWPM-based check infrastructure. It took three months to visit each domain once with our automated setup and check it for the presence of the four privacy issues. Overall we found 6,272,813 candidate sites (~21 %) with at least one privacy issue. Cases were not evenly distributed; most common was No Privacy Policy (17.44 %), followed by Before Consent (7.57 %) and No Consent (7.34 %). No HTTPS was rarest, with 2.85 % of sites. Checking all of these domains daily, let alone notifying all of them would have been infeasible given hardware restrictions, so we sampled 500,000 domains from the list of domains with at least one problem. Then, for each issue, we sampled up to 100,000 domains; since only about 1 in 10 problematic domains had No HTTPS, we only drew around 45,000 domains for this issue. This left us with 331,222 domains with privacy issues subject to further monitoring. Note that while this sampling was done in late September 2021, the set we sampled from contained all domains that had been identified as problematic once within the three preceding months. In addition, we found 58,715 domains with the Git issue, which we also added to the set of domains to check each day. In total, this yielded 388,825 domains for further consideration.

Notification Emails and Infrastructure
The notification process itself involved determining the email addresses to contact, the mail server setup, composition of the notification emails, and setting up a website that allowed participants to check the status of their website and learn about our study.

Contacted Email Addresses.
To identify points of contact with the monitored websites, we investigated a potential alternative to manually identified or generic email addresses: automatically finding email addresses on websites. As part of our daily Open-WPM checks, we searched the websites for email addresses likely to belong to people involved in the website's technical or legal administration. For this, we identified links to privacy policy pages as described in Section 3.1. Using a regular expression, we searched the HTML code of these subpages for email addresses. If none were found on a privacy policy page, we extended the search to subpages expected to contain generic contact information, such as "About" or "Contact us" pages. To remove false positives (e. g., file names containing '@') and to prevent emailing someone unrelated to the domain, we used only addresses with matching domains. If this procedure yielded at least one email address for the inspected domain, we emailed up to three discovered email addresses in the order served by our database and flagged the domain as being notified through (a) Parsed address(es). If no email address meeting the above criteria could be found, the domain was flagged as Generic and we sent our notifications to three generic aliases (RFC 2142 [9]): info@DOMAIN, the most frequently found email address in the first step, plus webmaster@DOMAIN, and abuse@DOMAIN, the two most commonly used email generics according to Soussi et al. [47].

Mailserver Setup.
To send notification emails, we used a designated server outside CISPA, i. e., hosted with an external server provider. This reduced the risk that our notifications negatively impacted our institution's normal email communication (e. g., in case we hit a spam trap). Both A and MX record of our subdomain notify.cispa.de point to this server. This subdomain was also used in the EHLO message. The server configuration followed best practices to increase the delivery rate, including SPF and DMARC records and DKIM signatures for outgoing emails. The policy in the DMARC record was set to 'none, ' the percentage to 100 %, and the address for aggregated reports to administration@notify.cispa.de. We also configured the reverse DNS to point to notify.cispa.de to create another clear connection to our institution and S/MIMEsigned all emails to enable validation by the receivers' email software. Finally, to reduce strain on receiving servers, we set the rate of delivery to our MX to at most one email every two seconds.

Notification Emails.
In our emails we openly identified ourselves as researchers and their purpose as being a scientific study. The sender name was composed of the name and institution of the author responsible for the notification setup. Mails were sent from a designated email address, notify@notify.cispa.de. The emails' subject line was "[Security and] data protection issue[s] on your website [DOMAIN]" or "Security issue on your website [DO-MAIN]", depending on the type of detected issues. Following prior findings that language did not significantly influence fix rates [61] and localization of notification messages may even increase the likelihood that recipients perceive them as malicious [35], all emails were sent in English. The message body introduced our project, the involved research groups and institutions, and the security and/or privacy issues identified on the respective website. We provided a description of the problem(s) and why they constituted a violation of privacy law or a potential security vulnerability. Appendix B contains an example notification email with all possible issues.

Report Website.
To aid operators in fixing their websites, we followed prior work [36,50] in providing a web interface that allowed them to track their website's status with regard to the investigated issues. Every email contained a link to a domain-specific online report, which again listed and described the issues found on the respective website, but was updated daily with the most recent check results. This allowed operators to learn if our checks still detected an issue or considered it fixed. To prevent incorrect feedback due to a flaky check, an issue's state was only reported as fixed if this was supported by the latest two checks. The online report also provided operators with the option to exclude their website from our checks and, for each detected issue, a form to report false positives. A screenshot of an example report is shown in Appendix D. The website serving the online reports was hosted at CISPA, from where the daily checks were conducted. It also contained an introduction to our research project (see Appendix C), an imprint, and a privacy policy explaining our data processing.

Research Ethics
To ensure our research followed ethical best practices, we requested approval from CISPA's IRB. We outlined that our measurement setup would not collect personal data beyond what was publicly available, i. e., email addresses found on websites or generic aliases. Beyond that we followed data minimization principles: Survey answers (see Section 5.1) were anonymized and did not contain any information that allowed us to identify the website or email address used for notification. The only information passed to the survey via URL parameters were the issues found on the website, notification round, email type, and study condition; see Sections 4.1 and 4.1.2. We received IRB approval without changes to the study protocol.
In addition, we followed best practices recommended by prior work for ethical network checks [13] and notification studies [37]. We communicated our identities and benign intentions at all points of contact with the monitored websites and their operators: In all notification emails and on the study website we identified ourselves as researchers and explained the purpose and scope of our checks and the whole research project. For the daily checks, we set the user agent of our OpenWPM crawler to "CISPA Web Analyzer (notify.cispa.de)" to point operators of the checked websites to our study website. Front office and IT staff at CISPA were briefed about the study, preparing them for operators potentially asking about the legitimacy of the study. Notified websites could use the opt-out functionality on the website with their report (see Appendix D) or send an email to be excluded from future checks. Web report accesses were collected in the report web application and disclosed in its privacy policy, drafted by our expert in data protection law. At the end of the study, we sent debriefing emails to still affected websites in the Control group (see Section 4.1), informing them about the detected issues and our study.
To ensure that the selected privacy issues were universally acknowledged violations of privacy law, the study was supervised by a legal expert with extensive knowledge of EU data protection law. In addition, we manually verified check results to ensure an as low as possible false positive (FP) rate. Still, sending email notifications for complex privacy issues at scale meant that we inevitably reached out to some domains that were FP cases for a privacy issue. When notification recipients emailed us about (presumed) FPs, we performed a timely manual inspection of their website and responded with the result to minimize recipients' time of uncertainty about the issue. We also routinely checked for FPs on domains whose operators had emailed us with an unrelated question. 75 out of 414 email conversations with recipients of privacy notifications concerned (presumed) false positives. On 33 of these 414 domains we manually found true FPs. The rest were presumed false positives, reasons for which we explore in Section 6.3.3. We assume that most recipients of a notification caused by an FP contacted us before investing a significant amount of time in the issue. Nearly 50 % of true FPs could also be quickly identified by non-experts, such as websites actually having a privacy policy or not targeting people in the EU. Thus, we believe that we did not cause undue burden on notification recipients and the benefit for the other notified operators outweighed the potential cost for the few true FPs.

Limitations
We defined technical constraints for privacy issues in collaboration with a legal expert, but since there are multiple steps involved that relied on external sources (e. g., for geolocation of IP addresses or classification of third-party requests), we can only aim to minimize, but not eliminate false positives. Our study design also did not focus on avoidance of false negatives, so we likely missed many websites that in fact did have privacy issues. For example, if a site provided a link to a privacy policy, we did not further investigate if that page actually contained the required disclosures. Due to use in a prestudy, we removed .de domains affected by the Git issue from the set of domains. However, we do not believe this had any effect on our findings. We openly communicated that the notification process was part of a scientific study, possibly prompting fixes that would not have taken place otherwise due to the observer effect [37].

MEASURED NOTIFICATION RESULTS
Evaluating the measurements first requires us to describe the final parameters we used when launching the notification campaign. After that, we present our results regarding website reachability, web interface usage, remediation rates, and the influence of warnings.

Final Study Parameters
4.1.1 Notified Domains. From the 388,825 domains initially considered, only 190,491 were still problematic when we started to send out notifications on October 20, 2021. The remainder was either fixed without our notification or could no longer be reached. To test our infrastructure, we sent out emails to 19,142 domains, which we removed from further consideration in our study. In this beta test, we noticed and fixed some minor issues. The full notification campaign started on November 3, 2021 and considered all 159,856 domains which were still flagged as problematic on November 1. and one that only received information about the issues but No Warning (40 %). The concrete wording of the warnings can be found in the example notification in Appendix B. In our analyses we also differentiate domains by email type, i. e., whether we contacted them via a Parsed or a Generic email address. We do not consider these "true" experimental conditions as we did not have any influence on whether we were able to find an email address on a website.

Reachability
As expected with generic and automatically extracted email addresses, not all emails reached their recipients. Only for 70,542 (55.5 %) of initially notified domains at least one email was successfully delivered, which we assumed if our mail server was able to hand over the email to the recipient's mail server. While such a successful handover does not mean that an email will reach the recipient's mailbox, this metric still provides an upper bound for the notification delivery rate. The difference between Parsed and Generic email addresses was quite high: While 87.8 % of initial notifications to Parsed addresses were successfully delivered, this was only the case for 33.8 % of emails to Generic addresses.

Web Interface Usage Statistics
As described in Section 3.3.4, each notification email contained a link to our study website with more information about the project and a personalized report for each domain that also let participants report false positives or opt out of the study. Over the course of the study, 5,731 reports were viewed, 259 false positive checks were reported, and for 466 domains the web interface was used to opt out of daily checks.

Remediation Rates
To determine if our notifications had any measurable effect, we checked the monitored websites daily over the course of two months.

Sliding Window Approach.
To avoid domains incorrectly flagged as fixed because of one-off checker timeouts or page maintenance, we implemented a sliding window approach to determine if an issue persisted: For a given day , we considered a domain to be problematic if at any point in time within a 7-day window ( , +6 ) our checker had identified the reported issue to still be present on the site. This 7-day window allowed us to obtain at least one real measurement result (i. e., a true/false evaluation of the checked issue, not a returned error or a missing data point) for 98.5 % of evaluated windows, resulting in a robust check.

Evolution of Problematic Domains
Over Time. Figure 1 (a)-(e) shows for each issue, experimental condition, and email group the percentage of domains considered problematic at a given point in time. We investigated the persistence of issues more closely at four distinct points in time: one and two weeks after both initial notifications and reminders. Table 2 in its % column lists problematic rates for November 10, 2021 and Appendix F for all four dates. Overall, we observe for a given privacy issue a similar downward trend in problematic domains across study conditions and email groups, with them mostly differing by only between 0-1 percentage points, though the Control group behaves as expected in yielding the highest rates of persisting issues. Git rates also follow this pattern but yield slightly higher differences to Control, in the dimension of 1-2 percentage points. This lack of a larger measurable effect is a direct result of our notifications' low delivery rates: With many domains in the treatment conditions never receiving a notification email, this subset is expected to behave similarly to the control group, exhibiting the same rates of natural decay of issues. For the two issues related to the use of third-party cookies, No Consent and Before Consent, the number of problematic domains steeply dropped between November 15 and 16, 2021. Inspecting the affected domains, we found the cause to be a Twitter cookie named lang, originating from cdn.syndication.twimg.com, that was no longer present from November 16. This coincides with major platform updates at Twitter, including migration to Twitter APIv2 1 . Figure 1 (f) compares the evolution of problematic domains by issue aggregated across all experimental conditions. Looking at that figure and the rates in Table 2 and Appendix F, we observe that if it were not for the Twitter drop, there would be a consistent difference of about 5 % between the plot for the Git security issue and those for the privacy issues. This suggests that either websites were more inclined to fix security vulnerabilities or privacy issues required more time to address. Given the two-month period of our experiments, we could not conclusively figure out the exact reason. With slightly higher effects (diff column in Table 2), No Consent appeared to be the privacy issue easiest to remediate.

Statistical Significance.
We also investigated the significance of differences in remediation rates at the aforementioned four points in time. We conducted Fisher's exact tests [19,20] on the null hypothesis that the number of problematic vs. no longer problematic websites in the experimental conditions (Warning and No Warning) does not significantly differ from the Control group, and in an identical fashion for email types. Table 2 in its column shows the results for November 10, 2021 and Appendix F for all dates. We used the Holm-Bonferroni method [27] to correct for multiple testing over time. While for November 10 the null hypothesis cannot be rejected for Before Consent and No HTTPS regardless of the presence of warnings, there is a significant difference in the distribution of problematic domains for No Consent and Git in both Warning and No Warning conditions. As shown in Appendix F, these observations largely also hold true over time. The only cases where differences between Control and the treatment groups emerged later was No Privacy Policy, for which the No Warning condition did not lead to rejection of the null hypothesis on Nov 10 (while it did on any other date and across all dates for the Warning case), and No HTTPS, which only yielded significant differences to the control group for the Warning condition on November 28, a week after the reminder. This could confirm an earlier security notification study that found a limited effect of reminders [49], but differences between issues suggest that some privacy issues take a longer time to be addressed. For email type, we observe similar tendencies over time. Across issues, Parsed email addresses more frequently resulted in statistically significant differences in fix rates compared to the control group, except for the surprising case of Git.

Remediation by Website Popularity.
Websites' willingness to remediate security and privacy shortcomings may depend on available human and monetary resources and how well a site is maintained in general. Hence, we investigated if issues are less likely to persist on more popular websites. To obtain popularity metrics for as many websites as possible, we queried the Google Chrome User Experience (CrUX) data set [24] via BigQuery as described by Durumeric [11] for the full domain rankings from November 2021, when we sent out notifications. For domains listed twice due to CrUX differentiating between HTTP and HTTPS, we used the higher rank. This yielded an overlap between the CrUX data  Contrary to our expectations, for most issues and points in time we found the rates of problematic domains for CrUX-ranked domains to exceed those for unranked ones by 0-1 percentage points. For Git, this was even more pronounced, with differences mostly between 2-3 percentage points, except for the Git-Parsed combination, which followed the overall 0-1 % pattern. We presume this difference to be mainly due to issues "fixing" themselves naturally, with unranked websites being less reliable to reach and more likely to be taken down permanently. Breaking this pattern, for No Privacy Policy notifications to Parsed email addresses, CrUX domains consistently exhibited lower rates of problematic checks, though the difference also mostly lay between 0-1 percentage points.

Influence of Warnings
For more insights into the effect of the presence of warnings on fix rates, we took a closer look at the set of domains for which at least one email could be successfully delivered according to our earlier definition, i. e., handed over to the next mail server. To determine the influence of warnings on fix rates, we computed logistic regression models [41,42] for each issue and four different points in time: one and two weeks after the start of sending initial notifications and reminders, respectively. Table 3 shows the regression models for Git and No Consent, both relative to the Control group; Appendix G for all investigated issues. For Git, No Privacy Policy, and No Consent, we observe a statistically significant influence of the Warning and No Warning conditions on fix rates compared to the Control group, while the models do not show such influence for the Before Consent and No HTTPS issues. Still, even in the case where Warning and No Warning perform significantly better than the Control group, we cannot observe any difference between the estimates for these two conditions that does not fall within the standard error. This highlights that while receiving a notification significantly improved remediation, the presence or absence of a warning does not. This contrasts with prior work [38], which claimed warnings did play a role in remediation success (albeit limited to German websites).

GATHERING RECIPIENT FEEDBACK
In order to gain further insights into the measured changes (or lack thereof), we leveraged two channels of communication with recipients: an online survey and email conversations.

Survey
For consistency and to make sure participants received the survey invitation when the decision how to react to the notification was still fresh in their minds, we sent the survey link with all emails, i. e., initial notification and reminder for domains in experimental conditions (Warning and No Warning) and with the debriefing message for the Control group. The survey was implemented using a LimeSurvey instance hosted at Ruhr University Bochum. We first asked participants to assess the correctness of our checks (Q1), about prior awareness of the detected issue(s) (Q2), and plans to address them (Q3-4). Participants with privacy issues on their website were asked about the applicability of the GDPR (Q5-6), past changes to their website due to privacy legislation (Q7-8), and the influence of GDPR-mandated fines (Q9-10). Next, we asked all participants what type of support they would find useful to fix the issue(s) (Q11). We asked for participants' role(s) with regard to the website (Q12) to determine if we had reached a person with a suitable background to address the issue(s). The survey concluded with the opportunity to provide general feedback about our study (Q13). The full questionnaire can be found in Appendix E. One of the authors conducted a manual thematic analysis on open-ended survey answers to inductively identify common themes and sentiments. We categorized the answers via labels informed by the survey questions and additional themes found in the answers. Note that survey responses are subject to self-selection bias, which includes people with a strong (negative) experience with our notification process being more likely to provide feedback.

Email Communication
Sending automated emails at large scale inevitably results in large volumes of incoming mail, including automated responses from ticketing systems, delivery status notifications, and "out of office" messages. To support participants with fixing identified issues and obtain more information how our notifications were perceived, we focused on incoming responses composed by humans. We answered emails in German or English, depending on the language used by the sender, and if requested, we also sent a German translation of our notification message. When asked for advice, we only referred to third-party resources that either had been published by the hosting company of a website, by a data protection authority in the website operator's country, or by the vendors of third-party software already used by the website. Upon request we provided the names of third-party cookies or URLs to Git repositories that had triggered a problematic check result.
The researcher who had signed the notifications answered incoming emails from notification recipients according to these guidelines and categorized them on the conversation level by identifying recurring themes. After about 50 conversations we found the topics to have reached saturation. From the resulting list we removed very rare codes, refined the definitions of the emerged categories, and  (3): Including sentiments that the notification could be spam, verification requests to other points of contact at our institution, and opt-out requests. The remaining emails were single-coded by two coders, with uncertainties resolved via discussion. Each conversation could be assigned an arbitrary number of codes. As we had not passed unique IDs to the survey, we could not identify overlap between survey and email respondents, so some individually reported sentiments from these analyses may originate from the same person or domain. 6 PARTICIPANT FEEDBACK 6.1 Overview of Survey and Email Responses 6.1.1 Survey Participants. The survey link was clicked 1,890 times. 1,556 people only accessed the welcome page, 121 provided partial responses, and 213 completed the survey. We discarded all incomplete responses, plus one response for which the survey parameters had not been passed and questions based on specific detected issues had not been displayed. This left us with 212 complete responses. Table 4 shows the sample of participants who took the survey by experimental condition, email type, detected issues on the website, and notification round. While full responses were roughly equally distributed between the Warning and No Warning conditions, as well as email address type, most responses were collected through initial notifications (including control group debriefing), as opposed to the reminder. This corresponds with earlier findings that notification recipients either tend to act upon the first received email or not at all [35]. Two thirds of survey participants had been notified due to an open Git repository, while privacy issues less frequently motivated people to take the survey. This hints at security notifications being either taken more seriously or at least leading to recipients' higher willingness to interact with us. Appendix E provides an overview of all survey questions with response counts.

Email Communication.
We received a total of 760 emails in 621 conversations with the operators of notified domains. 414 of these domains had been contacted because of a privacy issue and 167 because of a publicly accessible Git repository. 19 emails could not be assigned to a domain due to a lack of provided information. Most emails (662) had been sent to the two email addresses designated for this study. We also received 20 emails forwarded by the CISPA front office, and 85 had been sent to the institutional email address of the author who had signed the notification email. This was mainly done to verify if our notifications were legitimate. 7 emails had been sent to both the author and the project addresses. Appendix H shows how often each code (see Section 5.2) was assigned to email conversations with domains with security and privacy issues.

Who Did We Reach?
If emails were successfully delivered, they had significant impact on fix rates, so we wanted to understand who the recipients were.
In the multiple-choice Q12 the majority of survey participants reported technical roles (developer and similar 57.1 %; administrator/operator 60.4 %), followed by roles related to the website's content (19.8 %) and product/project management (13.7 %). As for people in legal advisory roles, data protection officer ranked fifth (9.9 %), while the role as legal counsel was rarer (2.8 %). The involvement of these two roles was equally distributed between privacy and the Git issue(s). While survey-takers only represent a fraction of emailed websites, this provides evidence that the people who ultimately felt incentivized to react to the notification mainly hold responsibility for a website's technical administration or content.
In email conversations we looked for how often the recipient referred the handling of the issue to another person (code: notified). This was the case in 15.9 % of privacy conversations and in 10.2 % of those about the Git issue. Explicitly mentioned people or entities for both types of issues included the IT department or webmaster, in the Git case the security team, and for privacy notifications a lawyer, the cookie plugin provider, or the marketing department.

When Do Recipients (Plan to) Remediate?
Beyond the daily website checks, we wanted to gain additional insights into notification recipients' remediation behavior and future plans to address the issue(s) (or not). For this, we analyzed recipients' reported awareness of the issues and willingness to remediate them. In survey Q2 we found that while most participants (72.6 %) reported to have been unaware of the issue(s) prior to receiving the notification, this number was higher for security (81.4 %) than for privacy issues (56.8 %). Participants' reported plans to make subsequent changes to the website (Q3) also differ: While overall 81.6 % planned to make changes, this applies to 90.7 % of participants notified of Git issues but only 64.9 % of people with privacy problems. Pairing these results from Q2 and Q3, it seems that privacy issues are more often knowingly ignored.
Email analysis revealed similar differences in remediation intentions. Privacy notification recipients mostly told us that the issue(s) would be handled in the future (37.9 %, code: will-handle), while only 14.5 % stated that they had already been fixed. For the security notification, we saw the opposite: For Git, in 16.2 % of conversations the recipients stated that the issue would be handled in the future, while 44.9 % reported that the issue had already been fixed. As in the survey answers, this could either mean that privacy issues are more likely to deliberately be left unfixed -or that fixes simply take longer as they are more complex and may require the involvement of legal professionals, for example, to draft a privacy policy.

Roadblocks to Notification Success
6.3.1 Language Barrier. As we emailed domains from various countries in English, we may have contacted recipients in a language they do not understand, as first indicated by 3 survey participants in Q13 who found it hard to understand or assess the trustworthiness of an email written in English. More concrete evidence were the 17 translation requests we received via email. Most requests were for German, but some also for French and Czech.

Notifications Perceived as Spam or Otherwise Malicious.
When asked for general feedback in survey Q13, 9 out of 76 participants (11.8 %) mentioned that they initially had been suspicious that the notification was spam or a scam attempt. To fight this impression, one participant suggested to point out the S/MIME signature in the email body, as "[s]pammers don't go out of their way to sign their emails from a public CA issued PEM certificate" (P1254).
Similarly, email analysis found in 7.2 % of conversations about Git-notified domains and in 12.1 % of privacy-related correspondence that the recipient was not sure if the notification email had been sent with benign intentions (code: unsure-scam). A special case were emails asking if the notification email had really been sent by our institution. 4 such verification requests were sent to dedicated project email addresses, 35 to the institutional address of the author who had signed the notification emails, and another 14 were sent to CISPA's front office. The language of the notifications may also have contributed to this. 6 email respondents wondered why we, as German research institutions, sent English notifications (code: expected-german); 3 of them stated they were not sure if our email was benign. This was also observed by Li et al. [35], who reported that emails the recipient expected to be in a different language (e. g., based on the sender's country of origin) were sometimes considered phishing or spam.

(Perceived) Incorrectness of Reports.
In survey Q1 the majority of participants answered that the report was correct, with rates highest for Git (87.9 %) and lowest for No Consent 2 (45.5 %). Correspondingly, reported incorrectness rates were lowest and highest for these two cases, with 5.7 % and 22.7 % respectively. Uncertainty about the correctness of the report was highest for the Before Consent (29.6 %) and No Consent (18.2 %) cases. This hints at notification recipients often finding it difficult to determine which third-party cookies were present on their website and if they required user consent. While not a true false positive for the Git check, 3 survey participants notified about it replied in Q4 that they did not intend 2 We do not consider No HTTPS here, as only 4 survey participants had this issue. to make changes because the issue did not pose a security risk, as their Git repository did not contain any sensitive information, was not under their control or not accessible.
In emails we also received feedback about check results being false positives, for 18.1 % of conversations about a privacy issue but also for 6.0 % of those about Git. 16 emails stated that no sensitive data was stored inside the Git repository, though 3 reported that they had still made the repository inaccessible.
Manual checks revealed the majority of false-positive claims for the Git case to be due to failure to reproduce the issue. Many recipients of Git notifications tried to access <domain>/.git/, saw a "Forbidden" error page from their web server, and falsely assumed that this meant the Git repository was inaccessible, while in fact directory listing was forbidden. It is likely that they then stopped to further investigate the issue, leaving it unfixed.
6.3.4 Perceived Inapplicability of Privacy Legislation. One recurring theme in both the answers to multiple survey questions and email conversations about privacy notifications was recipients' perception that the privacy legislation in question did not apply to their website. In survey Q5 the 74 participants notified about a privacy issue were asked whether they thought the GDPR applied to their website. 63.5 % thought it did, while 20.3 % did not think so and 9.5 % were not sure. When asked in Q6 why they thought the GDPR did not apply, we received 14 replies. 6 answered that they were not located in the EU ("Because the UK is no longer in the EU" (P349)), illustrating unawareness of the GDPR's extraterritorial applicability to non-EU websites with EU visitors. Interestingly, several of these respondents were based in the UK, which still has a verbatim copy of the GDPR in its national legislation, so the same legal requirements apply. Regarding the material scope of the GDPR, 7 participants claimed to not process any personal data, unaware that even temporary storage of IP addresses is considered processing of personal data under EU law (see Section 3.1.1).

Privacy
Indifference. Another demotivating factor that only emerged for privacy notifications was a general disdain for privacy legislation and its requirements. We found such sentiments in the answers to multiple open-ended survey questions. Asked in Q4 why they did not intend to add a privacy policy to their website, one participant replied "because these rules are plain stupid!" (P1131), and in Q8 another refused to make any changes at all due to the GDPR: "does not matter -GDPR is sucks [sic]" (P671).

Motivation for Remediation
The survey also investigated factors that motivated participants to (want to) take action, particularly awareness of fines mandated by the GDPR. In Q9 more than half of the 74 survey participants with privacy notifications (38, 51.4 %) had already been aware of these fines before the notification. Another 9 (12.2 %) had learned about them via the email, with 7 participants in the Warning and 2 in the No Warning condition. Still unaware of fines were 20 participants (27.0 %), 8 from the Warning and 12 from the No Warning group. Half of the 8 warned but unaware participants had only seen ePrivacy warnings, but the other 4 had (also) been warned about GDPR-mandated fines. This shows that notifications have limited educational impact about privacy legislation and potential fines.
Q10 explored how knowledge of GDPR-mandated fines had influenced participants' decision to fix the detected issue. 13 participants (27.7 %) answered that they had not been influenced by the risks of fines, stating as their motivation that they "wanted to be responsible" (P45) or "believe[d] that GDPR and compliance with it [was] important" (P1505). Another 13 (27.7 %) explicitly acknowledged that fines were a motivating factor in their decision ("want to prevent paying fines" [P973]). 7 of them had received a warning notification and 6 an email without a warning. Though these answers may suffer from social desirability bias, this hints at fines for GDPR noncompliance being known and influencing fix rates, regardless of whether they were explicitly mentioned in the notification.

How Can We Help Websites Fix Issues?
Survey Q11 asked what additional information recipients would have wished for to better understand and fix the notification issue(s).
We received 129 open-ended responses, many of which expressed generic sentiments: 37.7 % found the notification helpful and the information to be sufficient. 12.3 % would have appreciated more detailed guidelines or links to external resources on how to fix the issue(s). 16.5 % asked for additional documentation of our checks: 15 Git-notified participants suggested to add the URL for the repository in question, and 2 asked to include a check whether any sensitive information was present in the repository. While the first suggestion is an easy fix for future notifications, the latter would require extensive resources and would raise ethical concerns. Regarding the use of cookies without consent, we were repeatedly asked to add to the notification the names of the third-party cookie(s) that had triggered the problematic flag, which is also feasible.
Classification of email conversations confirmed this. A major category were requests for more information: 8.0 % of the 414 conversations regarding domains with privacy issues asked about privacy checks, especially the name of the problematic cookie (5.3 %); and questions concerning Git checks (10.8 % of the 167 Git conversations) most often were interested in the URL of the publicly accessible repository (5.4 %). We also received more generic requests, such as what to do in general, if certain changes would make the website compliant with privacy law, or about our research project.

How Were the Notifications Perceived?
It should be best research practice to notify affected parties about potential security or privacy issues on their systems, but there is a risk of backlash. Hence, in survey Q13 we asked for general feedback about our project and received 76 responses. Sentiments varied greatly between recipients of security and privacy notifications. 76.0 % of respondents notified about Git thanked us for the notification or voiced positive feedback ("This is an amazing project, please keep up the good work to make the internet a more secure place!" (P1675)), but only 50 % of respondents with privacy notifications did so ("Thank you for this hint! There are so much [sic] rules. For a little webmaster it's hard to know everything. It's really great to know there are some who help the little ones ;-)" (P840)). Negative sentiments were only expressed for privacy notifications ("the stated analysis is only 'may be' .... You have just wasted our time & energy" (P1685)). Repeated criticism included the privacy notifications being false positives, too threatening, unwanted, not sent in the participant's language, or sent with ill or monetary intentions. This confirms the sentiments reported in Section 6.3.
Email feedback contained similar differences in sentiment. Here people thanked us for the notification (code: thanks) in 74.9 % of conversations with recipients of security notifications, but only 56.0 % of privacy conversations. To distinguish between "thanks" and real enthusiasm for our project, we used the code great-project, assigned to 16.2 % of security and 2.9 % of privacy conversations. Correspondingly, the distribution of negative sentiments towards our notifications or project was reversed, assigned to 5.3 % of privacybut only 1.8 % of security-related email conversations.

DISCUSSION
Our study identified recommendations both for future research in web privacy as well as for the public entities tasked with the application and enforcement of privacy legislation.

Privacy vs. Security Notifications
We compared the effects of notifications about privacy issues with those about a security problem. Challenges faced by both types include how to reach the responsible parties, language barriers, and lack of a trustworthy messaging channel. These have already been identified by previous work on security notifications and continue to pose significant problems for any campaign that aims to reach people via email at scale, as it is hard for both computer systems and humans to differentiate automated emails sent with beneficial intent from those sent for malicious purposes. Specific to privacy notifications is the obstacle that many recipients are not aware that certain legal requirements apply to their website, either because of misconceptions regarding the territorial scope of privacy laws or the website's data processing operations. Future research in this area is encouraged to educate notification recipients in this regard and provide concrete information why the respective law was deemed to apply to the recipient's website. This is especially important given our observation that the motivation to make changes due to privacy notifications appears to be extrinsic (awareness of fines) more often, while fixes of security issues tend to be more intrinsically motivated.

Message Tone and Content
Security and privacy notifications were met with very different sentiments, which could be rooted in message content (mentioning of privacy laws) or tone (presence of warnings). Security notifications evoked more positive sentiments and fewer threats of legal action. To relieve recipients' anxiety and anger, we recommend researchers in future privacy notification studies to explicitly explain that they will not pursue any legal action against the recipients. Recipient feedback also indicated widespread problems to identify the third-party plugin or subpage that had triggered the placement of a third-party cookie before or without the visitor's consent. We recommend that future notification studies provide the necessary details to help recipients pinpoint the problem, in our case the Git URL or concrete third-party service or cookie and subpage that triggered the detection mechanism. Links to concrete guidelines by DPAs or courts (e. g., that pre-ticked checkboxes do not constitute informed consent [16]), could aid notification recipients in understanding the issue and why it applies to their website.

Call for Guidelines and Standardization
Our work shows that it is generally feasible to identify privacy issues on websites independent of specific vendors or consent frameworks, with a prevalence of false positive cases in the low single digits in our set of manually verified websites. Still, these checks were designed to minimize the number of false positives for the purposes of this study and false negatives were not a concern. It may well be a requirement in other contexts, such as automated privacy audits designed to take the burden off DPAs in enforcing privacy legislation. Identifying vendor-agnostic privacy issues at scale with low numbers of both false positives and false negatives is still a significant challenge, given the differences in implementation of, for example, privacy policies or consent to the processing of personal information. Regulators could aid privacy researchersand themselves in enforcing privacy laws -by issuing more concrete guidelines how to implement legal requirements, as in the example of the CCPA that requires a "Do Not Sell My Personal Information" link. The persistent challenge to enforce privacy laws online in the light of limited human and monetary resources requires long-term assistance via automated audits. The challenges with vendor-agnostic checks could be alleviated via standardization of how privacy-related information is presented on the Web. Hence, web researchers and standardization committees are encouraged to create new and build upon existing proposals how to unify the presentation and user control of a service's data processing practices.
To prevent any such standard from suffering the same fate as past web privacy mechanisms relying on voluntary adoption, such as DNT or P3P, regulators should make adherence mandatory.

The Challenge of Reachability
Using email addresses found on websites had promising results in terms of reachability -87.8 % of initial notifications for Parsed were successfully delivered, but only 33.8 % of emails to Generic addresses. While this does not guarantee that the correct person is reached and they act upon the notification, this approach can help overcome one of the obstacles in the way to reach the people responsible to fix security or privacy issues on websites. It needs to be noted that this approach can possibly introduce bias, as well-maintained websites are more likely to provide contact information, including an email address. The reachability problem also provides an opportunity for standardization: In the vein of security.txt [21,22], a proposed standard to help web security researchers identify points of contact, a file privacy.txt could serve this information for privacy-related issues -and also communicate a website's data processing practices in a standardized format or at least contain a link to its privacy policy. Until then, future privacy notification studies could also leverage security.txt to contact websites about privacy issues.

The Future of Privacy Notifications
Our results show limited success of our notification campaign, especially when weighting this outcome against the resources required to detect and notify websites about privacy issues at scale. Similarly, the improvements proposed above will be futile if recipients of notification emails do not trust the sender do not consider privacy violations a problem whose remediation is an urgent matter. Still, we believe that large-scale privacy notifications can be a valuable tool in improving web privacy, but they need to be accompanied by other measures to overcome these obstacles.
To increase sender credibility and authority, researchers could cooperate with data protection authorities for future notification campaigns about issues related to data protection and privacy. The role of the DPA would be to provide sender credibility via their legitimization as a public authority and to accompany the campaign with information and enforcement capabilities to raise awareness for the issues. They could communicate to the general public the goal of the notifications, participating research institutions, investigated issues, guidelines on how to fix them, territorial and material applicability of the relevant laws, and possible consequences of non-compliance. A DPA informing about the latter is likely to be met with less adversity, as enforcing applicable privacy laws is their core task. Researchers, in turn, could supply the personal and technical resources that public authorities often lack and provide the notification infrastructure, expertise to more reliably detect compliance issues at scale, and (limited) support with fixing them.
Past campaigns show that this could be a promising approach. Some DPAs already have experience with privacy-related web measurements, such as the "Cookie Sweep" [2] carried out by multiple national DPAs in 2014 to inform EU institutions about websites' use of cookies and obtain first evidence for ePrivacy compliance. The notification campaign by privacy NGO noyb [44] (see Section 2) shows how external entities can support authorities in enforcing privacy legislation. Manual analyses limited the scope of these campaigns, but they both illustrate where DPAs and privacy researchers could benefit from each other to help enforce privacy legislation at scale. These multi-modal campaigns might not only have a broader effect, but also provide opportunities for further research, such as evaluating the usability of guidelines to fix privacy issues.

CONCLUSION
We conducted a large-scale email notification campaign to investigate if this approach can also help websites fix more complex privacy issues like missing privacy policies and incorrectly implemented consent notices and to determine how they compare to notifications about security vulnerabilities. Though overall fix rates are higher for security than privacy issues and the latter show tendencies to be addressed at a later point in time, we still find a statistically significant influence of our notifications on remediation rates. To overcome the problem of websites being hard to reach, we identify a promising approach in automatically extracting contact information from websites. Qualitative feedback from email conversations with recipients and survey responses shows that website owners are less open towards notifications about privacy issues than a security vulnerability. Reasons include limited willingness to make changes for privacy compliance, widespread misconceptions about privacy laws' applicability, and often greater necessary effort to identify and fix the problem. Even though warnings about potential fines do not increase remediation rates, they do at times incur anxiety and anger with recipients and corresponding backlash towards the senders. Future work is encouraged to explore if more specific information about the privacy problems and assurance of benign intent can yield more positive reactions and make email notifications a tool that can support large-scale privacy compliance.

Hello,
We are a group of security and privacy researchers from the CISPA Helmholtz Center for Information Security and Ruhr University Bochum in Germany. As part of our current research project, we analysed potential security and data protection issues on websites.
We would like to raise your attention to the following security and data protection issue(s) on your website [DOMAIN]. Please note that we do not offer a conclusive legal assessment or consultancy on an individual website's legal compliance.
No privacy policy. For public websites that use European domains, are hosted in the EU, or may be used by European users, any collection of users' personal data is governed by the EU General Data Protection Regulation (GDPR). If a website meets these conditions, the operator is legally required by Article 13 of the GDPR to have a privacy policy explaining the use of their visitors' personal data. Personal data also encompasses the processing of communications data such as IP addresses of users even if no additional information is collected. The privacy policy has to inform users about the use of their personal data in a concise, transparent, intelligible, and easily accessible form.
Our automated analysis of your website did not detect a privacy policy, which may indicate noncompliance with the GDPR's information requirements.
Input fields for personal information without HTTPS. Article 32 of the GDPR requires data controllers such as website owners to implement appropriate technical and organisational measures to ensure a level of security appropriate to the risk, taking into account the state of the art. Protection of users' communication and interactions with your website via HTTPS is considered state of the art in data security.
Our automated analysis detected input fields on your website that allow users to enter personal data without using HTTPS secure communication to prevent eavesdropping and phishing. This may indicate noncompliance with the GDPR's data security requirements.
Use of third-party cookies without consent notice. Under Article 5 Paragraph 3 of the EU ePrivacy Directive (Directive 2009/ 136/EC) and respective implementations of the Directive into national law of the EU member states, the setting of individual cookies on the user's terminal equipment that are not strictly necessary for the functioning of the website is only allowed if the user has given his or her prior consent.
Our automated analysis did not detect such a consent form for the third-party cookies on your website. This may indicate noncompliance with EU ePrivacy requirements.
Third-party cookies set before interaction with consent notice. Under Article 5 Paragraph 3 of the EU ePrivacy Directive (Directive 2009/136/EC) and respective implementations of the Directive into national law of the EU member states, the setting of individual cookies on the user's terminal equipment that are not strictly necessary for the functioning of the website is only allowed if the user has given his or her prior consent. Such consent has to be given in advance via a meaningful interaction by the user. According to our automated analysis, your website does provide users with a cookie notice or consent form, but the cookies are set before any meaningful interaction of a user with the consent form takes place. This lack of explicit consent may indicate noncompliance with EU ePrivacy requirements.
Publicly accessible Git repository. If the configuration folder for Git (.git) is reachable through HTTP, an attacker may copy the content of this repository. This allows an attacker to access the source code versioned in this repository, including any credentials or other sensitive data possibly stored there. Our automated analysis detected a publicly accessible Git repository on your website. Note that we only check for the existence of a repository and do not attempt to download any actual content. Hence, we cannot state if it contains any sensitive information.
If in Warning condition: • If Git: Please note: In the worst case, access to configuration files with credentials could lead to an attacker taking over your entire website. in the email. The color of the accordion menu for each detected issue changed to display its current status: red for not detected as fixed, yellow for detected as fixed once within the last two days, and green for detected as fixed for at least two days. The red "Please note" box was only shown for websites in the Warning condition and displayed the corresponding warning message(s).

E SURVEY Survey Title
Survey on Security and Data Protection Notifications

Intro Text
We are security and privacy researchers from the CISPA Helmholtz Center for Information Security and Ruhr University Bochum in Germany. In our current research we are trying to better understand how to notify websites about security and data protection issues. We recently emailed you a security and data protection notification from notify@notify.cispa.de. You can help us improve our notification process through completing this survey. The survey is short and anonymous, and all questions are optional, so please answer the ones that you feel comfortable with. Your feedback is very valuable to us and we really appreciate your time.

Privacy Policy & Consent
We take great care in protecting our survey participants' privacy in accordance with the provisions of the General Data Protection Regulation (GDPR). Your answers to this survey will be stored securely on a server hosted by Ruhr University Bochum, Germany. Any of the survey data will only be accessible by the researchers involved in this project and will not be correlated with other data or otherwise used to identify individual participants. If we make data from this research available to the research community or the interested public, we will only publish it in an aggregated form that does not allow anyone to identify you or the website for which we sent you a notification email. You can find the contact information of the responsible data protection officers at https://notify.cispa.de/ privacy_en.html.
Participation. Your participation in this research is completely voluntary. Once you have started the survey, you may cancel at any time by clicking the "Exit and clear survey" button in the upper right part of the screen, and your answers will be discarded.
Contact Information. If you have any questions, comments, or concerns about the study either before, during, or after participation, please contact us at info@notify.cispa.de.

Survey Questions & Responses
First we would like to ask you about the security and data protection issues we found on your website. Here is a list of the issues we found: [of the following, only the detected issues were shown] • No privacy policy • Use of third-party cookies without consent notice • Third-party cookies set before interaction with consent notice • Input fields for personal information without HTTPS • Publicly accessible Git repository F RATES OF PROBLEMATIC DOMAINS AND FISHER'S EXACT TEST RESULTS OVER TIME