Transparency Guidelines for Human-Centered Security and Privacy Research

Transparency Guidelines for Human-Centered Security and Privacy Research Jan H. Klemmer, Juliane Schmüser, Jacques Suray, Jan-Ulrich Holtgrave, Byron M. Lowens, Florian Schaub, and Sascha Fahl v0.2, November 04, 2025 Request for Feedback This is a working paper of transparency guidelines for human-centered security and privacy research papers, on which we hope to collect as much community feedback as possible. Any feedback and comments are therefore appreciated, to ensure that these guidelines are relevant in practice. About the guidelines Transparent research reporting is crucial to make research understandable and replicable. The idea of this document is to provide guidelines for the usable privacy and security community on how to report our research transparently. Its focus is therefore human-subject security and privacy studies. SoKs/SLRs are not explicitly covered, though many items still apply - see the PRISMA statement for explicit reporting guidelines. These guidelines are based on research on the community’s research reporting expectations and practices [Klemmer et al., Klemmer et al.] and a workshop held at SOUPS 2025. The goal is to serve as a checklist for both authors and reviewers, ensuring that critical details are included in a paper. The guidelines aim to strengthen research quality further, improve replicability, and support newcomers in participating in our research community, both as authors and reviewers. How should these guidelines be used? - In general: This document provides two overview checklists with brief descriptions, one sorted by topic and one sorted by priority. Both link to more extensive descriptions for each criterion. - As an author: When designing and conducting your research and writing your paper, see what transparency criteria are relevant to your project. The guidelines provide instructions and examples that offer inspiration and can be adapted to a specific paper. - As a reviewer: When reviewing papers, you can check with these guidelines whether something is missing and should be added to a paper. Reviewers can reference these guidelines in their review to substantiate their critique and point authors to these guidelines for improving a paper’s reporting. - These guidelines should apply to the majority of papers, providing helpful advice on research reporting. However, there may be exceptional cases and exceptions that necessitate deviating from widely accepted practices. Therefore, these guidelines should be considered in the context of the respective paper. - These guidelines do not aim to prescribe how research should be conducted or should (not) be done in a research project. Instead, they aim to guide papers to report details that allow reviewers and others to assess the merit and rigor of the study and its execution. What is transparency? For research and its reporting, we define transparency as follows: Transparency: Reporting all relevant details, especially methodological details and artifacts, needed to (1) assess the validity of a study and its results and (2) to re-run reported studies independently. This aligns with the definition of the US Academy of Sciences. Contributors ● Timo Jagusch, University of Bonn ● Anna-Marie Ortloff, University of Bonn, ORCID: 0000-0002-5735-178X ● Lukas Struck, University of Bonn, ORCID: 0000-0001-5613-0646 ● Jenny Tang, Carnegie Mellon University, ORCID: 0009-0003-2840-1535 This guideline contains valuable input and feedback that we gathered at the Workshop on Developing Research Transparency Guidelines in Human-centered Privacy and Security at SOUPS 2025. Used Terms Priority ● Required: All studies the criterion applies to are expected to provide the respective information. If the information is not provided, a detailed and strong justification for omission is required. Examples for such justification are provided in the detailed description of criteria under “Reasons to Omit”. ● Strongly Recommended: Authors are strongly encouraged to include this information to ensure research clarity and assessability, or justify why they cannot include it. ● Recommended: Authors are encouraged to provide this information. Reporting Location ● In a dedicated (sub)section: The information should have a clearly marked (sub)section for easy findability. ● Within a xxx section: The information should be contained within the specified section (e.g., methodology), but does not warrant its own heading. ● Within the appendix: The information or material should be reported in a paper's appendix. ● As part of supplementary materials: The information or materials should be a part of a supplementary materials package in a suitable file format. The package should be hosted and published using an archival platform that assigns DOIs (such as the paper publisher, OSF or Zenodo). A link to the package as well as a description of its content should be included in a dedicated section directly before the references for easy findability. Authors can point to this section when referencing materials throughout the paper. Overview by Priority Title Description Applicable Studies Required ❐ Research Goal & Method Choice Explain the study objectives, the associated research questions, and the methods chosen to achieve/answer them. All ❐ Limitations & Threats to Validity Describe limitations of the paper and its methodology, and threats to validity that result from the study design. All ❐ Study Protocol Provide a study protocol that contains a description of all steps of the study execution and the order in which they were performed. All ❐ Condition Assignment Describe how conditions were assigned. Experimental studies with multiple conditions. ❐ Study Duration Describe how long the study or study sessions took, i.e., how long participants were involved, how long a measurement took, or the duration of data collection (if talking about the whole study). All ❐ Ethics Include a section describing the ethical implications of the research, as well as any mitigations of risks or negative impact. All ❐ IRB/ERB State whether an institutional review board (IRB) or ethics review board (ERB) (in some organizations those boards might have different names) reviewed the study and what the outcome was. All ❐ Consent Describe if and how participants’ consent for research participation was obtained, and provide details. Studies with participants ❐ Study Compensation State if, how, and how much participants were compensated. Studies with participants. ❐ Funding & Conflicts of Interest Declare who funded the project, as well as any other conflicts of interest. All ❐ Sampled Population Description of the target population that the study addresses and draws its sample from. All ❐ Sample Description/Demo graphics Describe any sample(s) used in analyses, including important sample characteristics such as demographic information. All ❐ Recruitment Approach/Samplin g Strategy Describe the procedure(s) with which the study’s participants were recruited, or otherwise the means through which the data set was acquired. Include changes made to the approach during the study. All ❐ Sample Size State the sample size of a study, i.e., the number of valid participants/data points. All ❐ Experiment Materials Describe all materials and infrastructure used to conduct experiments, including tested interventions. Experiment studies ❐ Interview Guide Describe and provide the interview guide or protocol used to conduct interviews during the research study. All studies involving qualitative interviews or (semi-)struct ured discussions/f ocus groups. ❐ Questionnaire/Sur vey Instrument Describe and provide all materials used to conduct experiments or interventions. This includes tasks, instructions, scripts, prototypes, software, hardware, or any other tools used as part of the study protocol. Studies with surveys or questionnaire s. ❐ Qualitative Data Preprocessing Describe how data was preprocessed prior to analysis, including transcription, translation, and anonymization. Qualitative studies ❐ Qualitative Data Analysis Procedure Outline the methods and steps of the analysis procedure, and the role of the involved researchers. Qualitative Studies ❐ Qualitative Analysis Reliability Discuss how reliability was ensured, such as handling disagreements among researchers, calculating Inter-rater Reliability (IRR), or coding consistency or explain why IRR was not reported. Qualitative studies ❐ Empirical Evidence Provide empirical evidence (e.g., quotes, field notes, excerpts, photographs) to support results. Qualitative studies ❐ Quantitative Data Preprocessing Describe any treatment of raw data prior to analysis, including outlier removal, variable transformation, data quality control. Quantitative studies ❐ Quantitative Data Analysis Procedure Outline the methods and steps of the analysis procedure, including statistical tests. Quantitative studies ❐ Variables Name and define all variables that are considered in the statistical analysis, and clearly label them as dependent or independent for each analysis they are a part of. Quantitative studies ❐ Hypotheses Explicitly state hypotheses for studies that use confirmatory inferential statistics. Quantitative studies with confirmatory inferential statistics. ❐ Assumptions State assumptions made for analyses, and how you tested or justified them. Quantitative studies ❐ Descriptive Statistics Provide measures that describe your data set, such as a mean or median for the central tendency, a measure for variability, or percentages for categorical data. Quantitative studies ❐ Statistical Results Report results of all statistical tests performed. Quantitative studies ❐ Confidence/Signifi cance Measure Provide measures for the statistical significance and confidence of statistical results. Quantitative studies ❐ Report and Interpret Effect Sizes Report and interpret effect sizes of statistical results. Quantitative studies Strongly Recommended ❐ Approach to LLMs/AI Describe if and how LLMs or other AI tools were used in the project. All ❐ Study Piloting Provide information about piloting and testing of the study protocol, as well as any associated adjustments. Studies with participants. ❐ Study Context Describe the context in which the study was conducted, including time, setting, and language(s). All ❐ Vulnerability Disclosure Describe if and how any vulnerabilities discovered during the research were disclosed. Studies that discovered vulnerabilities . ❐ Sampling Materials Provide the materials used for sampling/recruiting (e.g., recruitment emails or posts, scraping scripts). All ❐ Sample Size Justification Describe how the sample size number was arrived at, with references or reasoning explained (e.g., data saturation, power analysis). All ❐ Codebook Provide the final codebook. Qualitative studies Recommended ❐ Consent Form Provide the consent form used to collect written consent from participants. Studies with participants that used written consent. ❐ Anonymization/De identification Report utilized methods of data anonymization, de-identification or pseudonymization when applied to PII or otherwise sensitive data. This does not only apply to full datasets, but can also be necessary for aggregate data (e.g., demographics tables) or excerpts of data (e.g., a detailed quote from an interview) that might otherwise allow identifying someone. Both methods that are only used on published data and those used on all data before analysis should be described. Studies that collect PII or otherwise sensitive data. ❐ Sampling Success Rate If possible, provide numbers for the success of their sampling strategy, including number of invitations sent, accepted, rejected, and ignored, as well as drop-out rates during the study. Studies with applicable sampling strategies. ❐ Data Provide the data collected during the study, ideally complete, raw data, or aggregated or anonymized data if necessary. All ❐ Analyzed Software Describe, and when possible, provide access to the source code of any software developed, modified, or used during research, including dependencies, configurations, and version information. All studies involving software tools, algorithms, or user facing systems. ❐ Analyzed Hardware Specify and provide the details of any specialized hardware or devices used, developed, or modified in the research, enabling others to replicate or build upon the work. Studies that depend on specific hardware or use custom or modified hardware. ❐ Qualitative Analysis Software Share what software was used for data analysis. Qualitative studies ❐ Quantitative Analysis Software Provide (self-written) code and scripts or specifications of third-party software used for data analysis. Quantitative studies using software to analyze their data ❐ Correlations Describe correlations between the observed variables. Quantitative studies ❐ Explain Effect Sizes Explain effect sizes of statistical results. Quantitative Studies ❐ Insignificant Results Report statistically non-significant results. Quantitative studies ❐ Pre-Registration Pre-registering a study includes submitting a description of the research design, procedures, potential hypotheses and intended methods for data analysis before beginning the study execution. The description is immutable after finalization. Changes might be possible, but need to be traceable. Quantitative studies ❐ Positionality Statement Provide information about the authors’ domain-relevant characteristics, experiences, privileges, and reflections that may influence how they shape the research design or interpret the results. All Overview by Topic Condition Assignment Description Describe how conditions were assigned. Applicable Study Types Experimental studies with multiple conditions. Reasons to Omit — Priority Required Justification The description of how the study conditions were assigned is essential for replication and interpretation of the results. How-to Describe clearly how different conditions look like and are controlled within the study and how the condition assignment took place. Explicitly state whether participants are assigned to more than one condition in studies with repeated measures. Preferred Reporting Location In the methodology section. Example ● “The study employed a within-subject design. Participants tested a set of four conditions for one week each, resulting in a total study length of four weeks. The order of conditions was counterbalanced”. – Communicating Device Confidence Level and Upcoming Re-Authentications in Continuous Authentication Systems on Mobile Devices, SOUPS 2019 ● “Next, participants were randomly assigned to one of three groups (video, text, control).” – Comparing the Effectiveness of Text-based and Video-based Delivery in Motivating Users to Adopt a Password Manager, EUROUSEC 2021 Reviewer Guidance — Other remarks — Further resources — Study Piloting Description Provide information about piloting and testing of the study protocol, as well as any associated adjustments. Applicable Study Types Studies with participants. Reasons to Omit — Priority Strongly Recommended Justification To assess whether participants understood the tasks or questions, it is important to describe whether piloting took place and the extent to which changes were made as a result. How-to Provide information about the procedure, sample size, and sample composition of the pilot study, changes made between the pilot study and the formal study, and what researchers learned from the pilot. Preferred Reporting Location In the methodology section. Example — Reviewer Guidance — Other remarks — Further resources — Study Context Description Describe the context in which the study was conducted, including time, setting, and language(s). Applicable Study Types All Reasons to Omit — Priority Strongly Recommended Justification The context and environment a study takes place in can have significant influence on results and are thus needed to interpret them. How-to Explain the context of the conducted study and any factors that may have influenced participants or measurements, including, for example, the temporal context (e.g., year, time span of the evaluation/data collection), the setting in which the study took place (e.g., online or in-person), what language(s) the study was conducted in, and how any translations were handled. Preferred Reporting Location In the methodology section. Example — Reviewer Guidance — Other remarks — Further resources — Study Duration Description Describe how long the study or study sessions took, i.e., how long participants were involved, how long a measurement took, or the duration of data collection (if talking about the whole study). Applicable Study Types All Reasons to Omit — Priority Required Justification This is important to evaluate how long the study actually lasted, how high the burden on participants was, and the extent to which participants were discouraged by the duration. How-to Report an average duration and variability. Preferred Reporting Location In the methodology section. Example ● “To this end, we designed our surveys for experts to complete within 25–30 minutes of focused effort, in line with suggested best practices [35]. Actual completion time averaged 27.9 minutes (σ = 2.4 minutes).” – Above and Beyond: Organizational Efforts to Complement U.S. Digital Security Compliance Mandates, NDSS 2022 Reviewer Guidance — Other remarks The study (session) duration does not include time researchers spend on analysis. Further resources — Pre-Registration Description Pre-registering a study includes submitting a description of the research design, procedures, potential hypotheses and intended methods for data analysis before beginning the study execution. The description is immutable after finalization. Changes might be possible, but need to be traceable. Applicable Study Types Quantitative studies Reasons to Omit — Priority Recommended Justification The goal is to make research outcomes less malleable (e.g., avoiding p-value hacking) and help identify non-significant research outcomes that need to be reported. How-to The pre-registration will be an external submission that the paper should contain a link to. Describe and justify any changes between the pre-registered and actual method. Preferred Reporting Location In the methodology section. Example — Reviewer Guidance It is common to have to adapt to unexpected circumstances in research, and deviation from the pre-registered method should not be seen as negative as long as it is reported, well justified, and does not interfere with the research’s integrity. Other remarks Examples for pre-registration options include https://figshare.com/, OSF, Aspredicted Further resources — Positionality Statement Description Provide information about the authors’ domain-relevant characteristics, experiences, privileges, and reflections that may influence how they shape the research design or interpret the results. Applicable Study Types All Reasons to Omit The implementation and results are less susceptible to bias that emerges from the authors’ points of view, e.g. a large-scale closed-ended online survey, or the subject matter or domain does not involve a sensitive population, e.g. a comparison of mainstream password-management tools using task analysis. Priority Recommended Justification The goal is to be transparent about how the authors, their experiences, and their point of view may have shaped the research to provide context to the paper. How-to The statement should address the authors’ background and relationship to the research topic, their identities and potential biases, a discussion of power dynamics among the study team and the participants and their communities, strategies for mitigating biases and ensuring ethical conduct, and explicit statements of how the positionality has impacted on research design and findings. If you omit information to preserve anonymity during the review process, remember to add it for the final publication. Preferred Reporting Location A dedicated subsection in the methodology or discussion section. Example ● As researchers, we recognize that our personal and professional backgrounds inevitably shape our work on [topic in usable security and privacy]. Our expertise in [e.g., human-computer interaction, social psychology, governmental policy work] and our experience in [e.g., industry, advocacy] lead us to prioritize [these specific practices]. We acknowledge that our perspectives may introduce biases, such as [e.g., a focus on technical solutions or a specific user group]. To mitigate this, we have [e.g., employed a diverse research team of xxx, used a xxx approach, engaged in continuous self-reflection via xxx]. Our goal is to be transparent about our positionality so that readers can better understand the context in which this research was conducted and the participants’ data interpreted. ● The team ranged in experience from novices to researchers with 10 years’ experience. The first author, a former practitioner in the study domain, is a Phd student in HCI who led interviews and questionnaires, while the second and third authors assisted in qualitative coding and statistical analysis, and the fourth and fifth authors are the advisors and principal investigators for the project. Reviewer Guidance Positionality statements may have to be vague in certain aspects and should not be used to try to identify authors during the review process. Disclosing a position in relation to the research should also not be held against the authors, since it is helpful context for interpreting a paper that authors should be encouraged to share. Other remarks — Further resources Sections 7 and 8 in "Different Researchers, Different Results? Analyzing the Influence of Researcher Experience and Data Type During Qualitative Analysis of an Interview and Survey Study on Security Advice", CHI, 2023 Kale, Kay and Hullman (2019) Decision-Making Under Uncertainty in Research Synthesis: Designing for the Garden of Forking Paths, CHI 2019 Ethical Considerations Ethics Description Include a section describing the ethical implications of the research, as well as any mitigations of risks or negative impact. Applicable Study Types All Reasons to Omit — Priority Required Justification Conducting research ethically and outlining ethical considerations is important to prevent unintended or inappropriate harm to participants, researchers, or third parties. Describing ethics ensures it has been considered, benefits and harms resulting from a research project have been weighed, and helps alleviate concerns that reviewers and readers might have. How-to Outline ethical challenges, potential harm, and related considerations, e.g., how those were mitigated or why they are outweighed by the research’s benefits. This includes ethical considerations of the study design, execution, and results and their impact, and should highlight implications and mitigations. Preferred Reporting Location A dedicated ethics subsection in the paper’s methodology section or a dedicated section at the end of the paper if the conference allows/requires that. Example — (highly dependent on study) Example from a study with publicly available social media data: “This study does not constitute human subjects research because it exclusively analyzes publicly available data [9, 41]. However, we acknowledge that Redditors did not explicitly consent to the use of their posts for research purposes. All usernames, direct links, and identifiable metadata (e.g., timestamps granular to the hour) were permanently removed during data processing. Furthermore, while the dataset was aggregated and analyzed at scale, we will not publicly release the raw data and destroy our copy, as even anonymized posts could theoretically be traced back to original usernames, potentially exposing user identities [20, 42, 59]. To further protect user identities, excerpts mentioned in the paper have been paraphrased or re-worded so that a reverse search will not identify the post [43], following best practices [56] followed in the usable security and privacy community in reporting user data (https://www.workshopononlineabuse.com/policies.html)”; in Victims, Vigilantes, and Advice Givers: An Analysis of Scam-Related Discourse on Reddit, SOUPS 2025 Reviewer Guidance Authors should discuss ethical implications of study design, execution, results, and result impact. The discussion can be very brief if there are no specific associated risks. Other remarks The Menlo Report provides helpful guidance on research ethics. Further resources "Ethics in Computer Security Research: A Data-Driven Assessment of the Past, the Present, and the Possible Future", CCS, 2025 https://www.usenix.org/conference/usenixsecurity26/call-for-pape rs#ethics The Menlo Report "Common Pitfalls in Writing about Security and Privacy Human Subjects Experiments, and How to Avoid Them", SOUPS, 2010 “Flawed, but like democracy we don’t have a better system”: The Experts’ Insights on the Peer Review Process of Evaluating Security Papers", IEEE S&P, 2022 IRB/ERB Description State whether an institutional review board (IRB) or ethics review board (ERB) (in some organizations those boards might have different names) reviewed the study and what the outcome was. Applicable Study Types All Reasons to Omit — Priority Required Justification IRBs/ERBs provide an independent review of research, and knowing whether such a review was conducted and what the outcome was can provide context for reviewers’ and readers’ evaluation of the research ethics. How-to Outline whether IRB/ERB review was obtained. If yes, say what the outcome was. If not, explain why a review was not necessary or not possible (e.g., institution has no ERB) and how ethical research conduct was ensured. Preferred Reporting Location Within a dedicated ethics (sub)section (see Ethics). Example ● Ethical approval was obtained from the Institutional Review Board of ABC Hospital (Approval #IRB2022-xxx). ● The study was reviewed by the Institutional Review Board of XYZ University and was deemed exempt under category 4 for secondary research using publicly available data. Reviewer Guidance Research should not be rejected because a review was not possible – instead, the authors’ description of their ethical considerations should be used to determine if the project was conducted ethically. Other remarks The Menlo Report provides helpful guidance on research ethics. IRBs/ERBs should not be confused with ethics committees at conferences.) Further resources "Ethics in Computer Security Research: A Data-Driven Assessment of the Past, the Present, and the Possible Future", CCS, 2025 "Changes in Research Ethics, Openness, and Transparency in Empirical Studies between CHI 2017 and CHI 2022", CHI, 2023 "Common Pitfalls in Writing about Security and Privacy Human Subjects Experiments, and How to Avoid Them", SOUPS, 2010 "A Systematic Literature Review of Empirical Methods and Risk Representation in Usable Privacy and Security Research", ACM ToCHI, 2021 "Transparency in Usable Privacy and Security Research: Scholars' Perspectives, Practices, and Recommendations", IEEE S&P, 2025 Consent Description Describe if and how participants’ consent for research participation was obtained, and provide details. Applicable Study Types Studies with participants Reasons to Omit — Priority Required Justification It is important for both reviewers and readers to know if and how consent was obtained (e.g., for ethical clarity and as it might influence participant behavior or create a selection bias). How-to Describe the type of consent obtained (e.g., informed, written, none/deception…), what exactly the participants consented to (e.g. audio/video recording), and if any kind of deception study design was involved, include a description of the debriefing. Preferred Reporting Location In a dedicated ethics (sub)section or subsection (see Ethics), or Within a (sub)section that describes the recruitment or study procedure that consent was a part of. Example ● "All participants provided written informed consent prior to their inclusion in the study." ● "Verbal informed consent was obtained from all participants." ● "Written informed consent was obtained from the parents or legal guardians of all participating minors, and assent was obtained from children aged 7 and older." ● “The requirement for informed consent was waived by the IIRB due to the retrospective nature of the study and use of anonymized data." Reviewer Guidance — Other remarks If a consent form was used, providing it can provide details and clarity. Further resources "Changes in Research Ethics, Openness, and Transparency in Empirical Studies between CHI 2017 and CHI 2022", CHI, 2023 "A Systematic Literature Review of Empirical Methods and Risk Representation in Usable Privacy and Security Research", ACM ToCHI, 2021 "Transparency in Usable Privacy and Security Research: Scholars' Perspectives, Practices, and Recommendations", IEEE S&P, 2025 Consent Form Description Provide the consent form used to collect written consent from participants. Applicable Study Types Studies with participants that used written consent. Reasons to Omit A consent form may contain contact data of the researchers and thus have to be omitted or censored for the anonymous review process. Priority Recommended Justification The consent form can provide clarity on the exact terms that participants were presented with and agreed to, possibly indicating pre-study expectations or opt-out reasons of participants. How-to Provide the consent form as it was given to participants, ideally in PDF, Markdown, or similar format. Preferred Reporting Location Either within the appendix or as part of supplementary materials. Example — Reviewer Guidance — Other remarks — Further resources "Transparency in Usable Privacy and Security Research: Scholars' Perspectives, Practices, and Recommendations", IEEE S&P, 2025 Study Compensation Description State if, how, and how much participants were compensated. Applicable Study Types Studies with participants. Reasons to Omit — Priority Required Justification Compensation may influence who participates in a study and with what motivation, and therefore influence a study’s results. Such an influence is an important context when interpreting results. How-to Report if, how and how much compensation participants were offered, and potentially how many of them accepted it, and how it compares to the average income of the studied population. Explicitly state, if participants were not compensated. Preferred Reporting Location Either within the ethics section or subsection (see Ethics), or in a section that describes the recruitment procedure that compensation was a part of. Example — Reviewer Guidance — Other remarks — Further resources "Changes in Research Ethics, Openness, and Transparency in Empirical Studies between CHI 2017 and CHI 2022", CHI, 2023 "Common Pitfalls in Writing about Security and Privacy Human Subjects Experiments, and How to Avoid Them", SOUPS, 2010 "A Systematic Literature Review of Empirical Methods and Risk Representation in Usable Privacy and Security Research", ACM ToCHI, 2021 "Transparency in Usable Privacy and Security Research: Scholars' Perspectives, Practices, and Recommendations", IEEE S&P, 2025 Vulnerability Disclosure Description Describe if and how any vulnerabilities discovered during the research were disclosed. Applicable Study Types Studies that discovered vulnerabilities. Reasons to Omit In the case of ongoing disclosure processes it might be necessary to omit details – the fact that disclosure was considered/conducted should still be stated. Priority Strongly Recommended Justification Vulnerability discovery, disclosure, and remediation are an example of study results’ impact that may cause harm and should be mitigated. Therefore, it should be discussed as part of ethical considerations. They also underline the real-world relevance and impact of the findings. How-to Describe the disclosure process and its outcome(s). Preferred Reporting Location Either within the ethics section or subsection (see Ethics), or as a separate section. Example ● During the course of this research, a vulnerability was identified in [system/software]. The vendor was notified on [date], and disclosure followed a responsible disclosure process. As of the time of publication, a patch has been made available. ● Despite repeated attempts to contact the vendor regarding a privilege escalation vulnerability in XYZ OS, no response was received. We disclosed the vulnerability responsibly and provide limited technical details to avoid misuse. Reviewer Guidance Authors’ influence on the outcome of disclosure processes is limited, and vendor reactions are not necessarily a good indicator of research impact. Other remarks — Further resources "Transparency in Usable Privacy and Security Research: Scholars' Perspectives, Practices, and Recommendations", IEEE S&P, 2025 Anonymization/De-Identification Description Report utilized methods of data anonymization, de-identification or pseudonymization when applied to PII or otherwise sensitive data. This does not only apply to full datasets, but can also be necessary for aggregate data (e.g., demographics tables) or excerpts of data (e.g., a detailed quote from an interview) that might otherwise allow identifying someone. Both methods that are only used on published data and those used on all data before analysis should be described. Applicable Study Types Studies that collect PII or otherwise sensitive data. Reasons to Omit — including number of invitations sent, accepted, rejected, and ignored, as well as drop-out rates during the study. Applicable Study Types Studies with applicable sampling strategies. Reasons to Omit Not all sampling strategies enable reporting a success rate, e.g., public posts do not show how many people have seen them. Priority Recommended Justification The sampling success rate provides information on the sampling procedure’s effectiveness, and can provide insights on potential sampling biases. For future studies, insights into these numbers might help other researchers to plan their recruitment. How-to Share numbers for the success of their sampling strategy, including number of invitations sent, accepted, rejected, and ignored, as well as drop-out rates during the study. Preferred Reporting Location A “Sampling” or “Recruitment” subsection in the Method section. Example — Reviewer Guidance — Other remarks — Further resources "Preliminary guidelines for empirical research in software engineering", IEEE Transactions on Software Engineering, 2002 Sample Size Description State the sample size of a study, i.e., the number of valid participants/data points. Applicable Study Types All Reasons to Omit — Priority Required Justification The sample size is vital to understand the scope of a study, and assess the validity of statistical analyses. How-to Clearly state the size of the sample. If reporting only valid data points, also make sure to include how many invalid data points were removed before analysis. Preferred Reporting Location A “Sampling” or “Recruitment” subsection in the methodology section; can also be already stated in the Abstract or Introduction. Example ● We interviewed 26 IT professionals. ● Our final dataset includes 278 Reddit posts. Reviewer Guidance — Other remarks — Further resources "Changes in Research Ethics, Openness, and Transparency in Empirical Studies between CHI 2017 and CHI 2022", CHI, 2023 "Preliminary guidelines for empirical research in software engineering", IEEE Transactions on Software Engineering, 2002 Sample Size Justification Description Describe how the sample size number was arrived at, with references or reasoning explained (e.g., data saturation, power analysis). Applicable Study Types All Reasons to Omit — Priority Strongly Recommended Justification A rationale for the sample size can help readers understand how the sample size does or does not support certain conclusions (e.g., generalizability). How-to Explain the rationale for the sample size, and include references to methodological concepts as appropriate. For quantitative research, the sample size is typically based on a power analysis to estimate what sample size is needed to detect effects of a certain magnitude with planned statistical tests. For qualitative research, sample size may be determined based on when data saturation is reached or based on sample stratification. Preferred Reporting Location A “Sampling” or “Recruitment” subsection in the Method section. Example ● We stopped the recruitment process after 18 interviews, having reached theoretical saturation [cite]. ● We performed an a-priori power analysis to determine the required sample size. (Add explanations of parameters and assumptions used in the calculation.) Reviewer Guidance — Other remarks — Further resources "Preliminary guidelines for empirical research in software engineering", IEEE Transactions on Software Engineering, 2002 Study Instruments and Materials Experiment Materials Description Describe all materials and infrastructure used to conduct experiments, including tested interventions. Applicable Study Types Experiment studies Reasons to Omit — Priority Required Justification Providing detailed experiment materials is essential for ensuring transparency, replicability, and verifiability of results. It also helps other researchers understand and replicate experimental conditions accurately. How-to Describe all materials and infrastructure used to conduct experiments, including tested interventions. Preferred Reporting Location Descriptions of the materials should go in a dedicated subsection within the methodology section, or in the appendix. Artifacts should be provided as part of the supplementary materials. Example — Reviewer Guidance — Other remarks — Further resources "Changes in Research Ethics, Openness, and Transparency in Empirical Studies between CHI 2017 and CHI 2022", CHI, 2023 “Common Pitfalls in Writing about Security and Privacy Human Subjects Experiments, and How to Avoid Them", SOUPS, 2010 "Transparency of CHI Research Artifacts: Results of a Self-Reported Survey", CHI, 2020 Interview Guide Description Describe and provide the interview guide or protocol used to conduct interviews during the research study. Applicable Study Types All studies involving qualitative interviews or (semi-)structured discussions/focus groups. Reasons to Omit — Priority Required Justification Providing the interview guide ensures methodological transparency, aids replicability, enables external verification of the data collection approach. It also allows other researchers to adapt or refine these instruments for their own studies. How-to Clearly detail the interview questions, including the structure and format of the interviews (e.g., structured, semi-structured, unstructured). Describe the intended flow of the interview, specifying any probing questions or follow-up topics utilized. Additionally, explain the development process of the interview guide, for example, was derived from existing literature or refined through pilot testing. Finally, clearly indicate how and where the interview guide can be accessed, (e.g., appendix, supplementary materials, or an online repository). Preferred Reporting Location Dedicated subsection within the methodology or data collection sections, as an appendix or as part of supplementary materials. Example — Reviewer Guidance — Other remarks — Further resources "Changes in Research Ethics, Openness, and Transparency in Empirical Studies between CHI 2017 and CHI 2022", CHI, 2023 "Transparency of CHI Research Artifacts: Results of a Self-Reported Survey", CHI, 2020 "Transparency in Usable Privacy and Security Research: Scholars' Perspectives, Practices, and Recommendations", IEEE S&P, 2025 Questionnaire/Survey Instrument Description Describe and provide all materials used to conduct experiments or interventions. This includes tasks, instructions, scripts, prototypes, software, hardware, or any other tools used as part of the study protocol. Applicable Study Types Studies with surveys or questionnaires. Reasons to Omit — Priority Required Justification Transparency in surveys or questionnaires ensures replicability, facilitates the verification of results, and allows the research community to build upon validated instruments, thus enhancing the credibility of research findings. Moreover, it allows readers to contextualize results with the exact questions that were asked. How-to Include the full questionnaire/survey instrument in the appendix or online supplementary materials. Optionally, one can provide a short summary in the paper’s methodology section. Preferred Reporting Location As part of supplementary materials, or within the appendix. Example — Reviewer Guidance — Other remarks — Further resources "Changes in Research Ethics, Openness, and Transparency in Empirical Studies between CHI 2017 and CHI 2022", CHI, 2023 "Transparency of CHI Research Artifacts: Results of a Self-Reported Survey", CHI, 2020 "Transparency in Usable Privacy and Security Research: Scholars' Perspectives, Practices, and Recommendations", IEEE S&P, 2025 Data Description Provide the data collected during the study, ideally complete, raw data, or aggregated or anonymized data if necessary. Applicable Study Types All Reasons to Omit Raw data may contain PII or IP protected data, and cannot always be published. Protecting participants, their identity, and their data is very important, as is adhering to the procedure consent was initially obtained for. Data should not be published if it poses a risk to people or their rights. Priority Recommended Justification Data sets help to replicate analyses, and can be used for new analyses and comparisons of future research. How-to Format the data in a way that makes it easy to access and process, and provide accompanying guidance on how the data is structured and should be used if necessary or helpful to understand it. Provide a brief description of the dataset, e.g., what data is contained, what the values in a table column mean, etc. Ideally, provide the data in an open, reusable/human-readable format (e.g., CSV) if possible. Preferred Reporting Location In supplementary materials. Example — Reviewer Guidance — Other remarks — Further resources "Changes in Research Ethics, Openness, and Transparency in Empirical Studies between CHI 2017 and CHI 2022", CHI, 2023 "Promoting an open research culture", Science, 2015 "Transparency of CHI Research Artifacts: Results of a Self-Reported Survey", CHI, 2020 "Transparency in Usable Privacy and Security Research: Scholars' Perspectives, Practices, and Recommendations", IEEE S&P, 2025 Analyzed Software Description Describe, and when possible, provide access to the source code of any software developed, modified, or used during research, including dependencies, configurations, and version information. Applicable Study Types All studies involving software tools, algorithms, or user facing systems. Reasons to Omit Specific details of software may fall under IP rights and can therefore not be published. It falls under the reviewer’s discretion to decide if the paper’s contribution is enough without it in these cases. Priority Recommended Justification Sharing software enhances transparency research, allows others to reproduce results, identify bugs or limitations, and extend work. It contributes to methodological rigor and supports open science practices within the community. How-to Describe the role and function of the software used in the study. Clearly state whether the source code is available and under license. Include relevant technical details (e.g., programming languages, version numbers, dependencies, required environments, and installation instructions). Also provide a working link where code is hosted and cite release if references are possible. Include a README file or similar usage instructions. Preferred Reporting Location In supplementary materials. Example — Reviewer Guidance — Other remarks — Further resources "Changes in Research Ethics, Openness, and Transparency in Empirical Studies between CHI 2017 and CHI 2022", CHI, 2023 "Transparency of CHI Research Artifacts: Results of a Self-Reported Survey", CHI, 2020 "Transparency in Usable Privacy and Security Research: Scholars' Perspectives, Practices, and Recommendations", IEEE S&P, 2025 Analyzed Hardware Description Specify and provide the details of any specialized hardware or devices used, developed, or modified in the research, enabling others to replicate or build upon the work. Applicable Study Types Studies that depend on specific hardware or use custom or modified hardware. Reasons to Omit Specific details of hardware may fall under IP rights and can therefore not be published. It falls under the reviewer’s discretion to decide if the paper’s contribution is enough without it in these cases. Priority Recommended Justification Sharing hardware specifications ensures transparency, facilitates replication and verification of results, and supports researchers who seek to use, modify, or extend existing hardware-based studies. How-to Document the hardware components, specifications, or any related software dependencies required. Also, describe licensing or intellectual property considerations related to hardware sharing. Since actual hardware cannot be shared in digital format, schematics, 3D printing or assembly instructions can be ways to still share hardware. Include photos of hardware in the paper, if appropriate. Preferred Reporting Location In supplementary materials. Example — Reviewer Guidance — Other remarks — Further resources "Changes in Research Ethics, Openness, and Transparency in Empirical Studies between CHI 2017 and CHI 2022", CHI, 2023 "Transparency of CHI Research Artifacts: Results of a Self-Reported Survey", CHI, 2020 Qualitative Analysis & Results Qualitative Data Preprocessing Description Describe how data was preprocessed prior to analysis, including transcription, translation, and anonymization. Applicable Study Types Qualitative studies Reasons to Omit — Priority Required Justification Preprocessing transforms the data, and describing it in detail helps readers understand possible differences from the raw data. How-to Describe any preprocessing steps that were applied in sufficient detail to understand and replicate them, and software used for transcription or translation. This can also include inclusion or exclusion criteria when sampling from a larger set of data. Include if and how data generated by AI was detected and handled. Preferred Reporting Location Within the methodology section. Example Section 3.2 "During the analysis, we excluded 218 posts (8.58%) because they were irrelevant to online scams and were false positives" – Victims, Vigilantes, and Advice Givers: An Analysis of Scam-Related Discourse on Reddit, SOUPS 2025 Reviewer Guidance — Other remarks — Further resources — Qualitative Data Analysis Procedure Description Outline the methods and steps of the analysis procedure, and the role of the involved researchers. Applicable Study Types Qualitative Studies Reasons to Omit — Priority Required Justification Understanding the analysis procedure is essential to understand how the results were obtained. How-to State the methods used, providing references where possible, and describe the steps in detail. Maintain and describe a record of key decisions, changes, or analysis steps throughout the study. This allows others to trace the evolution of the analysis. Preferred Reporting Location Within the methodology section. Example — Reviewer Guidance — Other remarks — Further resources "Promoting an open research culture", Science, 2015: Table 1, “Analytic Methods (Code) Transparency” (p. 1423) Sections 4-7 in From repeatability to reproducibility and corroboration. ACM SIGOPS Operating Systems Review Vol 49 Variables Description Name and define all variables that are considered in the statistical analysis, and clearly label them as dependent or independent for each analysis they are a part of. Applicable Study Types Quantitative studies Reasons to Omit — Priority Required Justification Precisely naming and defining statistical variables ensures that the analysis is interpretable, that operationalizations are transparent, and that findings can be reproduced or compared across studies. How-to Clearly name, define, and label all variables considered in the analysis. For categorical variables, state the categories within each variable, and which variable is the baseline (when relevant, such as for regressions). Preferred Reporting Location Within the methodology section. Example ● The dependent variable was task completion time, measured in seconds from the start to the end of each authentication task. ● Independent variables included the security warning type (visual vs. textual) and user expertise level (novice, intermediate, expert), both treated as categorical variables. Reviewer Guidance — Other remarks — Further resources Section 5.2, specifically “Dependent and Independent Variables” in Misuse, Misreporting, Misinterpretation of Statistical Methods in Usable Privacy and Security Papers, SOUPS 2025 Hypotheses Description Explicitly state hypotheses for studies that use confirmatory inferential statistics. Applicable Study Types Quantitative studies with confirmatory inferential statistics. Reasons to Omit — Priority Required Justification Stating hypotheses increases transparency and allows for proper evaluation of study outcomes. They also distinguish exploratory from confirmatory analyses. How-to State hypotheses and link them to corresponding research questions and tests. Preferred Reporting Location Within the methodology section. Example ● H1: Increased password complexity leads to longer authentication times. ● H2: There is a difference in perceived usability between the baseline and enhanced warning designs. Reviewer Guidance — Other remarks — Further resources Section 3.5.2 calls for distinguishing between exploratory and confirmatory analyses and reporting the “specific a priori consideration of the statistical methods and planned comparisons, in Improving transparency and scientific rigor in academic publishing, Journal of Neuroscience Research 2018 Assumptions Description State assumptions made for analyses, and how you tested or justified them. Applicable Study Types Quantitative studies Reasons to Omit — Priority Required Justification Assumptions underlying statistical tests and if the data conforms to them provide important context to readers. How-to Explain all assumptions (such as normality of the data) made for each statistical test, as well as how you tested if they hold for the data you collected. Authors can also state that certain tests are robust to some assumptions (for example, two-sample t-tests tend to be robust to the normality assumption at sufficient sample sizes). Preferred Reporting Location Within the methodology section. Example — Reviewer Guidance — Other remarks — Further resources Section 3.5.2: “Report normalization procedures, tests for assumptions”, in Improving transparency and scientific rigor in academic publishing, Journal of Neuroscience Research 2018 Descriptive Statistics Description Provide measures that describe your data set, such as a mean or median for the central tendency, a measure for variability, or percentages for categorical data. Applicable Study Types Quantitative studies Reasons to Omit — Priority Required Justification The data set is a key result of a study, and should therefore be described in as much detail as possible. How-to Write a paragraph describing your data set and its key characteristics. This may include the central tendency, a measure for variability, or percentages for important variables You can support it with a graph showing the distribution of the data. Preferred Reporting Location Within the result section. Example — Reviewer Guidance — Other remarks — Further resources Section 5.2, specifically “Magnitude and Direction of Effects” and “Measures of Uncertainty, Variability, or Confidence” in Misuse, Misreporting, Misinterpretation of Statistical Methods in Usable Privacy and Security Papers, SOUPS 2025 Section 7, specifically recommendation to “Report non-standardized effect sizes and descriptives” in SoK: I Have the (Developer) Power! Sample Size Estimation for Fisher's Exact,Chi-Squared, McNemar's, Wilcoxon Rank-Sum, Wilcoxon Signed-Rank and t-tests in Developer-Centered Usable Security, SOUPS 2023 Correlations Description Describe correlations between the observed variables. Applicable Study Types Quantitative studies Reasons to Omit — Priority Recommended Justification Correlations between variables can lead to confounding effects, or might be interesting results themselves. They should therefore be identified and reported. Correlation or a lack thereof can be important for assumptions underlying statistical tests.Correlations in repeated-measures designs can also be used to calculate standardized effect sizes. How-to Describe which analyzed variables do and do not correlate, whether the correlation is positive or negative, and report the correlation strength. Preferred Reporting Location Within the result section. Additional correlations, especially those which are not so large that the authors believe they influence the interpretation of the results, can be reported in the appendix. Example — Reviewer Guidance — Other remarks — Further resources — Statistical Results Description Report results of all statistical tests performed. Applicable Study Types Quantitative studies Reasons to Omit — Priority Required Justification The outcomes of statistical tests are important analysis results, as well as context for the resulting interpretations and conclusions. How-to Report all test results and values (such as F-values, R²) for all tests pertaining to the research questions. A table can help to give readers a space efficient overview. Refer to the APA guidelines linked in further resources for reporting advice for specific statistical tests. Preferred Reporting Location Within the result section. Tests that have less relevance for the research questions and the paper’s main conclusions should be reported in the appendix. Example — Reviewer Guidance — Other remarks — Further resources Section 3.5.2 directs authors to include “a full account of the statistical outputs in the results section, in Improving transparency and scientific rigor in academic publishing, Journal of Neuroscience Research 2018 APA guidelines for quantitative research Confidence/Significance Measure Description Provide measures for the statistical significance and confidence of statistical results. Applicable Study Types Quantitative studies Reasons to Omit — Priority Required Justification Significance measures and confidence intervals provide valuable context for readers, and indicates the (un)certainty of statistical results. How-to Report measures such as exact p-values and confidence intervals around estimates of effect sizes for statistical results. Preferred Reporting Location Within the result section. Example — Reviewer Guidance Finding no significant effect can be a valuable result that should be reported. Only accepting papers with significant results might lead to a publishing bias. Other remarks — Further resources Section 5.2, specifically “Measures of Uncertainty, Variability, or Confidence” in Misuse, Misreporting, Misinterpretation of Statistical Methods in Usable Privacy and Security Papers, SOUPS 2025 Section 6 in Small, Medium, Large? A Meta-Study of Effect Sizes at CHI to Aid Interpretation of Effect Sizes and Power Calculation, CHI 2025 Report and Interpret Effect Sizes Description Report and interpret effect sizes of statistical results. Applicable Study Types Quantitative studies Reasons to Omit — Priority Required Justification The magnitude of an effect is important to indicate direction and whether it is relevant in practice. For example, there might be statistically significant effects but the corresponding effect size is so small that the practical relevance of the found effect is negligible. How-to Report the effect size together with statistical significance. Besides just stating the effect sizes, discuss and interpret them to set them into context. This can include e.g. using field-specific effect size guidelines [1], comparing to effects from related work [1] or highlighting practical consequences or limitations of effects [1, 3]. Preferred Reporting Location In the results section (or discussion). Less important effect sizes might be reported in an appendix. Example ● For reporting effect sizes: “From the figure we can observe that the differences in average comfort level between PII (µ = 2.4,σ = 1.45,95% CI [2.3,2.5]) and general objects (µ = 4.0,σ = 1.16,95% CI [3.9,4.1]) is large and significant (p < 0.0001).” – "I am uncomfortable sharing what I can't see": Privacy Concerns of the Visually Impaired with Camera Based Assistive Applications, Usenix Security 2020 ● For interpreting effect sizes, drawing conclusions from effect sizes: “We do not expect the observed gender effects of 𝐀male = 0.28 to impact the validity of our measurement as our study is in line with the current body of evidence for gender differences in self-efficacy, which further interact with cultural differences: Halevi et al. [ 56 ] reported a large gender difference in self-efficacy in the USA.” – Home Is Where the Smart Is: Development and Validation of the Cybersecurity Self-Efficacy in Smart Homes (CySESH) Scale, CHI 2023 Reviewer Guidance — Other remarks — Further resources Sections 5.6 and 6 in Small, Medium, Large? A Meta-Study of Effect Sizes at CHI to Aid Interpretation of Effect Sizes and Power Calculation, CHI 2025 [Section 8 in A Qualitative Study on How Usable Security and HCI Researchers Judge the Size and Importance of Odds Ratio and Cohen's d Effect Sizes, CHI 2025 Section 5.2, specifically “Magnitude and Direction of Effects” and “Model Fit” and Section 5.3 “Interpretation Recommendations” in Misuse, Misreporting, Misinterpretation of Statistical Methods in Usable Privacy and Security Papers, SOUPS 2025 Explain Effect Sizes Description Explain effect sizes of statistical results. Applicable Study Types Quantitative studies reporting effect sizes Reasons to Omit — Priority Recommended Justification Both readers, and authors from UPS papers may not be familiar with effect sizes (standardized or other) and as such have difficulties assessing effect sizes and drawing their own conclusions if they are reported without an explanation [2]. This is especially the case for effect size measures which are not commonly used, or which may not be recognizable as an effect size measure. How-to Provide a short definition of the effect size measures used for the hypothesis tests conducted in the paper. This could include with which tests or study designs the effect size measure is reported in the paper, an explanation connecting the standardized effect size to data characteristics like frequencies, mean differences or standard deviations, the minimum (no effect) and maximum possible values, if applicable, and whether directionality can be inferred from the effect size measure. Preferred Reporting Location In the data analysis or methodology section Example “Total Variation Distance (TVD), defined as TVD(P, Q) = 1/2 · Σi(Pi, Qi), is a standard metric for quantifying the distance between two distributions [24]. Intuitively, it corresponds to the fraction of respondents who answer differently between the two samples. A TVD of 0 indicates that two distributions are identical; as the distributions become increasingly disjoint the TVD approaches.1.” and remainder of subsection 3.3. Total Variation Distance. in Replication: How Well Do My Results Generalize Now? The External Validity of Online Privacy and Security Surveys, SOUPS 2022 “Cohen’s d measures the difference between two means, normalized using the standard deviation, so that a Cohen’s d of 1 represents a difference of 1 standard deviation between the two means. Cohen’s d of 0 means the means in both groups are the same. The higher Cohen’s d is, the larger the difference between the means is.” from Section 4.10 in A Qualitative Study on How Usable Security and HCI Researchers Judge the Size and Importance of Odds Ratio and Cohen's d Effect Sizes, CHI 2025 Reviewer Guidance — Other remarks — Further resources Section 5.3, specifically “Make Results Intuitive” in Misuse, Misreporting, Misinterpretation of Statistical Methods in Usable Privacy and Security Papers, SOUPS 2025 Section 8 in A Qualitative Study on How Usable Security and HCI Researchers Judge the Size and Importance of Odds Ratio and Cohen's d Effect Sizes, CHI 2025 Insignificant Results Description Report non-significant results. Applicable Study Types Quantitative studies Reasons to Omit — Priority Recommended Justification Not finding a statistically significant effect is a result that, while not providing evidence for an effect or a lack thereof, can help readers better understand a topic, contribute to meta studies, and inform the research and study design decisions of other researchers. How-to Report tests that produced non-significant results. Preferred Reporting Location Within the result section, or in an appendix. Example — Reviewer Guidance — Other remarks — Further resources Section 6 in Small, Medium, Large? A Meta-Study of Effect Sizes at CHI to Aid Interpretation of Effect Sizes and Power Calculation, CHI 2025