Materials for: Biased AI Writing Assistants Shift Users' Attitudes on Societal Issues

51 C. Full survey questionnaires C.1. Main task survey for Experiment 1 Page 1: Consent Form We are asking you to participate in a research study. The purpose of this study is to explore people’s opinions about common debate topics. We will ask you to read the instructions and write a short paragraph, no more than seven sentences. After the writing task, you will complete a demographic survey. Risks and discomforts We do not anticipate any risks from participating in this research. Benefits We do not expect you will gain other direct benefits from participating. The research will inform broader academic discourse and the future designs of writing tools Incentives for participation You will be compensated through the site that sent you here after you complete the writing task and demographic survey. You will be compensated at an approximate rate of $12/hr. Privacy/Confidentiality/Data Security We anticipate that your participation in this survey presents no greater risk than everyday use of the Internet. Your confidentiality will be kept to the degree permitted by the technology being used. We cannot guarantee against interception of data sent via the internet by third parties. Sharing De-identified Data Collected in this Research De-identified data from this study may be shared with the research community to advance science. We will remove any personal information that could identify you before files are shared 52 so that, by current known methods, your responses are fully anonymized. Despite these measures, we cannot guarantee anonymity of your personal data. Taking part is voluntary You do not have to be in this study if you don’t want to be. You can change your mind and leave the study at any time without giving any reason. You do not waive any of your legal rights by participating in this research. Please note that compensation will be given only to those who complete the study. If you have questions If you have any questions about the study, please contact the main investigator ([email protected]). Because emails may contain your name, we will delete the emails once the study is complete. If you have any questions or concerns regarding your rights as a subject in this study, you may contact the Institutional Review Board (IRB) for Human Participants at 607-255-5138 or access their website at https://researchservices.cornell.edu/offices/IRB. You may also report your concerns or complaints anonymously through Ethicspoint online at www.hotline.cornell.edu or by calling toll free at 1-866-293-3077. Ethicspoint is an independent organization that serves as a liaison between the University and the person bringing the complaint so that anonymity can be ensured. Statement of Consent: - I CONSENT to take part in the study. - I DO NOT CONSENT to take part in the study. 53 Page 2: Instructions On the next page, state your opinion on the following question: should standardized tests be used in education in America? Please support your position by writing at least 5 sentences. [Shown in the AI Treatment condition only]: A writing assistant supports you with suggestions. To accept a suggestion, press the TAB key. We encourage you to use the assistant but please also write yourself. 54 Page 3: Writing Task Interface shown in the Control condition: 55 Interface shown in the Static Text condition: 56 Interface shown in AI Treatment condition (suggestions were dynamically generated for each participant): 57 Page 4: Demographics Part 1 58 Page 5: Demographics Part 2 59 Page 6: Attitude and distractor questions (shown in a randomized order) 66 C.3. Main task survey for Experiment 2 Page 1: Consent form We are asking you to participate in a research study. The purpose of this study is to explore how people express their opinions about common debatable topics with and without the use of AI writing tools. We will ask you to read the instructions and write a short paragraph, no more than five sentences. After the writing task, you will complete a follow-up survey consisting of general questions about your demographics and attitudes. Risks and discomforts We do not anticipate any risks from participating in this research. Benefits We do not expect you will gain other direct benefits from participating. The research will inform broader academic discourse and the future designs of writing tools Incentives for participation You will be compensated through the site that sent you here after you complete the writing task and demographic survey. You will be compensated at an approximate rate of $12/hr. Privacy/Confidentiality/Data Security We anticipate that your participation in this survey presents no greater risk than everyday use of the Internet. Your confidentiality will be kept to the degree permitted by the technology being used. We cannot guarantee against interception of data sent via the internet by third parties. Sharing De-identified Data Collected in this Research De-identified data from this study may be shared with the research community to advance science. We will remove any personal information that could identify you before files are shared so that, by current known methods, your responses are fully anonymized. Despite these measures, we cannot guarantee anonymity of your personal data. 67 Taking part is voluntary You do not have to be in this study if you don’t want to be. You can change your mind and leave the study at any time without giving any reason. You do not waive any of your legal rights by participating in this research. Please note that compensation will be given only to those who complete the study. If you have questions If you have any questions about the study, please contact the main investigator ([email protected]). Because emails may contain your name, we will delete the emails once the study is complete. If you have any questions or concerns regarding your rights as a subject in this study, you may contact the Institutional Review Board (IRB) for Human Participants at 607-255-5138 or access their website at https://researchservices.cornell.edu/offices/IRB. You may also report your concerns or complaints anonymously through Ethicspoint online at www.hotline.cornell.edu or by calling toll free at 1-866-293-3077. Ethicspoint is an independent organization that serves as a liaison between the University and the person bringing the complaint so that anonymity can be ensured. Statement of Consent (please note: you must be using a desktop or laptop computer to participate) - I CONSENT to take part in the study and I am using a computer. - I DO NOT CONSENT to take part in the study and/or I am not using a computer. 68 Page 2: Instructions For participants who were in the Control condition: On the next page, state your opinion on the following question: - [insert one of the following for the participant’s randomly assigned topic: “Should the death penalty be illegal?”, “Should people who have completed felony sentences be allowed to vote?”, “Should genetically modified organisms (GMOs, such as crossbred vegetables) be grown?”, “Should the United States continue fracking?”]. Please write at least 5 sentences. For participants who were in the AI Treatment or the Debrief condition: On the next page, state your opinion on the following question: - [insert one of the following for the participant’s randomly assigned topic: “Should the death penalty be illegal?”, “Should people who have completed felony sentences be allowed to vote?”, “Should genetically modified organisms (GMOs, such as crossbred vegetables) be grown?”, “Should the United States continue fracking?”]. Please write at least 5 sentences. A writing assistant will offer suggestions. To accept a suggestion, press the TAB key. We encourage you to use the assistant but please also write yourself. 69 For participants who were in the Warning condition: On the next page, state your opinion on the following question: - [insert one of the following for the participant’s randomly assigned topic: “Should the death penalty be illegal?”, “Should people who have completed felony sentences be allowed to vote?”, “Should genetically modified organisms (GMOs, such as crossbred vegetables) be grown?”, “Should the United States continue fracking?”]. Please write at least 5 sentences. Note that the AI-generated suggestions might support a different view than your own. When this happens, it may bias your writing. A writing assistant will offer suggestions. To accept a suggestion, press the TAB key. We encourage you to use the assistant but please also write yourself. 70 For participants who were in the Write-in-favor condition: On the next page, regardless of your own opinion, write an essay arguing that - [Insert one of the following positions here, also used for the AI’s bias: “the death penalty should be illegal”, “people who have completed felony sentences should not be allowed to vote”, “the United States should continue fracking”, “genetically modified organisms (GMOs, such as crossbred vegetables) should be grown”.] Even if you do not agree with the argument that [insert argument here], please write an essay arguing for this position. Please write at least 5 sentences. A writing assistant will offer suggestions. To accept a suggestion, press the TAB key. We encourage you to use the assistant but please also write yourself. 71 Page 3: Attention check For participants in any condition except Write-in-favor: What are you being asked to do in this study? - Write about your opinion on a debatable issue - Write a song about a debatable issue - Categorize products from online retailers - Write an opinion about the weather For participants in the Write-in-favor condition: What are you being asked to do in this study? - Write an essay arguing for a specific position on a debatable issue regardless of your opinion - Write an essay arguing for your position about a debatable issue based on your personal views - Categorize products from online retailers - Do an online crossword puzzle 72 Page 4: Reminder instructions For participants in any condition except Write-in-favor: You have selected the correct answer: Write about your opinion on a debatable issue. Please click the arrow to continue to the writing task. Remember to write about your own opinion on [insert topic here]. For participants in the Write-in-favor condition: You selected the correct answer: Write an essay arguing for a specific position on a debatable issue regardless of your opinion. Please click the arrow to continue to the writing task. Remember to argue for why [insert topic argument here]. 73 Page 5: Writing task Interface shown to participants in the Control condition: - For participants who got assigned to write about the death penalty: 74 - For participants who got assigned to write about voting rights for felons: 75 - For participants who got assigned to write about GMOs: 82 Page 9: Instructions for attitude questions You will see 5 debatable issues on the following pages. Please indicate your opinion toward each issue by clicking the appropriate button on the scale provided. There are no right or wrong answers, so please answer honestly. You will be able to continue to the next issue after you have spent 5 seconds thinking about the current one. Click the arrow to begin. 83 Pages 10-14: Five attitude questions, shown in randomized order and separated by pages and with 5 second delays before being able to submit. Choices for each were the following: “Yes”, ”Probably yes”, “Not sure”, “Probably no”, “No”. - Should the death penalty be illegal? - Should people who have completed felony sentences be allowed to vote? - Should genetically modified organisms (GMOs, such as crossbred vegetables) be grown? - Should the United States continue fracking? - Is artificial intelligence good for society? Page 15: Political demographics 84 Page 16: AI experience questions part 1 (not shown to participants in the Control condition) 85 Page 17: AI experience questions part 2 (not shown to participants in the Control condition) Page 18: Knowledge of study purpose For participants in the Control condition: What do you think was the purpose of this study? "I don't know" is an acceptable answer. [Free response] For participants in the AI Treatment conditions: What do you think was the purpose of this study? "I don't know" is an acceptable answer, but if you guess correctly, you will receive a $0.25 bonus. [Free response] 86 Page 19: ChatGPT usage and technical issues inquiry Did you use or try to use ChatGPT (the external website that can answer questions) to assist with the writing task in this study? Your compensation will not be affected if you did. [Choices: Yes or No] If participant answered Yes: - Briefly describe how you used or tried to use ChatGPT for the writing task. [Free response] If you were exposed to a writing assistant during the writing task, did you experience any technical issues or glitches with it? [Choices: Yes or No] If participant answered Yes: - Briefly describe the technical issues or glitches you experienced with the writing assistant. [Free response] Page 20: Truthfulness Last question: Did you answer the questions in this survey truthfully? Your answer to this question will not affect your compensation, but will help our research. [Choices: Yes or No] Page 21: Optional feedback (end of survey). That's all we have! Do you have feedback for us? (optional). [Free response]