Chatbots: Security, privacy, data protection, and social aspects

Received: 15 January 2021 Revised: 30 March 2021 Accepted: 10 May 2021 DOI: 10.1002/cpe.6426 SPECIAL ISSUE PAPER Chatbots: Security, privacy, data protection, and social aspects Martin Hasal1Jana Nowaková2Khalifa Ahmed Saghair2Hussam Abdulla2 Václav Snášel2Lidia Ogiela3 1IT4Innovations, VSB—Technical University of Ostrava, Ostrava, Czech Republic 2Department of Computer Science, Faculty of Electrical Engineering and Computer Science, VSB—Technical University of Ostrava, Ostrava, Czech Republic 3Pedagogical University of Krakow, Krakow, Poland Correspondence Jana Nowaková, VSB-Technical University of Ostrava, 17. listopadu 2172/15, 708 00 Ostrava—Poruba, Czech Republic. Email: jana.nowako[email protected] Funding information European Regional Development Fund, Grant/Award Number: CZ.02.1.01/0.0/0.0/15_003/0000466; Technology Agency of the Czech Republic, Grant/Award Number: TN01000024; Vysoká Škola Bánská - Technická Univerzita Ostrava, Grant/Award Number: SP2021/24 Summary Chatbots are artificial communication systems becoming increasingly popular and not all their security questions are clearly solved. People use chatbots for assistance in shopping,bankcommunication,mealdelivery,healthcare,cars,andmanyotheractions. However, it brings an additional security risk and creates serious security challenges which have to be handled. Understanding the underlying problems requires defining the crucial steps in the techniques used to design chatbots related to security. There are many factors increasing security threats and vulnerabilities. All of them are comprehensively studied, and security practices to decrease security weaknesses are presented.Modernchatbotsarenolongerrule-basedmodels,buttheyemploymodern natural language and machine learning techniques. Such techniques learn from a conversation, which can contain personal information. The paper discusses circumstances under which such data can be used and how chatbots treat them. Many chatbots operate on a social/messaging platform, which has their terms and conditions about data. The paper aims to present a comprehensive study of security aspects in communication with chatbots. The article could open a discussion and highlight the problems of data storage and usage obtained from the communication user—chatbot and propose some standards to protect the user. KEYWORDS chat, chatbots, data protection, GDPR, security, virtual assistants 1INTRODUCTION A chatbot, short for chat robot, or bot*, short for chat robot, is an application that interacts with humans through user command. The interaction is done mostly through voice and text conversations. It is designed to replicate the pattern of human interaction, thus allowing human conversation with machines.1In 1966, Joseph Weizenbaum developed a computer program that demonstrated the possibilities of communication between a human and a computer via natural language, and, as such, he introduced the first generation of chatbots.2,3 A chatbot is supposed to work independently of the human operator at its end. It tries to understand the queries and provide appropriate responses. If, at any point, the conversation crosses its current knowledge, it deflects the conversation or passes it to human operators. However, modern chatbots are simultaneously learning from the conversation using machine learning algorithms to provide better answers in the future. Studies show thathumans have less experience in *A bot is sometimes referred to as a chatbot, but to be precise, a bot is a computer program (tool) that automates processes. A chatbot is a sub-genre of the bot environment with a focus on talking or conversation. Some companies instead of Chatbot use the name ‘Conversational AI’ or ‘AI chatbots’ to highlight that their chatbot is powered by machine learning and information retrieval techniques. In this article the term ‘chatbot’ is used for all types of chatbots. This is an open access article under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License, which permits use and distribution in any medium, provided the original work is properly cited, the use is non-commercial and no modifications or adaptations are made. © 2021 The Authors. Concurrency and Computation: Practice and Experience published by John Wiley & Sons Ltd. Concurrency Computat Pract Exper. 2021;e6426. wileyonlinelibrary.com/journal/cpe 1of13 https://doi.org/10.1002/cpe.6426 2of13 HASAL ET AL. FIGURE 1 The structure of chatbot interacting with chatbots than with other humans.4Therefore, the comfort level for the user is lower. Chatbots still face unpopularity among users; users often close the conversation when they find out that they are not talking with a real person. Modern chatbots, however, are built upon conversational data from multiple sources. It makes the conversation more natural for the person interacting with the chatbot, which can use typical typos, such as switched letters, and so forth. In essence, chatbots are nothing more than a programmed input-output system, where the output or input is presented in a pleasing way through natural language in the written or spoken form. Hence, many companies recognized that chatbots are a great tool for improving customer relationships as they can operate onmany platformssimultaneously, see Figure 1. Nowadays, many customers use digital communication channels and they appreciate a chatbot’s advantages, such as 24/7 customer service, personalized interaction, and no waiting time.Forcompanies, chatbots essentially mean costsavingsasmanyprocesses arerepeatedandcanbe automated, and employeescan bededicated to more complex tasks. Large technology companies such as Google, Facebook, and IBM have contributed to chatbot development and research in recent years. Naturally, they have access to a large amount of data on which chatbots can be trained. One of the most well-known examples are the question answering system named Watson by IBM, Apple’s Siri, Amazon’s Alexa, Microsoft’s Cortana, and Google’s recent chatbot Meena.5-7 As a matter of fact, all systems are rather open-domain chatbots,8which can engage in conversation on any topic and understand text and speech. The primary goal of these companies is to develop voice assistants5based on natural language processing (NLP), information retrieval techniques, and machine learning (ML) methods. Users can ask their assistants to control home automation devices, play back media via voice, manage other basic tasks such as email, to-do lists, and calendars with verbal commands.9The options of voice assistants increase everyday.7 Contrary to popular belief, many chatbots used by online stores, banks, industrial technical support, and so forth, are created by small or medium-sized companies (compared to the companies mentioned above, which also provide services to create closed-domain chatbots). Such chatbots are called closed-domain chatbots. Usually, they correspond to (or search for) keywords or intents to accomplish specific tasks. Technical solutions vary from company to company, but they generally use the client’s knowledge, such as a webpage, manuals, FAQs, documents, human–human transcripts, and so forth; and use this knowledge to create chatbots specific to tasks suited to the given company. The differentiation that big companies create open-domain chatbots and small companies create closed-domain chatbots is not exact. There are companies or scientific teams creating open-domain chatbots,10 but the creation of human-competitive chatbots requires a lot of data and experienced programmers. The advantage is that both types of chatbots share some methods and technical solutions,10 which can be used by people creating a knowledge base for other languages,11,12 and to get to level five in AI assistants†. Generally, all chatbots are designed to reduce the necessity of humans in classic conversation, for example, in e-shops, banking, medical advice, GDPR rules, psychotherapy,13 and so forth. Nowadays, the chatbots also have been shown to be very useful and time-saving in situations such as hotlines during the coronavirus (COVID-19) pandemic.14-16 On the other hand, pandemic situations make people less cautious about sharing personal data and revealing their fears.17 Chatbots work with user data; chatbots even ‘learn’ from these data. This means they could be considered to be a security problem. Users mostly do not know how their sensitive personal identifying information (PII) is treated, used, stored, or even shared. Generally, it is not a good situation and the new General Data Protection Regulation, well known as GDPR18‡, also raises the problem of security. The main aim of the current study is to highlight and discuss all the important security, privacy, data protection and social aspects in the usage of chatbots through the systematic review of the available literature, and to present a comprehensive view to the given problem. Further, the study indicates challenges in security issues and propose ways to minimize security problems that appear with chatbots’ usage. The concept of this paper is to provide comprehensive insight into the state-of-the-art of chatbots security issues and challenges. The next section,abriefhistoryofchatbots,presentsimportantstepsinchatbots evolution. It points out the steps relatedtothepresenttopic.Section3gives anoverviewofthepotential security issues andthreatsand describes responsivesteps.Thecomprehensivepart of Section 3coversthe problematic †https://www.oreilly.com/radar/the-next-generation-of-ai-assistants-in-enterprise/. ‡https://ec.europa.eu/info/law/law-topic/data-protection_en. HASAL ET AL.3of13 of data storage obtained during communication, which is closely related to the GDPR rules. The summary and identification of critical issues to the GDPR rules regarding chatbots are also discussed. Section 4 provides a discussion about the explored findings and social aspects and extends the security topic to the chatbots’ messaging platforms. Finally, in the last section, the entire study description, limitations and outline of future work are concluded with the authors’ suggestions. 2CHATBOTS IN HISTORY The history of chatbots or social bots (also: socialbots or socbots, agents that communicate more or less autonomously on social media19) concerns manyideasandtechnologicalsteps.Thewholehistoryofthechatbots10,20 isnotpresented,butcrucialchatbotsorstepsindevelopmentwithrespect to security and the knowledge base are mentioned. The discussion can be either by chatting by typing text or speech dialogue using a voice; or some chatbots can have a visual avatar§which adds facial expressions to the conversation. The processing of the information in both techniques is the same; speech is converted to text in the case of spoken dialogue; textual information is further analyzed. The first chatbot (called ELIZA2) was developed in 1966 at the Massachusetts Institute of Technology (MIT). It answered some very simple decision tree questions. A decision tree is a decision support tool that uses a tree-like model of decisions and their possible consequences.21 The grossprocedure programofELIZA and manyfollowingchatbots is thefollowing: It reads andinspects the text forthe presence ofkeywords.If such a wordisfound,thesentence istransformedaccordingtoaruleassociatedwiththekeyword;ifnot,acontent-free remarkor,undercertainconditions, earlier transformation is retrieved. After that, the answer (based on computed or retrieved text and decision tree probability) is printed out. Finding the right keyword is not an easy task. In detail, sentences can contain more than one keyword in a different form or position in a sentence. Such a situation requires techniques like parsing, pattern matching, AIML (Artificial Intelligence Markup Language), chat scripts, or even language tricks.20 Such techniques were developed and improved in recent decades.20 The chatbot A.L.I.C.E. (the Artificial Linguistic Internet Computer Entity) introduced AIML,22,23 which is a derivative of XML. AIML technology is responsible for pattern matching to relate a user input with a response of the chatbots’s Knowledge Base (KB).24,25 A.L.I.C.E. used AIML objects to simplify conversational modelling, in relation to a ‘stimulus-response’ process. The primary elements of an AIML object (input sentence) are categories, patterns, and templates. For each category a response template and set of conditions are prepared that give meaning to the template known ascontext.24 A.L.I.C.E. preprocesses AIMLobjects(performsword processing actions tofittheuser’s input to apre-establishedformat)and matches the input in a KB by pattern matching. If the input is matched, A.L.I.C.E. responds or performs a predefined action. One of the most widely used standalone human-like chatbots based on AIML is Mitsuku.26 It applies NLP with heuristic patterns and supervised ML. Mitsuku is a multi-lingual chatbot, but the new data are sent to the human manager for verification, and only accepted data are involved in KB. Mitsuku needs a large amount of training data to lead an effective dialog.10 The creation of a knowledge base by combining the attributes of two other chatbots is also possible.27 The authors created the Just.Chat platformcombining attributes of twoother Chatbots. Itprocessed the knowledgebases using three filtersto—discardoverlapping interactions,identify personal questions, and deal with interactions containing unwanted terms or topics. The present method was able to produce a chat corpus with around 7800 pairs of interactions in total. It helped to improve chatbot design techniques. The Cleverbot28 chatbot implements rule-based AI techniques.29 Cleverbot is capable of having a natural conversation with a human, whereas programmers do not predefine its responses. Its KB is based on conversational exchanges with people online through crowdsourcing. It means that Cleverbot’s responses are based on the user’s inputs and feedback. Cleverbot can learn from historical conversations,4so its responses can vary over time. Cleverbot has a few disadvantages, such as a sudden change of language (e.g., from English to Italy, if a user writes an Italian phrase), a change in the discussed topic, unpredictable responses and no ability to have a longer discussion on the given topic. To chat with Cleverbot¶,the user must confirm they are over 18 years old since Cleverbot learns many phrases from its users. Most of the AI chatbots are based on a Deep Learning leverage sequence to sequence (Seq2Seq) model.30 The Seq2Seq model showed great results in machine translations of other NLP related fields.31 The general strategy is to map the input sequence to a fixed-sized vector using one Recurrent Neural Network (RNN) and subsequently mapping the vector to the target sequence with another RNN. Most NLP is now based on languagemodelsproduced bymachine learningRNN, as the advantageis that RNN isable to effectivelyusedatafrom previoussteps32 andtomodel sequencedata. AnRNNtreatseachword ofasentenceasa separateinput occurringattime‘t’and alsousestheactivationvalue at‘t−1’asaninputin additiontothe inputattime‘t’.Consequently,RNNpredictsthe nextwordinasentencewithmore accuracy,basedonwhat the previouswordswere. In short, chatbots can predict an outcome (give an answer) based on historical conversations with similar previous texts—text generation. With an appropriate dataset, which is suitable for modelling long and open-domain dialogues close to spoken human language, open-domain conversational dialogue systems can be created.33 The more conversational data are available on a given topic, the better the chatbot is in this topic, but it fails in others. This is why the past conversations from any sources are so valuable. For instance, many companies use transcripts of human workers and §https://www.eviebot.com/en/. ¶https://cleverbot.com. 4of13 HASAL ET AL. promisethattheycan ‘automate’itoncethey’vecollected enough data. For now,thisisonlypossibleinspecificareas—like chat interfaces onbooking web-pages. On the other hand, systems based on RNN can help to assist human workers by proposing and correcting responses. Open-domain chatbots can ask, for example, the same question several times and subsequently evaluate the best answer. IBM started the DeepQA project to create a rule-based AI chatbot called Watson.34 Watson is used as an information-retrieval and question-answering system. It is based on NLP and hierarchical ML method. It finds and assigns feature values (values, names, dates, etc.) to generate responses based on the score. Watson’s cognitive computing technology involves dealing with complex and unstructured data, finding patterns, processing text on technologies such as Hadoop and the Apache Unstructured Information Management Architecture (UIMA) framework. Watson has almost limitless potential, but the complex technology makes it difficult to use without significant effort. Microsoft also created a few chatbots/assistants, such as LUIS,35 Cortana,36 Tay,37 Zo, Xiaoice, Rinna, and so forth; here only LUIS and Cortana are mentioned. LUIS is the acronym for Language Understanding Information Service. LUIS uses NLP in big data analyses to find intents from a sentence.AssoonasLUISrecognizestheuser’sintents,theusersuppliesexamplephrasescalledutterancesfortheintents.LUISrequiresaMicrosoft Azure subscription, as it is integrated with the Azure Bot Service. This makes it easy to create a chatbot which is connected to Azure pre-built apps and data lakes. Cortana is profiled as Microsoft’s personal productivity assistant, which recognizes natural language and processes all the users’ information, interests, routines, calendars, searches, and so forth, allowing to erase, suggest or improve the user’s search preferences. Cortana is connected to Microsoft products, Bing, Outlook, Skype, and so forth. Cortana is a working demonstration that chatbots can help users to speed up work in many domains. Cortana’s connection with Cortana Analytics Suite is interesting. It enables the chatbot to be driven by the outcome of the ML algorithm. Chatbots also serve in the healthcare domain, where they help to navigate patients through the treatment procedure. The ViDi (Virtual Dietician) chatbot interacts with diabetic patients as a virtual adviser.38 ViDi was designed to solve health problems and be the web-based chatbot as a redesigned version.39 ViDi uses a special pattern to remember past conversational paths. The path is divided into three levels of nine questions each, and this is done by the Vpath parameter, which determines the patient’s path. The redesigned version of ViDi used a relation database (SQL) and web programming languages such as PHP, HTML and XHR. Similarly, the MS Azure Bot Service can be used in medical applications. 3SECURITY Threats and vulnerabilities are the two main categories of security issues. A security threat is defined as a risk by which an organization and its systems can be compromised. Computer security threats40 are identified by a STRIDE model as Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, Elevation of Privileges, and many more; every security threat should be reduced by protective mechanisms ensuring the following properties as Authenticity, Integrity, Non-repudiation, Confidentiality, Availability, Authorization, and many others.41 System vulnerabilities are weakness which can be exploited by an attacker, to cross privilege boundaries (i.e., perform unauthorized actions) within a computer system.42 The system is vulnerable when it has weak coding, lacks current drivers on the hardware side, has a weak firewall, and so forth. Human errors mostly cause system vulnerabilities. SDL (Security Development Lifecycle) helps to avoid such errors.43 As many chatbots use cloud computing services, which have threats and vulnerabilities very well handled, to store the data,5the following text is focused on the communication part and different aspects of data manipulation. Companies not using cloud services usually also handle previous topics during the implementation phase of the chatbot system. Secure messaging can be divided into two domains. The fist domain concerns security in data transfers, that is, the secure transfer of messages, voice, and images to a server on which the chatbot is hosted. The second domain deals with the user’s data on the server (backend), that is, how the data is processed, stored, and shared. Both domains cover the lifecycle of the user’s domain. There are several threats to user messages in the first domain. The following text analyzes the methods to increase security in chatbot communication. Not all of them are generally used, as it is not necessary in many cases. If any company processes a user’s data, most of the following methods must be implemented. The following methods cover any communication with the chatbot, such as a pop-up chat window in a web-browser or mobile application. Authentication and authorization Confirmation of the user identity (authentication) is not always mandatory. When the user asks for help, for example, on a shopping website, usually noauthenticationis needed. In thiscase,thesystem does notrequirethe identification of theuseroraccess to theuser’sdata.The situation is differentwhentheuserasksforhelpandthechatbotworkswithuser’sdata.Atypicalexampleischattingwithabankingchatbotaboutanaccountbalance. In this case, authentication and authorization are necessary to identify that a user is verified with valid and secure login credentials. The credentials aremostlyausername,networkaddress,systemID,phonenumber,biometricidentification,certificate,password,andothermethodsofverification. Suchcredentialsaresentbytheusertothesystemandthesystemcreatesasecureauthenticationtoken,whichisusedthroughouttheuser’ssession. In the communication with a bank (and similarly protected services), tokens are temporary. The system must create a new token after a given period. HASAL ET AL.5of13 Thegreaterprotectionof a user’s data andcommunicationisensured bytwo-factor authentication,44 for example, the user isasked toverifythe authentication of account credentials through an email and text message. There are also (higher) multi-factor authentication schemes,45 but such methods are rarely used during the login and conversation with a chatbot. In summary, authorization ensures that the correct person has access to the right data and services and authorization is necessary when a chatbot works with a user’s data. The storage and sharing of personal data over the internet is never safe enough. While interacting with the bots, a personal authentication security (Personal Scan) layer would confirm the user’s confirmation by the bots. The personal scan would also ensure that in the case of interacting with a rogue chatbot, the information of the user would not be used by hackers or other phishing agents.46 Malicious chatbots operating on cross-platform instant messaging applications (Viber, WhatsApp, etc.) can contact the user and pretend to be, forexample, apizzaorderchatbot.Such a situation iscalledSmishing.47 From the momentwhenuserresponds to the attacker, the chatbotcontinues intheconversation. The goal is similar to other phishing schemes to steal personal data. Receiving messages from unknown people/services is a good indicationthat itis probably phishing. Manyonline phishing attacks aresent froma hacked accountinto theuser’s friends’ accounts. Not responding to unknown messages and authorizations can protect a user from a smishing attack. Authentication for user applications defends the users’ data and devices from misuse; that is, when a user loses a phone or computer or leaves them unlocked. If a chatbot window was open, then an attacker could just ask the chatbot for sensitive information. Here the temporary tokens are necessary. End-to-End Encryption End-to-End Encryption (E2EE) is a system of communication where only the communicating parties can read the messages. The conversation is encryptedinawaythatonlytheuniquerecipientofamessageisallowedtodecryptit,andnot anyone inbetween.Transported data can be tampered andspoofed byathird party. Hence, it isimportant to ensure that onlythe involvedpartieshaveaccess to thecryptographickeysrequiredto decrypt the conversation. In the case of public-key encryption, the user’s device generates a pair of keys—private and public. There are different protocols to provide encryption, like the RSA algorithm.48 The public key is used for encrypting the messages, and the corresponding private key can decrypt those messages. Naturally, the public key can be used by anyone who sends the message to the private key owner. Simply put, both sides of the chatbot can share the public key to encrypt the communication. Encryption is sometimes used with authentication and integrity protection schemes. It is very important to keep the user’s private key safe, otherwise an attacker can decrypt all the messages that are intended for this user. As in the previous case, chatbots which do not work with personal data mostly do not use E2EE. Many chatbots operating on a website use only Hypertext transfer protocol secure (HTTPS), which transfers data through an encrypted connection by a Secure Sockets Layer (SSL) or Transport Layer Security (TSL). HTTPS is ‘point-to-point’ encryption as opposed to ‘end-to-end’ encryption. HTTPS is secure on the user’s connection to a load balancer, but then the data are decrypted back to plain text. This makes the data open to attack all services after the load balancer#.OnlyE2EE ensures the safe data transfer between involved subjects. Article 32 (a) of the GDPR specifically requires that companies take measures to pseudonymize and encrypt personal data|| . Many chatbots are connected to WhatsApp, Facebook Messenger, Telegram, Slack, and so forth. As GDPR requires the encryption of personal data, many chatbots operating on these networks support data encryption. For example, Facebook Messenger has a feature called ‘Secret Conversations’ that enables E2EE based on Signal Protocol developed by Open Whisper Systems. Note that there is a ‘law versus privacy’ conflict49 and companies in some states must design their encrypted products and services whereby governments, acting with appropriate legal authority, can gain access to the data in a readable and usable format**. Self-destructing messages In many cases when Sensitive PII (Personally identifiable information) is transmitted, self-destructing messages are a practical solution. Messages containing PII are automatically erased after a set period. It can involve both sides—the user and the chatbot. Self-destructing messages are an important security practice in communication with financial (banking) and healthcare chatbots. Specifically, Article 5 (e) of the GDPR states that personal data shall be kept for no longer than is necessary for the purposes for which it is being processed. General conversation is not personal, but financial and healthcare data are. It is not easy to distinguish by chatbot what is and what is not personal information (e.g., in communication with a pharmacy chatbot), but this is also defined by GDPR, and Protected Health Information50 (PHI) defined under US law. According to PHI, any information about health status, the provision of health care, or payment for health care is created or collected by a Covered Entity and must not be #https://tozny.com/blog/end-to-end-encryption-vs-https/. || https://cai.tools.sap/blog/chatbots-security-measures-you-need-to-consider/. **https://www.justice.gov/opa/pr/international-statement-end-end-encryption-and-public-safety. 6of13 HASAL ET AL. linked to a specific individual. GDPR compliance requires an ‘intent level’ of privacy. This means that only the user’s intent will be logged and stored for audit purposes and the personal information of the user must never be revealed, not even from the backend. User communication data, backend side The previous topic deals with secure communication, which is necessary in many cases. Nevertheless, communications with chatbots are mostly stored on the backend side. The communication and the user’s data are private data which should be stored to resist potential security threats.41 The historical conversations are extremely valuable for chatbot developers and companies. When a customer calls a hotline, an interactive voice response (IVR) announces ‘this call may be monitored or recorded’. Contact centers can legally record and monitor phone calls in the EU and UK. The situation is more difficult to answer in the US, where the laws vary across states. The same ‘monitoring’ happens in chatbot conversations, but the approval is hidden in the website terms and conditions and disclaimers. Companies use the stored conversation for an analysis of their services and as protection in possible litigation. Chatbot developers analyze communications for the better improvement of a chatbot’s quality, as ML methods need the data to be trained on and the quality of the model generally grows with the amount of data.51 Most of the companies store the communication with a chatbot due to the aforementioned reasons. Companies handling PII according to the GDPR definition must take special actions to store data safely; see the following text for more details. For instance, banks use self-destructing messages, but the system contains logs and account balance records. Article 32 (a) of the GDPR specifically requires that companies take action to pseudonymize and encrypt personal data. The communication with a chatbot can be stored, but without a connection to a given user and user data, or it must be protected. In the first case, the database contains just the communication and can be shared with the development team; unless it contains information that identifies an individual either directly or indirectly.9Companies use regular expressions, pattern matching, and entity recognition methods to detect personal data. On the other hand, Cortana, for its proper functionality, needs to collect a lot of data about its user: it has access to contact lists, an overview of sent and received e-mails, short messages, and incoming and outgoing calls. In addition, it also tracks where the user is, what he says, what music he listens to, what he buys, what movies he watches, and what he does in the browser. This data is then sent to the operator’s server for analysis. For these reasons, it is recommendedthatanyonewhodoesnot intend tousethe personal assistant functionshouldturn it off ontheirdevice.However, even afterCortana is turned off, the data it has collected remains stored on the servers, and Bing returns to it when it searches for user requests. This is another aspect, chatbots can have access to any data on the given platform. Accessibility to many data sources is a potential security risk in companies’ chatbots; configuration of user control and access permissions is a must. Hence, the communication must be secure and the chat history must be well handled when personal data are involved. It is manageable in closed-domain chatbots. A little more problematic is the situation with open-domain chatbots, which are not as restricted as closed-domain chatbots. Note that closed-domain chatbots working with PII require authentication and authorization, and users agree with terms and conditions. On the contrary, a user can discuss any topic she/he likes with open-domain chatbots, but the chatbot’s providers discourage people from chatting about personal details. Someone can admit that free open-domain chatbots are mostly for fun, as they cannot yet lead a serious discussion on any topic. However, the social-bot52 application Replika, for example, allows you to become friends with the AI,53 then people can share their feelings, more than just PII by the GDPR definition. Table 1 summarizes the information which is shared with third parties by three popular chatbots—Replika, Kuki, and CleverBot. It can be seen that all companies share some user data (to third-party tracking and analytic tools, like Google Analytics54) and cookies. This helps companies evaluate the type of customers. Chatbots are mostly hosted on cloud-based services, which has their own privacy policy for user data.55 All companies can read all messages to improve their services. Hence, it is not a private chat by definition. Lastly, many chatbots can be hosted on many communication channels, see Figure 1; and messages can be duplicated on another server, for example, WhatsApp data are stored on Google Drive, see Section 4. Currently, on GooglePlay and in the AppStore there are hundreds of chatbot applications, which try to be the user’s best friends, medical helper, teacher of foreign language by chat, girl/boyfriend, and even oracle from Tarot cards. This is potentially dangerous, because people can send their PII to anonymous services or can get wrong information from untrustworthy services; and all these services collect data. 3.1 Chatbot security versus the new law in the European Union’s GDPR General Data Protection Regulation (GDPR) is considered by many to be the golden standard among data privacy regulations and as an example for other states. Chatbots have access to an increasing variety of PII and personal details on their user base. In this regard, the European Union formed the GDPR whereby the mandate constructs the people’s fundamental right of privacy and freedom to protect personal data from malicious use. Malicious use comprises unregulated data storage, authentication without the user’s permission, and the authorization and encryption of the personal data without the user permissions.56 If a chatbot can access the personal data of a user, the chatbots must have the GDPR mandates and regulations in place. Article 5 (e) of the GDPR orders that personal data shall be kept for no longer than is necessary for the purposes for which it is HASAL ET AL.7of13 TA B L E 1 Information shared by chatbot companies with third parties Stored data /chatbots Replika Kuki CleverBot Name S* S Other profile information birthday, gender, work status, and so forth N N Email address, phone number S* S Facts about you and your life, hobbies and interests N People mentioned in the chat N Images sent, voice messages S** Text chat N S** S (‘starred’) Usage data—button clicks, search queries, and similar S S S Cookies S S S Backend infrastructure Azure, AWS AWS UK company servers E2EE Standart SSL HTTPS Note: ‘S’ is used for shared and ‘N’ for not shared. Items marked with ‘*’ are mandatory and ‘**’ are items shared without any PII. Notation ‘starred’ means that popular text messages can be shared. Empty cells are not specified in the privacy policy. being processed. Note that GDPR is enforced only in companies based in the EU or in services processing EU citizens’ personal data. Article 4 (1) of the GDPR defines ‘1personal data’ as being any information relating to an identified or identifiable natural person (‘data subject’). The GDPR defines principles and the lawful bases for processing personal information and also specifies rights for individuals:57 Transparency in data procession, Data minimization (the personal data processed is adequate, relevant, and limited), Purpose limitation (personal data can be collected, but not further processed), Storage limitation (personal data are deleted when it is no longer needed), and Download (personal data are provided on demand), Change (right to change personal data), and Remove (sometimes called Right to be forgotten, i.e., delete all the user’s data). The new GDPR mandate mentions that every technology that uses personal user data must maintain strict security so that there is no malicious activity of encryption leading to the infringement of personal rights and personal data. The mandate also clearly mentions that the tool or the software must make sure that it has alternative options in cases of the accidentalor unlawful destruction of personal data, unauthorized disclosure of the personal data and its transmission thereof. Therefore, every chatbot must have ‘appropriate safeguards’ to avoid encryption-based security challenges. The authentication of personal data and its usage is also mandated in the GDPR. The authentication of personal data must be by biometric data and not photographs of the actual user. This also comes under the personal scan security threats of chatbots. It is the legal obligation that no personal data can be gathered or used in any form from the authentication process. The controller, the chatbot, in this case, is also obligated to gather all forms of authentication data entered by a person in order to match the degree of fundamental personal rights. The user’s intents are logged, can be kept for audit purposes and must be revealed on demand. Data storage as per the mandate of GDPR is a processing method and must not restrict any user from its usage unless the user violates the code of conduct (GDPR). A GDPR compliance measure is to have an ‘intent level’ of privacy. Besides, data storage by the controller must not use unlawful or unfair means to keep secondary storage of the original data. Personal data of any type must fully maintain its integrity and confidentiality. Every entity must ensure that only the authorized user is allowed to view or use the personal data stored. Therefore, aspects like biometric scans and authentication methods are a must. Only the user itself can exchange personal information, but the system must not reveal the data. Therefore, following the mandates of the GDPR in lieu of chatbot developers implicates the need for data protection by maintaining the lawfulness, fairness and transparency of the stored personal data. In addition, the mandate also indicates the need for accuracy of the personal data, accountability, confidentiality and storage limitations, thereby reducing the chances of security threats. The mandate of data governance obligations for authorization and authentication processes may reduce the security challenges of chatbots. Many companies creating chatbot building platforms added commands similar to ‘Request Download Data’, ‘Delete Personal Data’ or ‘Change Personal Data’, which give users some control over their personal data. There are still some open issues with GDPR and chatbots.57 The GDPR provides the ‘right to be informed’ about how the data are processed, butoverallalgorithmic transparencyis low. The basicfact about chatbots’ algorithmic constructionis known, see Section 2.On the other hand, many implementationdetailsand knowledge bases (createdfrom collectedconversations)will neverberevealedbycompanies.Chatbotsalso do mistakes as they learn from examples with mistakes. Chatbots based on RNN (see references in Section 2) do grammatical mistakes. Chatbots can go off the rails like Microsoft’s Tay did.37 Therapist chatbots, best-friend chatbots, finance chatbots, and the like can give wrong advice. Companies must somehow handle huge amounts of data to make generative models feasible, but so everything can be corrected. The question is who is responsible when something bad happens, the user blindly relying on a chatbot or incautious developers? 8of13 HASAL ET AL. Big data versus Data minimization and Storage limitation; many companies store data as long as possible, and time-limit is defined by the necessity to keep the data. The right to be forgotten is also complicated in the chatbot field. Past communications can be used to train AI chatbots, but once (e.g., RNN) a model is created; it is impossible to remove such communications. It should not generally be an issue. If a user can be identified indirectly or distinguished in the group by some data, then such data are personal according to GDPR (‘profiling’). Similarly, if a user uses a unique answer, greeting, word, and so forth, then the chatbot can use this phrase and the user’s content can be identified. There are special categories of personal data in GDPR (e.g., ethnicity or sexual orientation). A chatbot does not have to respect such aspects of conversation. It is difficult to find the correct strategy to provide reasonable replies and fully respect all aspects of conversation at the same time. The principal aim is to produce consistent answers to semantically identical inputs. One solution could be Persona-Based Conversation Models.58,59 Such models provide consistent answers and could be a solution for this type of conversation. Contesting decision (Art 22 GDPR) gives a user a right not to be subject to a decision based solely on automated processing and contest the result of automated individual decision-making. It means that users have the right not to be subject to automated decision-making (e.g., by analysis of conversation), especially in Persona-Based Conversation Models. Users are mostly not informed about being a subject of automated decision-making, which is quite natural in AI chatbots. By Art 13-15 (refers to Article 22(1) and (4)), data processors must inform about the existence of using automated systems, but not about the right of a user not to be subject to automated individual decision-making. Suppose the user in communication decides not to be subject to automated individual decision-making. In that case, it won’t be easy to continue in conversation as many AI chatbots automatically create the user’s profile, especiallyin open-domain chatbots and voice assistants. The last issue is about settings of cookies and other data collected by third parties. When a user visits a website, he/she is asked to actively agree to cookies, but this is not the case with many chatbots. Especially those operating on a mobile app, which can also have access to the device’s hardware information and other data. 4DISCUSSION Chatbotsarestartingtodominateinmanyareas,astheyareefficient,operate24/7,andrepresentapleasingwayofgivinginstructionstoacomputer. As was mentioned in the introduction, chatbots are nothing more than a programmed input–output system and basic ‘recipes’ for how to create and operate chatbots are known. The usage of chatbots comes with great responsibility and cyber-security risks. In many cases, chatbots work with ‘sensitive information’, and security practices should be implemented. Communication with chatbots is done via already implemented channels and protocols. For the most part, the communication does not present security issues that have not already been discovered and properly mitigated. But still, not every chatbot communicates through encrypted channels and users should not send any personal data during chatting. Some security issues come with stored communications and user data. The problem is that communication is extremely valuable and many companiesstorepastcommunication.Datacanbeencryptedonservers,but theMLalgorithmcannotbetaughtonencrypteddata.Theresultwould not make sense; moreover, NLP tools are not prepared to learn on encrypted data. This is one of the moments when communication is revealed and can be read. Companies must handle such situations according to the GDPR and other similar rules when PII are presented. Companies and employees would not share the discussion with PII. From a general perspective, chatbots record and learn from previous communication; they later re-use words, phrases, and complete utterances from conversations when talking with others later. Another issue is that chatbots can operate on third-party conversational interfaces/networks like Facebook Messenger, Viber, Facebook WhatsApp, Twitter, Facebook, LinkedIn, and so forth. Owners of those app/networks have different policies about what belongs to the user and what belongs to the company when the user uses their services. For example, there were more than 300,000 active chatbots on Facebook alone in 2018, which exchanged 8 billion messages between people and businesses. It is four times more than the number of messages exchanged in 2017††. Those and similar companies evaluate the price of data from chatbots.60 It is not only about using previous communication to train the chatbot’s ML algorithm; it is about cookies, user preferences, the main discussion topics of different groups, activity, language, positive/negative messages,and any other features which can be extracted from chatbots and sold to other customers. Note, any of the previous features belongs to the PII category, and still are very valuable for personalization and information retrieval.59 For instance, Facebook WhatsApp recently changed its terms and privacy policy to connect WhatsApp with other Facebook companies and use some information collected from WhatsApp‡‡. It means that business messages can be used for commercial purposes, and some of them are also stored on Facebook’s servers. As was mentioned at the beginning of this paragraph, many business messages are created by business chatbots. When a user chats with a chatbot on popular communication platforms, the platform operator gets information about it. To have a perspective about the user data stored by the terms policy, for example, WhatsApp stores—Device ID, User ID, Advertising data, Purchase history, Rough placement, Telephone number, Email address, Contacts, Product interaction, Failure data, Performance data, Additional diagnostic data, Payment info, Customer support, Product interaction, and Other user content. ††https://about.fb.com/news/2018/05/f8-2018-new-tools-for-businesses-and-people-to-deepen-connections-in-messenger/. ‡‡https://www.xda-developers.com/whatsapp-updates-terms-privacy-policy-mandate-data-sharing-facebook/. HASAL ET AL.9of13 FIGURE 2 Snapshot of conversation with kuki_ai chatbot Facebook Messenger collects Purchase history, Additional financial information Exact location, Rough placement, Physical address, Email address, Name, Telephone number, Additional user contact information, Contacts, Photos or videos, Game content, Other user content, Search history,Browsinghistory,UserID, DeviceID,Productinteraction,Advertisingdata,Furtherinformation onuse,FailuredataPerformancedata,Additional diagnostic data, Other types of data, Browsing history, Health, Fitness, Payment info, Photos or videos, Audio data, Game content, Customer support, Other user content, Search history Sensitive information, iMessages, Email address, Phone number Search history, and Device ID. Signal stores no information, except the Phone number (the only personal information) and Signal does not attempt to associate it with your identity. Telegram stores collected Contact information, Contacts, and User ID. User should know this, because in modern data driven world information channels can be connected. Possibly not now, but certainly in the future, people can be recognized by text; similarly to how a neural network identified the specific scenes in Henry VIII which were not written by William Shakespeare.61 The digital footprints60 of many users can be collected from many sources and compared for similarity. There are thousands of chatbots connected to webpages, operating on WhatsApp, Facebook, LinkedIn, Viber and other communications channels,hundredsofchatbotswiththeirownapplication,andafewverycomplexchatbotslikeCortanaoperatingsimultaneouslyinCortanaIntelligence Suite, Windows, and many other devices. It is not easy to compare the terms and privacy policy concerning the data processing of all companies. The issue is that people click to accept and do not read the terms and conditions in all services. There are some regulations (like GDPR) made by a government or other authority to control and protect people’s personal data which give some level of trustworthiness. People should avoid using the services which do not follow basic security rules discussed in Section 3. Even kuki_ai chatbot has an opinion about data sharing, see Figure 2. Social aspects This article focuses more on the technical solutions of security details around chatbots, but the authors also have to mention the social aspects of using chatbots to complete the security topic. Social aspects can enlarge because chatbots try to draw user attention and keep the user in communication. Communication in natural language is an important way of expressing feelings, ideas and concerns between humans. As chatbots learn from the user, they can take over the user’s rhetoric. One example is Microsoft’s social media-based chatbot ‘Tay’, who ‘evolved’ from a normal teenage girl to a chatbot displaying anti-Semitic, racist and sexist attitudes in less than sixteen hours.37 Microsoft’s social media-based chatbot ‘Zo’ also picked up some offensive habits.62 Despite the intention of their designers, many ML implementations have developed harmful human-like biases that cannot be easily removed.62-66 On the other hand, chatbots can be pre-biased and, for example, trained to influence public opinion and discourse.67-69 Shortly, social chatbots started to operate on many social platforms to spread pre-programmed information. Even chatbot assistants, who recommend a restaurant, flight, financial product, medicament, and so forth,canbepre-programmed to offer users prearrangedproducts.Thiscreatesadditionalincome for the chatbot creators from ‘product placement’ §§, but one has to realize that chatbots work 24/7 and people trust them. Chatbots are designed to draw a users’ attention by messaging because it is fast, easy, and actually feels like a conversation; and to use targeted messaging (precisely Conversational Marketing70) to help or sell you a product. Chatbot design techniques are continuously improved,20 and it is more and more difficult to distinguish conversations between a human or a chatbot.Some chatbotscanevenassistasmentalhealth support,71 whichrequiresareallygoodconversationallevelbychatbot.Suchsocial chatbots can obtain valuable information from the user or even make their user vulnerable to infiltration from other chatbots. The annual Loebner Prize1,72 is the world’s longest-running Turing-Test competition. It is a competition in artificial intelligence that awards computer programs considered by the judges to be the most human-like. No Chatbot has ever achieved the Loebner Prize’s golden medal (responses which cannot be distinguished from humans). However, some chatbots have scored as highly as 3 out of the 12 judges believing they were human.20 Huge technological companies have access to an enormous amount of conversational data and have resources to pay skilful programmers. This, with improving chatbot design §§Basically, this case is product placement, but this term has not been used before in this context. This topic also has not been publicly discussed by law or companies. Nowadays, voice assistants search for, for example, restaurant from open-source data, when user ask for a good restaurant. The question is what happens when some restaurant franchise will pay for directing the people to their restaurant.