Toward Designing Ethically Acceptable AI Security Systems Through Agent Modeling

This is a self-archived version of an original article. This version may differ from the original in pagination and typographic details. Author(s): Title: Year: Version: Copyright: Rights: Rights url: Please cite the original version: CC BY 4.0 https://creativecommons.org/licenses/by/4.0/ Toward Designing Ethically Acceptable AI Security Systems Through Agent Modeling © The Author(s) 2025 Published version Hallamaa, Jaana; Janhunen, Tomi; Nummenmaa, Jyrki; Nummenmaa, Timo; Saariluoma, Pertti; Zimina, Elizaveta Hallamaa, J., Janhunen, T., Nummenmaa, J., Nummenmaa, T., Saariluoma, P., & Zimina, E. (2024). Toward Designing Ethically Acceptable AI Security Systems Through Agent Modeling. In A. Autero, M. de Moraes Batista Simao, & I. Karppi (Eds.), Smart Urban Safety and Security : Interdisciplinary Perspectives (pp. 171-196). Palgrave Macmillan. https://doi.org/10.1007/978- 981-97-2196-2_9 2024 171 © The Author(s) 2025 A. Autero et al. (eds.), Smart Urban Safety and Security, https://doi.org/10.1007/978-981-97-2196-2_9 9 Toward Designing Ethically Acceptable AI Security Systems Through Agent Modeling JaanaHallamaa , TomiJanhunen , JyrkiNummenmaa , TimoNummenmaa , PerttiSaariluoma, andElizavetaZimina Introduction Security is a crucial concern in public places such as shopping malls. People need to feel safe and businesses should run smoothly; hence, the security measures should be sufficient but not too exaggerated. Overall, public places and shopping mall security present a complicated topic, as practically everything starting from the building design is relevant. Harmful events such as violent attacks or overreaction from guards will reduce interest in visiting a shopping mall. J. Hallamaa University of Helsinki, Helsinki, Finland e-mail: [email protected] T. Janhunen Tampere, Finland e-mail: [email protected] 172 AI has many components that make it useful in monitoring security in a public place. Conducting event analysis from videos, various sensor data, and voice data is a challenging task; hence, the utilization of AI becomes inevitable. Building AI systems requires considerable amounts of human and computational resources. Therefore, the suitability of such AI systems should be studied in advance. For this purpose, in this chapter, we propose the employment of modeling of relevant actors and shed light on the ethical concerns surrounding them as a multi-agent system (MAS). MASes serve as fundamental models for AI systems and their operating environments, offering flexible means for their definition, analysis, and implementation through agent languages. When understanding agents’ behavior, beliefs, desires, and intentions (BDI) are central concepts that have been widely applied in literature. In this chapter, the moral dimensions of BDI agents are considered. We approach them from the perspective of interaction and presuppose cooperation between agents that is based on social intentionality, thus initiating a framework for the socio-ethical modeling of agency. The framework utilizes three modes of social interaction that can be attributed to the intentions of participating agents. Throughout the chapter, social phenomena and scenarios arising within the context of a shopping mall are employed to drive discussion and analysis. In addition to theoretical considerations, the premises for practical implementations are defined in a GAMA model. Simulations and visualizations created using the proof-of-concept implementation serve to illustrate and to communicate the model for stakeholders. MASes provide abstract models of AI systems in action, ranging from complex societies of collaborative and/or competitive agents to simple single-agent problem-solving scenarios (see Woolridge, 2009, for a J. Nummenmaa (*) • T. Nummenmaa • E. Zimina Tampere University, Tampere, Finland e-mail: [email protected]; timo[email protected] P. Saariluoma University of Jyväskylä, Jyväskylä, Finland e-mail: [email protected] J. Hallamaa et al. 173 comprehensive introduction). Regardless of the application, individual agents in such systems perform actions in response to the perceptions they gather from their environments. An ideal and rational agent is expected to achieve its goals and maximize its expected utility in the long run (Russell & Norvig, 2020). To analyze moral aspects, agents should possess functionalities beyond perceiving and acting. A widely accepted approach defines beliefs (B), desires (D), and intentions (I) as fundamental elements of the behavioral description of agents (Rao & Georgeff, 1991). The BDI architecture serves as a solid foundation for addressing ethically relevant settings within MASes. Our empirical case concerns a shopping mall illustrated in Fig.9.1 and, more specifically, the security, monitoring, and maintenance activities of the mall. The example includes only an abstraction of a particular section of a real mall, with only a fraction of its services and activities. Nevertheless, it serves us as a challenging environment that is easy to understand in general but presents endless possibilities for refinement from the modeling perspective. In this light, our series of examples will Fig. 9.1 Illustration of the simulation environment: a shopping mall. Source: Authors (2023) 9 Toward Designing Ethically Acceptable AI Security Systems… 174 specifically concentrate on modeling the activities of customers and staff members (security guards in particular). In general, agents may participate in activities individually or by collaborating with others. Their actions may give rise to ethical concerns, which constitute the particular focus of our research. Some actions and activities in the mall involve groups of agents committed to joint goals, such as going to the movies together. The ways in which groups form and organize themselves vary, increasing the complexity of the process and the need for social interaction. While group formation is a complex process in itself, it is not the main focus of this chapter. Instead, we take into account the roles played by groups of agents (if present). In this chapter, we address the ethically relevant or moral aspects of MASes. Our overall goal is to find suitable primitives for the formalization of ethical principles and, in this manner, establish the premises for ethical modeling in multi-agent contexts. Ideally, ethical principles can be separated from operational details, and once formalized, they can be employed to analyze and answer ethically relevant questions. Our interdisciplinary approach emphasizes philosophical aspects, aiming to gain a fundamental understanding of ethically meaningful primitives from the analysis of multi-agent scenarios in the mall domain. Rather than concentrating on the representation of norms (see, e.g., Broersen etal., 2001; Neumann 2010), we take the modes of social intentionality (Tuomela, 2007) as a starting point for our analysis, thereby adopting a socio-ethical approach to modeling MASes. Our long-term goal is to facilitate the implementation of MASes and their simulation as well as promote the development of agent (specification) languages. However, we do not introduce new languages in this preliminary study and instead utilize an existing one, namely, GAMA (Taillandier etal., 2019), in our illustrations and proof-of-concept implementations. To summarize, this chapter initiates a socio-ethical viewpoint and approach to modeling BDI agency. Its main contributions include the following: 1. establishing the framework of modes of social intentionality for the analysis of BDI agency; 2. analyzing the grounds for moral action on the basis of the modes of action of the agents involved; J. Hallamaa et al. 175 3. applying the framework in the socio-ethical modeling of an openended application domain (the mall domain); and 4. addressing the limitations of traditional BDI models in the formalization of ethical principles. Multi-agent Systems A MAS constitutes an ecosystem of computing entities, namely, agents, each of which solves some sub-problem as part of a larger collective endeavor. The agents in a MAS form a network by virtue of sharing knowledge and communicating with each other. Other capacities, say the ability to follow if-then rules and core behaviors such as mobility, interaction, adaptation, and learning, have been listed as their characteristics (see, e.g., Balaji & Srinivasan, 2010; Rocha etal., 2017). Agents andEnvironments An agent A is an entity whose state consists of precisely defined mental components such as beliefs, capabilities, choices, and commitments that roughly correspond to their common-sense counterparts in humans (Shoham, 1993). Additionally, values that are more concrete may also be relevant when characterizing states. In the field of computer science, it is a common practice to formalize the states of agents by introducing state variables whose values range over particular domains of interest. The properties of the environment of a MAS are essential when it comes to designing a MAS in the first place, and they also determine how difficult it is for the MAS to achieve its goals. Environments can be roughly classified on the basis of their central characteristics, allowing us to define ranges such as static versus dynamic, or fully observable versus partially observable (Russell & Norvig, 2020). In simple MASes, it is also possible to view the environment as one agent hosting others (cf. typical master-slave architectures). 9 Toward Designing Ethically Acceptable AI Security Systems… 176 Actions Agents in a MAS interact with each other and their environment by performing actions. The actions serve two primary purposes: either observing or changing the state of the MAS, which includes the states of the individual agents as well as that of the environment. In addition to actions performed by agents, events occurring unexpectedly in the environment may also affect the state of the MAS.The (effects of) actions and events can be defined in different ways, for example, by assigning new values to state variables on the basis of old ones. In logic-oriented formalisms, such as STRIPS (Fikes & Nilsson, 1971), and the so-called action languages (Gelfond & Lifschitz, 1998), the states of a system can be described using state predicates, also known as fluents, whose truth values may change over time. The same applies to actions: A particular action can be performed if its preconditions are met. As the result of executing an action, certain fluents may receive truth values, thereby establishing the postconditions of the action. We describe changes such as these either as additions or deletions of predicates, which are sufficient to cover four possible cases for each fluent, namely, whether it stays/becomes true/false. Example 9.1 Consider a customer C entering the mall. Let e and l be the names denoting the entrance and the lobby of the mall, respectively. Furthermore, let predicates next/2 and in/2 describe whether a customer is next to something or in a particular space. Consider a particular customer c1 at the entrance, that is, next(c1, e) is true, thus enabling the action enter(c1). When executed, next(c1, e) is falsified while in(c1, l) becomes true. J. Hallamaa et al. 177 Group Actions In Example 9.1, the action involves a single agent. As a result, the state of the agent changes as reflected by the modified truth values of the fluent involved. In multi-agent scenarios, we consider group actions engaging several agents. Example 9.2 Continuing our examples, consider the act of one customer C1 approaching another (C2) in the same space S. As a result, both customers remain in the space S, but they appear next to each other afterward as encoded with the fluent next/2. As a result of a group action, the states of all agents involved may be updated. The action in Example 9.2 is asymmetric by nature, and the latter agent is merely treated as an object. The other agent might react by escaping from the situation by performing a counteraction escape(C2, C1), thus falsifying the fluents next(C1, C2) and next(C2, C1). These conditions are the natural pre- and postconditions for yet another group action: shake-hand(C1, C2). BDI Models Formalized Fluents describing a MAS essentially express the components of its state that are relevant for modeling. As usual, the meaning of such predicates can be decided on a case-by-case basis. For instance, in our shopping mall domain, if in(C, S) is true for a particular customer C and a space S, then 9 Toward Designing Ethically Acceptable AI Security Systems… 178 C is in S. Ethical aspects, however, cannot be directly formalized using state predicates, since the mental states of agents matter as well. To this end, one prevailing approach captures the beliefs, desires, and intentions of agents as meta-level concepts. In the sequel, we follow Labrou and Finin (1994) and formalize these concepts in terms of modal operators BA, DA, and IA associated with an agent A. For now, we restrict the application of these operators only to fluents or their negations, hence forbidding nesting. This is primarily to mitigate computational complexity and facilitate implementation. Example 9.3 Consider a customer C who wants to see a movie M in a particular theatre t of the mall. Ticket possession is one of the natural preconditions for seeing a movie. In the above, the mental state of the customer is updated accordingly, that is, the intention of seeing the movie is falsified. The management of desires is an independent aspect: Persistent desires can be maintained indefinitely, while those that are more one-time by nature can be abandoned by falsifying DC(D). J. Hallamaa et al. 185 of morally neutral acts such as making purchases, enjoying a meal, and resting one’s feet on a bench. Likewise, the customers should not (try to) do anything that would inhibit or hinder other customers and staff from setting their own goals and performing acts that are appropriate instances of behavior in the shopping mall context. In exceptional cases, morally forbidden acts may be permissible if A’s aim is to preserve something of (great) value, and the likely outcome of the harmful action is (expected to be) more positive than the anticipated outcome of A not performing the action. Example 9.10 A security guard S may use physical force to hinder customer C1 from punching C2. Some of the permissible acts are required or compulsory, and A has a moral obligation to perform them in a certain situation or context. A conceptual connection exists between what is permissible and compulsory in the following manner: All compulsory acts are permissible, and none of the impermissible acts are compulsory. Example 9.11 The customers must finish their purchases and leave the shopping mall when the closing time is approaching. The guards have an obligation based on their duties to ensure that the customers leave the premises. The same applies to emergency situations: The sounding of the fire alarm indicates that the customers must leave the mall immediately, disregarding what they are doing, and the guards must help them by showing the way out and making sure everyone is safe. The forbidden acts are, by definition, unfavorable, and this is why there is a common interest in curbing or preventing them. The permissible acts, for their part, can be categorized depending on how favorable they are in terms of their effects on others. Between the classes of forbidden and permissible acts, there lies a class of unfavorable acts. Moral acts contribute to the good of others, often enhancing their well-being. 9 Toward Designing Ethically Acceptable AI Security Systems… 186 Example 9.12 A security guard S assists a customer C who is looking for a place or an object P (e.g., a toilet, a garbage bin, the cinema). The different modes of social intentionality we have discussed imply certain moral features, as A is not able to engage itself in any positive cooperation with other agents without refraining from harming them and committing itself to doing its own part in the joint venture. To model the cognitive states and reasoning behind such an action would require a much more detailed BDI architecture than is possible to present within the scope of the present chapter. This might include implementing casebased reasoning in terms of the favorability of the probable outcomes of A’s actions and a structure of deontic logic covering the concepts of obligation, permission, and forbidden (see Honarvar & Ghasem- Aghaee, 2009). Modeling andImplementation Several agent languages and related tool sets are available for modeling and simulating MASes based on BDI agents (Adam & Gaudou, 2015). One of these toolsets is GAMA (Taillandier etal., 2019), a modeling and simulation environment that focuses on spatial modeling where specifications are written using the GAML language. The GAMA platform provides resources for building simulations within the framework of the classic BDI paradigm that is based on the philosophy of action (Bratman, 1987). Due to the provided support for spatial modeling and graphical visualization, we decided to implement our BDI models utilizing GAMA as our execution platform. These features are highly useful when it comes to modeling the shopping mall domain (see Fig.9.1). Figure9.1 illustrates the floor plan of a conventional shopping mall containing walkable areas: a lobby, a movie theater, a restaurant, and a toilet. A model’s entities, processes, and activities were formalized in GAML in terms of agents, which, in turn, were specified by their species, each with their own attributes, actions, and behaviors. An instance of a species J. Hallamaa et al. 187 can perform actions. The action is a function if it can return a value and a procedure if it cannot. A simple example of a procedure is the action of movement to some point: With a function, we assign the returned value to a variable (here referred to as the point type): The most critical feature of the BDI architecture is a plan, which defines an order of statements that are performed to fulfill some intention. Partial plans created at the time of designing can greatly reduce computational complexity (Bordini etal., 2007); hence, we used plans as offered by GAMA, although we do not touch plans in this chapter. The simplest plan in our simulation was wandering within the space limits: An agent can perceive the environment and change its behavior, mental state, social links, and the like on the basis of the knowledge it acquires. Agents can also interact with each other and change each other’s attributes and behavior by means of the ask statement (see Example 9.13). To manage time, GAMA operates using three global variables: cycle (an integer incremented by 1 at each step of the simulation), step (the modifiable duration of a simulation step; 1 second by default), and time (the actual time since the beginning). 9 Toward Designing Ethically Acceptable AI Security Systems… 188 To specify the examples described in the chapter, we first described our actions. We first identified the participants, preconditions, and possible additions and deletions. The GAMA implementation was required to follow the rules we had defined and act as an executable specification. Example 9.13 continues from the case of the security guard on a lunch break presented in Example 9.8. Example 9.13 Customer C1 is next to customer C2 and notices that C2 is smoking, that is, smoking(C2) is true, in a location L.Customer C1 stores this information as a belief in addition to the location of C2. Customer C1 also develops an intention of sharing information with a guard in wemode. Security guard S needs to be in progroup I-mode or we-mode to help a customer C.Thus, it may happen that S enters we-mode (and commits to the intention to guard the mall, depicted by the predicate patrol(S)) when S and C come next to one another if S is not in that mode at the time. In some cases, S may not be able to enter progroup I-mode or we-mode, and, thus, will not be able to help C.This alternate case is omitted here. J. Hallamaa et al. 189 Customer C1 informs security guard S about customer C2 smoking inlocation L. In the GAMA implementation, each customer is observing the area within its viewing distance. If customer C1 notices another customer C2 smoking (checking the Boolean smoking feature), C1 obtains a new belief containing C2’s location and develops a desire to approach the security S (if such a desire is not already present in C1). Customer C1 approaches S, and if the latter is in progroup I-mode or we-mode, C1 shares its belief about the smoker’s location within the inform_security plan. If S is in pure I-mode, C1 first attracts S’s attention and asks to receive the react_to_customer intention. S then decides whether he wants to abandon the pure I-mode and listen to the customer or not. With the probability of 50%, S shows that he is ready to be informed and asks the customer to proceed with the inform_security plan. Otherwise, S stops reacting to the customer, and the latter abandons the approach_security plan but does not receive an intention to share his knowledge. For the sake of brevity, the code below has been simplified. 9 Toward Designing Ethically Acceptable AI Security Systems… 190 J. Hallamaa et al. 191 Related Research Bosse etal. (2011) created a model for describing the reasoning process of other agents utilizing the BDI concepts, namely, beliefs, desires, and intentions, and the theory of mind. Norling (2004) utilized BDI features that resemble folk psychology to incorporate psychological abilities such as knowledge acquisition and decision-making into agent modeling. Adam etal. (2009) proposed a logical formalization to embed emotions into agent models. Cranefield and Dignum (2019) suggested a way to integrate social aspects into BDI agent systems by modeling social practices. To inhibit unwanted outcomes of actions, there must be constraints in place that rule out as many of such consequences as possible. Norms are deontic statements that are employed to define which (types of) desires and intentions A must not try to realize through actions. Traditional approaches to reasoning pertaining to norms are based on modal logic (Garson, 2021) and, in particular, deontic logic, which can be utilized to formalize obligations and permissions concerning conditions, in analogy to using modal operators in the description of BDI systems. Criado etal. (2010) extended BDI concepts to model agents that can make pragmatic, autonomous decisions by considering which norms to follow and how to apply them. Such extensions are possible in our approach, enriching the selection of conditions available for modeling. The same can be stated about aspects of time (see, e.g., Urlings etal., 2006) and temporal operators, since obligations and their fulfillment have implications for the past and future. When considering agent functionality in general, the ability to construct plans for the realization of goals and intentions is central, and the same holds true in the context of BDI systems (see, e.g., de Silva etal., 2009; Sardiña etal., 2006) for the hierarchical case. Since our approach is compatible with the traditional STRIPS-style planning (Fikes & Nilsson, 1971), we may cover scenarios involving concrete planning or related verification tasks. However, for the time being, we have concentrated more on reflexive agents and their use in simulations. A related concept is crowd simulation (Cho etal., 2008) that is also relevant to the shopping mall domain but beyond the purview of our focus for now. 9 Toward Designing Ethically Acceptable AI Security Systems… 192 Discussion andConclusions This chapter approaches the social dimension of actions performed by agents in terms of modes of social intentionality. The three modes, namely, pure I-mode, progroup I-mode, and we-mode, characterize the interacting agent’s intention toward engaging in social relationships with other agents that are relevant to the intended goal and the action being performed. The modes can be applied in various ways in the analysis, definition, and implementation of MASes. First of all, they can be used implicitly when modeling actions to understand their true nature and to ease their formalization in general. The models produced provide possibilities for analyzing, verifying, and simulating agents’ behavior. If modes are explicitly introduced as variables or conditions in modeling, then a more refined control over execution is enabled via the preconditions of actions. In addition, actions may also manipulate modes as needed if the agents’ social intentions change over time, for example, as reactions to other agent’s actions or events occurring in the environment. The three modes allow the analysis of positive instances of social action but do not lend themselves to model actions that are disruptive in terms of cooperation as such. In this respect, new conditions of intentionality could be taken into consideration as potential extensions of Tuomela’s research (Tuomela, 2007). Our chapter has, to some extent, been constrained by the limitations of the BDI model itself, which focuses on the three modalities involved, and there is no straightforward way to express the three modes of social intentionality with them. Rather, it was deemed necessary to incorporate modes as factual truths in terms of fluents (cf. the mode/3 predicate) as part of the agents’ states. In reality, agents have much more complex desires and social intentions that can be realized in a number of different ways, each of which could be modeled as a separate plan that further comprises steps involving intentions. Such a recursive structure seems extensive, but without it, a large amount of the specification moves to program code. A major step in our future work will be to tackle these limitations. There are also notable aspects in modeling that have been left unaddressed and will be considered in future work. Most importantly, J. Hallamaa et al. 193 the progroup-I-mode and the we-mode presume a group of peer agents. The group dynamics (forming and maintaining groups) and premises for trust are complicated issues in themselves that warrant further attention in the future. Acknowledgments All co-authors of this chapter have been partially supported by the Academy of Finland’s Strategic Research Council funded project Ethical AI for the Governance of Society (ETAIROS, grant #352441). References Adam, C., & Gaudou, B. (2015). BDI agents in social simulations: A survey (No. RR-LIG-050, LIG). Les rapports de recherche du Laboratoire d'Informatique de Grenoble. Adam, C., Herzig, A., & Longin, D. (2009). A logical formalization of the OCC theory of emotions. Synthese, 168(2), 201–248. https://doi. org/10.1007/s11229- 009- 9460- 9 Balaji, P. G., & Srinivasan, D. (2010). An introduction to multi-agent systems. In D. Srinivasan & L. C. Jain (Eds.), Innovations in multiagent systems and applications—1 (pp. 1–27). Springer. https://doi. org/10.1007/978- 3- 642- 14435- 6_1 Bordini, R.H., Hübner, J.F., & Wooldridge, M. (2007). Programming multiagent systems in AgentSpeak using Jason. John Wiley & Sons, Inc. Bosse, T., Memon, Z.A., & Treur, J. (2011). A recursive BDI agent model for theory of mind and its applications. Applied Artificial Intelligence, 25(1), 1–44. Bratman, M. (1987). Intention, plans, and practical reason. Harvard University Press. Broersen, J. M., Dastani, M., Hulstijn, J., Huang, Z., & van der Torre, L.W. N. (2001). The BOID architecture: Conflicts between beliefs, obligations, intentions and desires. In E.Andre, S.Sen, C.Frasson, & J.P. Mueller (Eds.), Proceedings of AGENTS 2001 (pp. 9–16). ACM. https://doi. org/10.1145/375735.375766 Cho, K., Iketani, N., Kikuchi, M., Nishimura, K., Hayashi, H., & Hattori, M. (2008). BDI model-based crowd simulation. In H.Prendinger, J.Lester, & M. Ishizuka (Eds.), Proceedings of IVA 2008 (pp. 364–371). Springer. https://doi.org/10.1007/978- 3- 540- 85483- 8_37 9 Toward Designing Ethically Acceptable AI Security Systems… 194 Cranefield, S., & Dignum, F. (2019). Incorporating social practices in BDI agent systems. In E.Elkind, M.Veloso, N.Agmon, & M.E. Taylor (Eds.), Proceedings of AAMAS 2019 (pp.1901–1903). IFAAMAS. Criado, N., Argente, E., & Botti, V.J. (2010). A BDI architecture for normative decision making. In W. van der Hoek, G.A. Kaminka, Y.Lesperance, M.Luck, & S.Sen (Eds.), Proceedings of AAMAS 2010 (pp.1383–1384). IFAAMAS. de Silva, L., Sardiña, S., & Padgham, L. (2009). First principles planning in BDI systems. In C.Sierra, C.Castelfranchi, K.S. Decker, & J.S. Sichman (Eds.), Proceedings of AAMAS (Vol. 2, pp.1105–1112). IFAAMAS. Fikes, R.E., & Nilsson, N.J. (1971). STRIPS: A new approach to the application of theorem proving to problem solving. In D. C. Cooper (Ed.), Proceedings of IJCAI’71 (pp.608–620). Morgan Kaufmann. Garson, J. (2021). Modal logic. In E.N. Zalta (Ed.), The Stanford encyclopedia of philosophy (Summer 2021 ed.). Stanford University. Gelfond, M., & Lifschitz, V. (1998). Action languages. Electronic Transactions on Artificial Intelligence, 2, 193–210. http://www.ep.liu.se/ej/etai/1998/007/ Hallamaa, J., & Kalliokoski, T. (2020). How AI systems challenge the conditions of moral agency? In M. Rauterberg (Ed.), Proceedings of culture and computing, C&C 2020 (pp. 54–64). Springer. https://doi. org/10.1007/978- 3- 030- 50267- 6_5 Honarvar, A. R., & Ghasem-Aghaee, N. (2009). Casuist BDI-agent: A new extended BDI architecture with the capability of ethical reasoning. In H.Deng, L.Wang, F.L. Wang, & J.Lei (Eds.), Artificial intelligence and computational intelligence (pp. 86–95). Springer. https://doi. org/10.1007/978- 3- 642- 05253- 8_10 Koster, A., Schorlemmer, W.M., & Sabater-Mir, J. (2013). Opening the black box of trust: Reasoning about trust models in a BDI agent. Journal of Logic and Computation, 23(1), 25–58. https://doi.org/10.1093/logcom/exs003 Labrou, Y., & Finin, T. (1994). A semantics approach for KQML—A general purpose communication language for software agents. In Proceedings CIKM’94 (pp.447–455). ACM. Neumann, M. (2010). Norm internalisation in human and artificial intelligence. Journal of Artificial Societies and Social Simulation, 13(1). https://doi. org/10.18564/jasss.1582 Norling, E. (2004). Folk psychology for human modelling: Extending the BDI paradigm. In Proceedings of AAMAS 2004 (pp. 202–209). IEEE Computer Society. J. Hallamaa et al.