Automatic linguistic reporting of customer activity patterns in open malls

https://doi.org/10.1007/s11042-021-11186-3 1188: ARTIFICIAL INTELLIGENCE FOR PHYSICAL AGENTS Automatic linguistic reporting of customer activity patterns in open malls Manuel Oca˜ na1·David Chapela-Campa2·Pedro ´ Alvarez3·Noelia Hern´ andez1· Manuel Mucientes2·Javier Fabra3·´ Angel Llamazares1·Manuel Lama2· Pedro A. Revenga1·Alberto Bugar´ ın2·Miguel A. Garc´ ıa-Garrido1·Jose M. Alonso2 Received: 2 February 2021 / Revised: 28 April 2021 / Accepted: 24 June 2021 / ©The Author(s) 2021 Abstract In this work, we present a complete system to produce an automatic linguistic reporting about the customer activity patterns inside open malls, a mixed distribution of classical malls joined with the shops on the street. These reports can assist to design marketing campaigns by means of identifying the best places to catch the attention of customers. Activity patterns are estimated with process mining techniques and the key information of localization. Localization is obtained with a parallelized solution based on WiFi fingerprint system to speed up the solution. In agreement with the best practices for human evaluation of natural language generation systems, the linguistic quality of the generated report was evaluated by 41 experts who filled in an online questionnaire. Results are encouraging, since the average global score of the linguistic quality dimension is 6.17 (0.76 of standard deviation) in a 7- point Likert scale. This expresses a high degree of satisfaction of the generated reports and validates the adequacy of automatic natural language textual reports as a complementary tool to process model visualization. Keywords Social workflows ·Localization ·Data mining techniques · Parallelization strategies ·Automatic linguistic reporting 1 Introduction SOcial Workflows (SOW) coordinate the activities carried out by a group of users [15]who either individually or in cooperation try to achieve a certain objective. The SOW are unstructured flows in which a large number of users carry out activities of a very diverse nature Manuel Oca˜ na [email protected] 1Universidad de Alcal´ a, Alcal´ a de Henares, Madrid, Spain 2Centro Singular de Investigaci´ on en Tecnolox´ ıas Intelixentes (CiTIUS), Universidade de Santiago de Compostela, Santiago de Compostela, Spain 3University of Zaragoza, Zaragoza, Spain Published online: 8 July 2021 Multimedia Tools and Applications (2022) 81:3369–3395 that they spread over time and typically consume few computing resources. An example of processes that are modeled through these types of workflows are the marketing campaigns that have the objective to motivate and persuade potential customers in the consumption of a certain product or service. However, from a technological point of view, the characteristics of the SOW are: (i) they model unstructured processes where users have multiple options and/or paths, and where they can even carry out additional activities to those initially planned in the workflow; (ii) a large number of users; and (iii) depending on the skill or motivation of each user, the same activity can be performed immediately or have a temporary duration much higher (in the order of hours or even days). These characteristics add a series of challenges when managing and monitoring the execution of SOW. On the one hand, the management of this type of workflow requires the handling of a large number of simultaneous executions whose duration extends over time. On the other hand, the analysis of the execution of the workflow requires the use of process mining techniques [44] to obtain the real workflow followed by the users and, in this way, understanding what really happens in the execution of the process, thereby allowing the improvement of the process and its (dynamic) adaptation to the needs of users. However, for unstructured processes with a huge number of executions, the results of the current process discovery algorithms are workflows that have a structure in spaghetti and therefore very difficult, if not impossible, to interpret. Therefore, it is necessary to develop techniques to extract valuable information about the discovered workflow. Furthermore, this information should be conveyed to the users who will consume it in an understandable way. To this aim, we combine in our reports both graphical information about the process as well as automatically generated linguistic reports which summarize the relevant information about the processes. This is the first axis around which pivots the project Business (Artificial) Intelligence for Social Workflows (BAI4SOW). Furthermore, the BAI4SOW project adds another dimension to the SOW: the activities to be carried out by users take place in geographic locations (e.g., an open shopping center, an area of tourist interest, an area where some activity is carried out or, in general, any area of a city) and they are also geolocated activities that users carry out through mobile devices (e.g., activities in a marketing campaign). Therefore, it is necessary to capture the events related to the execution of this type of activities and, for this, it is required the development of localization techniques and detection of the behavior of users in a given geographic area. This is the second axis of the project BAI4SOW. In addition, given the high number of users who can potentially participate in this type of flow, is necessary to run the algorithms in grid computing, cluster, and cloud infrastructures, minimizing the cost that this computation could have in business clouds like Amazon and Google. This constitutes the third axis of the BAI4SOW project. In this article, we present the complete system implemented in the project BAI4SOW, which enhances the proposal made in [26]. This system consists of a localization system to locate the customers and identify the activities that they are carrying out, and of the application of process mining and natural language generation techniques to this information in order to analyze the behavior of the customers to extract insights and linguistic reports from it. We present the results obtained in BAI4SOW project, whose objective is in the development of both process mining algorithms for the analysis of SOW containing geolocated activities, and automatic linguistic reports that can be helpful for marketing campaign design. The rest of this manuscript is organized as follows. Section 2presents the main structure of the BAI4SOW project and its framework. Then, the three main parts of the presented framework are described: Section 3introduces the localization system and its paralleliza- 3370 Multimedia Tools and Applications (2022) 81:3369–3395 tion; Section 4gives an overview of process mining and its application to this project; and Section 5describes the followed natural language generation approach. Finally, Section 7 presents the conclusions of the publication, and sketches some insights for future work. 2 BAI4SOW project The main objective of the BAI4SOW project is the development of intelligent techniques for automatic extraction and analysis of user behavior in processes modeled through social workflows. In particular, the target processes are those involving users moving around an outdoor area and carrying out the activities of the social workflow through their mobile devices. Figure 1depicts the conceptual framework which is developed to achieve this objective. It includes the following components: –Localization and behavior detection algorithms. To analyze social workflows is necessary to record the events generated by the activities that each user carries out. For each instance of the process —an execution of the process from start to end—, a sequence composed of these events is generated. In those processes involving users moving around a geographic area, this activities’ model when a user arrives at a certain point (to, for instance, make a purchase or take a photo), sits and moves inside an establishment, etc. Therefore, it is mandatory to locate the users in the geographic area where they are moving, as well as to detect users’ behavior of interest once they are in a given location. To record the execution of these activities, a set of algorithms have been developed using the information given by the users’ mobile devices such as the Global Positioning System (GPS) and Wi-Fi signals sensors. –Parallelization techniques. These techniques are used to design a technological infrastructure based on the integration of cloud resources that support the efficient and large-scale processing of the algorithms developed during the project. This infrastructure facilitates making a decision on what computation and storage resources are more appropriate for the execution of the algorithms. –Process Mining Algorithms. This set of algorithms allow to automatically discover the workflow —also referred to as process model— that has been actually executed by the users and, also, to extract relevant information about the process such as: i) frequent activity patterns, which indicate patterns of frequent user behavior; and ii) the hierarchical organization of unstructured and spaghetti workflows, which ease the visualization of the process reducing its complexity at different levels of abstraction. –Natural Language Generation Techniques. This set of techniques take as input the workflows and frequent activity patterns, as long as other analytics from the recorded Localization Process Mining Natural Language Generation Event log Process model Frequent patterns Linguistic report WiFi signals Parallelization techniques Fig. 1 Conceptual framework of BAI4SOW project 3371Multimedia Tools and Applications (2022) 81:3369–3395 behavior, and automatically generate textual descriptions providing textual information in natural language about their most relevant characteristics. This is an additional tool for conveying information to users, complementary to the visualization of workflows. The computational solution proposed in this project, which is depicted in Fig. 1,ismainly deployed in a cloud-based architecture. The mobile devices record the WiFi signals from the mall and send the data to the cloud-based localization system. This system processes the data and produces the event log with the activities that each customer carried out. The process mining and natural language generation algorithms, which are also deployed in a cloud system, access the event log to work with it. Process mining techniques produce the process model and the frequent activity patterns, and then the natural language generation techniques create the linguistic report. All these outputs are generated in the cloud to be accessible by an analyst from a client application, in order to analyze the behavior of the customers. It is worth noting that this conceptual framework is applicable to SOW containing activities related to the location and behavior of the users taking part in them. However, process mining algorithms and parallelization techniques are directly applicable to any workflow in general, and particularly to unstructured processes with many users taking part in them. In the following sections we will describe the three main blocks of this analysis: i) the implemented localization system, and its parallelization, to obtain the activities that the users are carrying out; ii) the process mining techniques used to discover the behavior of the customers; and iii) the natural language generation approaches applied to automatically generate linguistic reports. 3 Localization system In this project, we consider the customers’ behavior in the mall as a process, and the actions they carry out as the activities of the process. In this context, an execution of the process is the sequence of activities that a customer carries out in the mall, from the entrance to the end. For this reason, in order to analyze the behavior of the customers, it is mandatory to register the activities that they carry out while in the mall. This information is stored in what is called an event log, which constitutes the foundation of process mining analyses (as we will introduce in Section 4). For this reason, the process to locate the customers and identify what are they doing is crucial, so it must be ubiquitous, accurate, and in real-time. This section describes the localization system developed in this project and the parallelization techniques applied to increase its efficiency. 3.1 Indoor WiFi-based system Outdoor localization is usually obtained through the use of GPS but, unfortunately, it is not suitable for indoor environments – such as the environment of this analysis – due to the Non-Line-Of-Sight (NLOS) effect. To solve this issue, the proposed method uses a fingerprint WiFi-based localization as a highly-available, no-installation, no-cost solution: there are WiFi Access Points (APs) in almost every shop, almost every smartphone has a WiFi interface and there is no need to be connected to WiFi networks to measure WiFi signal, so it is free of charge even for private networks. This kind of methods relies on a two-stage approach: (1) the training stage, when the environment is divided into cells where real WiFi measurements are collected and stored 3372 Multimedia Tools and Applications (2022) 81:3369–3395 to build the localization system; and (2) the localization stage, when new measurements collected at an unknown location are compared with the stored ones to estimate the location of the device. Our localization system is based on the work proposed in [17]. It implements a method to increase the localization resolution without the need of increasing the number of cells to site-survey at the cost of a higher computational cost. Of course, real-time is a requirement for our system, so we have implemented a new version using parallel techniques (as will be explained in Section 3.2). Figure 2shows the general architecture of the system. As it can be seen, during the training stage a Support Vector Regresion (SVR) algorithm is used to extrapolate virtual WiFi measurements at positions where no real samples were collected using the real collected information from each existing WiFi AP. The trained localization system will be composed of all these surfaces. Then, during the localization stage, customers will collect new measurements in realtime at their unknown locations. These new measurements will be compared with the values stored in the corresponding surfaces to find their location inside the environment. This way, the coordinates where the difference between the Received Signal Strength (RSS) from an AP and the stored RSS for that coordinate is low will be marked with higher scores than the coordinates with greater differences. Finally, the scores for each coordinate will be summed up, to obtain a resulting surface where the coordinate with the highest score is the most likely location of the customer. In [17], this localization method was evaluated for determining the optimal size of cells when it is applied in a medium-size indoor scenario (around 2.500 m2). The results proved that 15 cm side cells are an efficient option using all the detected APs. With this configuration, and the size of the open mall (see Section 6), the execution time is 44.4 hours to compute all the surfaces during the training stage and 325 seconds to determine the current location of a customer. The experiments were conducted using a server with 8 Intel i7-4790K CPUs at 4.00GHz, a local SSD disk, and 32GB of RAM. Obviously, these performance results show the necessity of accelerating the training and location algorithms in order to be used in large real scenarios. Localization stage COORDINATE Discrete train data AP1 APi APn Training stage Search RSSs in the continuous surfaces ... Create RSS maps using discrete information for each AP Apply an environment mask Sum subsurfaces ... Measurement Estimate continuous Surfaces using -SVR Fig. 2 General architecture of the WiFi-based indoor localization system [17] 3373Multimedia Tools and Applications (2022) 81:3369–3395 3.2 A parallel version of the location algorithms In order to improve the performance of the previous process, new training and location algorithms have been programmed using parallelization techniques. The goal is to reduce the execution time of these algorithms in order to locate efficiently customers in a real scenario in a suitable way. The new version is based on a master-worker architecture [3], which has been developed to be deployed on a high-performance computing environment, specifically, in a set of cloud computing resources with the flexibility to support the dimensions and large-scale requirements of the scenario. The master-worker architecture follows a high-level design pattern that facilitates the parallel execution of applications composed of a set of independent tasks. The pattern consists of two kinds of processes: a master, and a pool of workers. The former is responsible for assigning tasks to workers and guaranteeing that all of them are correctly completed. On the other hand, the workers execute the assigned tasks provided by the master. The correct execution of these tasks is monitored and coordinated by the master, which also checks the termination status and the logs resulting from each process. If required, the master can execute a task again or even reassign it to a different node for its completion. The scenario of the problem depicted in this work requires a high level of flexibility when dealing with computing resources. This architectural model is highly scalable by increasing (or decreasing) the size of the pool of workers according to the execution requirements. On the other hand, the master and the workers usually interact among themselves using an asynchronous communication mechanism. Message brokers are usually used for this purpose, demonstrating their adaptability and effectiveness in this model of architectural solutions. The parallelization of the algorithms is separated into two different stages. First, a generic master-worker architecture has been developed in Java using RabbitMQ as the message broker [29]. Then, the training and location algorithms have been split into a set of independent tasks that can be executed in parallel using the master-worker implementation. The process of creating a continuous reference surface as part of the training algorithm has been programmed as independent tasks that can be computed from RSS measures. This way, the generation of the different surfaces can be easily parallelized. The location algorithm requires checking the sample of the customer’s unknown position against each of the reference surfaces. Each of these checks has been also developed as an independent task. Additionally, as part of the parallelization process, the MATLAB code used to create and process the reference surfaces was refactored and programmed using Octave [25]. This decision is based on economic criteria: the use of MATLAB in different cloud computing resources requires locating a license configured for cloud for each node or deploying a MATLAB Parallel Server, which represents an additional cost that would increase as the execution requirements did as well [23]. The parallel implementation of the localization system and its deployment in the Amazon cloud computing environment is depicted in Fig. 3. As shown, the system exposes its functionality through two Web-based interfaces as REST APIs. This allows easy and independent integration with the BAI4SOW applications. On the one hand, the Surface Creation Service provides a set of operations to submit training measures and compute the corresponding continuous reference surfaces. The master-worker version of the corresponding algorithm has been integrated into the logic of the service and it has been deployed into the Amazon EC2 infrastructure [2]. As will be detailed later, the flexibility of the master-worker architecture allows us to deploy the service over different configurations of virtual instances (used for the execution of workers, 3374 Multimedia Tools and Applications (2022) 81:3369–3395 Fig. 3 Cloud-based implementation of the localization system mainly). The results are then stored into the Surface database, which has been implemented using the Amazon S3 service [1]. This allows us to guarantee a robust, reliable, and durable data storage facility that can be easily scaled as well when required. On the other hand, the Location Service offers the functionality to estimate users’ location. The RSS samples measured by the user’s mobile are aggregated and sent to the service as input. Then, this information is processed by the master-worker version of the location algorithm, which checks them against the reference surfaces previously computed. This can be done because the Amazon S3 data storage is used by both services through the S3 service interface. Output The result of this system (as depicted in Fig. 1) is the identification of the activities the users carry out while in the mall, producing an event log such as the example shown in Fig. 4a. This information, consisting in the name of the activity being executed, the identification of the case —a visit of a customer, from the entrance to the end—, and the timestamp in which each activity took place, is later used by process mining techniques in order to perform the behavior analysis. 4 Process mining With the explosion of process-related data, the behavioral analysis and study of real processes have become more popular. Process mining is a research field offering techniques to discover, monitor, and enhance real processes by extracting knowledge from event logs recording information of the executed activities [43]. Through this analysis, we can understand what is really happening in a process [42], and improve the process based on this information. 3375Multimedia Tools and Applications (2022) 81:3369–3395 Fig. 4 Example of an event log, a trace, a process model and an activity pattern In this way, we can exploit the process mining potential to study and validate the customers’ behavior by taking the information obtained with the localization system explained in Section 3. This section introduces some basic notions about the process mining field, and describes the details of the process mining analysis performed in this paper. 4.1 Process discovery Process discovery plays a dominant role in all the stages of the process mining lifecycle [43]. The starting point is usually an event log recording the execution of the activities in the process, as formalized below. Definition 1 (Event, Trace, and Event log) Given a set α∈Arepresenting the activities of a process, an event εrepresents a particular execution of the activity, storing information such as the start and end timestamps, the resource, cost, etc. A trace is a list (sequence) 3376 Multimedia Tools and Applications (2022) 81:3369–3395 τ=ε1,...,ε nof events εioccurring at a time index irelative to the other events in τ. Each trace corresponds to an execution of the process, i.e., a process instance. We define an event log L=[τ1,...,τ m]as a multiset of traces τi. Figure 4a shows an example of an event log depicting 20 events of 7 different traces (trace 01 is depicted in Fig. 4d). With this information, process discovery techniques aim to build a process model —i.e. a social workflow— depicting the relations between the activities based on the behavior observed in the log. Definition 2 (Process model) Given a set α∈Arepresenting the activities of a process, aprocess model is a tuple M=(N, E) where V⊆A{Start,End}denotes the set of nodes in the process model representing the activities in the process, plus two artificial nodes to centralize the start and end of the process; and E⊆N×Ndenotes the set of edges representing a causal relation between two activities. Figure 4b shows an example of a process model depicting the behavior recorded in the event log in Fig. 4a. Discovering a good-quality and readable process model is crucial to maximizing the analysis and improvement of the process in the next stages. Nevertheless, in some scenarios where the executed activities of the process do not follow a clear structure, the discovered process model might be too complex to be analyzed directly. In these cases, post-processing techniques are used to extract as much information as possible from the process. A common practice is to focus on the parts of the model that are being executed more frequently, i.e., the frequent activity patterns. Definition 3 (Activity pattern) Given an event log Land a process model M=(N, E),an activity pattern P=(N,E)where N⊆N,andE⊆E, is a substructure of the process model depicting part of its behavior —a subprocess. Figure 4c depicts an example of an activity pattern of the event log and process model in Fig. 4. Given a frequency threshold, an activity pattern is said to be frequent when observed in a percentage of traces of the event log equal or higher than this threshold. For instance, for an event log of 100 traces, and a frequency threshold of 65%, the frequent activity patterns are those activity patterns observed in at least 65 traces. In this sense, the optimizations and improvements can be done w.r.t. the parts that are happening more often. 4.2 Conformance checking Another main stage of the process mining lifecycle is conformance checking. Conformance checking takes action once both the event log – recording the observed behavior – and the process model – either discovered or designed by hand – are available. The aim of conformance checking is to relate the events in the event log with the activities in the process model, and compare them to detect commonalities and discrepancies between them [43]. This conformance analysis is useful to detect if reality, as recorded in the log, conforms to the model and vice versa [44]. For instance, by applying conformance checking techniques, a path designed in the process model not being observed in the event log can be detected. This detection allows the managers to inspect why this expected behavior is not being recorded, and act accordingly. Conversely, a conformance analysis can also detect a path observed in the event log but not modeled by the process model. This phenomenon can 3377Multimedia Tools and Applications (2022) 81:3369–3395 Table 1 Paths in shopping mall App waypoint Path Device Distance Step Activity Zone Place 00 S6 /S6E 300 m 1 Eating Restaurants CoffeeShop 2 Walking Restaurants No Place 3 Walking Shops No Place 4 Shop window Shops ToyShop 01 S6 410 m 1 Eating Restaurants CoffeeShop 2 Walking Restaurants No Place 3 Walking Shops No Place 4 Walking Hypermarket No Place 5 Buying Hypermarket No Place 02 Tab2 250 m 1 Eating Restaurants CoffeeShop 2 Walking Restaurants No Place 3 Walking Outdoor No Place 4 Shop window Outdoor Restaurant 5 Buying Outdoor Restaurant 03 Tab 2 / S6 300 m 1 Shop window Restaurants CoffeeShop 2 Walking Restaurants No Place 3 Walking Shops No Place 4 Shop window Shops ToyShop 04 Tab 2 / S6E 410 m 1 Shop window Restaurants CoffeeShop 2 Walking Restaurants No Place 3 Walking Shops No Place 4 Walking Hypermarket No Place 5 Buying Hypermarket No Place 05 S6 / S6 E 250 m 1 Shop window Restaurants CoffeeShop 2 Walking Restaurants No Place 3 Walking Outdoor No Place 4 Shop window Outdoor Restaurant 5 Buying Outdoor Restaurant S6: Samsung Galaxy S6 S6 E: Samsung Galaxy S6 Edge Tab 2: Samsung Galaxy Tab 2 The summary of high-level activities that we have added to the basic activities recognized by Google’s API are: – Eating: we suppose that person is eating when he/she is localized at a restaurants’ zone during a higher time than a threshold. – Walking: accelerometers and localization are key features to determine that person is walking. 3384 Multimedia Tools and Applications (2022) 81:3369–3395 – Shop window: this activity means that a person is standing at the front of a shop but lower time than a threshold. – Buying: localization is a key feature to determine that person is buying inside a shop or hypermarket. 6.2 Analysis of the experimentation conducted with the cloud-based implementation Due to the fact that the services shown in Fig. 3are deployed in a cloud infrastructure, it is necessary to select the computing resources to be provisioned according to the BAI4SOW applications’ requirements. Let us now briefly analyze the execution times and costs of this approach. In order to conduct this evaluation, we have used the data reported in [17]ofauser who has freely walked in the environment described previously (a medium-size indoor scenario of around 2.500 m2). During the walk, the user’s mobile has measured 300 RSS values in different positions. Also, a location request has been sent for each of these values to determine her/his location. Previously, 100 reference surfaces with an accuracy of 1 cm were computed to be used by the location method in order to increase significantly the number of cells to be processed for the same surface (25 million cells in the scenario of 2,500m2). Besides, all the detected APs (a total of 105) and, as consequence, the 105 surfaces have been used to perform the localization. Two different trajectories were used to test the localization system, obtaining mean localization errors around 1,5 meters. As it was previously detailed, it took 44.4 hours to execute the sequential version of the algorithms to compute all the surfaces during the training stage, and 325 seconds to determine the current location of a customer using a server with 8 Intel i7-4790K CPUs at 4.00GHz, a local SSD disk and 32GB of RAM. Note that the experiment had no additional costs apart from the requirement of owning the required hardware and a correct configuration/administration of the operating system and the software. Now, different deployments of the architecture depicted in Section 3.2 are going to be configured and analyzed using the Amazon Elastic Compute Cloud infrastructure (Amazon EC2), which offers a comprehensive and complete elastic Web-Scale computing catalog of computing resources for different purposes. The experimentation is composed of two phases. First, the time and costs to create the surfaces are measured in Phase1. After that, the time and cost to serve the location requests and complete the experiment for each of the proposed deployments is measured in Phase2. The time per request can be calculated by dividing Time Phase2by 300. Table 2shows the execution times and costs of the different configuration deployments using the Amazon EC2 infrastructure. R5 instances have been chosen as the best option because they are optimized for memory-intensive applications and they can offer good performance for this type of processing problems. Also, R5 instances offer a better relation between memory per CPU and cost with respect to R4 instances. The on-demand hiring mechanism has been used for the experiments, and the number of cores, memory, and costs of the instances used are specified in Table 2as well for the North Virginia (USA) region. First, four R5 instances were used to conduct the concurrent execution of the algorithms. We started with a r5.xlarge instance (4 vCPUs and 32 GiB of memory) and twice the configuration up to a r5.16xlarge instance (64 vCPUs and 512 GiB of memory). As it can be seen in Table 2, the execution time in these instances presents a linear speedup according to the configuration of the machines (vCPUs and memory). The best execution time is then 3385Multimedia Tools and Applications (2022) 81:3369–3395 Table 2 Evaluation of execution time and costs for a cloud-based deployment of the algorithms EC2 instance vCPU Memory (GiB) Cost/hour (USD) Time Phase1(min) Time Phase2(min) Total time (h) Total cost (USD) Sequential 8 32 - 1085.2 1579.1 44.4 - Concurrent deployment r5.xlarge 4 32 0.252 161.3 783.6 15.7 4.03 r5.2xlarge 8 64 0.504 126.4 398.9 8.8 4.54 r5.4xlarge 16 128 1.008 102.3 202.2 5.1 6.05 r5.8xlarge 32 256 2.016 83.4 103.1 3.1 8.06 r5.16xlarge 64 512 4.032 66.72 53.4 2.0 12.01 Distributed deployment 3x r5.16xlarge 192 1536 12.096 24.9 22.1 0.8 12.01 6x r5.16xlarge 384 3072 24.192 13.2 11.5 0.4 24.19 3386 Multimedia Tools and Applications (2022) 81:3369–3395 obtained by the r5.16xlarge instance, which took 120.11 minutes to complete the experimentation. This means that the execution time is improved more than an order of magnitude with respect to the time of the sequential version (44.4 hours). The cost of experiments (column Total costs) considers the costs of computing resources and of storing the data into the Amazon S3 service. This is calculated by multiplying the cost per hour of the selected instances (column Cost/hour) by the total hours of execution time (column Total time). The cost of storing the data is the same for all the configurations, as the size of the used data is around 66 GB and the storage price is 0.023 USD/GB for the first 50TB/month. Therefore, the cost of the execution is proportional to the configuration of the instances in terms of the number of cores and memory. Finally, we also conducted the experimentation using two different distributed environments. These environments are composed of a pool of three and six r5.16xlarge instances, respectively. The worker nodes have been distributed between the vCPUs of the different instances, getting a maximum of 191/382 workers running simultaneously on the provisioned instances. The master is deployed on the remaining core. Table 2also depicts the results obtained for each configuration in the distributed environments. As shown, the execution time of the experiment continues following a linear speedup according to the configuration of the environment. The overhead due to the parallelization is minimum and the experiment can be completed in less than 25 minutes with a pool of six instances. However, as the cost of each instance is per hour, the cost of using six instances is twice the cost of using three (24.19 USD vs 12.01 USD, respectively), as with both configurations the experiment can be carried out in less than an hour. Therefore, the relationship between the need to complete the execution in less than 25 minutes versus the additional cost it requires must be carefully evaluated. Independent of the results shown in the table, the main conclusion of this experiment is that the master-worker architecture can be easily deployed in a cloud environment by applying different configurations (we are selected some of these configurations on the basis of our experience, but other different could be also considered) and that it is a good option to accelerate the location algorithms proposed in the paper. The architecture allows the process of location to be decomposed as much as possible and takes the advantage of the scalable nature of the cloud in order to optimise that process. 6.3 User behavioral analysis This section describes how we have applied process mining techniques to the activity and localization data obtained and described in the previous sections. The purpose of this analysis is twofold. First, to validate the recorded behavior using conformance checking techniques by comparing it against the ground truth data. Second, to inspect the customers’ behavior by applying process mining algorithms to the recorded event log, and to extract insights useful to the mall managers. The first step has been to build the process model (depicted in Fig. 7) describing the six paths of the ground truth in Table 1. With this model and the recorded event log, we have executed conformance checking techniques to analyze the deviations between the captured traces and the expected behavior. We have found that 79.66% of the recorded traces have shown no deviations w.r.t. the ground truth model, proving that they have been recorded correctly by the localization system and smartphone application. On the other hand, the remaining 20.34% presented some deviations in the analysis. Two different cases have been detected regarding these traces. On the one hand, the deviations present in 11.86% of the traces have been detected as missing 3387Multimedia Tools and Applications (2022) 81:3369–3395 Fig. 7 Process model depicting the behavior of the ground truth events from the ground truth paths, i.e., the recorded data corresponds to partial paths of the ground truth. In these cases, the most probable reason is a stop in the WiFi signal, either in the mall receptors or in the users’ devices, causing them to not record all events in the path. The remaining 8.48% of the traces have shown a deviation regarding the localization system. In these cases, a full path in the ground truth was recorded, but some of the events had been mislocalized. For instance, one of the traces corresponded with path 05, but recording the event ‘Buying Outdoor Burger’ as ‘Buying Restaurants Burger’, probably due to the proximity of the locations. After analyzing the recorded behavior w.r.t. the ground truth, we have discovered the process model depicting the users’ recorded behavior to extract insights about their behavior, and to show the potential of applying process mining techniques to this scenario. Figure 8 shows the process model obtained with ProDiGen [38]. We observe that the behavior of users doesn’t have a clear structure. Few conclusions can be extracted from an unstructured process as the one shown in Fig. 8. Nevertheless, this result is not surprising, as the users’ behavior in a broad area such as a mall is expected to be unstructured. To cope with this shortcoming, a more specific process mining analysis can be run, focusing on the parts of the process that the users are executing more frequently. We have used WoMine [5] to extract the frequent activity patterns – subprocesses – modeling the behavior commonly executed by the users. Figure 9shows two of the more relevant frequent activity patterns extracted by WoMine. Figure 9a highlights a behavioral pattern with a frequency of 42%. This behavior has been conducted by the 42% of the recorded customers, who have started in the shopwindow of the coffee shop, walking through the restaurants’ area, and later through the shops’ area without buying anything. This pattern can hint at a need in the 3388 Multimedia Tools and Applications (2022) 81:3369–3395 Fig. 8 Process model discovered by ProDiGen [38] depicting the behavior of the customers in the mall Coffee Shop to make changes in order to engage more customers who are doing window shopping (i.e, they stop briefly to look at the shop window but leave to other areas without coming in the shop). Figure 9b depicts another behavioral pattern being executed by 37% of the customers. This pattern shows that 1 out of 3 customers pass walking through the restaurants and shops areas, leaving the mall after staring at the shop window of the toy shop. This pattern hints that it might also be interesting to work on the ads at the more frequented areas, Restaurants, and Shops, to deal with those customers who leave the mall without consuming anything. In summary, the process mining analysis allows to discover the process model depicting the behavior of the customers, and to inspect in more detail their behavior through the frequent activity patterns. The behavior modeled by the two reported patterns – which were conducted by almost half of the customers – was useful to detect possible redesigns Fig. 9 Frequent patterns (highlighted structures) extracted by WoMine using the event log recording the activities of the customers in the mall 3389Multimedia Tools and Applications (2022) 81:3369–3395 in the marketing campaign of some stores to increase the attention of the customers. Furthermore, to improve the knowledge and insights obtained from the analysis, automatic linguistic reports can be extracted from this discovered behavior, as we will describe in the next section. 6.4 Automatic linguistic reporting evaluation and results We applied NLG techniques to automatically generate human-friendly linguistic reports associated with the previous process mining analysis. It is worth noting that both process mining (see Section 4) and report generation (see Section 5) were connected with the help of tools provided by the InVerbis Analytics service [18]. More precisely, we considered a sequential NLG pipeline for linguistic reporting, as depicted in Fig. 5in Section 5.WeshowinFig.10 the report generated for the test bed under consideration. As indicated in Section 5, this report pays special attention to the most frequent pattern (see Fig. 9a) extracted from the graph depicted in Fig. 8. The duration analysis in the report also refers to activities (and transition between activities) that deserve special consideration. For the first time in the application of NLG to process mining and analytics (at least to the best of our knowledge), we have conducted a human expert validation of the natural language reports in a real scenario. We carried out an online survey that follows the best practices in NLG evaluation, with the aim of evaluating the quality of the generated linguistic report (see a sketch of the questionnaire in Fig. 11). The questionnaire is inspired by the so-called Gricean conversational maxims which were first presented in [16]: (G1) Maxim of Quantity [“Make your contribution as informative as required for the current purposes of the exchange. Do not make your contribution more informative than is required”]; (G2) Maxim of Quality [“Do not say what you believe to be false. Do not say that for which you lack adequate evidence”]; (G3) Maxim of Relation “Be relevant”]; (G4) Maxim of Manner [“Avoid the obscurity of expressions, avoid ambiguity, be brief (avoid unnecessary prolixity), be orderly”]. These conversational maxims were studied by Dale and Reiter in the context of NLG systems [7]. Moreover, they were also considered in previous publications (e.g., [6]) for evaluating the quality of NLG systems. It is worth noting that, in the context of NLG, human evaluation is considered as the most suitable evaluation method [45], since it is able to capture adequacy, correctness, and fluidity of the texts, among other relevant features. Following the usual methodology in this field, we set up a questionnaire which is made up of six questions with the aim of covering properly all Gricean maxims. With the aim Fig. 10 Linguistic report in natural language generated using [18] for the real case considered 3390 Multimedia Tools and Applications (2022) 81:3369–3395 Fig. 11 Sketch of the questionnaire for assessing quality of linguistic reporting of minimizing the bias effect due to question order, the six questions were presented with shuffled order to each assessor. In addition, the answer to each question is given on a 7-point Likert scale [22]. This is a discrete scale where the assessor is expected to select the level Table 3 Evaluation of linguistic reporting quality (7-point Likert scale) Gricean Maxim Question Median Mean Std. Quantity Regarding the length of the report: is it adequate with respect to content? 6 5.51 1.15 Quality Regarding the truthfulness of the report: are results supported by adequate evidence? 6 5.47 1.45 Relation Regarding the relevance of the report: is the content relevant to the problem (estimating customer activity patterns in the mall)? 6 5.66 1.42 Manner Regarding the structure of the report: is content properly ordered to facilitate comprehension? 6 5.49 1.13 Regarding the clarity of the report: is the language correctly used? 6 6.10 0.91 Regarding the general linguistic quality of the report: how good are grammar, syntax, orthography, and punctuation? 6 6.17 0.76 3391Multimedia Tools and Applications (2022) 81:3369–3395 Fig. 12 Distribution of answers to the question about the general linguistic quality of the report of agreement with the given statement or question (e.g., from strongly disagree to strongly agree). The target audience was made up of people who may be somehow familiar with the problem under consideration, i.e., intelligent data analysis and linguistic description of estimated customer activity patterns in open malls. Accordingly, we sent the questionnaire to attendants of the 21st International Workshop of Physical Agents (WAF) where our preliminary work [26] was presented. We collected 41 answers to each question. Table 3summarizes the computed scores. The first column identifies the correspondence between the questions given in the second column and the Gricean maxims. There is one question related to each of the three first maxims and three additional questions to cover the fourth maxim. We included three questions related to the last Gricean maxim for the sake of completeness, and looking for evaluating exhaustively the use of the “everyday natural language” in the NLG templates used for linguistic realization. Thus, we asked about the order of the given pieces of information, about the correct use of vocabulary, and about the general linguistic quality of the report. Results are encouraging, being 6 the median of answers for all questions. In addition, mean values are always above 5 and standard deviation (Std.) is below 1.5 in all cases. Even more relevant is the fact that the assessors globally indicate a very high degree of satisfaction with the generated report, since the general linguistic quality of the report (see Fig. 12) achieves a value of 6.17 in mean (with 0.76 std.), as remarked in bold in the last row in Table 3.Thisisthe best indicator of the high quality of the texts generated automatically from the process data. 7 Conclusions and future work We have presented a method to estimate the activity patterns associated with customers in open malls. Such patterns are based on localization data which are provided by a specific smartphone application for logging information from sensors and process mining techniques to identify what kind of usual activities are made by customers. We use the localization key information joined with data mining, parallelization, and monitoring techniques. Our system has been tested in a challenging real scenario such as an open mall (mixed classical mall and isolated shops in the street) where location is obtained by mean of GPS and a WiFi localization system that is helped by a parallel computing method to speed up the whole process. From these localization data, we apply state of the art process mining techniques to first generate a process model (social workflow) describing the behavior made by customers in the mall. We have also analyzed the most common activity patterns performed 3392 Multimedia Tools and Applications (2022) 81:3369–3395 by customers. The hints provided by these process mining techniques may help managers to take specific actions considering the customers’ behavior and the activities they perform while in the mall. We have also integrated an automatic linguistic reporting module, which automatically describes in natural language the most relevant characteristics of the process, providing complementary information to the one conveyed by the visualization of the process model. Following the best practices for human evaluation of natural language generation systems, 41 experts validated this approach, with a global score of 6.17 on average (0.76 of standard deviation) in a 7-point Likert scale. As far as we know, this is the first time that this type of evaluation is conducted in the field of NLG applied to process mining and analytics reporting in a real case. These results empirically validated the adequacy of automatic natural language textual reporting as a useful tool to facilitate understanding of the usual graphical outputs given by process mining visualization tools. We strongly believe that automatic linguistic reporting and process mining visualization tools can act as complementary tools to provide managers with valuable and meaningful insights no matter their technical or mathematical background. This work has set a base in the analysis of customer behavior in open malls. Nevertheless, the analyses performed in this publication only take into account the control-flow information of the process, i.e. the sequence of activities that each user carries out. More interesting insights can be extracted if the analyses are performed taking into account other data related to the users such as age, gender, language, shopping interests, etc. Regarding process mining, many techniques have been proposed to perform analysis taking into account this type of personal data, known as multi-perspective process mining [20,24]. However, personal data must be carefully considered to avoiding bias as well as respecting privacy and ethical issues [10,11]. As future work, we plan to extend the mobile app to record properly this kind of information, and perform more precise analyses taking into account more attributes about the customers, while complying with legal and ethical issues. Acknowledgments This work has been partially supported by the Spanish Ministry of Science Innovation and Universities and the European Regional Development Fund (ERDF/FEDER) Grants RTI2018-099646-B- I00, TIN2017-84796-C2-1-R, TIN2017-90773-REDT, RED2018-102641-T and RYC-2016-19802 (Ram´ on y Cajal program, Jos´ e M. Alonso). Also by the Galician Ministry of Education, University and Professional Training and the ERDF/FEDER program (ED431F2018/02, ED431C2018/29, ED431G2019/04 grants). Funding Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. References 1. Amazon S3 - Simple Cloud Storage Service: https://aws.amazon.com/s3/. Online; accessed in January 2021 2. Amazon Web Services EC2 - Simple Cloud Hosting: https://aws.amazon.com/ec2/. Online; accessed in January 2021 3393Multimedia Tools and Applications (2022) 81:3369–3395