Human Performance Envelope as a set of interdependent factors for real-time monitoring of driver state

Human Performance Envelope as a set of interdependent factors for real-time monitoring of driver state Riccardo Ceriani1*[0009-0002-5191-7110], Andrea Castellano2[0000-0002-3636-2228], Lorenzo Uccello3[0009-0008-1080-3874], Gianpiero Mastinu3[0000-0001-5601-9059], Elisa Landini2, Claudio Lantieri1[0000-0003-2852-567X] 1 Department of Civil, Environmental and Material (DICAM) Engineering, University of Bologna, Bologna, Italy, 40136 2 Department of Physics, Informatics and Mathematics, University of Modena and Reggio Emilia, Modena, Italy, 41121 3 Department of Mechanical Engineering, Politecnico di Milano, Milan, Italy, 20156 CORRESPONDING [email protected] Abstract. Driving is a complex perceptual-motor skill that combines cognitive, psychomotor, and perceptual abilities to operate a vehicle safely and efficiently. It involves the driver’s ability to manage emotions, such as stress and frustration or cognitive states, like drowsiness, distraction and maintain focus. In this context, the Human Performance Envelope (HPE) is described as a construct use to monitor driver status based on the combination of a set of interdependent factors and represented with a spiderweb model. Its goal is to minimize human error by analysing behaviour and decision-making while optimizing systems and environments to enhance performance. The influence of the activity demand on the model needs to be considered. Assuming that an HPE can be characterized for a specific actor, there could be a number of acceptable HPEs for the same actor, each depending on the activities performed. One possible approach to addressing activity-based HPEs is to isolate specific tasks or situations, assess its factors in those contexts, and determine the feasibility of defining a specific HPE for each activity. Depending on the value of each factor, the resulting model could evolve from fully acceptable to not acceptable. Based on an in-depth bibliographic research this work aims to populate the three main “HPE parameters” (workload, stress, and situation awareness). with vehicular, behavioural and physiological parameters. Keywords: Human Performance Envelope, risk mitigation, driver monitoring 2 1 Introduction Numerous road traffic safety (RTS) studies carried out in different countries over the years have consistently shown that, of the three main components of the RTS system (road user, vehicle and environment) the human factor is the main cause of accidents [1]. The schematization of the three factors is shown in Figure 1. Fig. 1. Schematization of the three factors is RTS, taken from [2] Studies aimed at improving road safety are therefore mainly concerned in analyzing human driver: its personality, its behaviour on the road, its attitudes, its emotional tension, its knowledge, its experience and many other factors. These studies are of a cognitive nature and in simple terms it can be said that they are dedicated to identifying the reasons and circumstances of certain behaviours on the road [3]. The advent of heightened road safety requirements and the decline in the financial burden associated with data acquisition equipment has rendered the identification of driving behaviour a subject of considerable scholarly interest in recent years [4]. This technology has a broad spectrum of applications, including driver state monitoring and driving style analysis [5,6]. It is a commonly held assumption that single-factor evaluation is an inadequate means of assessing driver behaviour [7,8,9]; according to this assumption, the aim of this work is to propose a set of parameters to be applied to a set of interdependent factors to monitor the state of the driver, based on existing literature. In this particular case, the factors to be populated relate to a model called Human Performance Envelope (HPE). Originally developed in avionics, as part of the Future Sky Safety project, it has been described as a construct combining a set of nine interdependent factors, called “HPE factors” (workload (WL), stress, fatigue, situation awareness (SA), attention, vigilance, teamwork, communication and trust) and graphically represented with a spiderweb model. As shown in Figure 2, the outcome may range from fully acceptable (ensuring a nominal set of cognitive resources) to not acceptable (or degraded), where the environment becomes prone to errors both cognitively and physically. Although a single factor, such as low vigilance, may reach an unacceptable level, it is presumed that interactions among factors should allow for compensatory mechanisms to be identified, thereby demonstrating the overall acceptability of the model [10]. It was determined that the HPE should prioritise three critical aspects of human performance (workload, stress, and situation awareness) as the 3 most relevant measures to consider. While additional factors, such as fatigue and attention, were recognised as being significant, challenges in obtaining reliable indicators posed a potential limitation to the validity of the resulting model. Based on the existing literature, this work aims to populate the three main HPE factors with vehicular, behavioural and physiological parameters. This work also aims to lay the foundations for the application of the HPE concept to a controlled scenario in a simulated environment, needed to monitoring the evolution of the user's driving performance on the basis of a predefined sequence of events. Fig. 2. Representation of HPE spiderweb model (left) and example of event-related evolution of HPE parameters (right). Adapted from [9] Other researches on the subject manages to monitor individual “parameters” in the model using a fusion of data sources with excellent results. For example Liu (2020) [11] manage to monitor and prevent fatigue by monitoring heartbeat and driving posture; furthermore Schneegass et al., [12] monitor driver workload by measuring skin conductance, heartbeat, temperature sensors and driver camera; Lee at al., [13] estimated driver's stress employing the use of physiological signals and steering wheel motion analysis. Despite the evident efficacy of monitoring driving performance using a solitary “parameter”, it is the considered opinion of the authors that a combination of them and a fusion of instruments is required in order to achieve a more comprehensive and complete monitoring of the evolution of driving behaviour metrics [14]. In this respect, the objective of this work is to emphasise the possibility of tackling the problem of monitoring driver status from the opposite perspective. That is to say, it is proposed that the total potential of sensors be exploited by using the same instrument to monitor several parameters. In order to explore the concept of HPE in a simulated environment, it is necessary to employ questionnaires (pre and post simulation) in order to monitor the subjective perception of the user. The tools that are the subject of this article are illustrated in Figure 3. These include the following: questionnaires, eye tracker, electroencephalogram (ECG), electrocardiogram (EEG), instrumented steering wheel (ISW), and electrodermal activity (EDA). 4 Fig. 3. Proposed experimental setup The application of this approach type, has the potential to be particularly interesting especially in the context of urban driving scenarios. This is because, according to statistics, the occurrence of accident phenomena is more frequent in an urban environment and in general drivers are more stressed by external stimuli [15,16]. The urban environment is characterized by a heterogeneous traffic composition and roads are the public space in which this traffic flows. According to the Italian regulation, they are defined as “the area for public use intended for the circulation of pedestrians, vehicles and animals”. This definition implicitly implies the need to design the road according to the dictates and needs of different users. To ensure the safety of all of them, the NACTO guidelines [17] define “In an urban context, street design must meet the needs of people walking, driving, cycling, and taking transit, all in a constrained space. The best street design also adds to the value of businesses, offices, and schools located along the roadway”. Just as the design of the street takes on fundamental value in ensuring safety, the user's response to driving task and surrounding environment takes on importance at the same level and in the same way. 2 Literature review To support the theoretical foundation of the HPE in the context of real-time driver monitoring, a bibliometric analysis was performed using VOSviewer [18]. The search term "real-time driver monitoring" was applied to the PubMed database and a network visualisation was generated based on the co-occurrence of keywords in the relevant literature. As shown in Figure 4, “human” result as the centre of the research landscape and the major keyword adopted in the specific research field under analysis, acting as nexus between the different thematic cluster. As matter of fact this is in fact coherent with the aim of the HPE concept. Four kinds of clusters can be noticed: red, yellow, green and blue. The larger cluster is the red one and includes such as driving, monitoring, physiological, accidents, traffic, and machine learning, reflecting an ac- 5 tive body of research focused on physiological and cognitive monitoring of drivers. It is important to note that the HPE concept finds its natural collocation within this particular cluster. This red domain is in fact strongly aligned with the core principles of HPE, where behavioural and sensor-based inputs are critical for determining driver’s performance status. Adjacent clusters further enrich the HPE construct. A green cluster highlights factors related to demographics and clinical conditions, such as age, biomarkers and treatment outcomes, suggesting the relevance of inter-individual differences. The concept of HPE, being based on the individual's driving performance and condition, is intrinsically linked to personal conditions. It can thus be concluded that the 'optimal' or 'degraded' condition of parameters is influenced by these personal differences. A blue cluster addresses metabolic and psychophysical variables, such as blood glucose and hypoglycaemia, which may influence driver behaviour under certain conditions. In addition, peripheral clusters capture the broader ecosystem, including environmental factors (air pollutants, vehicle emissions) and technological dimensions (algorithms, real-time systems), highlighting the potential need to integrate external influences into the HPE model. This visualisation confirms that the literature addresses many of the interdependent factors envisaged in the HPE framework, albeit often in isolation. The identification of missing links between different clusters by means of the graphical tool VOXViewer indicates the possibility of intervening with research topics that fill precisely these gaps, thereby introducing innovation to the marker theme. By mapping these elements, the analysis supports the article's goal of populating the HPE dimensions with vehicular, behavioural and physiological variables and exploring their interrelationships. It also highlights the fragmented nature of current approaches and reinforces the need for a unifying model such as the HPE to integrate multiple data streams for improved driver state monitoring. This section presents also some of the contributions in the literature that link the instrumentation proposed in Figure 2 to the three HPE priority metrics (workload, stress and situational awareness). Fig. 4. VOSviewer representation of research framework 6 2.1 Eye tracker Bitkina et al., [19] by means of a study conducted in 2022 shows that ocular metrics such as fixation duration, gaze point and pupil diameter are strongly correlated with perceived workload while driving task, suggesting the usefulness of such metrics in workload’s level classification and prediction. Furthermore, Arias-Portela et al., (2023) [20] provided an articulated review on situation awareness assessment by means of eye tracking metrics. In this review, 38 scientific papers were considered, revealing a wide relationship between the eye-tracking metrics and situation awareness. 2.2 Electrocardiograph (ECG) A recent review carried out by Sriranga et al., (2023) [21] provided broad evidences of how ECG metrics (specifically heart rate variability (HRV)) can be employed to monitor driver mental workload. In the field of situation awareness, this study [22] developed a paradigm for measuring situational awareness in the context of highspeed train drivers and examined the relationships between SA levels and physiological measures, including ECG signals. Rastgoo et al., (2021) [23] developed a system to classify stress levels in real time employing ECG signals. Parameters’ optimization significantly improved system’s accuracy, enabling it to reach reach 77.78% on a dataset collected using advanced driving simulator. 2.3 Electroencephalograph (EEG) In research conducted by Di Fulmeri et al., (2018) [24] EEG-based index of mental workload to assess the impact of various factors on driver behaviour in real driving conditions. Results showed that traffic condition and the type of road had a significant impact on driver workload, with the EEG proving more sensitive metrics than subjective measures like questionnaires. The efficiency in the monitoring of situation awareness during driving tasks have been explored in a study [25] which made use of EEG and functional near-infrared spectroscopy (fNIRS); demonstrating its potential. 2.4 Instrumented steering wheel (ISW) Those two studies [26,27] used an instrumented steering wheel to analyse the forces through load cells moments applied by the driver during driving in a simulator. Results showed that analysis of these parameters can provide detailed information about the driver's workload during complex manoeuvres. In a contribution provided by Sahar et al., 2021 [28] they explore the feasibility of employed grip force on steering wheel as a measure of stress level. By showing strong positive correlation between the parameters, these results provide initial evidence that grip force can be used to measure stress in driving tasks. A research article proposed by [29] analysed the interaction between the driver and the steering wheel in emergency situations using an instrumented steering wheel. The results showed that the measurement of applied 7 forces and moments can provide useful information for understanding the driver's situation awareness during critical events. 2.5 Electrodermal activity (EDA) In a contribution carried out by Dogan et al., (2019) [30] the applicability of EDA in the monitoring of the stress level by comparing it with survey. Results showed a consistence of 87.5% between the EDA sensors results and the answers provided in the questionnaires. Moreover, [31] based on research conducted on a simulated environment, stated that ECG and EDA signals are sensitive to variations in workload. 2.6 Questionnaires For what concern the self-assessment of workload level there are different types of questionnaires which can be employed. Among these we can mention NASA Task Load Index (NASA-TLX), Overall Workload Scale (OW). The first one is multidimensional rating procedure that provides an overall WL score based on a weighted average of ratings on 6 subscales [32]. OW instead is a unidimensional measure rating of the subject's overall workload on a unidimensional scale of 0 to 100 [33]. Another valuable questionnaire which can be employed to measure driver condition is the Attentional Network Test (ANT). It is a computer-based test to measure users’ performance in three components of attention: alerting, orienting, and executive control. From its first development by Jin Fan et.al, [34] it has been widely employed in basic and applied research studies which have been conducted on various topics including driving behaviour, specifically to predict driving test scores [35]. Complementary to the original version, a new version in which a measure of vigilance is added is called Attention Network Test for Interactions and Vigilance (ANTI-V) [36]. It will be especially beneficial in evaluating specific hypotheses concerning vigilance, in conjunction with other attentional functioning metrics, such as phasic alertness, orienting, and executive control. The evolution of driver state condition over time is prone to modification inducted by external factors; those includes traffic conditions, traffic type and the driving context (e.g., motorways or urban) [16]. Depending on the previously mentioned factors, during the same trip undertaken by users, specific events or actions may be observed. These includes for instance follow the leading vehicle (also known as leader vehicle), overtaking, accelerations / harsh acceleration and braking / harsh braking. Each of these actions entails a characteristic cognitive load and has the capacity to induce fluctuations in emotional and attentional states, especially when they occur in rapid succession or in particularity demanding stress-level scenarios. The interplay of these contextual and behavioural factors is of paramount importance in the study of driver state dynamics, particularly in relation to the broader framework of HPE. It is imperative to comprehend the manner in which these components intersect to influence the driver's condition for the advancement of sophisti- 8 cated driver-assistance systems (ADAS), human-machine interfaces (HMI), and autonomous driving technologies that are attuned to real-world behavioural variability. Based on the above-mentioned contributions a schematization of physiological, vehicular and behavioural-related parameters is reported in (Tab. 1). From a methodological perspective, the adoption of simulated driving environments constitutes a powerful instrument for scientific investigation. Indeed, simulators offer to researchers a high-degree of experimental setup control, enabling the systematic manipulation of traffic conditions, environmental complexity, and task difficulty [37]. This, in turn, facilitates the generation of bespoke scenarios that simulate a broad spectrum of user engagement levels and stress-inducing situations. Through the implementation of such controlled experimentation, it becomes possible to isolate specific variables and examine their effects on driver behaviour, performance, and psychophysiological responses with a degree of precision that is often unattainable in naturalistic studies. Consequently, simulated environments facilitate the replication of real-world driving conditions in a safe and repeatable manner, as well as supporting the development of robust predictive models of driver state variation. Table 1. Differentiation between physiological, vehicular and behavioural-related parameters Physiological parameters Vehicular and Behavioural-Related Parameters Eye Tracker Acceleration (longitudinal/lateral) EEG Speed (longitudinal/lateral) ECG ISW EDA Braking activity 3 Materials and method In this section, based on existing literature and on the fundaments highlighted up to this point, authors propose a combination measure based on previously mentioned sensors capable to monitor driver’s evolution of HPE fundamental parameters, namely: situational awareness, workload and stress. In particular, authors propose this set of tools which can be applicable to evaluate the evolution of the parameters in case of controlled simulated test carried out through the usage of a dynamic simulator. The combination of physiological, behavioural, vehicular, and subjective measures is expected to enable a thorough evaluation of: 1. Situation Awareness, that involves perceiving, comprehending, and anticipating changes in the driving environment. It is influenced by vehicle performance parameters such as vehicle longitudinal speed, vehicle lateral speed, and vehicle rotations (pitch, roll, yaw). The position of the center of mass in relation to the global reference system can reflect how well the driver per- 9 ceives the dynamics of the vehicle, aiding in their situational understanding. While specific studies directly linking these vehicle dynamics to situation awareness are limited, ongoing research in driver behavior modeling continues to explore these associations. [38] 2. Workload, that reflects the cognitive demands on the driver and can be assessed using parameters such as heart rate variability (HRV), vehicle acceleration, and brake pedal position. Increases in vehicle longitudinal acceleration and brake pedal force have been linked with higher workload, as these parameters indicate the level of control a driver must apply to drive effectively [39]. 3. Stress, which is closely related to physiological responses like heart rate, heart rate variability, and grip force. HRV is widely studied as an indicator of stress, particularly through the analysis of the LF/HF ratio, where a shift towards lower frequencies is associated with higher stress levels. Stress responses are also reflected in vehicle dynamics; for example, sudden changes in vehicle longitudinal acceleration and brake pedal position can indicate moments of high emotional or cognitive stress. Each dataset might be mapped to key HPE Parameters, allowing for a quantitative and qualitative assessment of driver performance and the impact of human state on performance in driving situations across different levels of automation. By combining these parameters, the proposed dataset (Table 2) intends to identify patterns and thresholds that define different driver states, thereby facilitating the development of multi-parameter monitoring systems. Table 2. Mapping of HPE parameters and related measures HPE Parameter Sensor Measure Description Workload EEG 𝜃𝑓𝑟𝑜𝑛𝑡 / 𝛼𝑝𝑎𝑟 Relationship5between5frontal5theta5waves5and5parietal5alpha5waves5 𝜃𝑓𝑟𝑜𝑛𝑡 / 𝛽𝑓𝑟𝑜𝑛𝑡 Ratio5between5frontal5theta5and5frontal5beta5waves5 ISW Total driver grip force Measurement of the force exerted by the driver on the steering wheel monitored by load cells Eye tracker PERCLOS Percentage of time during which the eyes are partially or completely closed Fixation duration Average eye fixation time Dynamic driving simulator Vehicle longitudinal speed Measures needed to define the kinematic of the vehicle Vehicle lateral speed Vehicle longitudinal acceleration Vehicle lateral acceleration Brake pedal position Stress ECG Heart Rate (HR) The speed at which the heart beats Heart Rate Variability Mean NN refers to the average duration of the normal-