Integrating Logic Reasoning with AI for Lightweight Edge Based Video Monitoring

1 Integrating Logic Reasoning with AI for Lightweight Edge Based Video Monitoring Pavana Pradeep Kumar∗and Krishna Kant∗ I. INTRODUCTION Nonintrusive monitoring via imaging techniques is a widely deployed mechanism and is increasingly automated with advances in hardware capabilities and computer vision techniques. In particular, smart cameras can analyze video frames in real-time to detect certain objects (e.g., humans, cars, etc.) or even simple types of activities/anomalies. However, highly capable cameras are expensive and still limited to observing things within their area of coverage. For larger areas, we need an edge-computing infrastructure with multiple Edge Controllers (ECs), each managing multiple cameras in adjacent areas and interacting among themselves for cross-EC monitoring. The latter is common when large areas need to be covered, and the agents of interest (e.g., people, vehicles, robots, etc.) move across different coverage areas. The scenario can be considered an instance of edge computing, and applies to many IoT applications including traffic monitoring, monitoring of patients in a hospital, residents in a senior home, robot-human interaction in smart manufacturing, etc. The cameras in each local region send the partially processed video streams to Edge Controllers (ECs), which may do further analysis to detect anomalies or pieces of long-term behavior. Further interaction between ECs either directly or via a cloud, could then provide longer-duration situations aware. Such an environment must consider multiple factors for a smooth operation: (a) Real-time video processing capabilities of each camera, (b) Latency and bandwidth aspects of the interaction between a camera and ECs and across ECs, (c) Accuracy and timeliness of situational monitoring by ECs, and (d) EC resource limitations in terms of processing, storage, and latency of backend cloud interactions. II. INTEGRATION COMPUTER VISION AND LOGIC REASONING The current computer vision (CV) techniques are heavily focused on directly learning the desired activity from training examples. Such a method often needs extensive training data that may be difficult to obtain especially for situations that are considered unsafe or undesirable. Neural nets also have the usual problems of lack of explainability, potential bias, and poor performance in case of inadequate data. We believe that integrating these traditional techniques with explicit logical reasoning infrastructure can address many of these issues. The approach uses traditional CV techniques to recognize objects, their attributes, and explicitly tracks their movements. ∗CIS Department, Temple University, Philadelphia, PA, USA. (e-mail: pav[email protected], [email protected]) Such processing can be done within the camera itself, which can exploit it to detect scene changes at high level and thus intelligently determine redundant frames. Beyond this basic processing, both CV and spatio-temporal logic reasoning can be used in the ECs to detect and predict various behaviors and anomalies. We have used a discrete-time version of RTEC (run-time event logic) for this purpose [1]. Significant reductions in BW, latency, and processing needs can be achieved by transmitting only the most relevant frames or even the information extracted from them to the EC, as shown in [2]. Fig. 1: Avg Precision (AP) and Resolution time Margins As an example, Fig. 1 shows a comparison between our integrated AI + logic reasoning method against deep learning method for the task of detecting near collisions (i.e., unsafe distances) between vehicles both in the direction of travel and in the lateral direction. In this analysis, we effectively had full view of all the cars at a single EC, which not only detected the near collisions, but also tried to predict them [3]. The goal of the prediction is to allow time for resolution, i.e., actions to avoid the collision. Thus a larger resolution time is better provided that we maintain a high prediction accuracy. For this analysis, we used four traffic-related datasets, DAD [4], CADP [5], and AICity [6], and our own comprehensive dataset called TU-DAT [7]. The traditional CV methods that we found in the literature are YOLOv3+SVM [8], Driftnet (CNN based) [9], and DSA+LSTM [4]. It is seen that the integrated approach provides both the best accuracy and longest resolution margin without any specific training, and can be easily adapted to other situations by simply changing the logic assertions. Logic reasoning can also work across ECs and will reduce the amount of data transfer needed across ECs. Consider, for example, the problem of monitoring driving behaviors of drivers on the road. This would require watching each vehicle for at least several minutes, perhaps longer, which would span multiple ECs. Suppose that we want to designate drivers as aggressive, distracted, conservative, etc. With logic reasoning this can be done based on their lower-level driving behaviors that we call as microbehaviors or µBs. These µBs can be based on deep learning, but can also be recognized directly through logic reasoning. In the former case, training is essential and would become invalid if the µBs parameters are changed, which is not the case with the latter. Table I illustrates a set of µBs that we used in [10]. Microbehavior Meaning µwv : Weaving Swerving between lanes µss: Sudden Steer Abrupt heading changes on a straight road µhb: Hard Braking Braking deceleration is -8 m/sec µld: Lane Drift Not keeping in the center of the lane µst: Straddling Alternatively hugging left & right side of lane µos: Overspeeding Driving 25% above the speed limit µtg: Tailgating Following a vehicle at distance <2m Table I: Microbehaviors for Driving Characterization Fig. 2: Accuracy of microbehavior estimation with 20% gap between Cameras Since µBs may stretch across EC coverage boundaries, the behaviors from adjacent EC’s must be stitched together. With the DL approach the only practical solution is to train the model on stitched video stream, so that the behavior can be seen directly. This would require BW intensive video transfer across ECs (or collection of all video streams in the cloud). With the logic reasoning approach, we can do partial determination in each EC coverage area, followed by the aggregation, which is far cheaper in terms of communication bandwidth. For example, to monitor µwv (i.e., weaving through lanes), we need to collect together the lane change and changebacks in each EC coverage area and then put them together for each vehicle as it moves past EC coverage areas. If the EC coverage areas overlap, we need to track individual vehicle across ECs to ensure that the stitching is done for the same vehicle. This is rather straightforward. However, if there is a gap between coverage areas, the problem is a bit more difficult. In this case, we need to accurately track individual vehicles as they pass through the coverage gap. This requires recognizing static attributes of a vehicle (e.g., color, length, and height) and using them along with the dynamic parameters such as the position and speed. We also need to predict the trajectory of each vehicle (e.g., via Kalman filter). However, gaps could result in undercounts when a behavior starts in one EC but ends in the next. The impact depends on the length of the gap and how well the gap is handled in accounting for the microbehaviors. For experimentation we emulated a rather extreme scenario with a driving loop whose two ends are covered by two different cameras (and corresponding ECs) and the middle 20% of the loop uncovered. As the cars drive around the loop, they randomly assume the role of safe, aggressive or distracted driver for the whole lap, and engage in corresponding microbehaviors. It is seen that the accuracy is 93% or above in spite of rather extreme situation. It can be improved further by a more careful consideration of µBs that start before the gap and end after the gap. III. LOOKING AHEAD One key challenge in logic reasoning is the need to generate the appropriate assertions. These can be tedious for modeling complex situations. Therefore, a partial or full automation of the assertions is the need of the hour. We are currently examining how the popular large language models and corresponding video language models can help, since they are good at describing overall situation, but bad at the details. However, generating useful formal assertions using them is challenging. Another significant issue is the modeling of uncertainty. Formal logic inherently deals with crisp facts and most tools only support such reasoning. On the other hand, fuzziness is inherent in characterizing any real-world situation. Given an assertion of the type x > d which says that the distance between two objects (x) should be greater than some threshold value d, there are two types of uncertainties: (a) The threshold dmay be fuzzy with some lower bound along with the fuzzy interpretation of truth value of the constraint x > d, and (b) The constraint x > d may be nice to have rather than mandatory, as represented by a weight or confidence level indicating its importance. While the weights are easily handled in the formulation, a direct use of fuzzy logic in the reasoning infrastructure makes it extremely complex. Yet another issue in logic reasoning is ensuring that all necessary assertions have been included so that the deductive system is complete. REFERENCES [1] C. Vlassopoulos and A. Artikis, “Towards a simple event calculus for run-time reasoning (rtec),” in COMMONSENSE, 2017. [2] P. Pradeep, A. Pal, and K. Kant, “Resource efficient edge computing infrastructure for video surveillance,” IEEE Trans on Sustainable Computing, March 2021. [3] P. Pradeep, K. Kant, and A. Pal, “C-far: A compositional framework for anomaly resolution in intelligent transportation system,” IEEE Trans. on Intelligent Transportation Systems, Aug 2022. [4] F. Chan et al., “Anticipating accidents in dashcam videos,” in Asian Conference on CV. Springer, 2016, pp. 136–153. [5] A. Shah, J. B. Lamare, T. N. Anh, and A. Hauptmann, “CADP: A novel dataset for cctv traffic camera based accident analysis,” arXiv preprint arXiv:1809.05782, 2018. [6] M. Naphade, S. Wang, D. C. Anastasiu, Z. Tang, M.-C. Chang, X. Yang, Y. Yao, L. Zheng, P. Chakraborty, C. E. Lopez et al., “The 5th ai city challenge,” in Proceedings of the IEEE/CVPR, 2021, pp. 4263–4273. [7] P. P. Kumar., “Tu-dat dataset,” https://github.com/pavana27/TU-DAT, 2021. [8] V. M. Arceda and E. L. Riveros, “Fast car crash detection in video,” in 2018 CLEI. IEEE, 2018, pp. 632–637. [9] A. Noor, B. Benjdira, A. Ammar, and A. Koubaa, “Driftnet: Aggressive driving behaviour detection using 3D CNNs,” in 2020 SMARTTECH. IEEE, 2020, pp. 214–219. [10] P. Pradeep, K. Kant, and A. Pal, “Non-intrusive driver behavior characterization from road-side cameras,” IEEE IoT Journal, Sept 2023, accepted, Available at https://www.kkant.net/papers/Driver behavior characterization.pdf. 2