Inventory tracking for unstructured environments via probabilistic reasoning

Rajaraman, Mabaran; Bannerman, Kyle; Shimada, Kenji Article Inventory tracking for unstructured environments via probabilistic reasoning Logistics Provided in Cooperation with: MDPI – Multidisciplinary Digital Publishing Institute, Basel Suggested Citation: Rajaraman, Mabaran; Bannerman, Kyle; Shimada, Kenji (2020) : Inventory tracking for unstructured environments via probabilistic reasoning, Logistics, ISSN 2305-6290, MDPI, Basel, Vol. 4, Iss. 3, pp. 1-29, https://doi.org/10.3390/logistics4030016 This Version is available at: https://hdl.handle.net/10419/310107 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. https://creativecommons.org/licenses/by/4.0/ logistics Article Inventory Tracking for Unstructured Environments via Probabilistic Reasoning Mabaran Rajaraman , Kyle Bannerman and Kenji Shimada * Department of Mechanical Engineering, Carnegie Mellon University, Pittsburgh, PA 15213, USA; [email protected] (M.R.); [email protected] (K.B.) *Correspondence: [email protected] Received: 27 May 2020; Accepted: 17 June 2020; Published: 14 July 2020   Abstract: Workpiece location is critical to efficiently plan actions downstream in manufacturing processes. In labor-intensive heavy industries, like construction and shipbuilding, multiple stakeholders interact, stack and move workpieces in the absence of any system to log such actions. While track-by-detection approaches rely on sensing technologies such as Radio Frequency Identification (RFID) and Global Positioning System (GPS), cluttered environments and stacks of workpieces pose several limitations to their adaptation. These challenges limit the usage of such technology to presenting the last known position of a workpiece with no further guidance on a search strategy. In this work we show that a multi-hypothesis tracking approach that models human reasoning can provide a search strategy based on available observations of a workpiece. We show that inventory tracking problems under uncertainty can be approached like probabilistic inference approaches in localization to detect, estimate and update the belief of the workpiece locations. We present a practical Internet-of-Things (IoT) framework for information collection over which we build our reasoning. We also present the ability of our system to accommodate additional constraints to prune search locations. Finally, in our experiments we show that our approach can provide a significant reduction against the conventional search for missing workpieces, of up to 80% in workpieces to visit and 60% in distance traveled. In our experiments we highlight the critical nature of identifying stacking events and inferring locations using reasoning to aid searches even when direct observation of a workpiece is not available. Keywords: inventory management; industry 4.0; construction 4.0; IoT; smart manufacturing; construction technology; probability; graphs 1. Introduction Information of the state of various materials and processes within the shop-floor is critical to improve process efficiency in any manufacturing environment. This is reflected in the framework prescribed by the fourth industrial revolution, Industry 4.0, which is rooted in IoT (Internet of Things), which vastly expands the scope of available information for a process [ 1 – 3 ]. Information on the location of raw materials, process states or the state of any component that plays a part in the shop-floor is necessary to build a Digital Twin (DT) or a Cyber Physical System (CPS) [ 4 , 5 ] that presents a snapshot or model of the current state of the entire process pipeline. Availability of such DT or CPS of the shop-floor can help bridge the gap between the expected state of elements that workers plan tasks on, as well as the actual state [ 6 , 7 ] that is affected by various other agents and actors within the system. In the absence of such a system and given that any worker cannot manually monitor and account for the effect of every other actor, uncertainty creeps into process plans, creating unnecessary delays. Manufacturing industries that have adopted the Industry 4.0 framework have shown significant improvements in efficiency, and the manufacturing sector overall has since shown a healthy 6% annual Logistics 2020,4, 16; doi:10.3390/logistics4030016 www.mdpi.com/journal/logistics Logistics 2020,4, 16 2 of 29 growth rate [ 8 ]. However, heavy manufacturing industries, like construction and shipbuilding, face several challenges in adopting Industry 4.0 concepts [ 9 ] and show only a 2% growth, trailing far behind the industry average. Construction and shipbuilding industries, though considered to be a part of the manufacturing industry sector, present very different work environments compared to the rest of the industry [ 10 , 11 ]. One significant difference here is that tasks move around the product, whereas products move around tasks in most of the manufacturing industries like those involved in mass-manufacturing. This makes it difficult to adopt sensing and automation technology that was matured within the manufacturing industry sector, as the fundamental process structure is incompatible. This is particularly evident in the inventory tracking front, which is necessary to build a CPS of the project. Besides the lack of a CPS, nearly 50–60% of the entire project cost in these industries is spent on raw materials [ 12 ] that are consumed directly or after further processing. Since materials are consumed while the workspace is evolving, it limits available storage options. This makes it impractical to build structured storage solutions on-site for all the raw materials and leads to workpieces being stacked on by multiple stakeholders in the project. Long storage times, handling by multiple stakeholders and accidental displacement combined with the lack of an effortless information collection solution to log any displacements leads to poor pull times in these environments [ 4 , 8 , 13 – 15 ]. Since workpiece displacement and movement decisions are not centralized and occur locally at worker’s discretion, we focus on collecting information from their point of view and extracting the effect of such displacement decisions downstream through a reasoning model. The purpose of our proposed system is to reduce effort and time spent in searching for workpieces without adding any additional cognitive burden on the movers during workpiece displacements, accidental or otherwise. The remainder of this article is divided into four sections. Section 2discusses current work in literature that directly tackle the inventory tracking problem and those that deal with probabilistic and reasoning-based methods in the field of robotics from which we draw concepts. Section 3describes the motivation and design of various components of our proposed approach, focusing on the contributions over our previous work [ 15 ]. Section 4covers the setup, design and results for the experiments conducted to validate our approach. Finally, Section 5discusses the conclusion, limits, and future work for our system. 2. Related Work The prevalence of the inventory tracking problem in the shipbuilding and construction industries and the reasons for it are well-documented in the literature. A review of current shop-floor practices and cost of lost inventory to the entire process pipeline has been highlighted in [ 8 , 10 , 13 , 16 ]. A field review of a construction site by [ 4 ] highlighted the limitations in manual logging of inventory in terms of time spent on searching for lost inventory. They further showed that with an automated information collection system, processes showed significant reductions in completion time. Apart from documenting the problem and its causes, there are several suggested solutions for an information collection system to maintain up-to-date information on inventory movement [ 17 , 18 ] using techniques and sensors similar to those used in warehousing and mass-manufacturing industries. To our knowledge, previous work addressing the problem of inventory tracking in the construction and shipbuilding industries focused on modes of information collection. While these methods suggest solutions that are based on a track-by-detection approach, they break down in the absence of detection, which is common in cluttered and stacked environments. Our proposed approach explicitly relays ambiguities and borrows from a wider stream of research areas that deal with uncertainties, such as robotics and probabilistic modeling. While information collection is an integral part of our proposed approach, we expand on available information to provide suggestions in the absence of direct detection a workpiece. In our approach, apart from information collection, we employ a probabilistic multi-hypothesis approach to provide suggestions on workpiece location. Logistics 2020,4, 16 3 of 29 Information collection : Information collection is described as one of the building blocks of the Industry 4.0 paradigm. The need for such a system in industries operating in unstructured environments has been identified as one of the essential steps towards improving process efficiency and faster pull times [1,2,5,10,17] . While the need is well established, current research thrust, to our knowledge, is focused on improvements to and adaptation of current Radio Frequency Identification (RFID) technology [ 16 , 18 – 20 ]. With the warehousing industry pushing for RFID to be a standard mode of information collection, implementation costs have been reduced, making it more attractive for adoption by the construction and shipbuilding industries [ 21 ]. Since placing, logging, and pulling inventory are the primary purposes of warehouse workers, time spent in making sure inventory locations get registered through RFIDs is well justified. However, in the construction and shipbuilding industries, to further reduce burden to human workers, the authors of [ 22 , 23 ] explored an autonomous information collection system using drones with RFID and vision sensors to detect RFID and barcodes or Augmented Reality (AR) tags. Apart from detection based on AR tags and barcodes, recently, computer-vision based information collection methods have been explored as an alternate. Work by [ 24 – 26 ] explored the idea of using current state-of-the-art detection methods in computer vision to track specific events involving heavy machinery, equipment and personnel to actionable information so as to improve safety and track heavy equipment movement throughout work sites. While the objective of our system of improving inventory pull times without adding additional cognitive burden to workers is similar to the methods described here, the final information presented is based on tracking-by-detection. However, unstructured environments where workpieces are cluttered and stacked pose several challenges to detection by RFID and vision-based systems. While RFID does not need line-of-sight for detection, the penetration limits of the interrogation signal in passive RFID-based detection make for poor information collection [ 5 , 27 ]. While active RFID-based detection provides better detection, they pose several cost and implementation challenges for stacks of heavy and abrasive workpieces that are common in workspaces. These challenges make detection quite difficult in these environments, limiting the usefulness of a tracking-by-detection approach. In our approach, we further expand on available information to identify multiple case scenarios as hypotheses and propagate and prune them to provide actionable information to guide search for missing inventory. Probabilistic tracking : Probabilistic approaches to meter ambiguities in detections and robot states have been successful in research as well as practical applications in the field of robotics. For example, in particle filters [ 28 ], a popular method to track objects in images, multiple hypotheses as particles are populated to span possible locations in the hypothesis space. The weight or confidence in each of these particles is updated based on its fit to incoming observations. These approaches employ a Bayesian inference-based approach to make sequential observations while updating their estimate of state, given some model of the environment to correlate observations. While these approaches have been quite popular in the areas of robot localization [ 29 , 30 ] and computer vision, to our knowledge, there has been little work exploring such techniques that provide actionable information while preserving ambiguity for inventory tracking applications. In our work we show that these concepts can be customized for the purposes of inventory tracking by providing a model for identifying stacking events that could occur, from observations of workers interacting with workpieces. Multi-hypothesis tracking : Multi-hypothesis tracking approaches [ 31 ] gained popularity in the early 1990’s as a solution for data-association problems that involved ambiguities in detections and sensor readings. These methods required maintaining multiple hypotheses, as tracklets, where a graph of each hypothesis is spawned, updated, and pruned, so as to avoid strict pairwise data associations over time until ambiguities could be resolved from further observations. In recent years, improvements in sensor and detection algorithm accuracy have shifted focus away from these approaches [ 32 ]. However, they present an ideal framework to incorporate complex higher-order information into tracking when detection cannot be guaranteed due to occlusions and stacking, as is the case for inventory in unstructured storage environments. Logistics 2020,4, 16 4 of 29 Reasoning-based inference : Reasoning-based methods aim to model relations between events and actions and try to extract deeper inferences such as intent. Such methods aim to build causal relations between observed events through grammar defined as first order logic [ 33 ] or as probabilistic causal graphs [ 34 ]. These methods infer useful outcomes from observations through first order logic or event outcome maps that are solicited from human knowledge. Most algorithms expand this knowledge by further discovering patterns from observed events and outcomes. While these methods have been popular in cognitive science and artificial intelligence, to our knowledge, there is little work exploring their usage in industrial applications such as inventory tracking. Recently, work by [ 35 ] has shown that these methods can be adapted for use in real-world object tracking applications to determine the location or state of occluded pedestrians, expanding the scope of track-by-detection algorithms. Work by [ 36 ] has also shown that such methods can be used to predict intent and upcoming actions in scenarios if expert knowledge that models the behavior of the world is available. Such algorithms that build on causal grammar provide promising results to infer the hidden states of objects that are not otherwise visible, conditioned on the context set by the immediate environment. Since one of the challenges in inventory tracking is occlusion or lack of direct observation, we employ similar reasoning, to expand the hypothesis for workpiece location even when direct observations of a workpiece are not available. 3. Proposed Approach Our proposed approach aims to reason possible locations of workpieces as an effect of events that have taken place leading up to losing track of the workpiece locations. The reasoning model described in our approach mirrors the reasoning a human worker would follow to build a search strategy when given enough observations and time. For example, as shown in Figure 1a, given a snapshot of the state of workpieces in the workspace at any time t , one cannot do better than a random guess on the location of the missing workpiece 6. However, given its last-known-position in the workspace at t− 1 (Figure 1b) and further events in time (Figure 1c,d), the search strategy propagates based on the worker’s knowledge of the physical world and workpiece movement. In our approach we try to model this behavior of building a search strategy for missing workpieces based on observed events involving the workpiece. To accomplish this, we need two critical components as shown in Figure 1: an information collection system (Section 3.1) to present the observable workpiece scene across time; and a reasoning model (Section 3.2), similar to that of a worker, to infer possible locations and propagation of stacks. This work expands on our previous work described in [ 15 ] and focuses on addressing its limitation to prune search locations and extract more critical events from movers (Section 3.5). We also present methods to provide tighter search locations through knowledge insertion (Section 3.2), if available. While our previous work gives a detailed visual description of stack propagation and the concurrent operations in the event and dependency graphs, in this article we limit the discussion to strengthening our proposed approach by providing underlying probabilistic principles and additional contributions of this work. In the interest of completeness, algorithms for building and evolving each graph are described in Appendix A1. The remainder of this section covers the components involved in the proposed approach. The system is built following a modular design that allows for each component to be swapped out or configured independently as the environment demands. Section 3.1 describes the hardware and IoT-driven architecture used for information collection. Section 3.2 describes the underlying formulation based on discrete Bayesian inference, commonly employed in Markov localization methods [ 29 ], which is used to build probabilistic relations between workpieces. We show how observations are correlated to a model, based on expert knowledge of workpiece movement and stacking, to calculate weights for said relations. Logistics 2020,4, 16 5 of 29 ( a ) All workpieces are likely search locations for workpiece 6. ( b ) Workpieces 14 and 12 would be initial search locations based on further temporal information. ( c ) Workpiece 12’s new location is still a likely search location. ( d ) The movement and interaction of workpiece 12 affect the search for workpiece 6 Figure 1. A scene representing a worker’s reasoning to decide on search locations given temporal location information. ( a )With just the current snapshot of the workspace, search for a workpiece defaults to chance for any worker. ( b ) Information on events leading up to the disappearance of workpiece 6, however, immediately introduces priority search locations of workpieces 12 and 14 based on simple reasoning of proximity. ( c ) The search strategy evolves along with the movement of workpieces as the initial suspected stack location, workpiece 12, is moved ( d ) This process can be chained with further changes in workpieces based on workers’ reasoning for the propagation of stacks. 3.1. Information Collection The objective of the information collection module is to present the scene of observable workpieces through workers and fixed cameras at any time (Figure 1). This module presents the observable scene of the workspace across time for the reasoning module to build the beliefs for missing workpieces. In our system, we use wired USB-cameras to capture and transfer images to Raspberry-Pis attached to workers’ helmets (Figure 2) and machinery such as forklifts, collectively referred to as movers. As multiple movers move around the workspace to interact and displace workpieces, this module publishes images from their view at 30 frames per second (FPS). These images represent the raw data from which workpieces tagged with Augmented Reality (AR) markers (Figure 3) and events are identified by the detection and reasoning modules, respectively. AR marker setup : Static AR markers of known pose (translation and rotation) with respect to a predefined workspace origin are placed in expected storage locations. When observed, these AR markers provide the pose of the observer with respect to the workspace. While static AR markers localize the observer with respect to the workspace, workpieces tagged with AR markers localize the workpieces with respect to the observer. If the observer’s location is known, the location of the observed workpiece is calculated through chained transforms between workpiece and observer and observer and workspace. Locations for the placement of static AR markers can be determined based on the field of view of fixed cameras placed throughout the workspace monitoring likely storage locations. (Figure 4). While we use AR markers as the mode of detection for our work, any mode of detection that provides location and identity (ID) of the workpiece can be used in its place. The focus of our work is on inferring locations of stacked workpieces when a direct observation of workpieces, through any mode of detection, is unavailable. Logistics 2020,4, 16 6 of 29 IoT architecture : The objective of the observation module in our approach is to collect information from the viewpoint of movers and from static cameras covering common inventory storage areas. While publishing images through a Robotic Operating System (ROS) [ 37 ] network in smaller spaces with strong Wi-Fi, as described in [ 15 ], is a solution that provides live updates on inventory movement, this breaks down in larger work sites that have weak or no Wi-Fi signals. A system that can scale and guarantee that observations are not lost despite weak communication channels vastly expands its application and reduces the implementation burden on the environment, making it more attractive for industries to adopt. (a)hardhat with camera. ( b ) Power and data access underneath the hardhat. ( c ) Raspberry-Pi 3B+ and battery stored inside the hardhat. Figure 2. Cameras attached to a worker’s gear to observe the immediate environment. ( a ) A worker’s helmet fitted with USB cameras to provide the field of view of the worker to our system. ( b ) Easy access to power switches, charging ports and an SD card slot allows for quick access and charging when the hardhat is not in use. ( c ) Internally, the hardhat houses the battery and Raspberry-Pi. This allows each mover to behave as an IoT device for information collection. (a)(b) Figure 3. Workpieces tagged with AR markers used for experiments at the shipyard. ( a ) Real-world workpieces within the shipyard viewed by a worker. ( b ) Test environment where the workpieces were moved by workers to evaluate our system under real-world mover behavior. Logistics 2020,4, 16 7 of 29 ( a ) Map of the lab with fixed camera locations and monitored sites (blue). Areas where movers create impromptu stacks are highlighted in red. The terminal provides search locations for a queried workpiece. (b)Camera view from the six fixed cameras. Figure 4. Workspace scene setup in the lab. ( a ) Map of our lab with highlighted fixed camera and inventory storage locations. Areas being monitored by the fixed cameras are highlighted in blue whereas areas that are not constantly monitored but represent space where movers could make impromptu stacks are highlighted in red. ( b ) Images of storage locations containing workpieces (grey) and static markers (white) as viewed by the fixed cameras. In this work we use an Internet-of-Things (IoT) framework (Figure 5) where the movers and static cameras, with synchronized clocks, push timestamped images onto a remote database. Images from the database can then be pulled and processed by the detection and inference modules, through any terminal that can access the database to estimate possible locations for each workpiece. To make sure that data is not lost when movers are in areas with weak connections, images are stored locally with their timestamps and mover-identifier until a connection can be established (Figure 6). While this system may not be able to provide live updates, the system can run through all observations by the end of the day, when data from all movers are uploaded to a database. This way, data from all movers can be accumulated and processed in offline mode to provide a belief on the location of workpieces for workers on the next day. At the same time, if a strong wireless network for data transfer does exist, updates can be made immediately, and the system can be set to function in an online mode. Logistics 2020,4, 16 8 of 29 Figure 5. Scalable IoT-type architecture for information collection. Each mover pushes their observed images onto a database when a connection is available. This architecture allows for images from observers across vast work areas to pool their data, allowing for workpieces to be tracked across the entire scope of the work site. Figure 6. Offline information collection setup when Wi-Fi is unavailable. For each mover wearing the hardhat as an IoT device, observations as images are transferred to the database immediately when the Wi-Fi signals are strong. However, when a mover is out of range or under poor Wi-Fi coverage, data are locally stored and pushed to the database when a connection can be established. This makes sure that all observations are available at the end of the day when all hardhats are within Wi-Fi range. 3.2. Reasoning Observations Once observable information is available through the information collection module, possible locations of missing workpieces need to be hypothesized from available information. This is accomplished in the reasoning module that identifies stacking events and builds and evolves such hypotheses for the locations of missing workpieces. Similar approaches to build and evolve hypotheses as beliefs have been successfully employed in localization problems in robotics under ambiguous sensor observations. In this section we give a brief description of the underlying probabilistic concepts and our approach to utilize them for our case. Theoretical background: Like tracking-by-detection, single-hypothesis approaches in robotic and workpiece localization problems aim to provide a single best-fit solution based on observed data and their correlation to an available model, like geometry [ 38 – 40 ]. In cases where a single best-fit cannot be determined due to ambiguity, probabilistic localization approaches preserve ambiguities and maintain multiple relevant hypotheses as distributions and update them as more observations are received [ 29 , 41 ]. In such approaches, a belief of the state of the current system, representative of uncertainty, is maintained. The posterior probability distributions for such hypotheses are calculated Logistics 2020,4, 16 15 of 29 The situation depicted by Figure 13 is quite common when multiple movers make decisions to move, stack or split stacks based on their immediate requirements. While it may not be possible to predict what decisions are made and why, it is possible to record and reason the effect of such decisions from the viewpoint of the mover even if the mover’s location cannot be determined. Figure 13. Extracting critical information from local observations. ( a ) When a mover observes workpieces and stacks them while his/her position is unknown, dependencies between workpieces is locally extracted. ( b ) Since the positions of the worker and the workpieces are unknown, this information is not reflected in the dependency graph for the workspace. One way to extract and use local observations is to run multiple instances of the system under different scopes. A primary dependency graph, Gdependency , covers the scope of the workspace and is run as discussed in the previous sections. Whenever a mover, say m1 , observes workpieces whose position cannot be determined with respect to the workspace, a temporary graph Gm1 dependency is spawned. This graph extracts dependencies within the field of view of m1 assuming the camera origin to be the local workspace origin (Figure 13). If interactions were observed, creating edges in Gm1 dependency , then these edges, their weights and their associated nodes representing workpieces, are added to Gdependency without any location information. In this way, local knowledge of stacks observed by any mover gets transferred. Since these workpieces cannot be categorized as visible or occluded, they exist in an unknown-location state within the system. However, the edges hold critical stacking information as possible location of workpieces. This can be utilized in case any of the locally observed workpieces later become visible, as shown in Figure 14. Figure 14. Transferring locally observed stack information. When a mover observes the stacked workpieces when his/her own position with respect to the workspace is known, workpiece 23, the only visible member of the stack, is registered in the dependency graph. Additionally, since the earlier local dependency graph by the mover established that workpieces 12 and 13 were associated with 23, this information gets transferred to the workspace though they were never directly observed within the workspace. One challenge that needs to be addressed here is that since we cannot determine the pose or position of the mover, all observations are mapped into the same local coordinate system spanned by the mover’s camera. This leads to workpieces located at different positions in the workspace being mapped to overlapped positions within the view of the mover’s camera coordinate system. To overcome this issue, the graph Gmk dependency of any mover gets reset whenever there are no workpieces within view. All dependencies as edges are transferred to the primary dependency graph Gdependency Logistics 2020,4, 16 16 of 29 before clearing the local graph nodes and edges. This makes sure that when a worker moves away from or changes their view, a new instance for local observations are spawned for the new local workspace. Since movers make a variety of impromptu decisions to grab and stack workpieces, extracting such decisions and their effect is critical to build a tight belief on the location of workpieces. While it may not be possible to densely populate a workspace with fixed cameras to capture all events, using hardhats to extract events through the viewpoint of movers provides far more coverage of all critical events right at the source. The critical nature of extracting such local stack events from moves is highlighted in our experiments in the next section. 4. Experiments As discussed earlier, the purpose of our proposed system is to reduce effort and time spent in searching for workpieces without adding any additional cognitive burden on the movers during workpiece displacements, accidental or otherwise. One way to evaluate the system would be to compare inventory pull costs with our proposed approach against current industry practices. However, in our literature review, we could not find standard databases or test procedures to measure the performance of generic inventory tracking systems. Apart from lack of databases, standard simulation environments and tools to model complex workpiece movements and actor interactions, such as an Actor Based Model (ABM) environment, while of interest, are still in early stages of development [ 43 , 44 ]. Previous work investigating poor pull times [ 4 , 8 , 10 ] in the construction industry have compared the cost of pulling inventory when their position is known against cases where they have been misplaced without any information to guide a search. Since such observations have been the basis to investigate and show that excessive pull costs are the result of a lack of efficient inventory tracking systems, we use a similar approach to evaluate our proposed system within the confines of our lab (Figure 4). To show that such a system can be deployed in a real-world environment, we have also conducted simple experiments in a shipyard at Tsuneishi under real-world conditions such as lighting, workpieces and untrained movers (Figure 3). The following sections discuss the results of experiments conducted both in our lab and at the shipyard. 4.1. In-Lab Experiments In order to evaluate our system, we designed a workspace in our lab similar to unstructured storage spots used in the construction or shipbuilding industries for convenient and impromptu storage of workpieces. Our workspace setup consists of six fixed cameras, connected to four battery-powered Raspberry Pis, which monitor areas representing storage locations within the workspace. Note that while such storage locations are marked for monitoring by fixed cameras, there are no explicit rules or policies to drive incoming and outgoing inventory. This setup represents a workspace within which workpieces are stacked and displaced independently by movers wearing camera-mounted hardhats (Figure 2). As discussed in Section 3.5, the hardhats capture critical events involving workpieces that might have taken place away from the view of the fixed cameras but within the field of view of the mover. While the span of our experimental space might be much smaller than a typical environment in a construction or shipbuilding environment, with 25 different stacks populated throughout the space, search for a missing workpiece, given no further information, requires a lot of effort (Table 1). Experiment design: Evaluation of our proposed approach was conducted through a sequence of three staggered experiment phases set up to emulate workpiece population, stacking and displacement in an unstructured storage environment (Figure 4). During the experiment, workpiece 66 was displaced across the workspace by the movers. The performance of our system was based on the number of locations and distance a mover covers to find workpiece 66 using its suggested search locations. In the first phase of the experiment, the workspace was populated with 25 workpieces by two independent movers. Locations of the workpieces were predetermined for the repeatability of experiments and ground truth evaluation of our system. For the system, observations older than two seconds are Logistics 2020,4, 16 17 of 29 considered occluded. The value for Lmax , described in earlier sections, was set to 100. The function l( . ) to limit Lmax (Figure 9), discussed in Section 3.3, was defined as a linear hinge-loss function that decreases from 1.0 to 0.0 for distances, Dij , ranging from 0.0 to 0.5 m. The separation penalty γ that penalizes indirect stacking relations to dampen the effect of indirect stack events, was set to 0.5. A value of 1.0 for γ translates to all events, no matter how indirect, to have equal significance during a search. On the other extreme, a value of 0.0 for γ translates to no event being considered for search, reducing the system to present the last-seen position as the only search location. The effect and need for γ is explained in detail in [ 15 ]. Experiments were conducted under the same values for common parameters between the current and previous approach. Phase 1: Since the goal of our proposed approach is to passively collect and present workpiece location information to the rest of the movers, observed positions of workpieces were plotted on a map and checked against the ground truth location of each workpiece. Figure 15 shows the plot of all workpieces that were observed by our system. This experimental evaluation serves to qualify the information collection using AR markers for our system. While this experiment is simple with no stacking of workpieces, a lookup map of workpiece locations provides a CPS of the current state of the workpieces within the workspace. With such a system, workpiece placement decisions made by individual movers get conveyed to the rest of the workforce without adding any burden to the mover. While AR markers detection is used as the mode of information collection in our system, any other mode of detection that provides the workpiece ID and its location can be used as an alternative. This experiment, at the least, qualifies the sanity of the information collection module of our system. Phase 2: For the second phase of the experiment, more workpieces were introduced into the workspace wherein stacks were created to store new workpieces under limited storage space. This is a common mode of storage in unstructured environments, which presents challenges for direct visual searching due to clutter and lack of line-of-sight to the workpiece. As discussed earlier, such stacking also introduces challenges for the RFID-based mode of detection, as the interrogation signal cannot penetrate thick stacks. Apart from penetration issues, passive RFID tags, when closely stacked, introduce noise and are difficult to untangle by the reader [ 27 ]. In such stacked scenarios, our vision-based information collection system faces the same challenges and cannot extract any more information once a workpiece is occluded. The inference module of our system, described in Section 3.2, however, can hypothesize the positions of workpieces that are out of line-of-sight based on previously identified relevant events (Figure 16). While our proposed approach presents the possible location of any workpiece, visible or occluded, based on surrounding events, one could argue that a system that presents the last known position of a workpiece would produce the same results, provided the stack itself is not moved or disturbed. This condition was further perturbed in the next phase of our experiment. Phase 3: The third and final phase of our experiment covered the cases that are most challenging and highlights the advantages of our approach. This phase of the experiment represents cases where a mover stacks multiple stacks together and displaces the whole stack to other locations either accidentally or intentionally. Either way, such decisions of repeated stacking and displacement are not logged or taken account of in current workspaces to inform the rest of the workforce of the updated position of all the members of the stack [ 4 ]. Moreover, while making such impromptu decisions, the movers themselves are not aware of all the members of the stack they just displaced and cannot afford the time to make note of such information at every event. Such gaps in knowledge make it quite difficult for the worker searching for one of the members of the stack when it is no longer at its last seen position, rendering the workpiece lost as far as the shop-floor is concerned. With no further information to guide his/her search, a worker usually resorts to exhaustive time-consuming searches or just re-manufactures the workpiece. This scenario (Figure 17) is reproduced in our experimental workspace by: •Having a mover remove the stack containing workpieces 66 and 79. Logistics 2020,4, 16 18 of 29 • Stacking workpieces 33 and 35 on top of this stack, away from fixed camera view in areas highlighted in red on the map (Figure 4). • Placing the new stack, with workpiece 35 at the top of the stack, on top of another stack within the workspace. Table 1. Cost of search in terms of number of workpieces to be visited and distance covered to find workpiece 66 displaced to various locations P1–P6. Least number of workpieces to be visited and the least distance covered during the search are highlighted in green. Search Strategy Status of Workpiece 66 Average (P1–P6) Metric Visible Stacked and Hidden out of View in Location P1 P2 P3 P4 P5 P6 Last-seen location of workpiece 66 in known Attempts 1 1 7 12 16 19.5 23.5 13.2 Distance 0 m 0 m 2.5 m 6.5 m 10.5 m 14.0 m 20.0 m 8.9 m Previous method [15] Attempts 12.5 7 12 16 19.5 23.5 13.4 Distance 0 m 0 m 2.5 m 6.5 m 10.5 m 14.0 m 20.0 m 8.9 m Proposed approach Attempts 1 1 3 3 3 3 3 2.6 Distance 0 m 0 m 2.5 m 5.0 m 4.5 m 6.6 m 2.5 m 3.5 m Figure 15. Map of the workspace with observed workpieces after the first phase of the experiment. Expected (green) and observed (yellow) positions by our system plotted on a map. This information collection makes locations of workpieces available to the entire workforce and allows for fast pull times without need for manual logging. ( a ) Plot of all workpieces observed by the system. A yellow marker represents a visible workpiece whereas a grey marker, for workpiece 66, represents the last-seen position. Figure 16. Cont. Logistics 2020,4, 16 19 of 29 (b)Suggested search locations given by our system for workpiece 66 Figure 16. Snapshot of the workspace and belief of our system after the second phase of the experiment. ( a ) Current location of all workpieces plotted by our system. Locations of recently observed workpieces (within a duration of 3.0 seconds) are highlighted in yellow, whereas workpieces that are deemed missing or occluded are highlighted in grey. ( b ) A query for the missing workpiece, workpiece 66, produces a plot of likely locations and their probabilities in finding workpiece 66 stacked under there. Workpiece 79 gets the highest confidence due to its proximity to 66, making it the first location to search. Figure 17. Events, viewed by the hardhat camera of a mover, during the third and final phase of the experiment. The view at location P1 shows a mover picking up the stack with workpieces 79 and 66. The view at a random location (red) away from monitored areas shows the mover stacking workpieces 33 and 35 on the existing stack. The view at location P5 shows the mover placing the new stack with workpieces 66, 79, 33 and 35 on top of workpiece 77. These actions are repeated multiple times by movers to lose workpiece 66 throughout all storage locations, P1–P6, within our workspace. The entire experiment was conducted across a span of two weeks in total to present a case as close to the real world as possible where movers interact with workpieces over a longer duration of time. Evaluation criteria: Once the third phase of the experiment was completed by a mover, the entire workspace was reset to its original state defined in the first phase of the experiment. Table 2shows the suggested workpieces to look under, as search locations, for every location P1–P6 that workpiece 66 was displaced to. Table 1shows the cost, in workpiece visits and steps, that would be incurred under four different cases where a worker is given: • Last-seen information: This represents the case when a worker is aware of the original location where workpiece 66 was placed in the first phase of the experiment but has no information of further events that have taken place. Logistics 2020,4, 16 20 of 29 • Information using previous system [ 15 ]: This provides search locations based on events observed by fixed cameras monitoring storage locations. Local events observed by the workers away from fixed camera view are still lost. • Information from our proposed approach: In this case the worker solicits our system to provide search locations for workpiece 66. Table 2. Totem pole and confidence of search locations for each experiment where workpiece 66 is moved to each of the six locations P1 to P6. The true location of workpiece 66 is highlighted in green. The locations are listed in the order of search locations along with their probability values as suggested by our system for each displacement. Order of Search Locations for Workpiece 66 after Being Stacked and Displaced P1 P2 P3 P4 P5 P6 Order Prob. Order Prob. Order Prob. Order Prob. Order Prob. Order Prob. 79 0.47 79 0.36 79 0.36 79 0.33 79 0.35 79 0.35 70 0.33 70 0.25 70 0.23 70 0.25 70 0.24 70 0.23 71 0.10 35 0.15 35 0.16 35 0.16 35 0.17 35 0.17 67 0.09 71 0.08 55 0.09 71 0.09 71 0.09 71 0.08 62 0.07 71 0.08 47 0.08 77 0.09 65 0.08 67 0.06 67 0.06 67 0.06 67 0.06 67 0.07 As mentioned earlier, the objective of the proposed approach is to reduce the effort and cost of searching for workpieces that have been displaced by multiple stakeholders within the workspace. The evaluation criteria for inventory pull are based on the number of workpieces a worker would have to visit and the distance covered in such visits before finding workpiece 66. Since workpiece 66 is not directly visible, the complete search space spans the location of every other workpiece within our workspace. When a totem pole of locations to visit is available, the number of workpieces a worker would have to visit is based on the position of the true location within this list. When no totem pole exists, all workpieces within a location are considered equally likely. In this case, the expected number of workpieces one would have to visit can be shown to be (N+ 1 )/ 2, where N is the number of equally likely choices available. If workpiece 66 was not found in a location, a worker moves on to the next location based on the suggested search location or optimal search path for continued search. Distance covered is based on the distance a worker has to travel from location P1, as that is the last seen position for workpiece 66 and would be the first location a worker would start the search from. Our system was evaluated against our previous approach and search based on optimal search paths to cover all locations. The optimal path based on distance to visit all workpiece locations in the case of our workspace is to sequentially move from P1 to P6. Discussion: As seen from Table 1, the average number of workpieces to be visited by our proposed approach was 80% lower than manual uninformed searches and the previous approach. The average distance covered by a worker in searching for workpiece 66 was also lower by 60%. One critical observation to emphasize here is that the search cost of our proposed approach is not affected by workpieces that never interacted with it. While a manual search cost is directly influenced by the number of storage locations between workpiece 66’s last seen position and final position, our objective is to limit the search based only on events of interaction involving workpiece 66. This can be seen in the snapshot, as shown in Figure 18, of the event graph edge weights overlaid on the stack order graph and the dependency graph within our system that has captured the chain of events involving workpiece 66 when it was displaced to location P5. This highlights every possible workpiece that workpiece 66 might be stacked under along the way when it was moved to P5. Logistics 2020,4, 16 21 of 29 Capturing all possible interactions of the workpiece makes the search strategy of the proposed system independent of workpieces that could not have interacted with the missing workpiece. In the absence of this information, the search space for a missing workpiece expands to all locations within the workspace, requiring much more effort and time to search. While the objective of our previous approach in [ 15 ] is the same, as shown in Table 1, failure to capture events defaults the cost to manual uninformed search. Since the previous approach could not account for events taking place locally within the field of view of movers, critical information that could have helped in the search was discarded. The significance of such local observations from the point of view of movers that cause stacking events is highlighted in the experiments conducted in the lab. ( a ) Pairwise weights for events plotted on the graph monitoring stack order after workpiece 66 was displaced to P5. ( b ) Dependency graph with edge weights calculated with a separation penalty γ=0.5, representing confidence in workpiece 66 being stacked with every other workpiece based on its reachable ancestor and sibling nodes. Figure 18. Underlying graphs in our system calculating beliefs after displacement of workpiece 66 during the experiment. ( a ) A snapshot of the event weights and stack order of events that workpiece 66 could have encountered, as a directed graph, in the experiment run that displaced workpiece 66 to location P5. ( b ) The workpieces and their corresponding probability as search locations for the missing workpiece 66. Note that since workpiece 33’s location is unknown, it is not suggested as a search location. Additionally, workpieces 57, 45 and 90 are not considered, as there are no reasonable events connecting them to the directed graph either as an ancestor or a sibling. 4.2. Real-World Evaluation To test the system within a real-world environment outside of our lab, we conducted multiple experiments at the Tsuneishi shipyard. The goal of these experiments was to gauge the ability of our proposed system to collect local events and build the expected dependencies between workpieces from the view of movers using realistic workpieces, and under operation of workers who have never used the system (Figure 3). Since the objective here is to see if we can extract critical events and dependencies between workpieces, event weights are not highlighted as no search is being conducted and have no bearing in this exercise. In this experimental space, we were able evaluate our system under real-world conditions with poor lighting, sparse Wi-Fi and mover behavior. Our setup consisted of one hardhat camera and four Logistics 2020,4, 16 22 of 29 fixed cameras, two of which were hardwired into the main computer, and two others connected to battery-powered Raspberry-Pis (Figure 19). Figure 19. Experimental setup and layout of testing space in the shipyard environment. The experiment was broken down into three sub-experiments. The first portion of the experiment was to introduce workpieces within the staging area and perform a stacking operation under observation of the hardhat but before entering the fixed camera views. This was done by: •Having the first mover bring workpieces 11, 12 and 13 into the staging area. •Stacking them all together and forming the stack (11, 12, 13). •Moving the stack (11, 12, 13) into view of Camera 1. The result of this sub-experiment would have connections between workpieces 11, 12 and 13 because they were stacked together within view of the mover’s hardhat camera. Workpieces 11 and 12 will not have absolute locations and will be marked as so (pink node) in the graph. However, since workpiece 13 is directly visible, its position is known and marked as visible (yellow node). Figure 20 shows the expected dependency graph, showing that the stack relation between workpieces matches the actual relation as a graph produced by our system. Figure 20. Workpiece relations, as a graph, for the first experiment. The expected relation between workpieces from the scripted experiment ( left ) matches the relational graph published by our system (right). Another mover was then asked to perform the second sub-experiment, where they combined two separate stacks within view of a fixed camera. This was done by: •Having a new mover place workpieces 14 and 15 into the staging area. •Stacking workpiece 15 on top of 14, forming the stack (14, 15). • Moving the stack (14, 15) into the view of Camera 1, and placing it on top of the previous stack (11, 12, 13). •Another mover comes and moves the whole stack (11, 12, 13, 14, 15) into the staging area. Logistics 2020,4, 16 23 of 29 The resulting graph for this sub-experiment would be an interconnection between all of the workpieces because the two stacks were joined into one (Figure 21). Figure 21. Workpiece relations for the second experiment. Expected fully connected relation between workpieces ( left ) and the relational graph published by our system ( right ), which was able to infer the same relation. This new worker who came into the last sub-experiment was then asked to continue by: •Shuffling the stack (11, 12, 13, 14, 15), removing workpiece 11 during the process. •Splitting the remaining shuffled stack (12, 13, 14, 15) into stacks (12, 14) and (13, 15). •Placing workpiece 11 in view of Camera 1. •Placing the stack (12, 14) in view of Camera 0. •Placing the stack (13, 15) in view of Camera 4. The graph corresponding to this sequence of events shows that workpieces 11, 14 and 15 can all be seen and that their absolute locations are known. Workpieces 12 and 13 are not on the top of their stacks so they are not seen. The difference between workpieces 12 and 13 is that workpiece 12 was seen during the shuffling and re-stacking in the staging area, while workpiece 13 was never seen. Since the system cannot determine the order of shuffling or the members for the split stacks, workpiece 12 and 13 could be under any of the three visible workpieces 15, 11 and 14 (Figure 22). Figure 22. Workpiece relation for the third stage of the experiment. Expected relation between workpieces after the stack was shuffled and split ( left ), which matches the output of our system ( right ). Discussion: While the above experiments are much shorter and simpler than the experiments conducted in our lab, they serve to highlight the robustness and practicality of our proposed approach for industrial applications. The workers that participated in the experiments required no specific training or modifications to their regular behavior to utilize the system successfully. These experiments show that our proposed system can extract critical events regarding workpiece movement and stacking without adding any additional burden to the workers. 5. Conclusions and Future Work Current proposed solutions for inventory tracking in unstructured environments focus on conditioning and preparing the environment to adapt to mature track-by-detection technologies Logistics 2020,4, 16 24 of 29 like RFID. However, the gap between industries with unstructured storage environments and the technology for such adaptation is still challenging. In this work we proposed alternatives to the track-by-detection approach to improve over current practices of searching for lost workpieces without guidance. From experiments, we have shown that even with a simple mode of detection that has severe limitations with occlusions from stacking, reasoning events leading up to occlusion can significantly help narrow search locations. We have also shown that the proposed graph-based tiered approach of reasoning allows for easy additional knowledge insertion, if available, to further prune search locations. For application in adverse environments with sparse Wi-Fi, we have proposed an IoT architecture that enables the system to collect and process information in both offline and online modes of operation. While the proposed approach expands information collection over our previous approach to better estimate workpiece locations, the entire system is still dependent on observations and detection to build its belief. As shown in the experiments, extracting events is extremely critical, and missing an event would default the system to unguided search thereafter. Currently, our approach uses a single mode of information collection relying on AR markers and vision. However, the system can be further expanded if it can assimilate observations from multiple modes of detection spanning RFID, GPS, vision etc. since they also relay similar position and identity information. With regards to the reasoning algorithm, currently, observation of a completely empty space that was previously suspected to be the location of a missing workpiece has no effect on the system’s belief. By contrast, in the real world, when a worker visits a location that has no workpieces in it, this information does have an effect on the belief for every other workpiece that was previously suspected to be present there. In our current algorithm, while we can detect the presence of a workpiece with absolute certainty, we are not able to infer or translate the effect of absence of a workpiece at a location. We are currently exploring methods to identify and differentiate an obscured workspace in an image against an empty location that has no workpieces present. Another observation on the behavior of the system is that while it acts as an information sink, uncertainty or entropy of proposed search locations will tend to increase with more observed events involving a workpiece. Since the system actively maintains multiple hypotheses on the location of a missing workpiece, unless the workpiece is later directly observed, the system cannot inherently take action to prune the search locations. A further expansion of this approach would be to include humans in the loop to guide stacking actions or reduce entropy by actively asking workers to observe certain stacks to resolve uncertainty in the position of workpieces. Such a system could produce actionable information even before searching. A system that can communicate the current state of information gain and suggest actions to tighten its current confidence could maintain and provide tighter search suggestions. We are currently investigating these areas to expand the capabilities of our system. Author Contributions: Conceptualization, M.R. and K.S.; funding acquisition, K.S.; supervision, K.S.; methodology, M.R.; software, M.R.; validation, M.R. and K.B.; investigation, M.R. and K.B.; data curation, M.R. and K.B.; writing-original draft, M.R. and K.B.; writing-review and editing, all authors. All authors have read and agreed to the published version of the manuscript. Funding: This research was supported by funding from Tsuneishi Shipbuilding Co., Ltd. Acknowledgments: The authors thank Nishi Nobuaki, Shohei Okada, Satoshi Yamauchi and Hirotaka Inoue from Tsuneishi for their valuable inputs, and Austin Hsu and Spencer Persaud, from Carnegie Mellon University, for their time and assistance. Conflicts of Interest: The authors declare no conflicts of interest.