AMDCNet: An attentional multi-directional convolutional network for stereo matching

Displays 74 (2022) 102243 Available online 6 June 2022 0141-9382/© 2022 The Author(s). Published by Elsevier B.V. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/). Contents lists available at ScienceDirect Displays journal homepage: www.elsevier.com/locate/displa AMDCNet: An attentional multi-directional convolutional network for stereo matching✩ Hewei Wang a, Yijie Li a, Shijia Xi a, Shaofan Wang b, Muhammad Salman Pathan c,d, Soumyabrata Dev c,d,∗ aBeijing-Dublin International College, Beijing University of Technology, Beijing 100124, China bFaculty of Information Technology, Beijing University of Technology, Beijing 100124, China cThe ADAPT SFI Research Centre, Dublin D04V1W8, Ireland dSchool of Computer Science, University College Dublin, Dublin D04V1W8, Ireland ARTICLE INFO Keywords: Stereo matching Convolution aggregation network Visual sensitive Cost aggregation ABSTRACT Stereo matching refers to finding the correspondence of a point in the real world between two different storage mediums (e.g., intensity images, depth images, three-dimensional points). There are existing stereo matching methods in the literature, but they exhibit two shortcomings. Firstly, during the feature region extraction of stereo matching, these methods require measuring the distance of regions, but measuring the texture distribution of the region is difficult and might lead to the failure of matching. Secondly, the templates used in these methods are rectangles with a fixed size, while most of the natural images exhibit rich information and are more suitable for flexible templates. In this paper, we propose an attentional multi-directional convolutional network (AMDCNet) for circumventing these issues. Our AMDCNet approach consists of three stages: extract the visual sensitivity factor, construct the multi-directional aggregation template and utilize left–right consistency detection to optimize. We evaluate our approach using standard images in the Middlebury test dataset, Scene Flow and KITTI 2015. Experimental results show that AMDCNet can reduce the mismatch rate, and also show significant improvement in accuracy compared with some classical method. In some scenarios, it surpasses some advanced methods based on deep learning. The model code, dataset, and results of the experiments in this paper are available at: https://github.com/WangHewei16/Attentional-Multi-Directional-Convolution-Network. 1. Introduction Stereo matching, referred to as finding the correspondence of a point in the real world between two different storage mediums (e.g., intensity images, depth images, three-dimensional points), serves as one of the depth perception techniques for scene understanding, has wide applications in the fields of computer vision [1,2], robot industry [3, 4], earth observations [5,6] and automatic driving technology [7,8]. According to the statistics, the eyes are an important channel for us to obtain external information, and vision can bring us more abundant information. The visual information reflected by objects is transmitted to the brain through nerve tissue and will be effectively processed by the visual cortex of the brain in the way of content and space respectively. Human eyes have a strong ability to distinguish color information with a great subjective influence, whereas the perception of the gray level of the image is very poor. The emergence of machine ✩This paper was recommended for publication by Guangtao Zhai. ∗Corresponding author at: School of Computer Science, University College Dublin, Dublin D04V1W8, Ireland. E-mail addresses: [email protected] (H. Wang), [email protected] (Y. Li), [email protected] (S. Xi), [email protected] (S. Wang), [email protected] (M.S. Pathan), [email protected] (S. Dev). vision gets rid of the influence of human factors where the resolution of gray can reach 256 levels and has low requirements for the environment that intensifies our deep understanding of the image. Stereo vision technology in the field of computer vision has always been one of the hot spots of research. In recent years, stereo matching technology in stereo vision was the research direction of many scholars. The disparity map obtained by stereo matching technology is widely used in many intelligent fields. Such stereo matching can be used in the volumetric estimation of tumors in the domain of e-health analytics [9–11], insertion of 3D products [12,13] in scenes for product placements [14–16], reconstruction of cloud base height for atmospheric analysis [17,18], and amongst others. The disparity information contained in each pixel of the disparity map reflects the three-dimensional information of the scene, which has important application value for robot map construction (three-dimensional reconstruction), obstacle avoidance navigation https://doi.org/10.1016/j.displa.2022.102243 Received 25 January 2022; Received in revised form 2 May 2022; Accepted 25 May 2022 Displays 74 (2022) 102243 2 H. Wang et al. and many other fields. The image source of this technology is usually captured by a binocular camera [19,20], which captures two images from left and right perspectives, so as to carry out the stereo matching operation. Existing stereo matching methods can be roughly divided into two categories. The first category, named fixed template methods, are designed with the pre-defined convolution template parameters. The second category named adaptive template methods, only take the distribution of gray difference as one of the judgment conditions, with the limited cost value of calculation. Due to the simplicity of the efficiency, the second category is one of the mainstream methods of stereo matching. However, they exhibit the following shortcomings and our motivation is to solve these issues. •During the feature region extraction of stereo matching, these algorithms require measuring the distance of regions (viz., Hamming distance). However, it is difficult to accurately measure the texture distribution of the region. This may lead to difficulty during matching. •The templates used in such stereo matching algorithms are rectangles with a fixed size. However, most of the natural images exhibit rich information. Therefore, applying fixed size templates tend to lose some visual sensitive information, such as edge points, and contour points, leading to a reduced performance of stereo matching accuracy. To solve the aforementioned issues, we propose an attentional multi-directional convolutional network (AMDCNet) for stereo matching. AMDCNet first detects the edge points, contour points and other visual sensitive areas of an image, and analyzes the texture distribution difference between them and smooth areas. Then it builds the weight factor of visual sensitivity and guides the attentional direction of the degree of freedom extension in the subsequent aggregation stage. In the aggregation stage, AMDCNet constructs a multi-directional extension template by using the weight factor with vision induced attention and generates an image by processing covered pixels with the guidance of the template. The main contribution of AMDCNet is twofold: •AMDCNet introduces a visual sensitivity factor in the cost calculation stage and constructs a multi-directional aggregation template by using visual sensitivity factor as attention to improve the matching degree. •AMDCNet produces promising results in mismatching rate, disparity estimation, matching precision experiments, compared with several state-of-the-art stereo matching methods. The code related to this paper is available here: https://github.com/WangHewei16/Attentional-Multi-Directional -Convolution-Network. This paper is organized as follows. Section 2analyzes and discusses the relevant algorithms of stereo matching with their advantages and disadvantages. Section 3describes AMDCNet in details. Section 4conducts experiments on three datasets: Middlebury, Scene Flow and KITTI 2015 datasets. Section 5concludes this paper with future work. 2. Related works In this section, we review the related works on stereo matching from three angles: the process of stereo matching, region-to-region matching algorithm, and point-to-point matching algorithm. 2.1. The process of stereo matching The first stereo matching technology was proposed by Marr at the end of the 1960s [21]. This opened up the area for research of stereo matching technology. Stereo matching algorithms can be roughly divided into two categories: local matching and global matching. After decades of development, a variety of stereo matching research continues to emerge. The process of stereo matching can be divided into four stages: 1. Cost calculation; 2. Cost aggregation; 3. Calculation of initial disparity map; 4. Disparity optimization [22]. The process of cost calculation is usually to determine the initial generation value by calculating the gray value difference of three channels of a pixel in the left and right view images. The cost aggregation stage can be regarded as a filtering operation, in which the cost obtained in the previous stage is aggregated. In the first step, the parallax value of a single pixel is calculated at the cost, and these pixels are isolated points, often accompanied by a lot of noise, so a similar filtering process is needed to optimize. After aggregating the isolated parallax values in the previous step, we can get an image that is close to the real parallax image. In the next step, we need to select the most appropriate parallax value for the generated multiple parallax values, and often select the parallax corresponding to the minimum generation value. In global stereo matching, the energy function is used to generate a disparity map. The optimization of parallax can be considered a postprocessing process. It further optimizes the parallax value obtained above. This is one of the commonly used methods that includes left– right consistency detection, regional voting, linear interpolation and other methods, as well as relatively simple median filtering and other operations. The optimization phase can usually improve the matching effect. 2.2. Region-to-region matching algorithm With the development of stereo matching technology, many scholars have used this area for research purposes. Simoncelli et al. proposed Sum of Absolute Differences (SAD) stereo matching algorithm based on the gray difference [23]. Based on the former, Da et al. improved the algorithm and used the gray features of the image to find the optimal matching region [24]. Some scholars use the whole image of the characteristics of stereo matching to meet the requirements of the parallax map. Wang et al. in [25] proposed a novel stereo matching algorithm where they introduced the visual sensitivity factor in the cost calculation stage of the matching process for an matching. In [26], Sun et al. proposed a single direction stereo matching algorithm, which makes various limit constraints between the image and another image to be matched, and finds the optimal pixel in another image. Yin et al. used the threshold of color to segment the processing image, and then fitted the segmented region, and used the energy function minimization to optimize the result image [27]. However, the threshold used for color segmentation in his research is very strict, which is easy to cause over-segmentation and under segmentation. Similar to the SAD and Normalized Cross-Correlation (NCC) algorithm, Wang et al. extracted local operators and proposed a real-time matching method, which can quickly compute the result image, but its matching effect is poor, which affects the subsequent application [28]. Zabih et al. have first developed a nonlinear stereo matching algorithm that is based on the premise of less time consumption to obtain a better matching effect of the image [29]. The emergence of this method promotes the development of stereo matching technology, but its limitations are also relatively large, and the matching effect is poor in non-smooth areas. This transformation method is highly dependent on the only pixel in the region template. Noise has a great impact on the cost calculation process, resulting in the calculation of a large deviation of the parallax value that greatly reduces the image matching effect. After this method was proposed, many scholars have improved it, mainly from the overall image matching effect. For Displays 74 (2022) 102243 3 H. Wang et al. instance, Fan et al. divided the noise fluctuation range in the matching process to prevent excessive noise from affecting the image matching effect [30]. Similarly, Men et al. used the gradient value, changed direction and gray information as the cost of image matching [31]. Guo et al. constructed the energy function of local information based on census stereo matching algorithm, which improved the matching effect [32]. Some scholars put forward new ideas according to the limitations of the algorithm proposed by Zabih et al. [29] in the stage of cost aggregation and post-processing optimization. For example, Chai and Cao [33] fused the SAD algorithm with census algorithm according to the characteristics of better local matching effect, so that the gray difference can guide the cost calculation process. In the process of parallax refinement, a median filter is mainly used to eliminate isolated noise and ‘‘bad points’’, and to reduce the cost, quadratic linear interpolation is used to further improve the image matching effect. However, the initial cost of the above method is relatively simple and the robustness is low, so the matching effect of the initial disparity map is not high, and the improvement of the overall image quality is limited. In the process of cost aggregation, the rectangular template is used to aggregate the generation value. This fixed template usually loses a lot of important information and affects the parallax effect. Therefore, on this basis, Zhang et al. [34] proposed a relatively simple cross extension template, which sets the extension condition as gray difference and distance difference to guide the cost aggregation process. Mei et al. [35] enhanced the robustness of automatic growth based on the former and added a third constraint condition, which made the template construction closer to the distribution of texture features in the image region, and improved the matching degree of the image to a certain extent. However, the construction of constraint conditions in the aggregation process was relatively simple, and the isolated points in the disparity map were lacking. The matching accuracy still needs to be improved. Based on the improvement of disparity optimization and cost aggregation in stereo matching, more new matching methods are proposed, such as the sub-pixel difference method proposed by an et al. From the perspective of disparity optimization, which can improve the disparity map matching effect by 1%–2% [36], for instances, Zhang et al. [37] and Shan et al. [38] improve the matching effect from the aggregation process. One is to improve the matching accuracy and introduce the regularization term in the aggregation stage whose advantage is to combine a variety of aggregation algorithms; the other is to improve the time-saving performance of stereo matching, using micro adaptive template window for aggregation, and using parallel processing for the whole process to save time and cost. De-Maeztu et al. [39] introduced the concept of the gradient similarity-matching in the aggregation stage to further constrain the calculation method of the initial cost to improve the matching effect. 2.3. Point-to-point matching algorithm In addition to the region based matching method, the method based on feature point extraction can also obtain the corresponding parallax image. At present, the widely used feature matching methods mainly include ORB [40], SIFT [41] and SURF [42], whose principle is basically to establish the scale by extracting the feature points on the image (often with obvious structural features such as edge contour, corner, inflection point, etc.). The degree space guarantees its scale in-variance, and then the descriptors are generated by the gradient direction of the feature points. Finally, the matching results are obtained by matching feature vectors, but the feature-based matching methods often cause disparity discontinuity due to the sparse nature of feature points. There are many mismatched points, and the generated 3D point cloud is sparse, which affects the reconstruction effect. In this paper, the processing method of feature points is not described in detail. To be specific, we briefly introduce the classic census algorithm, which is a local region matching algorithm based on nonlinear transformation. It is mainly judged according to the gray difference of the Fig. 1. Census Principle. Clockwise comparison begins in the position of upper left corner. The values in the 3 ×3 template represent the pixel distribution of a certain area in the image, and each area can be represented by a string of binary codes. Finally, the Hamming distance of the two binary strings is calculated to be 2. neighboring pixels around the target point. Suppose that in a pixel covered by a 3 ×3 rectangular template, the gray difference between the central point and the surrounding pixels is made when the pixel is larger than the target point. When it is smaller than the target point, it is marked with 0. The principle formula is as follows: 𝑙(𝑝, 𝑝′) = {0, 𝑖𝑓 (𝑝>𝑝′) 1, 𝑖𝑓 (𝑝<𝑝′) 𝐶𝑒𝑛𝑠(𝑝) = ⊗ 𝑝′∈𝑊(𝑢,𝑣) 𝑙(𝑝, 𝑝′) (1) where 𝑝represents the target point under the template, that is, the center pixel, 𝑝′represents the neighboring pixels, and 𝑊(𝑢, 𝑣)which is assumed to be a 3 ×3 rectangular template, represents the central pixel of the window. Through the above model, the pixels under the template can be reclassified, and then the template can be moved to traverse the whole image. Finally, many binary strings similar to bit code can be obtained. These strings reflect the texture distribution of the template region to a certain extent, and can roughly represent the feature information of the region. Then, from another image to be matched, we plan the same operation as above: find the region similar to the feature information of the region, complete the matching and calculate the parallax. The judging condition is selected by the Hamming distance. The specific schematic diagram is shown in Fig. 1. The census algorithm calculation is as follows: from clockwise from the top left, we compare each number to the center number. If the number is larger than the center number, the bit calculation result is 0. If the number is smaller or equal to the center number, the bit calculation result is 1. However, this method also has many drawbacks. For instance, it only takes the distribution of gray difference as one of the judgment conditions, and the cost value of calculation is limited. Moreover, the template used in this method is a rectangle, and most of the images processed are rich in information, so it is easy to lose some visual sensitive information, such as edge points, contour points and so on. According to the algorithm principle, two groups of binary string codes are obtained as shown in Fig. 2, which are the same as those in Fig. 1. The calculated value of Hamming distance is also figured out, but the texture distribution of the region they represent is vastly different, so it is easy to cause the phenomenon of matching failure. 3. AMDCNet AMDCNet consists of three stages (see Fig. 3 as a flowchart of AMDCNet). In the sensitive points extraction stage, AMDCNet detects Displays 74 (2022) 102243 4 H. Wang et al. Fig. 2. Different regions of images. The values in the 3 ×3 template represent the pixel distribution of a certain area in the image, and each area can be represented by a string of binary codes. Finally, the Hamming distance of the two binary strings is calculated to be 0. the edge points, contour points and other visual sensitive areas in the image, analyzes the texture distribution difference between them and smooth areas, and guides the judgment weight of degree of freedom extension. In the multi-directional template matching stage, AMDCNet constructs a multi-directional extension template, then, extended pixel length is guided according to the attention, which is induced by visual sensitivity, finally, AMDCNet completes the whole image processing. In the optimization stage, AMDCNet uses the left–right consistency detection method to further reduce the error matching rate. 3.1. Extract the visual sensitivity factor As shown in Fig. 4, the template is divided into two directions: horizontal 𝐻𝑥and vertical 𝐻𝑦, and convolution operation is performed on the covered pixels through the two directions, and two candidate factors are extracted subsequently. One represents the brightness difference in each direction, and the other represents the directivity of the gradient. We use this factor to calculate the visual sensitive area. The brightness difference is calculated as follows: 𝐺𝑥=𝐻𝑥∗𝐴 𝐺𝑦=𝐻𝑦∗𝐴(2) where ∗denotes the convolution operator, 𝐴is the input image, 𝐺𝑥is the transverse brightness value, and 𝐺𝑦is the longitudinal brightness value. By summing the absolute values of the two, the gray value of a ‘‘point’’ representing the region can be obtained as follows: 𝐺=√𝐺2 𝑥+𝐺2 𝑦(3) The calculated 𝐺is used to match and guide the two images. We use the Hamming distance between the binary strings to represent the first generation value to highlight the grayscale changes in the two regions. Then, we further judge whether the difference of 𝐺between the two regions is less than a certain threshold. If two regions exhibit different gray levels but with the same sensitivity, then we set it as a candidate matching, and further obtain the parallax between the two images. The process of sequential judgment by two generations of values is expressed by: 𝑓𝑠(𝑝, 𝑑) = 1 − exp [−𝐶𝑠(𝑝, 𝑑) 𝜆𝑠] 𝑓𝑔(𝑝, 𝑑) = 1 − exp [− 𝐶𝑔(𝑝, 𝑑) 𝜆𝑔](4) where 𝑝represents the current pixel, 𝑑represents the horizontal displacement difference of the points in the two images, 𝐶𝑠,𝐶𝑔represent the cost of the points in the two positions, respectively, and 𝑓𝑠is the mapping value of 𝐶𝑠after a certain transformation and gray approximation. After gray screening, the second cost value, also visual sensitivity, 𝑓𝑔is calculated and screened. 𝜆is an empirical parameter, which makes these two kinds of generational values project to the interval of 0 and 1 using an index. The above method can be used to calculate the initial disparity map of the two images, but this is not the best result. This paper proposes a cost aggregation algorithm based on the non-directional growth template controlled by the visually sensitive directional factor. 3.2. Construct an attentional multi-directional aggregation template We construct an attentional multi-dimensional aggregation template in this section. Aggregation of AMDCNet is not a simple superposition, but through visual sensitivity factor proposed in Section 3.1 as attention. The visual sensitivity direction of a certain area can be easily obtained. The calculation is as follows: 𝜃= arctan( 𝐺𝑦 𝐺𝑥 )(5) where the angle value 𝜃obtained by the above formula is called the visual acuity direction factor, and can be regarded as the sensitivity direction especially when the gray level of images changes greatly. The pixels and areas in the image will produce such direction factors. Therefore, we can use this factor to guide the aggregation results in the next stage of aggregation. This paper proposes a multi-degree-of-freedom template based on orientation factors. AMDCNet builds an aggregation template for pixels. This template is not a fixed rectangular window, but an adaptive template with a constantly changing shape. Such an adaptive template will meet the four-degree-of-freedom growth decision condition. The specific process is as follows: ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ 𝐷𝑠(𝑝1, 𝑝)< 𝐿1 𝐷𝑐(𝑝1, 𝑝)< 𝑓(𝜏1, 𝜃) 𝐷𝑐(𝑝1, 𝑝1+ (1,0)) < 𝜏2 𝐷𝑐(𝑝1, 𝑝)< 𝑓(𝜏3, 𝜃), 𝑖𝑓 𝐿1< 𝐷3(𝑝1, 𝑝)< 𝐿2 with 𝜏3< 𝜏2<=𝜏1, 𝐿1< 𝐿2 (6) where 𝑝represents the central pixel, and 𝑝1represents a point extending a certain distance. The function 𝐷𝑠is used to generate the distance between two points and ensure that its value is less than a certain threshold 𝐿1or 𝐿2.𝐷𝑐represents the color similarity of the two points, which is also related to the sensitive direction. When the direction is similar to the extended distance, it means that the color difference between the point and the reference point is greater. We set a gray threshold 𝛾1on this basis and judge the gray scale difference between the two points. If it is less than its threshold, it continues to extend in this direction, and 𝐿1represents a distance threshold. The third degree of freedom judgment is to compare the forward time during the extension process. When a newly extended point still out of the threshold with the reference point, it also needs to be compared with the point at the previous moment in grayscale. Similarly, it sets a gray scale threshold as the criterion for the two points. If only this condition is met, it is still can continue to extend. The judgment criterion for the fourth degree of freedom is to mutually restrict the distance and color. When the extension point reaches a certain distance but does not exceed the maximum distance limit, we reduce the previous color threshold 𝛾3for more stringent screening. Similarly, when the extension continues, it is still adjusted by the direction factor, and the maximum distance parameter will be set at this time. When the extension distance exceeds the threshold, the extension will be terminated immediately. Fig. 5 shows a growth polymerization template of the degree of freedom Displays 74 (2022) 102243 5 H. Wang et al. Fig. 3. Architecture overview of proposed AMDCNet. We perform three operations on the image: (a) Perform convolution template to get the visual sensitivity factor; (b) Calculate the initial cost of the image and construct the multi-directional aggregation template. (c) Detect left–right consistency to optimize. Fig. 4. Convolution template graph. The template has vertical and horizontal directions. Left denotes the horizontal convolution template 𝐻𝑥and right denotes the vertical convolution template 𝐻𝑦. Convolute a region of the image in two directions to get a new factor representing the region. based on the direction factor control. The leftmost figure is the cross template extended from each reference point, and the rightmost figure is the result of further expansion in the upper and lower directions according to the above rules based on each polymerization arm extended from the left figure. The template in the right subfigure of Fig. 5 is the cost aggregation template that should be used in the end. Through the above aggregation template, the initial cost calculated at the beginning is aggregated to obtain a more refined parallax image. 3.3. Parallax optimization of AMDCNet: left–right consistency detection We utilize an optimization algorithm named left–right consistency detection to further reduce the error matching rate. For the processed parallax image, AMDCNet carries out the final optimization algorithm, to make the test parallax value closer to the real value. The purpose of the optimization is to further propose some redundant points and improve the matching accuracy. This process is often considered the post-processing stage of the stereo matching process. The optimization algorithm used is the combination of left–right consistency detection and region voting: first, the left–right consistency detection is used to distinguish the occluded point and the mismatched point in the image. For the occluded point 𝑃, the first non-occluded point appears in the left and right directions, which is recorded as 𝑝𝑙and 𝑝𝑟. The minimum value of the two is assigned to the 𝑃point to complete the filling. For the mismatched points, the region voting method is adopted. The gray distribution of all pixels in the support window area is counted, and the parallax value with the most frequent occurrence is selected to replace the parallax value of the point. Finally, the image is processed with quadratic linear difference operation, and the parallax value of a point is interpolated with the following formula: 𝑑𝑝=𝑑−𝐶(𝑝, 𝑑 + 1) − 𝐶(𝑝, 𝑑 − 1) 2(𝐶(𝑝, 𝑑 + 1) + 𝐶(𝑝, 𝑑 − 1) − 2𝐶(𝑝, 𝑑)) (7) where 𝐶represents the replacement value of 𝑝-pixel points calculated in this method, and 𝑑is the horizontal displacement interpolation for 𝑝points in the left and right view images. 4. Experiments and discussion In this section, we discuss the benchmarking experiments that are conducted in this research. 4.1. Datasets The experimental datasets used in this paper are from Middlebury official test datasets, Scene Flow dataset and KITTI 2015 dataset. These datasets have also been used in many pieces of literature [36,37,43– 46]. To be specific, Middlebury datasets are mainly used for vision processing of binocular images and can be used for image restoration, stereo matching, image recognition and other operations as recommended by the majority of scholars. We have used the Middlebury dataset for benchmarking classical stereo matching algorithms. This is because most of these classical algorithms are compared in this dataset. In order to benchmark the more up-to-date methods, we have chosen the KITTI and Scene Flow datasets. We have provided a subjective evaluation of the recent stereo methods in the latest datasets. We have proceeded in this manner, because this helps in a fair comparison amongst all the benchmarking stereo matching algorithms. In this paper, we select four kinds of images in Middlebury 2.0 for experiments, which contains 24 kinds of left and right view images, including the color images and the real parallax images. AMDCNet has tested on Win7 64 bit system by using Visual Studio software and OpenCV2.4.9 framework. 4.2. Parameter setting Firstly, we do quantitative experiments on distance threshold 𝐿1 and color threshold 𝛾1and choose a more appropriate parameter value. In this paper, four kinds of images in the dataset are selected for experiments, which are Tsukuba, Cones, Venus and Teddy. As shown in Fig. 6, we assign different sizes to the distance threshold and the color threshold and judge whether the selected parameters meet most of the images and are worth being selected as empirical parameters according to the error matching rate of the aggregation results. Mismatch rate represents the percentage of the number of mismatches between the test result disparity map and the real disparity map in the whole image. From the information in Fig. 6, it can be found that when the color threshold 𝛾1is between 20 and 21, the efficiency of image matching is better, and when the threshold is further increased, there is no more obvious effect. The distance threshold between 17 and Displays 74 (2022) 102243 6 H. Wang et al. Fig. 5. The process of constructing the multi-directional aggregation template. For a cross template that each pixel may extend into at a single time (left), we adopt orientation factors to instruct the growth of template (middle), and then construct the final aggregation template once (right) for the target pixel. Fig. 6. Mismatch rate with different threshold. The experiment is carried out from the (a) image: color threshold and the (b) image: distance threshold. 21 has a good matching performance of the image, starting from 17, and the increase of distance threshold 𝐿2has no great impact on the results. Therefore, this paper will control the color threshold at 20 to 21, the distance threshold 𝐿2choose 17 or 18. 4.3. Experiments of AMDCNet versus some non-deep learning methods In order to prove the effectiveness of AMDCNet, we display the processed images, using Cones, Tsukuba, Teddy, and Venus in the dataset. Fig. 7 shows qualitative results of AMDCNet versus Census on Cones, Tsukuba, Teddy, and Venus (top to bottom). According to the results, AMDCNet can effectively calculate and generate the disparity of the original image, and the matching effect is better than that of classical Census method. Moreover, Fig. 8 shows qualitative results of visual sensitive area retention of AMDCNet versus classical Census method [29]. Through the display of subjective images, we can see that AMDCNet can effectively calculate and generate the disparity of images, and produce better matching effect than classical Census method [29]. Moreover, it has been improved and preserved in many visual sensitive areas, such as ‘‘peak’’, ‘‘lamp’’ and ‘‘table’’. Next, this paper combines the objective data to prove the effectiveness of AMDCNet. Table 1 shows the error matching rate of the above four types of images. These values are obtained by comparing nonoccluded areas. The non-occluded area refers to the image under two viewing angles. There must be a certain area that is invisible from the other viewing angle, which indicates that a certain displacement exists in the horizontal direction. Such invisible areas are called non-occluded areas. In general, the matching effect of comparison of non-occluded areas is better than the matching effect of all image areas. As shown Table 1 The error matching rate obtained by various methods. Algorithms cones tsukuba teddy venus avg% Census [29] 21.89 27.41 23.43 27.78 25.13 AMDCNet 4.08 2.01 7.82 1.34 3.80 Ref. [30] 4.04 2.53 7.57 1.60 3.93 SG-C [47] 12.92 4.80 8.05 1.91 6.92 Mp-C [37] 6.78 4.50 11.32 3.55 6.53 Table 2 Mismatch rate under different Gaussian noise concentrations. Noise none 2% 5% 10% 15% Ref. [33] 4.54 6.67 11.73 26.88 48.63 AMDCNet 3.81 5.01 8.59 17.32 38.41 in Table 1, in the non-occluded area, the matching performance of AMDCNet is better than the others. AMDCNet is also more robust than other types of methods. In order to prove this, we add Gaussian noise of different concentrations to the four original images and show the comparative results in Table 2, where ‘‘None’’ means that there is no input noise. AMDCNet in the table also uses an improved adaptive aggregation template. In addition, the results in the table show that when the noise concentration increases, the change degree of the error matching rate of this method is lower, and the sensitivity to noise is lower. For the above-processed images, AMDCNet optimizes these images by left–right consistency detection in order to make the test parallax value closer to the real value. The picture effect of the above four kinds of images after optimization is shown in Fig. 9. From the figure, we Displays 74 (2022) 102243 7 H. Wang et al. Fig. 7. Results of disparity estimation for Middlebury test images. (a) left image of stereo image pair, (b) ground truth disparity. (c) and (d) disparity map estimated using Census. (e) and (f) disparity map estimated using AMDCNet. The error area is shown in red. The more red, the worse the matching effect. Fig. 8. Visual sensitive area retention. It shows the effect of the method for the visual sensitive area in the image, and compares with other methods. top: Census method, bottom: AMDCNet. Table 3 Comparison results of different methods. Algorithms cones tsukuba teddy venus avg% all nocc all nocc all nocc all nocc AMDCNet 8.85 3.46 1.26 0.92 10.72 6.09 0.86 0.34 4.06 Ref. [31] 9.26 3.79 4.15 3.62 11.6 5.68 1.87 1.08 5.13 Seg-CT [48] 13.77 6.15 5.40 4.57 11.44 6.27 1.93 1.19 6.34 SAD+CT [33] 7.65 4.09 1.98 1.29 12.60 6.02 0.74 0.53 4.36 GRD [37] 16.13 4.45 3.59 2.72 17.56 7.45 4.12 1.68 7.21 MCADSR [38] 11.1 3.51 4.15 3.62 14.7 7.57 0.87 0.48 5.75 GradAdaptWt [39] 7.67 2.61 2.63 2.26 13.10 8.00 6.99 1.39 5.58 Displays 74 (2022) 102243 8 H. Wang et al. Fig. 9. Disparity images before and after optimization. can see that the quality is significantly improved after optimization. After optimization, not only the large-scale ‘‘hole point’’ is removed for the non-occluded area, but also the isolated incorrect matching points are compensated by the quadratic linear interpolation, and the original boundary contour and other regional structures are maintained. This paper further compares the error matching rate of the optimized disparity map, and the results are shown in Table 3. We observe that GradAdaptWt’s algorithm is suitable for processing and recognizing objects with shape features like cone. From Table 3, AMDCNet generally has better results, in addition to the optimization effect of algorithm GradAdaptWt on cones is better than that of AMDCNet. This is because the algorithm of GradAdaptWt is more sensitive to the shape of cone, but AMDCNet has better performance in tsukuba, teddy, venus and average values. From Table 3, we can see the difference between NOCC and all, where the NOCC is the non-occluded area mentioned above, and ‘‘all’’ means the matching of all areas of the image, including the blind area of parallax. The data of mismatch rate in the table objectively proves the effectiveness of the AMDCNet, which has more efficiency than the same kind of traditional methods. AMDCNet is not only applied to the mentioned four types of images but also experimented with other images in the dataset. The experimental images are shown in Fig. 10. From this figure, we can observe that AMDCNet produces a good matching effect for most of the binocular vision images. 4.4. Experiments of AMDCNet versus state-of-the-art deep learning methods Next, we compare AMDCNet with some advanced deep learning methods. Wang et al. [44] proposed a multi-task attention stereo network (MASNet) to integrate the feature information from stereo image pairs for disparity estimation. Compared with the traditional CNN algorithm, AMDCNet deepens the number of network layers, embeds the pooling layer into the convolution layer, and provides a better measure for the calculation of parallax. However, this method is more inclined to the optimization process in stereo matching, and cannot provide a more detailed description for the calculation of early cost characteristics. Chang et al. [45] proposed a pyramid stereo matching network (PSMNet) consisting of two modules: spatial pyramid pooling and 3D CNN. The spatial pyramid pooling module takes the advantage of the capacity of global context information by the aggregating context in different scales and locations to form a cost volume. PSMNet is similar to the way of constructing an image pyramid, but for highresolution images, its matching effect will be limited because the influence of the CNN convolution layer is ignored in the process of network construction. Liu et al. [46] proposed a richer convolutional feature network architecture (RCF), increasing the detection of edge features, having a good matching effect for high-resolution images, and using convolution mode to refine and extract features. However, this method does not have much workload for parallax optimization. In [49], the authors proposed a local stereo matching algorithm that aims to improve the quality of stereo matching algorithm by solving the multi-label problems. Hong et al. in [50] proposed an almost near real-time local stereo matching algorithm possessing fast computational time. Similar fast computational time is achieved in [51], that performs depth enhancement using RGB-D guided filtering. In [52], the authors have described a generic framework for parallelizing dense matching procedures using CUDA (Compute Unified Device Architecture) programming. Furthermore, the idea of fast guided image filtering is not restricted to images, but extended to real-time 3-dimensional (3-D) video services with reduced computational complexity [53]. Displays 74 (2022) 102243 9 H. Wang et al. Fig. 10. Performance evaluation using Middlebury test data. (a) left image of stereo image pair, (b) ground truth disparity, and (c) disparity map estimated using AMDCNet. Top to bottom: four types of images are art, baby, dools and book. Table 4 Matching precision of different methods before and after optimization in non-occluded areas. Algorithms cones tsukuba teddy venus avg% No-refine refined No-refine refined No-refine refined No-refine refined AMDCNet 4.10 3.46 2.01 0.92 7.84 6.09 1.35 0.34 3.26 MASNet [44] 4.85 3.25 2.58 1.04 8.04 4.59 1.86 0.64 3.35 PSMNet [45] 4.21 3.98 1.86 0.99 9.51 8.45 0.89 0.29 3.77 RCF [46] 3.97 3.89 2.13 1.29 6.84 6.02 1.73 0.61 3.35 4.4.1. Middlebury AMDCNet is compared with these advanced algorithms from before optimization and after optimization. The targeted area is non-occluded. The comparison of the error matching rate is shown in Table 4. It can be seen from the table that before optimization, the effect of stereo matching depends more on the description of detailed features. The false matching rate of MASNet is higher than AMDCNet. However, after adopting the optimization algorithm, the optimization effect is slightly better for some images. Although the optimization algorithm is better than that of AMDCNet, it needs to learn the features in advance, AMDCNet can be optimized directly through the left–right consistency of features, and the efficiency is relatively high. Through the image Teddy, we can see that the effect of PSMNet will be significantly reduced in high-resolution images, which shows that its general performance is poor. While the RCF has certain advantages in feature convolution learning before optimization, but its optimization stage is carried out by using the optimization algorithm of AMDCNet. Therefore, comprehensively speaking, AMDCNet is better than other methods. Moreover, AMDCNet does not need to rely on high hardware requirements, reduces the feature learning stage, and the equalization ability is more prominent. Fig. 11 shows the results of disparity estimation for Middlebury test images. It can be seen from the baby figure that the results produced by PSMNet show that the baby’s arm and head are obviously connected, and its treatment of detail effect is not good. Although the results obtained by RCF can better retain the details, thanks to its relatively perfect convolution network, it has lost a certain optimization effect, resulting in many unnecessary ‘‘details’’ being exposed, thus reducing the overall matching effect. Other images have similar phenomena. Combined with these comparative experiments, it can be seen that although the method used in deep learning can greatly