HSI-MSER: Hyperspectral Image Registration Algorithm based on MSER and SIFT

IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, VOL. 14, 2021 12061 HSI-MSER: Hyperspectral Image Registration Algorithm Based on MSER and SIFT Álvaro Ordóñez , Álvaro Acción , Francisco Argüello , and Dora B. Heras , Member, IEEE Abstract—Image alignment is an essential task in many applications of hyperspectral remote sensing images. Before any processing, the images must be registered. Maximally stable extremal regions (MSER) is a feature detection algorithm that extracts regions by thresholding the image at different grey levels. These extremal regions are invariant to image transformations making them ideal for registration. The scale-invariant feature transform (SIFT) is a well-known keypoint detector and descriptor based on the construction of a Gaussian scalespace. This article presents a hyperspectral remote sensing image registration method based on MSER for feature detection and SIFT for feature description. It efficiently exploits the information contained in the different spectral bands to improve the image alignment. The experimental results over nine hyperspectral images show that the proposed method achieves a higher number of correct registration cases using less computational resources than other hyperspectral registration methods. Results are evaluated in terms of accuracy of the registration and also in terms of execution time. Index Terms—Hyperspectral imaging, image registration, maximally stable extremal regions (MSER), remote sensing, scaleinvariant feature transform (SIFT). I. INTRODUCTION CURRENTLY, hyperspectral image (HSI) from remote sensing are widely available thanks to the increased availability of sensors. This allows us to obtain images of the same region of the Earth taken from different viewpoints at different times. The series of remote-sensing images are used in applications where it is essential to compare, study, or find differences betweenimages.Automaticchangedetection[1],environmental monitoring [2], or super-resolution image creation [3], among others are applications in which registration is a fundamental Manuscript received May 31, 2021; revised August 24, 2021 and October 8, 2021; accepted November 16, 2021. Date of publication November 18, 2021; date of current version December 8, 2021. This work was supported in part by the Ministerio de Ciencia e Innovación, Government of Spain, under Grant PID2019-104834GB-I00, in part by the Consellería de Cultura, Educación e Universidade under Grant ED431C 2018/19 and Grant accreditation 2019-2022 ED431G-2019/04, and in part by the Junta de Castilla y León under Project VA226P20 (PROPHET II Project). All are co-funded by the European Regional Development Fund (ERDF). The work of Álvaro Ordóñez was also supported by Ministerio de Universidades, Government of Spain, under a FPU Grant FPU16/03537. (Corresponding author: Álvaro Acción.) Álvaro Ordóñez, Álvaro Acción, and Dora B. Heras are with the Centro Singular de Investigación en Tecnoloxías Intelixentes (CiTIUS), Universidade de Santiago de Compostela, 15782 Santiago de Compostela, Spain (e-mail: alv[email protected]; alv[email protected]; [email protected]). Francisco Argüello is with the Departamento de Electrónica e Computación, Universidade de Santiago de Compostela, 15782 Santiago de Compostela, Spain (e-mail: [email protected]). Digital Object Identifier 10.1109/JSTARS.2021.3129099 prior task [4]. The images must be previously aligned in order to work with them afterwards. Imageregistrationalgorithmscanbeclassifiedintoarea-based and feature-based methods [4]. Methods in the first group work directly with image intensity, e.g., Fourier Transform [5] and mutualinformation [6],while thosein thesecond group,featurebased methods, look for information at a higher level, e.g., at the level of regions, lines, or points [7]–[10]. This property makes feature-based methods more suitable for images with illumination changes, which it is the case of remote sensing HSIs of the earth surface. For these images, the atmospheric conditions usually vary from one capture to another. Generally, feature-based methods consist of the following fourstages:feature detection, featuredescription,featurematching, and image transformation [11]. These methods rely on extracting the same features in the images to be registered. Knowing a number of corresponding features, an image transformation that aligns one image with respect to the other can be calculated. Maximally stable extremal regions (MSER) [7] is a featurebased method for region detection in images that can be used for extracting the features needed for a later registration process. This method extracts regions, called extremal regions (ERs), by thresholding the image at different grey levels and according to a stability criterion. If MSER is applied to a pair of images, the extracted and matched regions of both images can be used to compute an image transformation to register them. MSER is resilient to changes of scale, rotation, translation, and illumination conditions. Other well-known feature detector methods in the literature are features from accelerated segment test (FAST) [8] and speeded up robust features scale-invariant feature transform (SURF) [9]. Both build a scale-space where scale-invariant points are detected. But the most popular feature detector and descriptor algorithm is the scale-invariant feature transform (SIFT) [10]. SIFT extracts keypoints from a multiresolution pyramid of the images created performing Gaussian convolutionsandinterpolations.Itsdescriptorstandsoutforbeinghighly distinctive and invariant to illumination and distortion changes, making it a widely used method in the literature [12], [13]. The literature indicates that MSER and SIFT are two of the best region detector [14] and descriptor algorithms [15], respectively. They are used not only for image registration [16] but also in many other applications such as object recognition [17], [18], image retrieval [19], [20], and robot localization [21], [22], among others. MSER is also often used in the literature with the local affine frames (LAF) descriptor [18], [23]. This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ 12062 IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, VOL. 14, 2021 SIFT and MSER-based methods have been previously proposed for image registration [24]–[27]. However, the exploitation of the whole spectral information of HSIs in order to improve the accuracy of the registration process has not been explored and analysed in-depth. To the best of authors’ knowledge, the publications that used SIFT and MSER to register HSIs, as will be detailed in the following, separately reduce each of the images to a single band. New algorithms to deal with the spectral information in HSIs in an efficient way from the point of view of registration accuracy and computational cost need to be designed [28], [29]. The use of spectral information also allows registering images that cannot be registered considering only one band. A band selection method is required for selecting those bands that are most relevant for registration, i.e., it should avoid bands with redundant or low-quality information. Another feature of previous works is that they validate the algorithms considering a few scale factors and rotation angles. Unlike them, in this article, an exhaustive evaluation of the registration algorithms is carried out considering different hyperspectral datasets (urban, rural, crops, and nature scenes) and a wide variety of registration parameters (scale factors and rotation angles). One example of a method that uses the multispectral image as a one-band image, and it is only evaluated on one pair of images is proposed in [24]. The authors propose a method to register a pair of multispectral and visible spectrum images using the keypoints detected and described by SIFT and the regions independently detected and described by MSER and LAF, respectively. Guo et al. [25] presented a multispectral remote sensing algorithm based on MSER for region detection. The regions are described twice using the SIFT descriptor and using a shape descriptor based on the Fourier transform. The method is evaluated on a SPOT2 multispectral and a SPOT2 pan image with different resolution. Moreover, the rotational invariance is not extensively tested, as the only angle difference considered is 30◦. No exploitation of spectral information is performed. Zhanget al. [26]proposedamultisensor registrationmethodthat combines MSER and SIFT, and takes into account the number of matches used and their distribution in order to improve the final transformation. For the experimental analysis, they register two bands of Landsat multispectral images, and a panchromatic image with respect to a RADARSAT SAR image. The first pair has translation changes, and the second has a different scale factorandrotationangle.Liuetal.[27]presentededge-enhanced MSER, a method for multisensor image registration also based onMSERandSIFT. Before detectinganddescribingtheregions, anedge enhancement isapplied to theimage pairs toobtain more stable matches. The method is evaluated on a pair of images with a scale factor of 0.7×and a rotation angle of 35◦.Nouseofthe information from the different bands is detailed, as in previous works. In this work, a registration method for HSIs based on MSER forregiondetectionand SIFT forregiondescriptionispresented. It exploits the spectral information available in the HSIs by performing feature detection and description in several preselected bands of the images to be registered and by incorporating spectral information into the descriptor. The algorithm was designed to deal with extreme situations in terms of scale factor and rotation angle. The main contributions of this article are the following. 1) An efficient registration method that adapts MSER and SIFT to efficiently exploits the spectral information available in HSIs is proposed. The method exploits spectral information in two ways. First, by extracting regions in several bands. Second, the regions are described with a descriptor composed of a spatial and a spectral part as it helps to discard false matches. 2) Theproposalincludesabandselectionmethodspecifically designed for the HSI registration problem. It selects bands according to their entropy and wavelength, and achieves better results than other band selection methods in the literature. 3) An exhaustive histogram-based search is used to estimate the registration parameters. All the possible combinations between the matched regions are taken into account. The method selects the best transformation considering all candidates. 4) An in-depth evaluation of the method is carried out. The evaluation is performed over nine pairs of HSIs taken by different sensors at different dates. Moreover, the set of images is extended by applying a wide range of scale factors and rotation angles. The rest of this article is organized as follows. Section II describes the different stages of the proposed method; the results are discussed in Section III. Finally, Section IV concludes this article. II. HYPERSPECTRAL MSER Inthissection, we presenthyperspectralMSER(HSI-MSER), a registration method to align two hyperspectral remote sensing images based on MSER as region detector followed by SIFT as feature descriptor. The method exploits the spectral information availableintheimages.First,thestandardversionsofMSERand SIFT are described. Then, the proposed method is presented. A. Maximally Stable Extremal Region MSER is a method for region detection in greyscale images [7]. It has been successfully applied to a large number of applications such as image recognition, tracking, and image registration [30], [31]. The algorithm extracts a number of regions called MSERs by thresholding the image at different grey levels and according to a stability criterion. An ER is a region in which all pixels within the ER have higher intensity values (for the bright ERs) or lower intensity values (for the dark ERs) than all the pixels on the outer boundary of the region. The outerboundary ofa regionis definedas theset ofpixelsthat meet two conditions: being adjacent to one, two, or three pixels in the region and not belonging to the region. An ER is considered stable (an MSER) when it does not change substantially as the grey level threshold is varied [7]. MSER presents two properties that make it ideal for image registration [14]. First, linear or affine transformations do not ORDÓÑEZ et al.: HSI–MSER: HYPERSPECTRAL IMAGE REGISTRATION ALGORITHM BASED ON MSER AND SIFT 12063 Fig. 1. Proposed HSI-MSER schematic for registration of two HSIs. affecttheextractedERs becausetheyonlydependon pixelintensities that are preserved under these monotonic transformations. Second, a set of regions is preserved after applying geometric and photometric changes because an ER will continue to be an ER after these transformations. A reference implementation of MSER can be found in [32] although it only considers RGB images in the range [0,255]. In this work, the original range of the input HSIs is considered and the algorithm is extended to an arbitrary number of bands. B. SIFT Descriptor The SIFT is one of the most popular feature detectors and descriptors. In this work, we use the descriptor part to calculate the description of each region previously detected by MSER. The steps to compute the SIFT descriptor of each region are the following. First, the dominant angles are calculated for each region to achieve invariance to image rotation. An orientation histogram with 36 bins covering 360◦is created from the gradient orientations within the surface of each region. Then, it is weighted by gradient magnitude and by a Gaussian-weighted circular window. The highest peak in the histogram is selected as the dominant orientation. Moreover, any peak above 80% of it is also taken into account. That means that we will have regions with the same location but different orientations. The next step is the descriptor construction. First, an area of size 16 ×16 pixels centred around each region is selected. The area is divided into subareas of 4×4pixels. For each subarea, an orientation histogram with 8 bins is created. Finally, a128-parameter descriptorforeach regionis generatedfromthis set of weighted histograms. To reduce the influence of boundary effects, brightness and illumination changes, the descriptor values are thresholded and normalized to unit length. C. HSI-MSER In this section, we present HSI-MSER, a registration algorithm for HSIs based on MSER and SIFT. HSI-MSER seeks to find a similarity transformation that successfully aligns two hyperspectral remote sensing images. One of the images is called the reference image. The other, the image that we want to register with respect to the reference image, is called the target image. The method consists of the following six stages: band selection,regionextraction,regiondescription,regionmatching, band combination, and registration. A schematic of the proposed algorithm can be seen in Fig. 1. 1) Band Selection: It is common that contiguous bands do not differ in relevant information for the registration process. Band selection allows reducing the computational cost with respect to considering all the bands of the image but keeping onlytherelevantinformation.Forthisreasoninthefirststage,the most relevant spectral bands of the reference and target images are selected. Out of the methods available in the literature for band selection of HSIs, principal component analysis (PCA) [33], BandClust[34], andward’slinkage strategyusing mutualinformation (WaluMI) [35] were evaluated. All perform the selection by considering each HSI individually. The entropy-based selection method used in this work considers both images of the dataset jointly. The method consists in selecting the Nbands with the highest entropy but separated by at least Dconsecutive bands. Asthe bandsformingan HSI areordered by wavelength,Disthe minimum number of bands between each pair of selected ones. This ensures that the selected bands differ in both entropy and wavelength. We call this method entropy-based band selection (EBS). First, the entropy of each band is calculated for both images, i.e., two entropy values are obtained for each band, and the minimum value of each pair is assigned to the band. Then, the bands are ordered according to decreasing values of entropy. The first band selected is the one with the maximum entropy. The next band selected will be the next with the highest entropy but separated by, at least, a distance of Dbands, as indicated earlier. This step is repeated until Nbands are selected. If it is notpossible tofinda bandthatfulfils thiscondition, Disreduced by one band and the process is restarted. 2) Region Extraction: The second stage consists in extracting regions from the HSIs. The region extraction is applied to each band selected in the previous stage. 12064 IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, VOL. 14, 2021 Let Ibe an HSI consisting of H×Wpixels indexed by the variable xand Bspectral bands. Let Bb(x)bethegreylevel value of a pixel xin the selected spectral band b. Let also L= [min(Bb(x)),max(Bb(x))],x∈Ibe the grey level range in the bband. The extracted regions in the bband for the greyscale level l∈Lare transformed into ellipses μl=1 |Rl| x∈Rl x, Σl=1 |Rl| x∈Rl (x−μl)(x−μl)(1) where μland Σlare the mean and variance of the pixels composing the region, and Rlis an ER detected in this band for the greyscale level l[36]. The aim of this stage is to detect a large number of common structures in both HSIs that will then be used to calculate the transformation to align the images. HSI-MSER is specifically designed to deal with spectral information because some structures are only perceptible in some bands. 3) Region Description: In the third stage, the extracted regions are described. The algorithm is designed to register HSIs. Thisrequiresthatthedescriptoris madeupofspatialandspectral information in order to achieve better alignments. The SIFT descriptor is used for the spatial part. For each ellipse, the coordinates of its centre are considered as the coordinates of the region. The SIFT descriptor is computed on the surface of the region bounded by the ellipse as explained in Section II-B. Thisspatialdescriptorisenrichedbyaspectralpart,inparticular, the spectral signature of the keypoint, which is defined as the pixelvectorofthecentreofthe ellipsesince theregionsextracted by MSER are homogeneous. Both parts are concatenated to form a descriptor that takes into account both the spatial and the spectral information. The descriptor is a vector made up of 128 components for the spatial part plus the number of bands of the original HSIs as the spectral part. 4) Region Matching: Then, in the fourth stage, the regions of each pair of bands (one band for each HSI) are matched independently, i.e., withouttaking intoaccountthe regionsof the other pairs of selected bands. Although the bands are matched independently, the spectral information of the other bands has been taken into account. In particular, the Euclidean distance is used for the spatial part of the descriptor, and the cosine similarity for the spectral part. The process for matching the regions of both images consists in calculating the distances between their descriptors. Given a region in the reference image, the best candidate match in the target image is the one with the closest distance. However, some regionsarenot detectedin the targetimage, whichmeans that we will get a false match. Therefore, a method is needed to discard false matches. The method for region matching consists of two steps. First, the Euclidean distances between each region of the reference band and all of the regions of the target band are computed. The Euclidean distances are calculated on the spatial part of the descriptor, i.e., the SIFT descriptor. Given a region in the reference image, the region closest to it in the target image is considered a possible match if the ratio between the distances to thetwoclosest regionsissmaller thana distanceDspatial.Second, to finally be considered a match, the cosine similarity between the spectral signatures of the centre of the regions must be higher than Sspectral. The spectral information allows discarding false matches in this second step. Dspatial and Sspectral were experimentally set at 0.7 and 0.95, respectively. These are the tradeoff values that achieve good results in terms of number of successfully registered cases for the whole dataset with a moderate computational cost. 5) Band Combination: As some regions are only detected in some bands, all matched regions extracted from the selected bands are joined in the fifth stage, i.e., all the regions extracted from the different bands are considered in the same pool. Thus, regions that are only present in some bands are used to compute the transformation, i.e., all the spectral information is considered together. 6) Registration: Finally, in the sixth stage, an exhaustive histogram-based search is performed to register the images. The method computes a possible transformation for each combination of two matched regions. A selection is then carried out based on all the rotation angles and scale factors obtained. The procedure is as follows. First, a scale factor, a rotation angle, and translation parameters are computed from each combinationoftwomatchedregions,ascanbeseeninFig.1.Second, a histogram with the rotation values is calculated. As we want a robust method against rotations, the 360◦have been divided into 72 bins, i.e., the bin size is 5◦. It was selected based on experiments and following the recommendations by [37]. This allows obtaining bins with a considerable number of elements (transformations), higher accuracy in terms of rotation angle for a first estimation, and a well-defined peak. Bin sizes of 2.5◦and 10◦were also considered, obtaining worse results in terms of number of successfully registered cases. Moreover, a 2.5◦overlap between bins has been defined. The overlap of 2.5◦allows having a flexible boundary between bins, so each angle could contribute to different bins, for example, an angle of 7.5◦contributes to the bins centred in 5◦and 10◦. Once the histogram is built, the elements (transformations) of the bin of highest frequency are sorted by the scale factor to obtain the median, which is a measure that is more robust to outliers than the mean. Finally, the scale factor ρ, rotation angle θ, and translation parameters (x, y)of the median element are selected to register the HSIs. III. RESULTS In this section, the results obtained by the HSI-MSER method using different hyperspectral remote sensing images are presented. First, the experimental conditions and test images are described in Section III-A. In Section III-B, the proposed band selection method is compared to other methods in the literature. Then, in Section III-C, an analysis exploiting different numbers of bands in the first stage of the algorithm (see Fig. 1) is carried out. In Section III-D, HSI-MSER is compared to other hyperspectral registration algorithms in the literature in terms of number of successfully registered cases, ORDÓÑEZ et al.: HSI–MSER: HYPERSPECTRAL IMAGE REGISTRATION ALGORITHM BASED ON MSER AND SIFT 12065 Fig. 2. HSIs commonly used for testing in remote sensing: (a) Pavia University, (b) Pavia Centre, (c) Indian Pines, (d) Salinas. correct number of matches and root-mean-square error (RMSE) among other common measures of registration accuracy. Finally, in Section III-E, the execution times of HSI-MSER are presented and compared to those of other algorithms in the literature. A. Experimental Conditions and Dataset This section presents the experimental conditions and test images as well as some experimental results. The experiments were carried out on a PC with a quad-core Intel i7-4790 CPU at 3.60 GHz and 24 GB of RAM. The code was written in C and compiled using the gcc and the g++ 7.5.0 versions under Ubuntu 18.04. The test procedure consists in registering one image, called reference image, with respect to a second image, called target image,whichpresentschangesofscale,rotation,andtranslation. The evaluation of the proposed method was performed over nine hyperspectral remote-sensing scenes [38] that can be divided into two groups: images frequently used in the literature, for which only one image is available, and pairs of images taken by the airborne visible/infrared imaging spectrometer (AVIRIS) sensor at different dates. The first group contains scenes of rural places and cities. The target image, the image we want to align with respect to the original, is generated by scaling and rotating the original images (called reference images). In this way, we can investigate all the registration details in controlled conditions. The generation of the target images will be explained in more detail later. A colour composition of these images is presented in Fig. 2. 1) Pavia University: It is an urban area surrounding the University of Pavia, Italy, taken by the reflective optics system imaging spectrometer (ROSIS) sensor and it is made up of 103 spectral bands. The image size is 610 ×340 pixels with a spatial resolution of 1.3 m/pixel. 2) Pavia Centre:Itisan ROSISimageof102bands ofthecity of Pavia, Italy. The image size is 1096 ×715 pixels with a spatial resolution of 1.3 m/pixel. A 381-pixel-wide black vertical band in the middle of the image was removed. 3) Salinas: It is a rural scene taken by the AVIRIS sensor in the Salinas Valley, California. This image has a size of 512 ×217 pixels with a spatial resolution of 3.7 m/pixel and 204 spectral bands because 20 bands covering the region of water absorption were removed [39]: 108–112, 154–167, and 224 4) Indian Pines: It was collected by the AVIRIS sensor over a rural area in NW Tippecanoe County, Indiana. It is a 220band image because four noisy bands were removed [40]: 1, 33, 97, and 161. The image size is 145 ×145 pixels with a spatial resolution of 20 m/pixel. The second group consists of pairs of images of the same urban or rural scene taken at different dates by the AVIRIS sensor. Due to this, changes in infrastructure, vegetation, buildings, crops, and others, are present, as well as, different scale factors, rotation angles, and translations. The oldest image of each pair is used as reference image and the most recent one as target image. A colour composition of these images is presented in Fig. 3. 1) Jasper Ridge: As their name implies, these images cover a region of the Jasper Ridge Biological Preserve, California. They include a forest, a lake, some buildings, vegetation, roads, and parts of bare soil, among others. The first image was taken in 2006 with a spatial resolution of 3.3 m/pixel and the second one in 2007 with a spatial resolution of 3.4 m/pixel. The second image has a drier appearance with respect the first one. The size of the images is 1286 ×588 pixels and 224 spectral bands. 2) Santa Barbara Box: These images were taken in a cultivation area near the city of Santa Barbara, California. The first image was taken in 2009 with clear weather and the second one in 2010 with a few clouds. Changes in crops are also present. There have been changes in the crops from one year to the next. Both images have a size of 1024 ×769 pixelswith aspatialresolutionof 15.2m/pixel and 224 spectral bands. 3) Santa Barbara Front: These images were collected over the coastal city of Santa Barbara, California. They include an urban area with numerous buildings, some vegetation, as well as the coastline. They are 224-band images with a 12066 IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, VOL. 14, 2021 Fig. 3. Jasper Ridge, Santa Barbara, Baraboo Hills, and Crown Point scenes taken for the AVIRIS sensor. (a) Jasper image taken on May 12, 2006, (b) Jasper image taken on August 13, 2007, (c) Santa Barbara Front taken on March 30, 2009, and (d) fragment of the second Santa Barbara Front image taken on April 30, 2010, (e) Santa Barbara Box image taken on April 11, 2013, (f) Santa Barbara Box image taken on April 16, 2014, (g) Baraboo Hills image taken on July 13, 2008, (h) Baraboo Hills image taken on August 26, 2010, (i) Crown Point image taken on September 30, 2010, (j) Crown Point image taken on November 16, 2011. size of 900 ×470. The first image was taken in 2013 and the second one in 2014. Both images differ in the spatial resolution, which is 16.4 m/pixel for the first image and 11.3 m/pixel for the second one. 4) Baraboo Hills: As the name indicates, these images were taken in Baraboo, Wisconsin. In the image, we can see a part of the city of Baraboo, as well as the Devil’s Lake State. They also include forests, vegetation, crops, clouds, androads,amongothers.Thefirstimagewastakenin2008 with a spatial resolution of 16.9 m/pixel and the second one in 2010 with a spatial resolution of 11.7 m/pixel. Both are 224-band images with a size of 1200 ×710. 5) Crown Point: These images were taken in Crown Point, which is a rural neighborhood located in Marrero, Louisiana. In addition to the urban area, they also include vegetation, a river, canals, and roads. Both images have asizeof1400 ×540 pixels and 224 spectral bands. The first image was taken in 2010 with a spatial resolution of 3.5 m/pixel and the second one in 2011 with a spatial resolution of 3.4 m/pixel. In order to evaluate the registration capabilities under extreme scaling and rotating conditions, and investigate all the registration details also in controlled conditions, a comprehensive set of target images was created. This set was generated applying a range of scale factors from 1/9×to 16.5×(40 scale factors) and rotation angles from 0◦to 360◦in increments of 5◦(72 angles). From 1.0×to 16.5×the images are scaled up in steps of 0.5×, as shown in Fig. 4, while from 1/2×to 1/9×the images are scaled down (scale factors of 1/i×,i=2,3,...,9). Inthecase of theimagesinthe secondgroup,theseparameters are applied to the most recent image, as mentioned above. In all cases, the target images are trimmed on the central region to keep the same size as the original images. The test consists in registering each target image (the generated ones) with respect to the reference image. Theregistrationalgorithmobtainsangle,scale,andtranslation parametersas output.The registrationis consideredcorrect ifthe parametersobtained by thealgorithmare thesameas the original values. B. Evaluation of Band Selection Methods Inthissection, theevaluation ofdifferentbandselectionmethods in the first stage of the proposed algorithm is presented. EBS is compared to PCA [33], BandClust [34], and WaluMI [35]. PCA is a well-known dimensionality reduction method. It generates a new set of linearly uncorrelated variables where the first few retain most of the data variation [33]. The idea is to ORDÓÑEZ et al.: HSI–MSER: HYPERSPECTRAL IMAGE REGISTRATION ALGORITHM BASED ON MSER AND SIFT 12067 TABLE I SUCCESSFULLY REGISTERED CASES FOR EACH SCENE USING DIFFERENT BAND SELECTION METHODS IN THE FIRST STAGE OF HSI-MSER: A RANDOM BAND OF EACH SCENE (IN THIS CASE BAND 88), PCA, BANDCLUST,WALUMI, AND EBS The range indicates the scales successfully registered for the 72 angles. The numbers in parentheses summarize the number of scales that were correctly registered for all angles. If an angle is incorrectly registered, the whole scale factor is considered incorrect, i.e., this case is not included in the table. The registration is considered correct if the parameters obtained by the algorithm are the same as the original values. Fig.4. Methodforcreatingthecomprehensivesetof targetimages byapplying different scales and rotations. eliminate data redundancy while preserving relevant information. BandClust is an unsupervised recursive binary band-splitting algorithm [34]. It iteratively splits a band interval into two disjoint contiguous sets based upon a criterion of minimization of the mutual information, i.e., the method automatically determines the optimal number of bands. Finally, the bands of each set are averaged. WaluMI performs a hierarchical clustering based on the Ward’s linkage method [35]. It groups bands to minimize the intraclustervarianceandmaximizetheinterclustervariance.The distance used is based on the mutual information between each pair of bands. In the end, WaluMI chooses the most representative band of each cluster. Table I summarizes the cases that were correctly registered for each scene using a single band, randomly selected for each image, or a set of bands extracted by these methods. As explained in Section III-A, the registration is considered correct if the parameters obtained by the algorithm are the same as the original values. In the case of the band selection method used in the proposal, EBS, two parameters must be fixed: the number of bands to be selected Nand the minimum distance between the selected bands D.Nis set to 8, while Dis set to 20, as we want to select bands with different wavelengths to keep all the relevant information. This configuration is called EBS 8. The same number of bands have been selected for the state-of-the-art methods, with the exception of BandClust in which the method itself determines the optimal number of bands. The last row of Table I shows the average number of scalings, i.e., the sum of the number of scales per scene that were correctly registered for all angles divided by the number of scenes. As shown in Table I, better results are obtained when the spectral information is exploited by considering several bands. This allows detecting features that are only present in some bands. Fig. 5 illustrates this statement. It shows an example of matching for two pairs of bands selected by EBS for the Jasper Ridge images. It can be observed that some features are only present and detected in some spectral bands. PCA is the exception to the rule. It provides worse results than using only one band. The reason is that PCA applies different transformations to the reference image and to the target images obtaining 8 different principal components (PCs) for each one. This results in a small number of common regions. The results in Table I show that using the proposed band selection method, EBS, 20.86 cases are correctly registered on average (for all the scenes), more than twice the number of cases achieved using only one band (9.71 cases). C. Results Exploiting Different Numbers of Bands As explained in Section II-C, HSI-MSER exploits spectral informationin twoways.First, bysearching fordifferentregions in selected bands, and second, by incorporating the spectral information into the descriptor. In this section, we analyze the effect of selecting a different number of bands in the first stage, i.e., different values for Nin EBS will be evaluated. Dis set to 20 as in the previous section. The test procedure is as explained in the previous section. A total of 40 scale factors from 1/9×to 16.5×and 72 rotation angles from 0◦to 360◦are applied to the target image. Table II 12068 IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, VOL. 14, 2021 Fig. 5. Matched ellipses detected in two pairs of bands of the Jasper Ridge images. Matches (blue), and matches used in registration (green). (a) Band 8. (b) Band 20. TABLE II SUCCESSFULLY REGISTERED CASES FOR EACH SCENE USING HSI-MSER EXPLOITING DIFFERENT NUMBERS OF BANDS: 2, 4, 6, 8, 10, 12, 14, AND 16 The range indicates the scales successfully registered for the 72 angles. The numbers in parentheses summarize the number of scales that were correctly registered for all angles. If an angle is incorrectly registered, the whole scale factor is considered incorrect, i.e., this case is not included in the table. The registration is considered correct if the parameters obtained by the algorithm are the same as the original values. summarizes the cases that were correctly registered for each scene by exploiting different numbers of bands selected by EBS, from 2 to 16 bands in steps of 2. It shows that the more bands we used, the better the results. The HSI-MSER using 16 selected bands by EBS provides the best results on average, correctly registering 21.22 cases as compared to 11.78 cases registered using 2 bands. For EBS 8, the number of cases correctly registered is 19.00, which are very close in quality to the results obtained by EBS 16 but with lower computational cost. For that reason, we chose EBS 8 as the default configuration for the proposed method. The computational cost will be evaluated in Section III-E. ORDÓÑEZ et al.: HSI–MSER: HYPERSPECTRAL IMAGE REGISTRATION ALGORITHM BASED ON MSER AND SIFT 12069 TABLE III COMPARISON OF DIFFERENT HYPERSPECTRAL REMOTE-SENSING IMAGE REGISTRATION METHODS REGARDING THE SUCCESSFULLY REGISTERED CASES FOR EACH SCENE: HYFM, HSI-SURF, AND THE PROPOSED METHOD HSI-MSER The range indicates the scales successfully registered for the 72 angles. The numbers in parentheses summarize the number of scales that were correctly registered for all angles. If an angle is incorrectly registered, the whole scale factor is considered incorrect, i.e., this case is not included in the table. The registration is considered correct if the parameters obtained by the algorithm are the same as the original values. TABLE IV REFERENCE REGISTRATION PARAMETERS FOR THE SECOND GROUP OF TEST HSIS TABLE V COMPARISONS OF HSI-SURF AND THE PROPOSED METHOD HSI-MSER (8 BANDS)REGARDING THE NUMBER OF MATCHES OBTAINED FOR EACH SCENE HYFM is not included as it does not look for matches. D. Comparison to Other Methods in the Literature In this section, the proposed method is compared to other hyperspectral registration algorithms in the literature: the hyperspectral Fourier–Mellin (HYFM) [41], and the hyperspectral SURF (HSI-SURF) algorithm [42]. Both algorithms exploit 8 spectral bands to register two HSIs. The comparison is made in terms of range of successfully registered cases for each scene, number of matches, number of correct matches, RMSE, registration error, and computational time. HYFM is an area-based method, which performs a PCA to reduce the dimensionality and extracts 8 PCs for each HSI [41]. One log-polar grid for each pair of PCs is computed using the TABLE VI REGISTRATION RESULTS FOR HSI-SURF AND HSI-MSER (8 BANDS) REGARDING THE NUMBER OF MATCHES REALLY USED TO COMPUTE THE REGISTRATION PARAMETERS HYFM is not included as it does not look for matches. adaptable multilayer fractional Fourier transform. The different log-polar grids are combined to integrate the information from the different PCs. The highest peaks in the combined log-polar grid are examined to determine the scaling, rotation, and translation parameters. HSI-SURF is a feature-based method [42]. It extracts keypoints in 8 selected bands for each image. The method is based on SURF [9] algorithm as keypoint detector and descriptor, and considers the spectral information of the images in the band selection, keypoint description, and keypoint matching stages. It uses a band selection method based on entropy as in the HSI-MSER. A comparison between HYFM, HSI-SURF, and HSI-MSER regarding the number of successfully registered cases is presented in Table III for the same number of extracted bands.