Overcoming segmentation confusions in PRISMA hyperspectral images

Overcoming segmentation confusions in PRISMA hyperspectral images Serban Oprisescu1, Andrei Racoviteanu1,2, Mihai Ivanovici1 1 Electronics and Computers Department, Transilvania University of Brașov, România {serban.oprisescu, mihai.ivanovici}@unitbv.ro 2 Image Processing and Analysis Laboratory, National University of Sci. & Tech. Politehnica, București, România [email protected] Abstract—Hyperspectral satellite images (HSI) which contain hundreds of spectral bands, offer the possibility of a much finer characterization of ground surfaces. Agricultural crop identification and segmentation remains one of the most used techniques for characterization of land use. Even though HSI allows better segmentation results than in the case of multispectral data, there are still challenges when different crops look the same at certain development stages. This paper proposes two methods for segmentation confusion elimination. The first method relies in analyzing the histogram of SAM values obtained by comparing different parcels. The second method relies on the tdistributed Stochastic Neighbor Embedding (t-SNE). We illustrate the efficiency of our proposed methods with PRISMA hyperspectral images from the region of Brasov, Romania. I. INTRODUCTION Smart agriculture is one of the main research areas today, considering the challenges introduced by climate change. Artificial intelligence applied on the data provided by satellites and ground sensors aims to obtain better yield, to detect diseases, drought etc. Optical satellite imaging has evolved from multispectral (which is still used today within satellites such as Sentinel 2) to hyperspectral. Hyperspectral satellite imaging (HSI) offers ground images in hundreds of narrow spectral bands. For instance, the PRISMA (Hyperspectral Precursor of the Application Mission) satellite built by ASI (Agenzia Spaziale Italiana) [1] offers 239 spectral bands with a spectral resolution lower than 12nm, in the range 400 – 2500 nm. Agricultural crop identification and classification based on hyperspectral images is an ongoing research field. Mainly there are two approaches: crop identification based on a single satellite image or based on a time series (several satellite images taken at some time intervals). The time series crop identification tries to classify different crops based on the knowledge of their growth characteristics in time. In this paper we focus on the agricultural crop identification based on a single hyperspectral satellite image. As shown in [2], there are many techniques which can be used for hyperspectral image segmentation, such as: Support Vector Machine (SVM), Extended Morphological Profile (EMP), Joint Sparse Representation (JSR), 3D Convolutional Neural Network (3D-CNN), CNN with Point Pair Features (CNN-PFF), Gabor-based Convolutional Neural Network (Gabor-CNN), 3D Generative Adversarial Network (3D-GAN), Deep Feature Fusion Network (DFFN) etc. The accuracy of the segmentation obtained using any of these methods varies [2] function of the number of classes, the characteristic of the image itself, the amount of training samples etc. There is no universal method, and the results depend on each classification scenario. In our segmentation experiments on PRISMA hyperspectral images, we noticed that the accuracy strongly depends on the selected classes (or the selected crop types) and their development stage. There are crops which can be easily discerned within the first weeks of development, such as wheat and rapeseed. And there are crops that can be misclassified such as the confusion between wheat and alfalfa in the first stage of development. In this paper we address the problem of crop segmentation confusion by proposing a method based on histograms of spectral angle mapper (SAM). The rest of the paper is structured as follows. Section II describes the segmentation confusions and challenges and depicts the proposed methodology. Section III includes the results obtained on selected hyperspectral crops. Section IV concludes the paper. II. THE PROPOSED METHOD First of all, we would like to illustrate a case when pixel-wise segmentation fails because of the high similarity between the spectral signatures belonging to two different crops. Fig. 1 shows a crop (RGB band selection) from a PRISMA image – the area of Brasov, Romania, 23 Mars 2024. Two parcels are highlighted: grassland (P1) and rapeseed (P2). Figure 1. A grassland parcel (P1) and a rapeseed parcel (P2) Fig. 2. shows the spectral signatures (the spectral reflectance curves – SRC) for all the pixels within the two highlighted parcels from Fig. 1. Figure 2. The SRCs for all pixels from the parcels of Fig. 1 One notices that the SRCs for the pixels from the two parcels are very similar, which leads to segmentation confusions. To show that confusions appear between the two parcels, we performed a Random Forest (RF) segmentation on the PRISMA crop, choosing five classes: rapeseed, grassland, wheat, buildings and bare soil. Fig. 3.a) shows the training areas. Figure 3. a) Training areas for the five classes (rapeseed, grassland, wheat, buildings and bare soil); b) RF segmentation result. The confusion area is marked with a circle The result of the RF segmentation is shown in Fig. 3. b). One notices, as expected, that a rapeseed parcel (marked with a circle) is segmented as grassland. That area should be red. These confusions appear if we try other segmentation methods such as K-means or Fuzzy C means (FCM). We cannot try deep learning methods on the selected image because of the very limited training set. But the cause of the confusion is clearly the similarity between the SRCs in this development stage of the two crops. To overcome the confusion, we propose in this paper the use of histograms of Spectral Angle Mapper (SAM) [3] values. SAM compares two spectra; in our case, as shown in (1), r denotes a reference SRC and t a test SRC. The value of SAM should be zero if the two SRCs are identical. The proposed method is explained in Algorithm 1. The first step in our algorithm is the image transformation using the Minimum Noise Fraction transform (MNF), reducing the number of bands to 30. We then choose two parcels, a reference parcel P1 and a test parcel P2, and build a matrix denoted ISAM which contains the SAM values computed on all pairs of pixels between the two parcels. And then we compute the histogram of the ISAM matrix. Algorithm 1 The proposed method Input: the hyperspectral image I Output: H (the histogram of SAM values) I = MNF(I) manually choose two parcels P1 and P2 from I for each pixel i of P1 for each pixel j of P2 denote r the SRC of the i pixel denote t the SRC of the j pixel compute ISAM(i,j) = the SAM value between r and t compute H = histogram(ISAM) (1) III. EXPERIMENTAL RESULTS We give here an example where we selected four parcels as shown in Fig. 4.a) P1 is the reference parcel (rapeseed). The test parcels are: P2 (rapeseed), P3 (grassland) and P4 (wheat). The crop is from the same PRISMA hyperspectral image, Brasov region, 23 Mars 2024. Fig. 5 shows the obtained histogram of SAM values. Algorithm 1 was applied on the three pairs of parcels, P1-P2, P1-P3 and P1-P4 and all obtained SAM values were concatenated in a single vector. The purpose of such experiments is to propose a method for crop type identification. Thus, one takes a reference known parcel P1 and then select many other parcels (P2, P3, …, Pn). When we obtain the histogram of SAM values from all P1-Pi pairs of parcels, the parcels with the same crop will be the first. Because SAM computes the similarities between all parcels pixels, and smaller SAM values indicate identical crops in our scenario. This is easily noticed in Fig. 5 where the P1-P2 (rapeseed-rapeseed) histogram is the first. The proposed method resembles the image indexing methods, where the system receives a query image and returns all similar images in decreasing order of similarity. Figure 4. a) The reference parcel P1 (rapeseed) and the test parcels P2 (rapeseed), P3 (grassland) and P4 (wheat); b) Four parcels: P1 (grassland – lower right), P2 (grassland – right), P3 (rapeseed) and P4 (rapeseed) Figure 5. The SAM histograms between P1-P2, P1-P3 and P1-P4 As we mentioned at the beginning of this study, we addressed the confusions which may appear between crops at certain development stages. We also applied for these cases the tdistributed Stochastic Neighbor Embedding (t-SNE) [5] which is a powerful dimensionality reduction and data visualization technique. t-SNE is an unsupervised non-linear method that focuses on preserving the similarities and relationships between data points in a lower dimensional space. We applied t-SNE on the four parcels from Fig. 4.b) The result is shown in Fig. 7. The two grassland parcels (P1 and P2) group together, while the two rapeseed parcels (P3 and P4) are separated because of the difference in development stage. Anyway, one can see in Fig. 7 that the confusion between grassland and rapeseed that was obtained after RF segmentation (Fig. 3 b)) is no longer visible after t-SNE. Hence, t-SNE could also be used for discriminating between different crop parcels, but as we saw, it is sensitive also to intra-class variations that are caused by development stages. Fig. 6 shows the proposed method applied on the pairs of parcels from Fig. 4 b) when we take P1 (grassland) as reference. The separation between grassland and rapeseed is visible in Fig. 6. Figure 6. Histogram of the pairs of parcels from Fig. 4.b) Figure 7. t-SNE of the pairs of parcels from Fig. 4.b) Finally, in Fig. 8 one shows a selection of five parcels that have much lower confusion between them. These parcels are: P1 (wheat), P2 (rapeseed), P3 (rapeseed), P4 (wheat) and P5 (wheat). We applied our algorithm on these parcels, taking as reference the wheat parcel (P1). The obtained histogram is shown on Fig. 9. One clearly sees the much bigger separation between the pairs of wheat parcels (P1-P4 and P1-P5) and the pairs of different crop parcels. Figure 8. The reference parcel P1 (wheat) and the test parcels: P2 (rapeseed), P3 (rapeseed), P4 (wheat) and P5 (wheat) Figure 9. Histogram of pairs of parcels from Fig. 8 Fig. 10 shows the empirical cumulative distribution functions (ECDF) of the SAM values corresponding to the parcels from Fig. 8. One notices that the wheat plots (P1-P4 and P1-P5) group together i.e. the distance on the X-axis between them is small and they are near the origin of the X-axis. The combinations between different crops (wheat-rapeseed) P1-P2 and P1-P3 also group together but at a greater distance than the curves plotted for the same crop (wheat-wheat). To have a numerical measure of the similarity between these plotted distributions, we computed the two-sample Kolmogorov-Smirnov test (kstest2) and the Jensen–Shannon divergence (JSD). Ktest2 is a great statistical measure that tells if two samples came from the same distribution (in our case if the crop type is the same). The value of the test is called k1. A value for k1 close to zero affirms the null hypothesis (that the samples are from the same distribution); a value of k1 close to one rejects the null hypothesis (the parcels contain different crop types). Table I shows the obtained values for Fig. 10. The JSD is an entropic measure of divergence between two probability distributions based on the Kullback-Leibler divergence. The obtained values are shown in Table I and have the same meaning as the k1 values. Hence, by a simple threshold on k1 or the JSD value we are able to decide if the compared parcels have the same crop (small values) or not. Figure 10. ECDF of parcels from Fig. 8 TABLE I. SAM DISTRIBUTIONS SIMILARITY METRICS Pairs of parcels K 1 value JSD value P1-P2 vs P1-P3 0.36 0.16 P1-P3 vs P1-P4 1.00 0.99 P1-P4 vs P1-P5 0.26 0.07 P1-P2 vs P1-P4 1.00 0.99 IV. CONCLUSIONS The identification and segmentation of agricultural crops applied on satellite images still has its challenges mainly due to the similarity of certain crops within some development stages. In this paper we proposed two methods for crop identification and discrimination. The first method relies on analyzing the histogram of SAM values computed pixelwise on pairs of SRCs from selected parcels. The proposed method resembles image indexing and searching techniques, by taking a known reference parcel and compare it with a number of different parcels. The parcels having the same crops as the reference parcel will be plotted first in the obtained histogram. The second method consists of visualizing the 2D t-SNE of the selected parcels. This later method, even though it clearly separates the different crops, also separates the same type crops that are in slightly different development stages. Kstest2 and JSE have been used to prove that our method discriminates well between crops. ACKNOWLEDGMENT This work was funded by the European Union, from the AI4AGRI project. The AI4AGRI project entitled “Romanian Excellence Center on Artificial Intelligence on Earth Observation Data for Agriculture” received funding from the European Union’s Horizon Europe research and innovation program under grant agreement no. 101079136. REFERENCES [1] ASI website: www.asi.it/en/earth-science/prisma [2] S. Li, W. Song, L. Fang, Y. Chen, P. Ghamisi and J. A. Benediktsson, "Deep Learning for Hyperspectral Image Classification: An Overview," in IEEE Transactions on Geoscience and Remote Sensing, vol. 57, no. 9, pp. 6690-6709, Sept. 2019, doi: 10.1109/TGRS.2019.2907932. [3] F.A. Kruse et al, "The Spectral Image Processing System (SIPS) - Interactive Visualization and Analysis of Imaging spectrometer Data." Remote Sensing of Environment 44 (1993): 145-163. [4] A. A. Green, M. Berman, P. Switzer, and M. D. Craig, 1988, A transformation for ordering multispectral data in terms of image quality with implications for noise removal: IEEE Trans. on Geos. and Remote Sensing, v. 26, no. 1, p. 65-74. [5] L. Van der Maaten and G. Hinton, Visualizing Data Using t-SNE. Journal of Machine Learning Research, 1, 1-48, 2008