Identification of clutter echoes using a fuzzy logic technique

IDENTIFICATION OF CLUTTER ECHOES USING A FUZZY LOGIC TECHNIQUE Marc Berenguer1,*, Daniel Sempere-Torres1, Rafael Sánchez-Diezma1, Ramon Pascual2 (1) Grup de Recerca Aplicada en Hidrometeorologia. Universitat Politècnica de Catalunya, Barcelona (Spain) (2) Centro Meteorológico Territorial. Instituto Nacional de Meteorología, Barcelona (Spain) 1. INTRODUCTION Quality Control of radar measurements is one of the most important issues before implementing them for quantitative uses. In particular, one of the sources of error that affect radar data is the presence of echoes caused by nonmeteorological targets (such as important buildings or the orography). If clutter echoes are not identified and removed, they may cause an overestimation of rainfall and this would also affect the performance of automatic algorithms based on radar data (i.e. radarbased nowcasting algorithms). On the other hand, the Anomalous Propagation (AP) of the radar beam significantly enhances the magnitude of this problem: in AP conditions, the size and intensity of mean clutter echoes change and new (sometimes very important) echoes may appear. Propagation conditions mainly vary with atmospheric conditions (they are frequently associated to temperature inversions and/or negative vertical gradients of humidity -see Battan 1973; Steiner and Smith 2002-). In this study we propose an algorithm for clutter identification based on the analysis of different features of the radar field (the echotop, the vertical gradient of reflectivity,...), using fuzzy logic concepts (see Kosko, 1992; Mendel, 1995). 2. RADAR DATA Radar data used in this study were measured with the C-band radar of the Spanish Meteorological Institute (INM) located in Corbera de Llobregat (near Barcelona -see Fig. 1-). Barcelona -200 -100 0 100 200 distance to radar (km) -200 -100 0 100 200 distance to radar (km) Fig. 1.Domain covered by the Corbera de Llobregat Cband radar (the black triangle shows its location) A database of 200 radar scans corresponding to a wide variety of meteorological and propagation situations has been manually analysed by an expert, who labelled all bins as “clutter”, “precipitation” or ”clear air”. This database has been used to adjust the different parts of the algorithm. 3. CLUTTER CHARACTERISTICS One of the objectives of the present study is the statistical characterization of clutter echoes. A number of authors (e.g. Moszkowicz et al. 1994; Bellon and Kilambi 1999; Steiner et al. 1999; Grecu and Krajewski 2000; Steiner and Smith 2002; Kessinger et al. 2003) have shown that clutter presents the following characteristics: • short vertical extension (clutter uses to affect only the lowest tilts). • high degree of spatial variability. • Doppler velocities close to 0. precipitation ground clutter sea clutter 0 102030405060 0.0 0.1 0.2 0.3 0.4 0.5 h k (x) -20 -10 0 10 20 0.0 0.1 0.2 0.3 0.4 0.5 Z (dBZ) radial velocity (m/s) vertical gradient (dB/º) Reflectivity (dBZ) Radial velocity (m/s) Vertical gradient of reflectivity (dB/º) -30 -20 -10 0 10 20 0.0 0.1 0.2 0.4 0.5 h k (x) h k (x) 0.3 h k (x) 0.0 0.1 0.2 0.3 0.4 0.5 0 100 200 300 400 500 0.0 0.1 0.2 0.4 0.5 texture (dB 2 ) Texture of reflectivity (dB2) 0 20 40 60 80 100 0.0 0.1 0.2 0.3 0.4 0.5 spin(%) Spin change (%) h k (x) 0.3 h k (x) 0 5 10 15 20 25 0.0 0.1 0.2 0.3 0.4 0.5 echotop(º) Echotop (º) h k (x) Fig. 2.Feature histograms, hk,t(x), corresponding to precipitation (orange line), ground clutter (dashed purple line) and sea clutter (green line), derived from a database of 200 radar scans, which have been manually analysed by an expert. Figure 2 shows the histograms, hk,t, of some features derived from radar data, corresponding to echoes of nature t (that is, precipitation, ground clutter or sea clutter), which have been derived according to P4R.1 *Corresponding author address: Marc Berenguer, GRAHI-UPC Gran Capità 2-4 Ed. Nexus D-102. E08034-Barcelona (Spain). e-mail: [email protected] equation 1, from the manually analysed dataset mentioned above. hk,tx () =pX k=xecho type =t [] =nX k=xecho type =t () n echotype =t () (1) n(Xk=x  echotype=t) stands for the number of radar bins where the feature Xk=x and this echo has been classified of type t; n(echotype=t) is the number of bins affected of echoes labelled with type t. It is worth noting that the curves presented on Fig. 2 are quite similar to the results presented by Steiner et al. 1999, based on the analysis of 8 radar scans measured in regions with significantly different climatologies. Similarly, Fig. 3 shows the conditional probability fk,t(x) of every bin to be affected by a certain echo type t (precipitation, ground or sea clutter) when the feature Xk takes a certain value, x (see equation 2). f k,t x () =p echo type =t X k =x [] =nX k =xecho type =t () nX k =x () (2) n(Xk=x) stands for the number of bins where Xk=x. From these results, we can conclude that some of the analysed features may have potentiality to discriminate precipitation and clutter echoes (as their curves present significant differences). precipitation ground clutter sea clutter Reflectivity (dBZ) Radial velocity (m/s) Echotop (º) 0.0 0 102030405060 Z (dBZ) 0.0 0.2 0.4 0.6 0.8 1.0 radial velocity (m/s) -20 -10 0 10 20 0 5 10 15 20 25 echotop(º) f k (x) vertical gradient (dB/º) 0.0 0.2 0.4 0.6 0.8 1.0 Vertical gradient of reflectivity (dB/º) -30 -20 -10 0 10 20 f k (x) 0 20 40 60 80 100 spin(%) Spin change (%) texture (dB 2 ) 0 100 200 300 400 500 Texture of reflectivity (dB2) 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 f k (x) f k (x) 0.0 0.2 0.4 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.6 f k (x) f k (x) Fig. 3. Conditional probability curves, fk,t(x), corresponding to precipitation (orange line), ground clutter (dashed purple line) and sea clutter (green line), derived from a database of 200 radar scans, which have been manually analysed by an expert. 4. THE PROPOSED ALGORITHM Last years, some authors (see Bellon and Kilambi 1999 and Kessinger et al., 2003) have proposed using fuzzy logic concepts to characterize clutter through the analysis of different features. Fuzzy logic algorithms are based on assigning a value in the range [0,1] to each radar bin, quantifying the possibility of the bin to be contaminated by clutter. This is done through the analysis of the fields of features, Xk, using a set of curves, µ k(x), known as “membership functions” (presented in Fig. 5). These functions assess the possibility of a bin (ri,  j) where Xk(ri,  j)=x of being affected by clutter. This allows us to derive a set of fields Yk(ri,  j)= µ k(Xk(ri,  j)). These fields Yk are, then, averaged according to a set of weights, wk, to obtain a unique field, Y(ri,  j): Yr i ,  j () =w k Y k r i ,  j () k=1 n  (3) Finally, the bins (ri,  j) where Y(ri,  j)>0.5 are labelled as “clutter” and, therefore, removed. 4.1 Ground and sea clutter Figures 2 and 3 agree with the results shown by some authors (Andersson et al. 1997; Steiner et al. 1999) in the sense that ground and sea clutter present significantly different characteristics. Therefore, we have decided applying two different configurations of the algorithm over the ground and over the sea. 4.2 Used features We have chosen features that have already been implemented by other authors to discriminate between precipitation and clutter echoes. Although we have evaluated other statistics (not presented in this paper), most effective features have been found to be: • to characterize ground clutter: the Doppler velocity (m/s), the vertical gradient of reflectivity (calculated from the two lowest elevations; i.e. in dB/º), the spin change (%; as defined by Steiner and Smith 2002) and the texture of reflectivity (dB2; see Kessinger et al. 2003). • to characterize sea clutter: the reflectivity (dBZ), the echotop (expressed as the elevation angle, in º), the vertical gradient of reflectivity (dB/º) and the spin change (%). 4.3 Clutter frequency map Jointly with the statistics presented above, we have included a map with the frequency with which different areas are affected by clutter as additional feature of the algorithm (see Fig. 4). This map has been derived, from the manually analysed database presented above, as the quotient between the number of scans where each bin has been labelled as clutter and the total of analysed scans. Because of the relatively small frequency with which AP clutter affects radar measurements (in comparison with fix ground echoes) the associated membership functions should enhance the importance of areas frequently affected by AP clutter. -200 -100 0 100 200 distance to radar (km) -200 -100 0 100 200 00.5 525 50 90 frequency (%) distance to radar (km) Fig. 4.Clutter frequency map derived from a database of 200 radar scans, which have been manually analysed by an expert. 4.4 Membership functions Somehow, functions fk,t(x), allow us to quantify the confidence that can be given to an echo where the feature Xk=x of being of a certain nature (clutter or precipitation). This is, in fact, the purpose of membership functions, µ k,t(x). It is, thus, intuitive that the shape of membership functions, µ k,t(x), should not be far from the shape of functions fk,t(x). In the present study, we have chosen piecewise linear membership functions (Fig. 5) reproducing the shape of the sample functions fk,t(x). However, due to the low relative frequencies of sea clutter, the shape has been exaggerated with values of µ k,t(x) significantly higher than fk,t(x). 4.5 Adjustment of weights Table 1 presents the weights wk used to average the fields Yk associated to the different features. They have been subjectively chosen after the analysis of the fields of the manually edited database presented above. TABLE 1. Weights assigned to the features implemented in the presented fuzzy logic algorithm. Weight (%) Feature Ground clutter Sea clutter Reflectivity (dBZ) 0 15 Radial velocity (m/s) 15 0 Echotop (º) 0 20 Vertical gradient of reflectivity (dB/º) 25 25 Spin change 15 25 Texture of reflectivity 25 0 Clutter climatology 20 15 10 20 30 40 50 60 Z (dBZ) 0.0 0.2 0.4 0.6 0.8 radial velocity (m/s) -20 -10 0 10 20 0.0 0.2 0.4 0.6 0.8 Reflectivity (dBZ) Radial velocity (m/s) 0 -30 -20 -10 0 10 0 5 10 15 20 25 echotop(º) 0.0 0.2 0.4 0.6 0.8 1.0 Echotop (º) 1.0 µ k (x) vertical gradient (dB/º) 0.0 0.2 0.4 0.6 0.8 1.0 Vertical gradient of reflectivity (dB/ ) µ k (x) 1.0 0 20406080100 spin(%) 0.0 0.2 0.4 0.6 0.8 1.0 Spin change (%) 0.0 0.2 0.4 0.6 0.8 1.0 texture (dB 2 ) 0 100 200 300 400 500 0.0 0.2 0.4 0.6 0.8 Texture of reflectivity (dB2) 1.0 µ k (x) 0.0 0.2 0.4 0.6 0.8 1.0 Clutter climatology (%) 0 20 40 60 80 100 freq(%) ground clutter sea clutter µ k (x) µ k (x) µ k (x) µ k (x) Fig. 5.Membership functions, µ k,t(x), used in the proposed algorithm to identify ground clutter (purple dashed line) and sea clutter (green line). 5. RESULTS OF IMPLEMENTATION In this paper we present the results of implementing the proposed algorithm over 5 selected radar scans, representative of different meteorological situations (radar loops of these cases may be found at http://www.grahi.upc.edu/events.php). The corrected fields presented below have been processed with the algorithm proposed by Delrieu and Creutin (1995) to mitigate the effects of mountain beam screening, the presented fuzzy logic technique and resulting gaps have been filled using the method proposed by Sánchez-Diezma et al. (2001). 17 July 2001 2000 UTC The scan shown in Figure 6 corresponds to a severe AP situation, which lasted for more than 12 hours. In this case, all echoes of this clear-air scan correspond to ground and sea clutter (ground clutter affects up to the fourth tilt as can be appreciated in the vertical cross section of Fig. 6d). In this case, the fuzzy logic algorithm has been able to classify most of echoes as clutter and only very weak and small nonprecipitating echoes remain after the correction. -200 -100 0 100 200 distance to radar (km) -200 -100 0 100 200 distance to radar (km) -200 -100 0 100 200 distance to radar (km) -200 -100 0 100 200 distance to radar (km) (c) Corrected Z map (0.5º elevation)(b) Identified clutter (0.5º elevation)(a) Raw Z map (0.5º elevation) A B (d) 10 20 30 40 50 60 70 Z (dBZ) distance (km) height (km) 10 8 6 4 2 010050 250200150 AB Fig. 6. Identification of non-meteorological echoes in the radar scan measured on 17 July 2001 2000 UTC. (a) Raw reflectivity map corresponding to the first tilt (0.5º elevation). (b) In black, bins identified as clutter in the raw reflectivity field (areas where Z>5dBZ are depicted in grey). (c) Corrected reflectivity field (after removing clutter bins and filling the resulting gaps). (d) Vertical cross section interpolated from raw volumetric radar measurements along the line A-B of (b). Dashed line encircles most relevant clutter echoes. Beam paths have been calculated assuming Normal Propagation conditions. 02 January 2002 1230 UTC In the radar scan of Fig. 7 a widespread precipitation system affects mountain ranges close to the radar (the bright band enhancement can be appreciated in Fig. 7d at a height of around 2.5 km). As propagation conditions may be considered as mean conditions, clutter is only due to the fix orographic echoes. Therefore, we have to verify if the algorithm is only removing fix ground echoes and leaving precipitation echoes untouched. This is indeed the case except for a short-developed echo of very low intensity located over the sea, which has been erroneously removed by the fuzzy logic algorithm (shown with a dashed line ellipse in Fig. 7b). 23 August 2004 1730 UTC Figure 8 shows a radar scan where a number of convective cells affect the mountainous area in AP conditions. At the same time some sea clutter echoes, caused by beam trapping over the sea, were also measured. However, the fuzzy logic algorithm is able -200 -100 0 100 200 distance to radar (km) -200 -100 0 100 200 distance to radar (km) -200 -100 0 100 200 distance to radar (km) -200 -100 0 100 200 distance to radar (km) (c) Corrected Z map (0.5º elevation)(b) Identified clutter (0.5º elevation)(a) Raw Z map (0.5º elevation) A B A height (km) 6 4 2 0 B 10050 250200150 distance (km) 10 20 30 40 50 60 70 Z (dBZ) (d) Fig. 7. Same as Fig. 6 but for the radar scan measured on 02 January 2002 at 1230 UTC. -200 -100 0 100 200 distance to radar (km) -200 -100 0 100 200 distance to radar (km) -200 -100 0 100 200 distance to radar (km) -200 -100 0 100 200 distance to radar (km) (c) Corrected Z map (0.5º elevation)(b) Identified clutter (0.5º elevation)(a) Raw Z map (0.5º elevation) A B (d) distance (km) height (km) 50 200150 6 4 2 0 10 8 AB 100 10 20 30 40 50 60 70 Z (dBZ) Fig. 8. Same as Fig. 6 but for the radar scan measured on 23 August 2004 at 1730 UTC. to identify most of fix ground echoes and sea clutter, while all convective cells (such as the patterns of the left area of Fig. 8d) remain untouched. Therefore, the performance of the proposed fuzzy logic algorithm can be considered as satisfactory. 14 August 2001 2010 UTC Figure 9 shows a situation of some convective cells embedded in a stratiform system. We can also observe some sea clutter and very intense AP echoes affecting the southern coast. In this case, the algorithm has been able to discriminate most of sea and ground clutter without affecting rainfall patterns, but some residual weak echoes still remain in the vicinity of most important ground clutter echoes (near the coast). 23 August 2004 1130 UTC In this radar scan (Fig. 10), some areas are affected by light precipitation in the West and in the Northeast. In the Southeast, sea clutter echoes affect the first tilt. However, this scan has been chosen because of the presence of some ground clutter embedded in precipitation in the southeastern part of the scan. After correcting this radar scan, fix ground echoes have been perfectly removed and, over the sea, the proposed algorithm has been able to discriminate sea clutter from precipitation and ground clutter embedded in rainfall has also been well identified and removed properly. -200 -100 0 100 200 distance to radar (km) -200 -100 0 100 200 distance to radar (km) -200 -100 0 100 200 distance to radar (km) -200 -100 0 100 200 distance to radar (km) (c) Corrected Z map (0.5º elevation)(b) Identified clutter (0.5º elevation)(a) Raw Z map (0.5º elevation) A B (d) 10 20 30 40 50 60 70 Z (dBZ) distance (km) height (km) 50 200150 6 4 2 0 10 8 B 100 250 A Fig. 9. Same as Fig. 6 but for the radar scan measured on 14 August 2001 at 2010 UTC. -200 -100 0 100 200 distance to radar (km) -200 -100 0 100 200 distance to radar (km) -200 -100 0 100 200 distance to radar (km) -200 -100 0 100 200 distance to radar (km) (c) Corrected Z map (0.5º elevation)(b) Identified clutter (0.5º elevation)(a) Raw Z map (0.5º elevation) A B (d) 10 20 30 40 50 60 70 Z (dBZ) distance (km) height (km) 50 250200150 6 4 2 0 10 8 150 300 A B Fig. 10. Same as Fig. 6 but for the radar scan measured on 23 August 2004 at 1130 UTC. 6. CONCLUSIONS Clutter echoes contaminate radar precipitation measurements, which results not only in errors in rainfall estimates but may also affect the performance of some automatic algorithms based on radar information. The first aim of this study has consisted on statistically characterizing clutter echoes and significant differences between clutter measured over the ground and over the sea have been obtained. On the other hand, we have also proposed and evaluated an algorithm for clutter identification based on fuzzy logic concepts. This fuzzy logic algorithm has been validated from a qualitative point of view over a set of characteristic radar scans measured in different meteorological situations and it has demonstrated a good performance, particularly in AP conditions. Future work will include the systematic implementation of the algorithm over longer sets of radar data and its validation in more quantitative terms. Acknowledgements: This project has been carried out in the framework of the EC projects VOLTAIRE (EVK2-CT-200200155) and FLOODSITE (GOCE-CT-2004-505420). Radar data were provided by the Spanish Institute of Meteorology (INM). References Andersson, T., P. P. Alberoni, P. Mezzasalma, D. B. Michelson, and S. Nanni, 1997: Anomalous propagation: identification from terrain and sea waves using vertical reflectivity profile analysis. Preprints, 28th Conf. on Radar Meteorology, Austin, TX, Amer. Meteor. Soc., 93-94. Battan, L. J., 1973: Radar observation of the atmosphere. University of Chicago Press, 324 pp. Bellon, A. and A. Kilambi, 1999: Updates to the McGill RAPID (Radar Data Analysis, Processing and Interactive Display) system. Preprints, 29th Int. Conf. on Radar Meteorology, Montreal, Canada, Amer. Meteor. Soc., 121-124. Delrieu, G. and J. D. Creutin, 1995: Simulation of radar mountain returns using a digitized terrain model. J. Atmos. Oceanic Technol., 1038-1049. Grecu, M. and W. E. Krajewski, 2000: An efficient methodology for detection of anomalous propagation echoes in radar reflectivity data using neural networks. J. Atmos. Oceanic Technol., 17, 121-129. Kessinger, C., S. Ellis, and J. Van Andel, 2003: The Radar Echo Classifier: A fuzzy logic algorithm for the WSR88D. Preprints, 3rd Conf. on Artificial Intelligence Applications to the Environmental Science, Long Beach, CA, Amer. Meteor. Soc., 1-11. Kosko, B., 1992: Neural networks and fuzzy systems :A dynamical systems approach to machine intelligence. Prentice Hall, Inc ed. Mendel, J. M., 1995: Fuzzy logic sustems for engineering: A tutorial. Proc. IEEE, 83, 345-377. Moszkowicz, S., G. J. Ciach, and W. F. Krajewski, 1994: Statistical Detection of Anomalous Propagation in Radar Reflectivity Patterns. J. Atmos. Oceanic Technol., 11, 1026-1034. Sánchez-Diezma, R., D. Sempere-Torres, G. Delrieu, and I. Zawadzki, 2001: An Improved Methodology for ground clutter substitution based on a pre-classification of precipitation types. Preprints, 30th Int. Conf. on Radar Meteorology, Munich, Germany, Amer. Meteor. Soc., 271-273. Steiner, M., Smith, J. A., Kessinger, C., Ferrier, B. S., 1999: Evaluation of algorithm parameters for radar data quality control. Preprints, 29th Conf. on Radar Meteorology, Montreal, Canada, Amer. Meteor. Soc., 582-585. Steiner, M. and J. A. Smith 2002: Use of three-dimensional reflectivity structure for automated detection and removal of non-precipitating echoes in radar data. J. Atmos. Oceanic Technol., 19, 673-686.