Evaluation of the lightning imaging sensor aboard the International Space Station using Lightning Mapping Array
Abstract
In this work the data from a Lightning Mapping Array (LMA Ebro Delta) and the Lightning Imaging Sensor (ISS-LIS) has been used with the aim to explore which properties of the former make it more likely to have simultaneous ISS-LIS detections associated. To do so, the data has been divided in 10ms time bins; the ISS-LIS and LMA data points inside a same time bin have been associated as a sole emission. Then, the properties of each sensors’ detections have been transferred to the bins, to which have been assigned the properties of both kinds of detections. The analysis has shown a clear relation between the amount of LMA data points in a bin and the likelihood of that bin to have a simultaneous LIS detection. This relation is probably due to the sum of powers of LMA data points; the study has not shown any clear relation between the maximum power recorded in a bin with its simultaneous ISS-LIS detection. Other parameters such as the height of LMA detections has been taken into account; which has shown no clear influence on generating orbit detectable emissions. In this work is also presented an introductory technical report on ISS-LIS and its current data products, as well as the way how they have been treated for acquiring the mentioned results. Plus, it has been performed a literature research with the aim to find a relation between the voltage applied to the open air in order to generate an electrical arc and its emissions on the 777.4 nm band; research that has shown unsatisfactory results.
Full text
Universitat Polit` ecnica de Catalunya Escola Superior d’Enginyeria Industrial, Aeroespacial i Audiovisual de Terrassa Engineering in Aerospace Technologies Evaluation of the Lightning Imaging Sensor aboard the International Space Station using Lightning Mapping Array Bachelor Thesis – REPORT – Author: ´ Icar Fontcuberta Supervisor: Joan Montany` a Delivered: 10 June 2019
ABSTRACT In this work the data from a Lightning Mapping Array (LMA Ebro Delta) and the Lightning Imaging Sensor (ISS-LIS) has been used with the aim to explore which properties of the former make it more likely to have simultaneous ISS-LIS detections associated. To do so, the data has been divided in 10ms time bins; the ISS-LIS and LMA data points inside a same time bin have been associated as a sole emission. Then, the properties of each sensors’ detections have been transferred to the bins, to which have been assigned the properties of both kinds of detections. The analysis has shown a clear relation between the amount of LMA data points in a bin and the likelihood of that bin to have a simultaneous LIS detection. This relation is probably due to the sum of powers of LMA data points; the study has not shown any clear relation between the maximum power recorded in a bin with its simultaneous ISS-LIS detection. Other parameters such as the height of LMA detections has been taken into account; which has shown no clear influence on generating orbit detectable emissions. In this work is also presented an introductory technical report on ISS-LIS and its current data products, as well as the way how they have been treated for acquiring the mentioned results. Plus, it has been performed a literature research with the aim to find a relation between the voltage applied to the open air in order to generate an electrical arc and its emissions on the 777.4 nm band; research that has shown unsatisfactory results. 1
Contents 1 Introduction 8 2 Description of Sensors 10 2.1 INTERNATIONAL SPACE STATION - LIGHTNING IMAGING SENSOR . . . . . 10 2.1.1 International Space Station - LIS . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.1.2 LIS: instrument description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.1.3 Event Grouping Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.1.4 Technicalproperties ................................. 16 2.2 LIGHTNING MAPPING ARRAY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.2.1 PrincipleofOperation................................ 17 2.2.2 Ebre Lightning Mapping Array . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3 Sensors’ Data Handling 19 3.1 ISS-LIS HIERARCHICAL DATA FORMAT FILES . . . . . . . . . . . . . . . . . . . 19 3.1.1 LIS HDF processor.m ............................... 20 3.1.2 LIS vs LMA comparator.m . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 3.2 LMADATA.......................................... 25 3.2.1 LMA zoom7.sci ................................... 25 4 LIS data analysis using LMA as reference 26 4.1 OVERVIEW OF GATHERED DATA . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 4.1.1 Space-Time distributions of detections . . . . . . . . . . . . . . . . . . . . . . . 26 4.1.2 Influence of excited pixels’ position on the CCD . . . . . . . . . . . . . . . . . 37 4.1.3 Sectionsummary................................... 38 4.2 INFLUENCE OF VHF SOURCES’ PROPERTIES ON ITS LIS DETECTIVITY . . . 39 4.2.1 Hypothesis and analysis description . . . . . . . . . . . . . . . . . . . . . . . . 39 4.2.2 Sources’ Height Influence on Detectivity . . . . . . . . . . . . . . . . . . . . . . 40 4.2.3 Sources’ Maximum Power influence on Detectivity . . . . . . . . . . . . . . . . 45 4.2.4 Density of sources in the time bins . . . . . . . . . . . . . . . . . . . . . . . . . 47 4.2.5 SectionSummary................................... 50 4.3 ANALYSIS OF THE FLASH DURATION CONCORDANCE BETWEEN LIS AND LMASENSORS ....................................... 51 4.3.1 Flash Duration Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 4.3.2 Sectionsummary................................... 54 5 Electric arc emissions for LIS calibration 55 5.1 Minimum Radiance Emissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 5.2 Relation peak voltage – radiance emitted . . . . . . . . . . . . . . . . . . . . . . . . . 58 5.2.1 Directrelation .................................... 58 5.2.2 Otherapproaches................................... 59 6 Conclusion and further work 62 3
CONTENTS Appendix A APPENDIX 64 A.1 Geostationary Lightning Mapper description . . . . . . . . . . . . . . . . . . . . . . . 64 A.2 Night vs. Day distribution of LIS during 2017 period . . . . . . . . . . . . . . . . . . . 65 A.3 Detections’ properties influence on LIS detectivity from a typical value approach . . . 66 A.3.1 Sourcesdensity.................................... 66 A.3.2 Sources’height .................................... 68 A.3.3 Sources’Power .................................... 70 A.3.4 Sectionsummary................................... 74 A.4 ExtraFigures......................................... 74 A.4.1 Meansources’height................................. 74 A.4.2 Histogram distributions from a data-point point of view . . . . . . . . . . . . . 76 A.5 User Guide for data the processing codes . . . . . . . . . . . . . . . . . . . . . . . . . 77 A.6 Codes for data processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 A.6.1 LISHDFprocessor.m ................................ 89 A.6.2 LIS vs LMA comparator.m . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 ACRONYMS AND ABBREVIATIONS In alphabetical order ASIM Atmosphere-space Interaction Monitor CCD Charged-Coupled Device CG Cloud to ground (flash) EM Electromagnetic ESEIAAT Escolta Superior d’Enginyeria Industrial, Aeroespacial i Audiovisual de Terrassa FOV Field of View HDF Hierarchical Data Format IC Intra Cloud (flash) IFOV Instantaneous Field of View ISS International Space Station LAT Latitude LON Longitude LMA Lightning Mapping Array LINET Lightning detection Network LIS Lightning Imaging Sensor LRG Lightning Research Group OTD Optical Transient Detector oe only sources pwr Power RF Radio-frequency se sources + events SNR Signal-to-Noise Ratio TLE Transient Luminous Event 4
TOA Time of Arrival TRMM Tropical Rainfall Measuring Mission txt Text UTC Universal Coordinated Time VHF Very High Frequency List of Figures 2.1 Measuring range of OTD and TRMM imaging sensors. Source: [2] . . . . . . . . . . . 11 2.2 ISS-LIS observable area. Source: [2] . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.3 Lightning Spectra Information. Source: [6] . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.4 Example of grouping process. Source: [8]. . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.5 3-D models of the ISS-LIS sensor. Source: [7] . . . . . . . . . . . . . . . . . . . . . . . 16 2.6 Lightning Mapping System antennae locations. Source: [11] . . . . . . . . . . . . . . . 17 2.7 Position of LMA antennae in Ebre delta (white circles). Source: [13]. . . . . . . . . . . 18 3.1 ISS-LIS HDF files organisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 3.2 Screenshot of a output LIS txt file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 4.1 Space and time distribution of LIS and LMA detections during 17-10-18 10:30. . . . . 27 4.2 Height[m] vs. time [sec. from start of 10 min. period]. Zoom on time distribution of LIS and LMA detections during 17-10-18 10:30. . . . . . . . . . . . . . . . . . . . . . . 28 4.3 Space and time distribution of LIS and LMA detections during 17-10 18 17:00. Top: height[m] vs. time[sec.]. Bottom left: LAT/LON [deg.] Bottom right: height[m] vs. LAT/LON [deg.]. Sources are dots, events are circles. Sources coloured by time, events bygroup. ........................................... 28 4.4 Space and time distribution of LIS and LMA detections during 17-10 18 17:00. Focus on LIS view time. ..................................... 29 4.5 Sources and Events detections printed on height and time . . . . . . . . . . . . . . . . 30 4.6 Space and time distribution of LIS and LMA detections during 18-04-27 13:10. . . . . 31 4.7 Space and time distribution of LIS and LMA detections during 18-04-29 11:30 view time. Top: height[m] vs. time[sec.]. Bottom left: LAT/LON [deg.] The smaller picture displays the whole 10 min period. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 4.8 Space and time distribution of LIS and LMA detections during 18-05-25 01:40 view time. 33 4.9 Space and time distribution of LIS and LMA detections during 18-06-05 15:10 view time. 33 4.10 Space and time distribution of LIS and LMA detections during 18-06-06 14:20 view time. 34 4.11 Space and time distribution of LIS and LMA detections during 18-06-13 17:50 view time. 35 4.12 Height and time distribution of LIS and LMA detections during 2018-08-09 18:50 view time............................................... 35 4.13 Height and time distribution of LIS and LMA detections during 2018-08-09 04:40 view time............................................... 36 4.14 Space and time distribution of LIS and LMA detections during 2018-09-18 03:30 view time............................................... 36 4.15 Space and time distribution of LIS and LMA detections during 2018-10-18 15:10 view time............................................... 37 4.16 Excited pixels distribution on the CCD for bad data cases. . . . . . . . . . . . . . . . 37 4.17 Excited pixels distribution on the CCD for two cases with good data: 17-10-18 . . . . 38 5
4.18 Mean height histograms for the cases of the 2017 fall . . . . . . . . . . . . . . . . . . . 41 4.19 Median of Sources Heights Histograms for all periods of data . . . . . . . . . . . . . . 42 4.20 Example for symmetrical distribution of power . . . . . . . . . . . . . . . . . . . . . . 43 4.21 Power-weightened average sources’ height . . . . . . . . . . . . . . . . . . . . . . . . . 44 4.22 Maximum power histograms for all periods of data . . . . . . . . . . . . . . . . . . . . 46 4.23 Distribution of Density of sources in a bin for all periods . . . . . . . . . . . . . . . . . 48 4.24 Distributions of the sum of power for all periods . . . . . . . . . . . . . . . . . . . . . 49 4.25 Sources (+) and events (x) detected during the 171018 1030 time period printed over time,colouredbyflash. ................................... 52 4.26 Zoom on a flash of the 171018 1030 time period. Sources (+) and events (x) printed overtime. .......................................... 52 5.1 MODTRAN output Flux and Radiance . . . . . . . . . . . . . . . . . . . . . . . . . . 57 5.2 Atmospheric transmittance depending on the wavelength by MODTRAN . . . . . . . 57 5.3 Emission properties of controlled electric arc discharges. Extracted from [19] . . . . . 58 5.4 Emission properties of controlled electric arc discharges (second). Extracted from [19] 59 5.5 Emission intensity of different spectral lines with the plasma temperature. Source: [21] 60 5.6 Intensity of emissions depending on the wavelength. Source: [23] . . . . . . . . . . . . 61 A.1 GEOS satellite and its FOV on the Earth surface. Source: [10] . . . . . . . . . . . . . 65 A.2 GLM sensor performance properties. Source: [10] . . . . . . . . . . . . . . . . . . . . . 65 A.3 Night-Day presence of lightning detected by LIS around Deltebre area from March 2017 toJuly2018. ......................................... 66 A.4 Typical values of sources densities in bins for each 10 min period. . . . . . . . . . . . . 67 A.5 Typical values of sources’ height in each 10 min. time period, achieved from the typical height values inside each bin of the same period. . . . . . . . . . . . . . . . . . . . . . 68 A.6 Space-time detections distribution of 2017-10-18 with LINET included (X and +). ”x” are negative strokes and ”+” are positive strokes. Circles are events (coloured by flash) and dots are events (coloured by time). . . . . . . . . . . . . . . . . . . . . . . . . . . 69 A.7 Sumofpowersforeachbin ................................. 70 A.8 Typical values for power-weighted centroid positions in each time period. . . . . . . . 71 A.9 Maximum power for each bin during 171018 1030 . . . . . . . . . . . . . . . . . . . . . 72 A.10 Typical values of maximum power for each inside-FOV time period . . . . . . . . . . . 72 A.11 Data-point POV histograms for 171018 10:30 . . . . . . . . . . . . . . . . . . . . . . . 76 A.12 Data-point POV histograms for 17-10-18 17:00 . . . . . . . . . . . . . . . . . . . . . . 76 A.13 Data-point POV histograms for 18-08-09 . . . . . . . . . . . . . . . . . . . . . . . . . . 76 A.14 Data-point POV histograms for 18-08-31 . . . . . . . . . . . . . . . . . . . . . . . . . . 76 A.15 Data-point POV histograms for 18-09-18 . . . . . . . . . . . . . . . . . . . . . . . . . . 77 A.16 Data-point POV histograms for 18-10-18 . . . . . . . . . . . . . . . . . . . . . . . . . . 77 List of Tables 2.1 ISS-LIS sensor technical properties. The frame rate is of 2ms and the total FOV is of 600x600km.Source:[9] .................................. 16 4.1 Duration of LMA and LIS flashes (seconds). In the table there are not the flashes that were not detected by LIS. The rows with zeroes represent those flashes that LIS detectedbutnotLMA..................................... 53 6
LIST OF TABLES 5.1 Typical radiance values of events with sources associated at 2500m, with a tolerance of 50m in [ µJ sr2m2µm ]....................................... 56 7
Chapter 1 Introduction Motivation Lightning are rather puzzling phenomena: beams of light that fall from the sky, announcing the arrival of a sound blast that will frighten men and animals for equal. It is not surprising that they have sparkled mythologies, stories and the human mind an imagination from millennia. Through observation and reasoning we have mostly unraveled the physical mechanisms that generate such beautiful elements; yet their measurement and observation has been far from losing interest. In the modern times lightning have gained a huge interest as indicators of thunderstorms: they are wonderful electromagnetic signal generators. Their print is widely used to generate thunderstorm maps by weather-forecasting institutions all over the globe. Scientists and engineers have pushed themselves along the years to design a number of systems that can map the presence of lightning. One of those are the Lightning Mapping Arrays (LMA), systems of antennae that can detect lightning in a radius of the order of 100km. Despite the LMA being extremely useful to generate local lightning maps, since the 90s some space agencies such as NASA have put their focus on orbit-based systems, which have an extremely wider range of detection. These systems, although not having the same precision or being as effective as the LMAs, have proven their utility by allowing the scientific institutions study thunderstorm populations where other sensors could not ”dream” to reach. Monitorin the oceanic storms, for instance, is not only mandatory for understanding the weather in situ, which has a gigantic economic impact in maritime trade, but is also important for being able to prescience the weather else were. That being said, the success and utility of orbit-based lightning detectors is currently circumscribed by their relatively short time of experience. Validation and calibration are major operations that each one of these sensors has to face and can be done, for instance, from a cross-sensor approach. The LMA systems gain here a new role: they can provide very accurate local data to less accurate global detectors. This approach is relevant since the systems are already installed and the technology is well known. The only perk, nonetheless, is that orbit-based systems, being the Lightning Imaging System at the International Space Station (ISS-LIS) the most promising at the moment, don’t read the same same signal: while LMA captures VHF RF pulses (at the band of 60MHz approx.), the ISS-LIS detects the light pulses of much greater frequency (wavelength of about 777 nm). In order to be able to compare the data, for example with the aim of calibrating a LIS sensor, it is mandatory to understand if both detections can be associated and in which way. Assuming that a lightning emits both types of signals, the first step could be to explore which properties of the LMA detections have an impact on the likelihood of generating a simultaneous detection from LIS. 8
Finally, as commented above, the newness of orbit-based systems, and in particular ISS-LIS, leads to a lack of information about the systems’ properties and data products. Moreover, there is little knowledge and analysis on data crossing from LIS-LMA systems. Aim and Scope Accordingly at what has been said above, this work aims to provide: 1. A comprehensive report about the technical properties and the fundamental structure of the LIS and LMA systems 2. A description about the ISS-LIS data products and how they have been processed in this work 3. A report on the exploration of the influence of the LMA sources’ properties on its likelihood to be detected with ISS-LIS and other cross-sensor assessments Out of the scope will be left: •An exhaustive technical description of the systems •Data analysis with sensors other than ISS-LIS and LMA •Production of statistical models for prediction of detection by ISS-LIS of LMA sources In summary, the main objective of this work is to evaluate the detections of lightning from space by means of high-resolution Lightning Mapping Array systems (LMA); and as partial objectives will be done reviews of lightning and lightning imagers, definition of the evaluation, develop codes for reading data and develop codes for the evaluation. 9
2.1. INTERNATIONAL SPACE STATION - LIGHTNING IMAGING SENSOR 2.1.4 Technical properties In the table 2.1 below are displayed the most relevant technical properties of the ISS-LIS sensor. PARAMETER VALUE PARAMETER VALUE FOV 80ºx 80º B Location 1 pixel IFOV (nadir) 4 km intensity 10% Awavelength 777.4 nm time tag at frame rate bandwidth 1 nm Csensor head assembly 20 x 37 cm Detection Threshold 4.7 \micro J m electronics box 31 x 22 x 27 cm SNR 6Weight 20 kg CCD Array Size 128 x 128 pixels Power 30 Watts Dynamic Range >100 Ddata rate 8 kb/s Detection Efficiency ∼90% format PCM False Event Rate <5% EIntegration time (I.T.) 2 ms Convention for time assignation 1 ms before end of I.T. Table 2.1: ISS-LIS sensor technical properties. The frame rate is of 2ms and the total FOV is of 600 x 600 km. Source: [9] Area Ais ”Interference Filter”, Bis ”Measurement Accuracy”, Cis ”Dimensions”, Dis ”Telemetry” and Eis ”Time labeling at the CCD”. Figure 2.5: 3-D models of the ISS-LIS sensor. Source: [7] . 16
2.2. LIGHTNING MAPPING ARRAY 2.2 LIGHTNING MAPPING ARRAY The Lightning Mapping Array or LMA is a system that locates in 3-D and time very high frequency (VHF) emissions from lightning. The system consists of a set of sensors (radio frequency antennas), located at a distance of kilometres from each other. The term ”LMA” is used widely to refer all the systems that follow the same design in the world. Indeed, there are LMA systems installed in different places, by different institutions. For instance, the Lightning Research Group (LRG) at Polytechnic University of Catalonia (UPC) has control over three of those: one in the south of Catalonia, called ”Ebro”; and the others in Colombia, in Santa Marta and Barrancabermeja. Despite the introduction of satellite-based lightning measurement systems such as LIS, which provide a global coverage, LMA systems still remain relevant. Since they have a much higher detection efficiency, their measurements can be used as a reference for satellite-based sensors calibrations. 2.2.1 Principle of Operation It is known that lightning emit VHF electromagnetic pulses. These pulses can be measured by radio antennas that, when set up in a network (fig. 2.6), can track the position of the VHF source. The principle of operation relies on computing the time of arrival (TOA) of the each signal and then computing the origin by modeling the light velocity in the medium. Figure 2.6: Lightning Mapping System antennae locations. Source: [11] Each antenna computes the TOA of its received pulse, and then – knowing the propagation velocity of the EM wave – the distance of the source from the antenna. Afterwards, with the correlation of all distances computed by all the antennas and their position, the source coordinates can be obtained. The source position, its time of detection and power will be saved. Each source is registered for the antennas as an electromagnetic pulse (EMP) with its corresponding energy, or power. The detected sources’ power may vary widely: using the LMA system they have been registered to range from 1W up to 10-30kW [12]. 17
2.2. LIGHTNING MAPPING ARRAY 2.2.2 Ebre Lightning Mapping Array The LRG of UPC has installed over the Ebre delta a set of sensors, forming the Ebre LMA. It is the LMA system from which the data used in this work is generated. In the west Mediterranean sea, few storms -compared to tropics in South Americatake place. Furthermore, during the cold season they occur over the sea and during the hot season over the mainland. Then, placing the LMA system throughout the Ebre delta has revealed a great solution for capturing thunderstorms from both seasons, as its measuring range reaches both the sea and the mainland. Moreover, the presence of the LINET network in the area provides the possibility of complementary observations. In the figure 2.7 below the position of LMA antennae is displayed. Currently 7 of them are operative. Figure 2.7: Position of LMA antennae in Ebre delta (white circles). Source: [13]. There is a town in the delta’s centre named Deltebre, so the area may be called Deltebre Area as well. 18
Chapter 3 Sensors’ Data Handling Contents 3.1 ISS-LIS HIERARCHICAL DATA FORMAT FILES . . . . . . . . . . . 19 3.1.1 LIS HDF processor.m . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 3.1.2 LIS vs LMA comparator.m . . . . . . . . . . . . . . . . . . . . . . . . . . 24 3.2 LMA DATA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3.2.1 LMA zoom7.sci . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 The objective of this chapter is to give an introduction to the reader regarding what data formats and programs have been used to get the results displayed further in this document. Here will not be presented a technical description of the codes nor the data, but it should be regarded more as a comprehensive description that will allow the reader to understand the outputs of the codes and how the data from the sensors is treated. 3.1 ISS-LIS HIERARCHICAL DATA FORMAT FILES The Hierarchical Data Format (HDF4) files are items formatted in a self-describing way (i.e. they contain metadata that describes the data that is stored within the file), and they are designed to store large amount of very different scientific data1. Basically, these files can contain the following types of data, also represented in the figure 3.1a below: •Raster Images: Graphical representation of data; in this case, the orbit. •Palette: Information about graphical display of the data in the raster images (e.g. colour distribution). •Scientific Data Set: N-Dimension matrix of data (includes the attributes of the data points, which are the axis of the matrix). •Annotation: Text data that provides info about the file •Vdata: data stored in table format. •Vgroup: group of data items. A Vgroup can contain one or more of the previous data items. This way of information organisation allows the storage, in one file, of lots of data that is indeed related but not necessarily has the same format, nor refers to the same physical phenomena. For instance, in ISS-LIS case each file contains information of a whole ISS orbit. Nonetheless, each file contains information about the orbit itself (start time, start coordinates in Geo. Coordinates System...); information about the LIS sensor in every moment along the orbit; information about lightning phenomena recorded by LIS etc. All these different types of data will be stored under different Vgroups, which will contain various HDF items (Sci. data sets, Vdata tables...), depending on the respective requirements. In the figure 3.1b 1Most information in this section has been extracted from [14]. 19
3.1. ISS-LIS HIERARCHICAL DATA FORMAT FILES (a) HDF Structure. Source: [14] (b) Screen picture of HDFView when accessing an ISSLIS file. Figure 3.1: ISS-LIS HDF files organisation this is clearly displayed. Each ”folder” icon represents a Vgroup item, than contains other items. In this case, all information except the Raster Imager is stored in Vdata tables. Notice also how for the selected Vdata set, ”event”, some metadata is displayed on the menu on the right: this Vdata table has 21 fields that contain data of different sizes, from 7749 events or ”rows”; the class of the data arrays etc. More precise information about this file format can be extracted from creators’ website ([14]). Although the HDF4 files can be accessed through a number of programs (e.g. HDFView), they do are not useful for massive data extraction. Thus, in this project a custom MALTLAB code will be used, named ”LIS HDF processor.m”. MATLAB uses its own commands for accessing the HDF4 files, so a particular description for this case is required. The MATLAB codes developed for this project are available at the appendix A.6 and at https://github.com/icarfontcu/LIS-LMA-data-reading-codes. 3.1.1 LIS HDF processor.m The program was made to satisfy the need of creating files of LIS data that can be an input of the program ”LMA zoom7.sci”, a Scilab code made by van der Velde, O. that plots data from LMA, LINET, LIS and other sensors in the space-time domain. As it was made, ”LMA zoom7.sci” receives LIS data from txt files. Considering that the LIS full data is stored in HDF4 files, the need of a program that converts HDF4 to txt files arises. Furthermore, some relevant issues appeared, regarding the bulk data downloading and selection processes. They will be discussed later. All the data is available online and can be download through Earthdata website client. When this program was developed, the data was not yet downloaded on any UPC server so some assumptions, mainly the organization of said data, had to be taken. The program has a total of 7 functionalities, to be selected trough a ”switch” function when the program starts. Some of them more relevant than others. Following is a description of those functions. READ AND STORE INFORMATION FROM HDF FILES This case will write txt files from HDF4 files. To fulfil this task, some other functions will be previously executed. 20
3.1. ISS-LIS HIERARCHICAL DATA FORMAT FILES ”Check for folders” This program is written bearing in mind the files organization of Earth Data HTLM server2. This is: 1. Year folders (a) Orbit ID folders i. Bulk data Inside the latter are all the .hdf files that contain the data of the events detected during the orbits. Supposing that the data, once downloaded, would be organized through: 1. General folder (a) Subgeneral folder (b) Subgeneral folder i. subsubgeneral folder ii. subsubgeneral folder and so on until the last folders, where there would only be .hdf data items. The first task that the code shall bear is to find the data inside whatever server it is. This task is given to the function ”check for folders”. What this function does is, given a general directory on the computer (e.g. C:/Users/Name/Desktop/Data), go through all the subfolders of the directory until finding those folders with only data files (or not-folders, as it is coded). Precisely, this folders use the MATLAB function ”folderinfo” to get the information about the items that are inside the folder where the function currently is set. Using folderinfo.isdir, the user gets a logical comma-separated-list of which items inside the folder are directories. With that information we can do some conditional commands. In our case, as the info. is organized, the code will set the current file as a final reading directory only if no item in the folder is a folder. To do so, we can transform the comma separated list to a logical array ”cell2mat()” and check for 1 (directory) or 0 (not directory). If the folder is a final reading directory its address is saved in an size-increasing string array, ”dir list”, which will return to the main program as an output of the function. If the folder is not a final reading directory (it does contain other folders) the function recalls itself and explore the subfolders that the current folder has. ”Select interestingfiles” Once we have the list directory for the data that we want to acquire, we shall extract that data. It is supposed that the user will be searching data with some space and time boundary conditions (they must be specified in the first lines of the codes). Supposing also that the UPC server will have a lot of bulk data that may not be interesting, the program will search in the folders of the directory file for the HDF4 files containing data that is interesting (events within the space-time conditions). Those files will be called ”interestingfiles”, and those events ”interestingevents”. The utility of this method is that a list of ”interestingfiles” can be stored for later use: in case the user would want to operate again with those files, the program wouldn’t need to search trough all the data base To check if the files contain any ”interestingevents”, the MATLAB functions ”interesect(a,b)” and ”find(x>y)” have been used, in order to select the array indexes where the data that is within the space-time boundaries is. Previously another method was used (line-by-line and logical true/false analysis) was made, but the use of ”intersect” and ”find” turned up to be much faster. 2https://ghrc.nsstc.nasa.gov/pub/lis/iss/data/science/nqc/hdf/ 21
3.1. ISS-LIS HIERARCHICAL DATA FORMAT FILES ISS-LIS HDF4 files data organisation In order to get the data arrays where MATLAB can search for interesting data it is previously required to import that data. Then, the knowledge about the LIS HDF4 data becomes important. IN the ISSLIS case, each HDF contains: 1. Orbit Information (groupdata) (a) Point information (groupdata) i. Lightning data (groupdata) A. area (vdata) B. flash (vdata) C. group (vdata) D. event (vdata) ii. point summary (vdata) iii. viewtime (vdata) iv. background summary (vdata) (b) Orbit summary (vdata) (c) one second (vdata) 2. A raster Image of the Orbit through the world map MATLAB can access the structure and meta-data of these files with the command ”hdfinfo”, which will give the names, properties (size, class...) etc. of the data. Once you read a groupdata address MATLAB will show a structure containing different field with labels, and data characteristics of that data group. The ”sub folders” (vdata or groupdata) will be labelled as 2 different fields. Accessing it as a structure field will lead to an opening of a new structure with fields or directly data, respectively. The vdata files within will be regarded as a field, that will contain at the same time data in the structure form. E.g., we can access the lightning data groupdata through: vdata address(i) = fileinfo.V group.V group.V group.V data(i) We can read the vdata that we want with the command hdfread(vdata address). With i (1,2,3 or 4) we can select the area, flash, group or event vdata, cell arrays where each cell are cell arrays containing columns of same-class and size data (i.e. event vdata=hdfread(vdata address(4)) will give a group of group of cell arrays. We can store them in a new structure element called ”event”: event.coordinates=event vdata(3). (The third column of this vdata address(i) contains 1x2 elements which represent the coordinates of the event)). Now, with this data organization, more clear and manageable, we can introduce the space-time boundaries and know if the hdf file we are reading is an interesting file. This last information will be stored in a vector that will remain in the global workspace during the rest of the program. PRINT txt FILES WITH EVENTS’ INFORMATION ”w txt files” This function will read all the interestingfiles and plot its interesting events to a txt file with the similar name: ”ISS LIS 20171018 1032 1034 events”, where the 20171018 indicates the date time through the format YYYYMMDD and the 1032 and 1034 indicate the time (HHMM) of the first and last event in the file. To do so, we will introduce again the space-time boundaries and with the indexes of the interesting events apply the MATLAB function ”fprintf()” to print row-by-row the relevant information. The data will be separated with a tab and the file will contain a header with the titles of each column (e.g. TAI933Latitude, Longitude...). 3Time of the measurement in seconds from 1 Jan 1993. To convert from TAI93 to UTC the former has to be summed to the date (1 Jan 1993) as follows: (7.8248e+08)[s] + (1 Jan 1993) = (18 Oct 2017 10:31:56). 22
3.1. ISS-LIS HIERARCHICAL DATA FORMAT FILES At this stage, it was required to make a decision on which data from the HDF4 file will be stored in the txt files (remember that we can access area data, flash data, group data, event data, orbit data...). Regarding that the interesting parameters to measure are radiance, space-time coordinates of the events and its grouping with other events, the stored parameters are: •TAI93 time: TAI93 time in seconds resolution of the event •event LAT and LON: space coordinates of the event •radiance: event’s radiance •group address: The group to which the event belongs •group LAT and LON: the group’s radiance-weighted centroid •flash address: The flash to which the event belongs •flash LAT and LON: the flash’s radiance-weighted centroid •area address: The area to which the event belongs •area LAT and LON: the area’s radiance-weighted centroid •exited pixels’ position on the CCD •observe-time: the time during which the LIS sensor was focused on the are The reader should note that the addresses of each group, flash and area are renewed for different interesting –txt– files, so events from two different files may have the same group address, which doesn’t mean they do belong to the same group. This is an aspect to improve in the near future, but at the moment it doesn’t represent a problem, as for this documents the txt files have been processed (and displayed) separately. The output txt files will have a column for each saved parameter, and a header containing the title of each column. A screen shot of a txt file is displayed in the figure 3.2. Figure 3.2: Screenshot of a output LIS txt file EVENT PLOTTING This task is optional and the writing of general events txt files may be done without plotting the data. Nonetheless, if the plotting is activated, the function plot-events is activated. It simply plots on a gridded map the events, sized by radiance and coloured by group, of each file. It also draws the area where the data is selected, and plots a low resolution coastline loaded from the MALTAB mapping toolbox. 23
3.1. ISS-LIS HIERARCHICAL DATA FORMAT FILES HDF4 FILE NAMES TO URLs To download LIS bulk data you need a txt files with the URLs of the HDF4 files that you want to download from Earthdata server. These URLs have the following shape: https://ghrc.nsstc.nasa.gov/pub/lis/iss/data/science/nqc/hdf/2018/0613/ISS LIS SC P0.2 20180613 NQC 08218.hdf On the LIS data website you can download a txt file with the URLs from all the HDF4 files whose orbit is within a period of time specified by the user. Nonetheless, this generates a problem: the selected files will correspond to orbits in the time period, but that orbit may not pass (actually will be the common case) through the area that we want to analyse. Therefore, the amount of data downloaded would be much higher than the required and the process of selecting interestingfiles way much longer. A workaround to this problem was found by using the space-time domain search engine from the LIS website. This tool lets you specify the time period as well as the area were you want to look for, and displays a list with the names of the HDF4 files that contain data in this space-time domain. Then, it is required to select ”by hand” this list and copy it to a txt file. This txt file will be later processed by the function webfilenames2urls(), that will construct URLs from the files names taken from the website list. Then, these URLs are written on a new txt file, and finally we are able to download the interesting HDF4 files with the cmd command: wget –user username –ask-password –auth-no-challenge –no-check-certificate -i URLsfile.txt This, provided by instructions from the LIS website will download all the HDF files listed in the URLsfile.txt. For an optimal usage of the LIS data with the program LIS vs LMA comparator.m, the best praxis is to join all txt files for the interesting period of time with the aim to have a large sole file containing all the events. This can be done manually, since the number of txt files that the studies presented in section 4.2 require is of the order of 5. Also, a posteriori Prof 3.1.2 LIS vs LMA comparator.m In order to compare the data from LMA and LIS a MATLAB program has been made. Such code compares data from both sensors (fundamentally events and sources) and associates them by time proximity. The output of the program are statistical result (typical value distributions, histograms...) that contain information about such associations. For instance, a source and an event that are separated in time less than 10 ms are considered to be associated. Since events and sources have different properties, via this association we can compare events properties (e.g. radiance) with sources properties (e.g. height) and produce distributions that allow a study of, say, the radiance evolution with height. A more exhaustive explanation of this approach is done in the section 4.2 and in this section only the structure and behaviour is to be discussed. To use the program, the user should edit the code (first lines, indicated in MATLAB) to introduce the proper variables: LIS total filename The txt file containing all the LIS data LMA filename The txt file containing the LMA data for of the interesting time period timestep Maximum time between an event and a source to associate them Plotting Options Histograms/plotting/sources and events. Set variable as 1 or 0 to plot histograms, typical values of each time bin or display all sources and events printed over time. comparing length section This option enables the execution of a section that is used to compare the length of lightning computed from LIS or LMA data; results described in section 4.3. 24
3.2. LMA DATA A more detailed explanation of the user-introduced variables can be found in the user guide, at A.5. As the reader might infer, the fundamental output of this program is graphical displays of the evolution of some properties with others (e.g. radiance vs counts, height vs counts, radiance vs height etc). Its results can be seen in section 4.2. Another section of the code is dedicated to assess which is the typical radiance of events that had sources associated at a given height. For this functionality, the user should set the variables DISCHARGE-ALT [m] and tolerance [m] accordingly. The results of this study are displayed in the Chapter 4. 3.2 LMA DATA The LMA data is generated by each LMA antenna, by dumping ”.dat” files each 10 min. Each antenna dumps its files respectively, containing information about the distance and timing of the sources detected by the antenna. In order to get the 3-D position of each source, a post-processing is required, made by an UPC-LRG4algorithm. This algorithm crosses the information about various antennas regarding the same time, and dumps ”.dat” files for each 10 min period, containing the position; timing; power and other values of the sources detected during the said period. This crossover of information may be done more or less exhaustively. For instance, the user can select to cross information from 5 to all 7 antennas. The number of antennas is the number of simultaneous, separated detections that the algorithm requires to declare a source in the respective location. More antennas will give results with less noise but the user might lose information, as some VHF emissions might be only detected by less antennas that the specified. In the other hand, too few antennas will produce a very noisy result, where a lot of the declared sources may be errors or detections of other phenomena. The LMA .dat files used in this document have been computed using a 5 (out of 7) antenna restriction, a parameter that has been found to give a nice equilibrium between quality and quantity. In order to process the said .dat files a program has been made with Scilab, LMA zoom7.sci (Van der Velde. 2017). This program displays the sources in space and time domain, as well as other information such as their power, spread velocity etc. Alongside the sources the program can also display the LIS events, LINET detections, Meteorage detections and other data coming from other sensors. For this document, the LMA zoom7.sci will be used mainly to compare LIS vs. LMA detections, and only a part of all the program applications will be executed. Below are described the capabilities of the Scilab program that are of use in this document. 3.2.1 LMA zoom7.sci The main goal of this program is to display the information about position and timing of detections in a comprehensive way. To do so, the user must select a 10 min LMA file and the program will display a panel with 4 graphs. In this panel there is information regarding 1. Time-height position: top panel 2. LAT-LON position: bottom left panel 3. LAT/LON vs height position: bottom right panels The program allows the user to colour the detections under a number of parameters (power, time, flash...). An example of the output figures is fig. 4.1 below. Note that the sources have been coloured by time. As the reader may notice, the LIS detections are also displayed as a circles, coloured by group. To do so the user must select a corresponding LIS txt file. Symmetrically, the same can be done with LINET, if data is available. The usefulness of this display is that allows the user to have a generic view on the data gathered at a given time. It allows a search for space or time offsets and other particularities of the data set that with a less graphic approach could go unnoticed. 4Lightning Research Group at Universitat Polit`ecnica de Catalunya 25
4.1. OVERVIEW OF GATHERED DATA Figure 4.7: Space and time distribution of LIS and LMA detections during 18-04-29 11:30 view time. Top: height[m] vs. time[sec.]. Bottom left: LAT/LON [deg.] The smaller picture displays the whole 10 min period. 2018-05-25 01:40 During this time period, as seen in the figure 4.7, LIS detected some lightning in front of the coast, at S.W. from the delta. In the time-space domain window it is possible to observe that this LIS detection is not simultaneous with any other LMA detections. As a very good signal from LIS was registered (the dots correspond to the same flash but come from a number of groups), LINET data was checked and a simultaneous stroke was found in the same place. Therefore, a discharge did happen but was not detected by LMA. 32
4.1. OVERVIEW OF GATHERED DATA Figure 4.8: Space and time distribution of LIS and LMA detections during 18-05-25 01:40 view time. The colourful inside-figure displays the strokes detected by LINET. height/LAT and height/LAT distributions are not displayed because the non-validity of the data. 2018-06-05 15:10 In this case, displayed in the figure 4.9, only one lightning was detected. The time window of the LIS FOV was, approximately, from second 360 to 430, so probably the detected flash was the only one that LIS could had seen. Figure 4.9: Space and time distribution of LIS and LMA detections during 18-06-05 15:10 view time. Top: height[m] vs. time[sec.]. Bottom left: LAT/LON [deg.] Bottom right: height vs. LAT / height vs. LON 33
4.1. OVERVIEW OF GATHERED DATA 2018-06-06 14:20 During this period few data was recorded by LMA: only what it seems to be a portion of a full-scale lightning, far S.W. from Deltebre (fig. 4.10). Furthermore, it was detected at the edge of the interesting Deltebre area, so information about the possible development of discharges outside the space boundaries lacks. Moreover, LIS didn’t detect that phenomena either: it only detected two lone events far to the North. This data has no interest for methodical analysis. Figure 4.10: Space and time distribution of LIS and LMA detections during 18-06-06 14:20 view time. Top: height[m] vs. time[sec.]. Bottom left: LAT/LON [deg.] 2018-06-13 17:50 In this date LIS detected a lightning simultaneously with LMA, as displayed on the figure 4.11 below. LIS seems to have a relatively large offset in S.W. direction, but the time domain windows shows how LIS detections matches perfectly with a lightning detected by LMA. 34
4.1. OVERVIEW OF GATHERED DATA Figure 4.11: Space and time distribution of LIS and LMA detections during 18-06-13 17:50 view time. Top: height[m] vs. time[sec.]. Bottom left: LAT/LON [deg.] Fall of 2018 During this period 4 valuable episodes were recorded, that provided a huge number of detections. The following data only contains sources that were under the LIS FOV at the given time. Since they were observed to be concurrent in space and show a very high activity, only the outputs from LMA vs LIS comparator.m will be displayed. 2018-08-09 18:50 In this case, as in the following, a relevant quantity of synchronized sources and events were recorded. Below are printed over time. In the figure 4.12 it can be seen how the activity, despite only occupying 2 min in time, is intense. A minimum of 10 separated flashes were detected by the LMA and virtually all of them had synchronised LIS detections. Figure 4.12: Height and time distribution of LIS and LMA detections during 2018-08-09 18:50 view time. 35
4.1. OVERVIEW OF GATHERED DATA 2018-08-09 04:40 In the figure 4.13 it can be seen how an intense activity was detected from both sensors during the LIS pass. Although more homogeneously distributed over time, here discrete lightning discharges can also be appreciated during the whole pass. Figure 4.13: Height and time distribution of LIS and LMA detections during 2018-08-09 04:40 view time. 2018-09-18 03:30 In this pass (fig. 4.14) a remarkable intense activity was registered. In the figure it can be seen the extremely dense cluster of LMA detections and their LIS couple. Figure 4.14: Space and time distribution of LIS and LMA detections during 2018-09-18 03:30 view time. 36
4.1. OVERVIEW OF GATHERED DATA 2018-10-18 15:10 In this pass several flashes were detected. They can be appreciated discretely in the fig. 4.15 alongside with their synchronous LIS detections. In this case, at the contrary of the 2018 09 18 one, the lightning are clearly separated in time. Figure 4.15: Space and time distribution of LIS and LMA detections during 2018-10-18 15:10 view time. 4.1.2 Influence of excited pixels’ position on the CCD The LIS CCD has a surface of 128x128 pixels, and it is possible to made a plot where to represent pixels, given their xy coordinates. To explore if the position of the exited pixels on the CCD has some influence on the three cases with deficient data (18-04-29,18-05-25 and 18-06-06), the position of said pixels have been printed. They are displayed in the fig. 4.16, and there it can be seen that only few pixels were activated. In 18-04-29 and 18-06-14 cases the excited pixels were in the boundaries of the CCD whereas in the 18-05-25 they were more centred. (a) 18-04-29 (b) 18-05-25 (c) 18-06-14 Figure 4.16: Excited pixels distribution on the CCD for bad data cases. For comparison purposes, also the pixel excitation distribution of some good data has been displayed 37
4.1. OVERVIEW OF GATHERED DATA (fig. 4.17). It shows the two cases of 17-10-18, from where more data is available.In fig. 4.17 it can be seen how there are excitation both in through the centre of the CCD and in the boundaries. (a) 10:30 (b) 17:00 Figure 4.17: Excited pixels distribution on the CCD for two cases with good data: 17-10-18 It is therefore acceptable to sustain that there is no relation between the quality of the gathered data and the position of excited pixels on the LIS CCD. 4.1.3 Section summary From the section 4.1.1 and 4.1.2 some conclusions arise: •The periods of more data are the ones of the 2017-10-18 and the cases of the 2018 fall. This is concordant, since in the coast of Catalonia strong thunderstorms occur typically at the summer’s end. Other cases where data is not extremely mediocre are 180427 13:10, 180605 15:10 and 180613 17:50, where although LIS only detected one flash per case, there is a simultaneous LIS detection with low space offset from LMA data. Because they do not provide a high amount of data, compared to the more populated observations, they will not be analysed. The deficient data cases are of two kinds: 1. In the 181525 01:40 period, LIS clearly detected a flash near the coast, as did LINET; who detected a stroke in the same position at the same time. Nonetheless, LMA did not detect anything so the detected flash (by LIS) is not useful for comparison with VHF presence. 2. The other two cases LIS and LMA detected activity, but with a very high offset in space. This means that either they detected different phenomena or the signals that they recorded are background noise. •The validity of the data does not seem to have a clear relation with the relative position of the sensor to the event (and therefore the position of the exited pixels on the CCD). It has been seen how in both good and bad data cases the exited pixels can be on the edge of the CCD or near the centre. 38
4.2. INFLUENCE OF VHF SOURCES’ PROPERTIES ON ITS LIS DETECTIVITY 4.2 INFLUENCE OF VHF SOURCES’ PROPERTIES ON ITS LIS DETECTIVITY The scope of this section is to assess what (if any) properties of the VHF RF sources emitted by lightning discharges have an impact on the lightning detection by LIS. Since effects of the excited pixels’ position on the LIS CCD have already been discussed, below this will not be discussed again. In this document a statistical analysis of data containing dozens of lightning will be done, an approach that has not been done yet with ISS-LIS. Some works have been done around the matter that concerns this section but only at a scale of studying some flashes –of the order of ten. This section is meant to be a results report, so statistical statements will only regard the used data and they are not intended to be valid about LIS detection worldwide. 4.2.1 Hypothesis and analysis description The presented data can be analysed from different perspectives. One of these is to check the LIS efficiency to detect lightning discharges. A priori, LMA is a better detector of lightning than LIS (i.e that it usually detects the discharges that LIS sees and some that LIS doesn’t); so to measure the effectiveness of LIS in detecting lightning we can use LMA data as a reliable source of information on what LIS should detect. Following this criteria, an hypothesis has been made: The luminosity detected by LIS is part of the same physical process that generates the VHF emissions recorded by LMA, i.e. leader propagating through the air. Then, with the appropriate time corrections for possible offsets between LIS instrument measuring (TAI93 time) and LMA (UTC time), detections should be roughly simultaneous. It is understood that both detectors may have offsets of microseconds and that the VHF emissions and LIS detections, although part of the same exact physical process, may not be generated at the same exact time. For instance, it is possible that in order for LIS to get the minimum luminosity to activate its threshold a number of near-simultaneous VHF sources must be closely generated. Actually, the data preview seems to support this last suggestion, since there is a much greater of LMA detections than LIS events, which might suggest that the sources must have some characteristics (e.g. minimum power, height ...) in order to be detected. The data will be analysed in order to find those characteristics. As stated in section 3.1.2 for this analysis a MATLAB code named ”LMA vs LIS comparator.m”1 was made. Basically it divides the time period in time bins, in order to assess the presence of events and/or sources in the mentioned bins. Then, every time bin may or not contain event and sources, that at the same time will have some physical properties associated, for instance the power of the source. Moreover, the program will compute some physical properties of the bins (e.g height-weighted centroid of sources’ power, mean height of the sources in the bin...). To do so , it processes text files dumped from the Scilab program, which contain information about sources, flashes, power... displayed in whatever time period the user is displaying. Afterwards, ”LMA vs LIS comparator.m” loads events text files of the same period and compare the data. For instance, the bins will be separated in ”bins with only sources”, ”bins with events and sources” etc. These labels refer to the content of the bin; bins with only events will, for the moment, not been taken into account. It is appropriate to comment that preparing the data for its analysis with the LMA vs LIS comparator.m program has not been a strait forward task. The final approach has been proposed and implemented by Prof. Montany`a. To put it in few words, the LIS FOV time of a certain area has been extracted from the LIS HDF4 files, and a filter has been set in order to eliminate all the LMA data that did not match the space-time criteria of the LIS sensor. Only the sources that were under the LIS FOV have been used for the analysis. 1The code is available at at appendix A.6 and https://github.com/icarfontcu/LIS-LMA-data-reading-codes. 39
4.2. INFLUENCE OF VHF SOURCES’ PROPERTIES ON ITS LIS DETECTIVITY The size (in time) of the bins that divide the time period is called ”timestep”. The size of the timestep should be big enough to ignore the possible micro-offset in time and small enough to be able to correctly separate different events detections in different bins. A timestep of 10 ms was selected, after observing with the Scilab program the time distribution of events and sources detections within flashes: there’s rarely a gap of more of 10 ms between sources in the same flash, so inside the said flash we shouldn’t get empty bins. In this section the properties of the VHF sources that influence its detectivity with the LIS instrument will be explored. This analysis will be done from an histogram approach. A typical value approach has also been done and can be found in the appendix A.3. It is not presented in the report because it provides less complete information than the histogram approach. For an histogram point of view, only some of the observed periods have enough data to produce significant results. Below are the results of such cases. 4.2.2 Sources’ Height Influence on Detectivity The first source’s parameter that has been studied is the source’s height. Since LIS is an orbital imager system, one should consider some simple aspects like the atmospheric attenuation of the radiation emitted by lightning or its distortion by the clouds. Such parameters might have an impact on the final detection depending on the source’s height since, the higher is the source, the less atmosphere and (probably) clouds will be between it and the detector. Comment About the State of the Art The most significant work on the matter can be read on the paper published by Thomas et al. [18]. In this paper are explained the results of a thunderstorm observation with a LMA system at New Mexico. In the only pass of the ISS, 128 lightning discharges were observed. With this data, a relation between height and LIS detectivity is observed: most of lightning that occurred in the higher part of the cloud were detected, not like discharges that happened in lower parts. In the said paper, this is attributed to the higher discharges being closer to the cloud boundaries since in previous work it had been found that light was more likely to escape the cloud if it was originated near the surface. Cases of 2017 fall The 171018 periods sources distribution are displayed in figure 4.5a. These histograms show the counts of a representative value of a time bin. I.e. the data (sources, events) has been grouped in time bins of 10ms, and a representative value of the elements in each bin has been computed. For instance, the mean heights of the sources within each bin has been assessed, and in the histogram is represented the counts of that last value. For this case, we can observe a higher distribution of detected sources on the higher spectrum of heights, from 8 to 12km. In the figure 4.18 is represented the main height distribution of the sources of that period. There, it is clear how sources detections are distributed in two groups, the upper and the lower one. One could argue that they could correspond to sources detected in the upper and lower part of a cloud, but this is hardly defensible, as the centre of the ”blocks” is separated roughly 6km. Then, it is assertive to state that LMA has detected two groups of flashes –as we have already seen in fig. 4.5a: one with lower flashes and one with higher. What is interesting here is to how the ratio of sources also detected by LIS was much higher in high altitude flashes than it was in lower altitude flashes. The reason of that detectivity difference can be a matter of discussion. The median histogram was also made but showed the same results. If we take a look at the period 171018 17:00 (its detections are displayed in fig. 4.5b) we find quite a different result. In this case, where only one flash was measured, one cannot appreciate any kind of special distribution. Of course, as can be seen in the fig. 4.5b this was a low flash (it stroke the ground) so we cannot compare the detectivity with higher sources. Nonetheless, we can see that whithin a flash it is not obvious that the height matters on the detectivity. This may be because of the difference in heights within a flash is too small to have a real impact in the detection. 40
4.2. INFLUENCE OF VHF SOURCES’ PROPERTIES ON ITS LIS DETECTIVITY From another point of view, following the line of thought of Thomas et al. [18] one could extract from this observation that VHF sources’ distances to the closest surface, inside the surveyed flash, were homogeneously distributed, or even more closer in the central region –around 4.6km. (a) Mean height distribution of sources for the 171018 10:30 period . (b) Mean height distribution of sources for the 171018 17:00 period Figure 4.18: Mean height histograms for the cases of the 2017 fall The darker red is the same as the blue (indicated in the legend) All cases These observations don’t seem to hold when looking at the data from a more global scale, with all periods included. The histograms for the median heights of the sources detected during all the periods is displayed in the fig. 4.19 below. There, it can be seen how there is a main difference between the cases of 2017 and those of 2018. In the first place, in the 2017 cases the cluster of detections can be clearly differentiated in two groups at different heights, while in the 2018 cases the observations approach more to a single-peak distribution. The data of the 171018 17:30 period could be disregarded, since it displays only one flash and that might interfere with the results. Nonetheless, even doing so, the 171018 10:30 period still would present an anomalous distribution of detections in the lower heights. The 181018 period is the other one to present a significant amount of detections in that spectrum of heights but in this particular case a extremely relative greater number of detections were made. On the other hand, one thing that can be observed from the data is that there seems to be a huge decrease of LIS detections of sources for heights lower than 4km, even if there have been detected sources below. On the other hand, an important aspect can be inferred from this graphs2: all the sources that were also detected by LIS group up around 10km. Even in the cases where there was a higher number of sources detected in lower heights (i.e. 171018 10:30 and 181018) the peak ofsources+events bins is around the said height. This could reveal that height might have an impact in limiting the LIS detection of sources, for example, it could be extracted that the efficiency in detecting sources decreases with altitude. In fact, this again is concurrent with the Thomas et al. [18] observations of some lightning in south U.S.A. The measurements shown on figure 4.19 are the medians of the sources. The median has been chosen as the representative value of the height over the mean since it is less affected by outliers. In the lightning parameter evaluation the outliers can be particularly ”dangerous”, since noise can generate detections far away from where the physical phenomenon is taking place, which will deviate the data significantly. An analogous figure but with means as the representative value can be found in the appendix A.4. 2Disregarding the single-flash data 41
4.2. INFLUENCE OF VHF SOURCES’ PROPERTIES ON ITS LIS DETECTIVITY Figure 4.23: Distribution of Density of sources in a bin for all periods From left to right, and top to bottom: 171018 10:30 171018 17:00 (single flash) 180809 18:50 180831 04:40 180918 03:30 181018 15:10 48
4.2. INFLUENCE OF VHF SOURCES’ PROPERTIES ON ITS LIS DETECTIVITY Figure 4.24: Distributions of the sum of power for all periods From left to right, and top to bottom: 171018 10:30 171018 17:00 (single flash) 180809 18:50 180831 04:40 180918 03:30 181018 15:10 49
4.2. INFLUENCE OF VHF SOURCES’ PROPERTIES ON ITS LIS DETECTIVITY 4.2.5 Section Summary In this section the Height, Power and Density (and cross-overs) of sources in the time bins have been studied in order to see their influence on its LIS detectivity. •In the height case, the median has been taken as a representative value for each bin, which has not shown any remarkable tendency. With the aim to account the possible effect of a source power to influence on the detection, also a weighing of the height median values has been done, using the power of each source as a factor. This latter approach has shown similar distributions to the without-weighting approach. And thus not providing any valuable result. •The influence of the maximum power of a sources cluster in its LIS detectivity has also been studied. In this approach it has been seen for the first time how the power of the VHF RF sources might have an impact in its detection by LIS: the bins where also events were detected usually had a higher maximum power recorded. •The density of sources in the bins has also been studied, it has shown how both only sources and sources+events bins are more numerous in the low-density spectrum. Nonetheless, it has been observed how the increase rate is not the same, and the proportion of bins detected by LIS in relation to the not–detected diminishes in the low-density spectrum. Thus, the data shows that LIS is more likely to detect high populated source clusters. •Finally, the distributions of sum of power for each bin have been studied and have shown that LIS has been more likely to detect VHF source clusters with a higher sum of power. In all the cases recorded, the peak of the sources+events bins detected is centered at the same or at higher power than the only sources bins detection distribution. Summing up the last two points above, an effect of the sources’ power into its LIS detection has been recorded. It has been observed how source clusters more populated –and therefore more likely to have a higher total power– have been more likely detected by LIS than their counterparts. 50
4.3. ANALYSIS OF THE FLASH DURATION CONCORDANCE BETWEEN LIS AND LMA SENSORS 4.3 ANALYSIS OF THE FLASH DURATION CONCORDANCE BETWEEN LIS AND LMA SENSORS Each sensor measures electromagnetic waves coming from lightning, and more than just providing information about that phenomena, it is interesting to extract information about the whole flash. The scope of this section is to compare the information that the user would extract about the flash from both sensors. For instance, the flash duration or its size is something that a post processing of the events’/sources’ time and position could provide. Nonetheless, if for example one would compute the lightning duration from LIS and LMA by saying: ”flash duration =max(events.time)−min(events.time)” or ”flash duration =max(sources.time)−min(sources.time)” Where ”events.time” and ”sources.time” are the vectors of events/times of each flash. It is clear that different results could be extracted for the same flash, as LIS could detect events only in the middle of the real flash and so on. 4.3.1 Flash Duration Analysis Comparing flash durations (with the previously proposed method) has not been trivial. Even if in the sources’ txt files4it is specified the ”seconds-in-flash” of each source (so we have already the flash duration) and the flash number; the comparison process is not as simple as it may seem. This is because 1. in both txt files (LIS and LMA) the flashes are not numbered following the same ”coordinates”. 2. a simple time comparison can not be made as different start/end times in the two files could relate to the same flash Therefore, in order to be able to compare the flash durations provided by both sensors, an extension of the LMA-vs-LIS-comparator.m has been made. This extension creates vectors with the start/end times computed from both files; and then compares them by considering that a LMA flash and a LIS file are the same if: 1. at least the start time of the LIS flash is comprehended in time in the LMA flash 2. at least the end time of the LIS flash is comprehended in time in the LMA flash 3. both ends in time of the LMA flash are comprehended in the LIS flash Please note that the case were the LIS flash is comprehended inside the LMA flash is implemented in one of the 2 first options. 2017-10-18 10:30 In the studied data, this is the best period to check if the comparator works, as is the only one with several flashes in it. the LMA-vs-LIS-comparator.m displays the following (fig. 4.25): In this time period, the LMA5detected 23 different flashes, whereas the LIS detected 17. This can be attributed, for example, to the various flashes that LIS did not detect. In the fig. 4.25 can be seen how approximately at 10:33:37, 10:34:03, 10:34:8, 10:34:16, 10:34:25 and 10:34:29 there are LMA flashes that were not detected by LIS6. There are also some flashes that LIS detected but LMA did not. This 4As previously said, they are extracted from the Scilab program for a given 10 min period 5Its criteria is established in LMA-zoom7.sci and it follows a simple imposition of the maximum amount of time between two subsequent sources; if they are more separated than the established time they will be considered from two different flashes. 6Please note how the LMA criteria is considering to be the same flash the first two clusters near the 10:32:45. 51
4.3. ANALYSIS OF THE FLASH DURATION CONCORDANCE BETWEEN LIS AND LMA SENSORS Figure 4.25: Sources (+) and events (x) detected during the 171018 1030 time period printed over time, coloured by flash. does not mean that LIS detected more real flashes than LMA but it can be attributed to the different (probably more restricted) LIS criteria when it comes to separate various detections in different flashes. In this time period, under a closer look to the events-sources distribution the problem about flash duration arises, as LIS and LMA detections in a flash have a very different span in time. A clear example of that is shown on the figure 4.26. Figure 4.26: Zoom on a flash of the 171018 1030 time period. Sources (+) and events (x) printed over time. 52
4.3. ANALYSIS OF THE FLASH DURATION CONCORDANCE BETWEEN LIS AND LMA SENSORS The results of computing the flash duration from data of both sensors are displayed in table 4.1 below. LIS flash# LMA flash# LIS flash duration LMA flash duration 295 0 0 0 296 31 0.4799 0.0753 297 0 0 0 298 0 0 0 299 33 0.3913 0.3516 300 33 0.0734 0.3516 301 35 0.3010 0.1796 302 37 0.0990 0.1307 303 42 0.2241 0.1971 304 43 0.2487 0.2754 305 48 0.1809 0.3528 330 32 0.1543 0.4110 331 34 0.1487 0.0736 332 36 0.3598 0.3711 333 39 0.1503 0.2809 334 41 0.3312 0.4899 335 47 0.5030 0.5481 Table 4.1: Duration of LMA and LIS flashes (seconds). In the table there are not the flashes that were not detected by LIS. The rows with zeroes represent those flashes that LIS detected but not LMA. For the cases with complete data (2 simultaneous detections of flashes) the subtraction of LIS durations to LMA durations is shown in the following set of data: -0.4046 -0.0397 0.2782 -0.1214 0.0317 -0.0271 0.0267 0.1719 0.2567 -0.0751 0.0114 0.1307 0.1587 0.0451 Set of data extracted from table 4.1 The typical values of this set are: mean = 0.0261 median = 0.014 Therefore, at least in this set of data, LIS tends to provide results where lightning appear to be shorter than they really are. Although as it can be seen in the data this is not an absolute statement, as there are cases where the duration values for LIS are higher than the ones from LMA. This result is not very surprising, because it is know that LMA detects a higher number of events7 than LIS, so a higher probability of finding earlier and later phenomena of a flash seems reasonable. Also, the median and mean of the flash durations are: LIS :mean = 0.2144 median = 0.1809 LMA :mean = 0.2405 median = 0.2754 which is in the other of 10 times bigger than the median/mean value of the subtraction. This is quite surprising, because in the example (shown in fig. 4.26) the LIS flash is way shorter in time than the LIS one, at the order of a half. Despite being surprising, this is a positive result, because it shows how, at least in this set of data, typically the sensor does not have a very high influence on the typical lightning duration. 7Not in the sense of sources/events notation, but referring to the fact that it detects an event as as a generalisation of ”EM pulse”. 53
4.3. ANALYSIS OF THE FLASH DURATION CONCORDANCE BETWEEN LIS AND LMA SENSORS 4.3.2 Section summary In this section the issue for lightning duration with LIS and LMA has been addressed, by comparing simultaneous flashes between them. For the computation of the flash duration, the time of the earliest and latest event/source of a flash (coming from respective txt files) have been subtracted. Afterwards, typical values of duration and subtraction have been computed in order to see the influence of the sensor in the calculus. It has been found that 1. Typically LMA flashes last more than the LIS flashes, if the said criteria when computing the duration is used. 2. The typical value of the difference in time between LMA flashes and LIS flashes is 10 times smaller than the typical duration of respective flashes, so the influence of the sensor over the computed typical duration is rather small. 54
Chapter 5 Electric arc emissions for LIS calibration Contents 5.1 Minimum Radiance Emissions . . . . . . . . . . . . . . . . . . . . . . . . . 55 5.2 Relation peak voltage – radiance emitted . . . . . . . . . . . . . . . . . . 58 5.2.1 Direct relation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 5.2.2 Other approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 Although LMA can be very useful for calibration of orbit lightning sensors, other methods can be used. One of those can be the creation of an electric arc discharge near the ground, such that produces radiance emissions observable by LIS. This technique would clearly increase the precision of the calibration since the position of the light source would be completely known. In order to do this project, three main tackles should be overcome: •Setting the minimum radiance emission at the source in order to make it detectable by LIS •Establishing the relation between the peak voltage for the electric arc generation and the radiance emission at the 777.4 nm line. •Constructing the system that enables an effective discharge This chapter is meant to be a report on the bibliographical research about the two first stages, being the second a topic that is far from trivial in plasma physics. 5.1 Minimum Radiance Emissions The location of the discharge generator has been supposed to be the station at El Niu de l’` Aliga, situated in the mid-longitude Pyrenees of north Catalonia. It has also been supposed that the experiment would be done in a Summer night. The data regarding the weather during such periods has been extracted from [20]. The web application for MODTRAN (http://modtran.spectral.com/) has been used to model the atmosphere. The solver has worked with the following parameters set: •Atmosphere Model: Mid-Latitude Summer •Ground Temperature as average temperature of Summer months (Jun-Sep): 283.01 K •Sensor altitude: 99 km (maximum) •Sensor Zenith: 180º(the sensor is on top of the discharge) •Spectral range: 0.7-0.8 µm •Resolution: 0.00192 µm (maximum) 55
5.1. MINIMUM RADIANCE EMISSIONS •Water Column (atm-cm): 3635.9 •Ozone Column (atm-cm): 0.33176 •CO2 (ppmv): 400 •CO (ppmv): 0.15 •CH4 (ppmv): 1.8 •Ground Albedo: 0 •Aerosol Model: rural •Visibility: 23 km In the figure 5.1 are displayed the radiance and radioactive flux registered in the conditions above. In the figure figure 5.2 below, the transmittance for each wavelength on the same conditions is displayed. It is remarkable to notice how for the wavelength of the radiation emitted by the atomic oxygen the transmittance is of approximately 0.79; where T=Iwλ I0wλ (5.1) where [I] = µJ sr2m2µm In order to set a minimum radiance emission that enables its detection from orbit, at a height of approx. 2500m a selection of the events that had sources associated at that height has been done. Then, a representative value of these events’ radiance has been extracted. The values obtained are below: Period 171018 10:30 171018 17:00 180809 180831 (t = 200m) 180918 181018 avg 18.23 14.48 18.07 10.5 15.76 16.75 m14 12.5 14 10.5 12 13.5 Table 5.1: Typical radiance values of events with sources associated at 2500m, with a tolerance of 50m in [ µJ sr2m2µm ] For the period 181031, where no sources took place at 2500m with t =50. Results with a t=200m are displayed instead. In the table 5.1 it can be seen how the the average and the median differ significantly, showing a high variance since the averaging is greatly influenced by outliers. Also, note that for the period of 1810831 no sources were detected withing the tolerance (2500m ±50m), so the latter was augmented to 200m. Still, the data shows a high deviation from the other cases so it will not be considered. 56
5.1. MINIMUM RADIANCE EMISSIONS (a) Atmospheric Radiance depending on the wavelength by MODTRAN (b) Atmospheric radiation Flux depending on the wavelength by MODTRAN Figure 5.1: MODTRAN output Flux and Radiance Figure 5.2: Atmospheric transmittance depending on the wavelength by MODTRAN For the other cases, an average of the medians will be taken as a representative value, since the medians only show a standard deviation of only 0.9083 between them. Thus, an acceptable representative value is: Iwλ = 13.2µJ sr2m2µm Thus, using the data provided by MODTRAN (see fig. 5.2) the radiance emitted at the source1should be, approximately: Isource =Isensor Tλ=777.4nm = 16.71 µJ sr2m2µm 1Not as ”VHF source” but as ”the origin of the emission” 57
Appendix A APPENDIX Contents A.1 Geostationary Lightning Mapper description . . . . . . . . . . . . . . . . 64 A.2 Night vs. Day distribution of LIS during 2017 period . . . . . . . . . . 65 A.3 Detections’ properties influence on LIS detectivity from a typical value approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 A.3.1 Sources density . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 A.3.2 Sources’ height . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 A.3.3 Sources’ Power . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 A.3.4 Section summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 A.4 Extra Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 A.4.1 Mean sources’ height . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 A.4.2 Histogram distributions from a data-point point of view . . . . . . . . . . . 76 A.5 User Guide for data the processing codes . . . . . . . . . . . . . . . . . . 77 A.6 Codes for data processing . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 A.6.1 LIS HDF processor.m . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 A.6.2 LIS vs LMA comparator.m . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 A.1 Geostationary Lightning Mapper description The Geostationary Lightning Mapper (GLM) is a sensor integrated on satellites that constantly monitor the Americas (fig. A.1b) for weather study [10]. A system of GLM sensors is being implemented with a new series of satellites (fig. A.1a) within the Geostationary Environmental Satellite Network (GEOS) mission; which has been developed by NASA and NOAA. This mission is launching to geostationary orbit a constellation of 4 satellites (GEOS-R,S,T,U), with a GLM instrument each, that will cover a huge part of Earth surface, centred in the Americas. The improvement in weather forecasting and monitoring with these new sensors come from the new capabilities for detecting cloud- to-ground (CG) and intra-cloud (IC) lightning; as well as from the fact that the satellites also will have a Advanced Baseline Imager that observes the atmospheric moisture with improved scale resolution from previous, similar instruments, and provides fresh data five times faster than its predecessors. 64
A.2. NIGHT VS. DAY DISTRIBUTION OF LIS DURING 2017 PERIOD (a) Representation of a satellite from the GEOS-R series. (b) Combined FOV view from GOES-R series constellation superimposed on 10-yr of lightning observation from TRMM-LIS and OTD. Figure A.1: GEOS satellite and its FOV on the Earth surface. Source: [10] Sensor Description The GLM, as the LIS, measures the radiance that lightning emit through the top of clouds with an imaging device. Actually, the sensor’s principles are very similar to those from LIS1and will not be explained another time. It is important to remark that it can measure CG, CI and CC lightning during day and night; the latter being crucial as the satellite is stationary and will suffer from continuous daytime cycles over its FOV. As the GLM orbit is much higher than the one of ISS-LIS (35786km vs. 400km) it is clear that it will suffer in its spatial resolution. In fact, it goes from 8km at the NADIR of the CCD to 14km at the edge. This decrease would be much higher, but it has been lowered by the implementation of a CCD that has smaller pixels on the edges, allowing an homogeneous distribution of resolution along the image. To achieve a acceptable spatial resolution from such a high orbit, a CCD of 1372 x 1300 pixels has been implemented (for the record LIS had a 128 x 128 pixel CCD). Also, a higher telemetry speed (and therefore the amount of processable data) has had a great impact on increasing the detection efficiency, due to the possibility to establish a lower threshold to detect weaker luminous events. The overall result is a geostationary imaging sensor that has a flash detection of at least 86% and a false alarm rate of 5%. Regarding the detections classifications the GLM uses, just as LIS, a groping structure that clusters the luminous events in groups, flashes and areas. Information about this parent-child algorithm can be found on [8]. Other technical properties are described in the table below (fig. A.2). Figure A.2: GLM sensor performance properties. Source: [10] A.2 Night vs. Day distribution of LIS during 2017 period As previously mentioned, the thunderstorms in the west Mediterranean weather (and therefore over Deltebre) are not very numerous. This, added to the fact that the ISS orbit does not always pass over Catalonia; and that for a approximately 100 x 100 km area the view time at maximum of 2 min; makes the amount of lightning observations over the Ebre delta not be very high. Then, it is relevant to generate some statistics about the probability of LIS recording a thunderstorm in the place. It is 1I.e. a wide FOV, focusing radiation beam in a CCD with an interference filter at 777.4 nm, and finally a data processing subsystem. 65
A.3. DETECTIONS’ PROPERTIES INFLUENCE ON LIS DETECTIVITY FROM A TYPICAL VALUE APPROACH also important to remark that LIS has been operative only since March 2017 and the data of these observations in particular reach June 2018. While the LIS was operative (from March 2017) the ISS passed over Deltebre 1998 times. It only captured activity in 8 of those. Most of activity occurred during daytime, and the magnitude order of detected flashes has been of tenths. (a) 2017 (b) 2018 Figure A.3: Night-Day presence of lightning detected by LIS around Deltebre area from March 2017 to July 2018. In the figure A.3 is displayed the day-night distribution of LIS detected flashes during 2017 and 2018 periods. It is in this moment that the background remover and the LIS post-processing unit detailed in section 2.1.2 shows its importance, as a good managing of the background radiance is critical during daytime. A.3 Detections’ properties influence on LIS detectivity from a typical value approach In this appendix the reader will find an analysis of the influence of some lightning properties on its detectivity using a representative value approach. This approach has been only added to the appendix because it was only after it was produced that it was found that the Histogram method provides clearer information about the matter. A.3.1 Sources density In this section will be analysed the effects of the number of sources in each time bin on the LIS detection. To do so, a typical value for bins with only sources and bins with sources+events has been extracted from each 10 min. time period, if the said period had simultaneous (LIS and LMA) detections to compare on. These typical values have been made with the average and also the median, as in every set of data appear to be some heavy outliers that will very much effect the mean. They have been displayed in figure A.4; along with the number of bins with both events and sources, as the number of data available may play some role in the results. In the figure A.4 are displayed both median and mean typical values, for the mean values there seems to be no tendency (in reference to comparison between bins with LIS detection and bins with LIS+LMA detections); but a further inspection, with the median reveals more interesting. It shows that for the cases with significant data (the first and the second in time), the bins that also had LIS detections tend to more source-populated. The mean and the median in the 171018 17:00 period differ so much because the outliers: the standard deviation for the bins with events+sources and only sources in this period is 4.5682 and 6.9174, 66
A.3. DETECTIONS’ PROPERTIES INFLUENCE ON LIS DETECTIVITY FROM A TYPICAL VALUE APPROACH respectively. This is a lot, considering that the the average is situated at approximately 8 sources in each case, so the median seems indeed a better indicator of the typical value for this case. Figure A.4: Typical values of sources densities in bins for each 10 min period. This tendency (LIS detections come from more populated bins) is not so clear in the following cases, but as the figure shows, the first case has 128 simultaneous detections and the second 42, whereas the third has only 7 and the next period with data 3. Also, it is mandatory to bear in mind what it is known from section 4.1.1: the size of the thunderstorms recorded during 2017-10-18 was much higher than during the other dates, and this is reflected by the number of bins with simultaneous detections as well; from the first to the third time period the number of simultaneous detections is reduced a 94.53%. So if there is not a permanent tendency on all the data, it is acceptable to state that detection rate increases with source density in time, as displayed by the data that provides more information. 67
A.3. DETECTIONS’ PROPERTIES INFLUENCE ON LIS DETECTIVITY FROM A TYPICAL VALUE APPROACH A.3.2 Sources’ height Typical values of height in each bin have been computed, and then averaged again to get the typical values of the typical heights for each 10 min. time period. This information is displayed on the figure A.5, where it is show the mean and median values for each case. Figure A.5: Typical values of sources’ height in each 10 min. time period, achieved from the typical height values inside each bin of the same period. In the figure of this section there is 6 variables per period of time, as the following values have been computed for each type of bin: 1. Mean of the typical heights from bins with ; where the typical heights were obtained with simple mean. 2. Median of the typical heights; where the typical heights were obtained with simple mean. 3. Mean of the typical heights from bins with ; where the typical heights were obtained with the median. The goal of this diversity of values is to check the effects of the data dispersion on the results, both of sources’ heights and the variability of their typical values.As it can be seen in the figure, dispersion does not play an important role in heights, as medians and means show the same result: there is no tendency observable for the height. Indeed, in the first case the typical height values of the events+sources bins are higher than those that only have sources, but in the second time period this relation is twisted -and very accentuatedthe other way around. In fact, in this 171018 17:00 time period the detected flash was one that stoke the ground, as commented during the data overview (section 4.1.1). LINET effect One could argue that return stroke can be the dominant variable in this second case. Let’s compare the data gathered by LINET in the two time periods of 171018. 68
A.3. DETECTIONS’ PROPERTIES INFLUENCE ON LIS DETECTIVITY FROM A TYPICAL VALUE APPROACH (a) 10:30 (b) 17:00 Figure A.6: Space-time detections distribution of 2017-10-18 with LINET included (X and +). ”x” are negative strokes and ”+” are positive strokes. Circles are events (coloured by flash) and dots are events (coloured by time). In the figure A.6 are displayed the LIS, LMA and LINET detections for the time periods of 2017-10- 18. As it can be seen, virtually all the detection clusters -flashes- have some stroke associated; both flashes that LIS did and did not detect. This means that the sole presence of strokes has not a direct implication on the LIS detection of its flash. 69
A.3. DETECTIONS’ PROPERTIES INFLUENCE ON LIS DETECTIVITY FROM A TYPICAL VALUE APPROACH A.3.3 Sources’ Power One of the main sources’ characteristics that may influence in LIS detection is the source’s power. Moreover, it may do it in various ways. For instance, the summed up power in a bin time, the maximum power registered in that bin or the height of the power centroid may have a role in LIS detection of the said bin. Sum of sources’ powers This case is a crossover between the influence of power in LIS detection and the density of sources in a given bin. Obviously, it will have a strong relation with the section A.3.1, but it may have some amplifying effects on some of the samples. It is displayed in the figure A.7, where de mean and median of powers’ sums for all types of bins are printed. Figure A.7: Sum of powers for each bin In the said figure, some unexpected effects arise: whereas in the first, third and last time period the relation observed in section A.3.1 seems to be enhanced, in the other cases its appearance is drastically reduced. For instance, the mean value for the only-sources bins in the period 171018 17:00 sky-rockets way above the other values for the same period; while in the first case the relation between sources+events bins and sources bins is maintained. This actually means that, while in the 171018 10:30 period the sources of both types of bins have similar powers, in the second case the power of the sources that have no events associated is higher than those from bins where events were detected; so this may indicate that sources’ power level - or at least, its sum - does not have an impact and detections. Power-weighted sources’ centroid Even if in section A.3.2 it has been revealed that the height by itself does not have a clear role in LIS detection, its effect combined with power might do. I.e. that higher source with lower power might be more likely detected by LIS than a lower source with higher power. To reflect this effect, the power-weighted centroid (in height coordinates) for each bin has been made; and then computed 70
A.3. DETECTIONS’ PROPERTIES INFLUENCE ON LIS DETECTIVITY FROM A TYPICAL VALUE APPROACH its typical value for each 10 min. time period. The said power-weighted centroid height has been computed applying the following formula in each bin: P−i= 1n−srcs(pwr −i∗height −i) P−i= 1n−srcspwr −i Figure A.8: Typical values for power-weighted centroid positions in each time period. The information displayed in the fig. A.8 shows similar results to those we had from section A.3.2, figure A.5. The second case has a very strong variation between sources+events bins and sources bins: it has already been discussed how in that case the heights of detected sources were very low, as the lone detected had its sources close to ground. This, computed now with powered centroid, stays roughly equal, even if now there are big differences in mean and median; which means that in each bin there is usually no deviation in heights (a flash develops closely in space) but there is high deviation in power (a flash produces a high range of VHF emissions). Again, in the first case (where there is information available for a number of flashes) shows some difference between sources+events and sources bins: it seems that in that period the detected sources were higher and more powerful. Maximum power Supposing that a number of sources take place roughly simultaneously (separated less than 10 msec.), and are detected by LIS; one can hypothesize that not all of these simultaneous sources have been detected by LIS, and in fact LIS has detected the one with maximum power. To explore this option, the maximum power values in each time bin have been recorded, and will be compared to those from bins where no events where recorded by LIS. The figure A.9 displays a point (dot or asterisk) for each bin. Each point represents the power of the source that has the maximum power inside the bin. The blue asterisks correspond to bins where events also were detected, and black dots to bins where no events were detected. It is clear that the vertical -so almost simultaneoussources clusters correspond to separated lightning. 71
A.3. DETECTIONS’ PROPERTIES INFLUENCE ON LIS DETECTIVITY FROM A TYPICAL VALUE APPROACH Figure A.9: Maximum power for each bin during 171018 1030 Taking in account that the area came under LIS FOV at 10:32:46, we can observe how the sources that were accompanied by events group up in higher power clusters. With the exception of the clusters at min. 32.95 and 33.3, the sources’ power of clusters that LIS also detected are above 20 dbW. This tendency is well observable in the mentioned case. The figure A.10 shows results of the same approach for the other 10 min. sets. Figure A.10: Typical values of maximum power for each inside-FOV time period There, it can be seen how this tendency is hardly maintained through the other cases. If both the 180427 and the 180613 periods have sources+events bins with higher maximum power than the other 72
A.3. DETECTIONS’ PROPERTIES INFLUENCE ON LIS DETECTIVITY FROM A TYPICAL VALUE APPROACH bins, this is not the case for the second and sixth time periods, where it is the other way around. Even if in the first period the amount of lightning evaluated is way higher than the second, the data shows how both the mean and median of only sources bins remains roughly equal. This could mean that virtually all undetected bins on the day had an average maximum power value, but it had no effect on its detection; as in the detected cases the maximum power varies without restriction. 73
(a) LIS website file selection (b) LIS space-time domain searcher Figure 1: Screenshots of LIS website Figure 2: space-time domain parameters in the code 3
1.2 Read interesting information from HDF files and print it into txt files This application reads the LIS HDF files that the user has stored and print its information (if relevant) to txt files with adequate names. These txt files will contain a number (it can be changed) of columns with one type of data each, e.g. LAT, LON, time... Note that the code is prepared to access a big collection of folders with data, as it has functions implemented that register each subdirectory for other subdirectories until it finds where the HDF files are. This process is completely automatic excepting for the specification of the space-time domain selected. The steps required are described below. 1. Introduce the required directories in the first lines of the code. (a) write dir: where the txt files with information will be written (b) read dir: the directory from the program will start looking for HDF files. 2. Introduce the space-time domain as shown in the fig. 2. 3. Run the code and select the option 1 when asked by the command window. The option 2 also plots the interesting events for validation purposes. The option 5 only plots the interesting events, without writing txt files. Code projection and future improvements One possible improvement is to make the downloading of HDF files fully automated. The option 6 of the code’s menu is a first approach to do so: when downloading LIS bulk data, the user gets a txt files with a huge amount of HDF URLs. These can be processed to get only URLs from files that contain info from a certain period. What option 6 does is to read the mention big URLs files and generate a similar, new one with only the relevant URLs. 2 LIS vs LMA comparator The LIS-vs-LMA-comparator.m has the objective of generating statistical distribution and moments for a study of LIS and LMA data products together. Fundamentally, it pierces the given measurement time with time bins of a given size (e.g. 10 ms). Then, it associates the LIS and LMA data points to each time bin for finally working on bin properties (e.g. mean radiance of all the LIS events inside a bin) instead of working with properties of the data points. This approach is useful since the aim is to associate LMA and LIS detections by time proximity, and in the end generate relations a la ”this LMA source had this LIS radiance associated”, i.e. giving a scalar value for a given detection (or bin). The optimal way to work with this code is to provide it with two text files. First, a text file where all LIS detections are listed. Second, another text file where are listed the LMA detections that were under the LIS FOV during the measurement time. The LMA detections that were not under 4
the LIS field of view at a given time can be filtered out with the help of a code made by Prof. Montany`a (Electrical Engineering Dpt. UPC). Once the user has these files, to use the code: 1. Set the variable LIS total filename as the name (with the path) of the text file where all the LIS events are; line 21 of the code. 2. Set the variable LMA filename as the name (with the path) of the text file where the LMA sources are, line 22 of the code. 3. Set the variable examined day as the day where the measurements where done, line 23 of the code. Use the format: datetime(yyyy,mm,dd). 4. Set the variable timestep as the time width of the time bins with which the data will be divided, line 28 of the code. A recommended value is 10ms. 5. Set plotting options by changing the following variable values (1 = on / 0 = off): •histograms: plot histograms of several bins properties (e.g. mean height of the bin, mean power of the bean...) •plotting: display plots of bin properties in their position in time (i.e. a point represents a value of a property where the coordinate is the time coordinate of the bin and the ordinate the value of the property). Also plot the excited pixels on the LIS CCD •sources and events: plot the sources and events detected over time. For LMA ordinate is the height, for LIS is a random value fixed at 100 m. •comparing length section: activate the section where assessments regarding which is the length of a given flash are done. Do not activate this if you are not particularly going to use this section. •save workspace: save the workspace in the current LMA file directory •savingcsvfile: save a csv file on the current LMA file directory containing moments of the statistical distributions of detections’ properties. If an assessment of the power of events that had a source at a given height associated the discharge altitude has to be set at the line 576 of the code and the tolerance at the line 577. In the figure 3 below the variables listed above can be seen. In the figure 4 are displayed the lines of code where the user should introduce the necessary variables for computing the power at a given height. 5
Figure 3: Introductory section of the LIS vs LMA comparator code Figure 4: User introduced variables for power at a given altitude assessment 6
3 ANNEX 3.1 LIS HDF processor 3.1.1 Download HDF files 1. Search for interesting at iss data websiteHDF files using ISS-LIS search tool 2. Select the list of emerging HDF files 7
3. Copy it to a text file (say yes to any dialogue appearing when closing the txt file) 4. Select space-time domain in MATLAB code 8
5. Set the variable ”website filename” as the directory where you stored the txt file with the HDF files’ names. 6. Execute the code and select option 7 when asked by the command window. This will create a file named interesting URLs.txt in the directory where the txt file with the names was. 7. Open the Windows CMD and introduce the following command: wget –user EarthDatauser –ask-password –auth-no-challenge –no-check-certificate -i interestingURLSfile.txt2 The HDF files will be downloaded in your current directory. 3.2 Process HDF files to txt files 1. Set the directories: write dir, read dir 2. Select space-time domain in MATLAB code 2To use this command a user login has to be made into the Earth Data website 9
3. Execute the code and enter option ”1” when asked by command window 3.3 Process NC files 1. Set the reading/writing directories 2. select space-time domain 3. Execute the program 3.4 LMA vs LIS comparator 1. Set the Reading directories 10
2. Set the time domain 3. Set the time step (etimestep) 4. Select Modes: Recommended setup as shown in the figure below 5. Execute the program 11
A.6. CODES FOR DATA PROCESSING A.6 Codes for data processing Below are displayed the MATLAB codes produced for this project. They are also available at https://github.com/icarfontcu/LIS-LMA-data-reading-codes. A.6.1 LIS HDF processor.m 1%//////////////////////////////////////////// 2%// LIS HDF4 data processor // 3%// MATLAB converter // 4%// Universitat Polit cnica de Catalunya // 5%// [email protected] // 6%// // 7%// Requires: MATLAB R2017b at least // 8%// // 9%// // 10 %//////////////////////////////////////////// 11 12 13 14 15 clear all; 16 close all; 17 clc; 18 19 % #################USER MODIFIABLE VARIABLES #################### 20 read dir='/Users/Icar/Desktop/transfer/EUMETSAT/1 DE/HDFprova/';%from where to read ... HDF files 21 write dir scilab='C:\Users\icar\Desktop\LIS HDF files\Barrancabermeja\Barranca txt Jun Jul'; 22 write dir='C:\Users\icar\Desktop\LIS HDF files\Barrancabermeja\Barranca txt Jun Jul'; 23 read dir txtfiles=write dir; %atm we read the txt files from the same place we store ... them 24 urls file='C:\Users\icar\Google ... Drive\PRACTIQUES\HDF reading\reading txt files\GHRC URLs.txt'; 25 corrected urls file write dir='C:\Users\icar\Desktop\LIS HDF files\Barrancabermeja\';... %here will also go the file twith urls from the website names file 26 relevant orbits file='C:\Users\icar\Google ... Drive\PRACTIQUES\HDF reading\reading txt files\relevant orbits.txt'; 27 website filename='C:\Users\icar\Desktop\LIS HDF files\Barrancabermeja\filenames website.txt'; 28 map folder='C:\Users\icar\Google Drive\PRACTIQUES\LMAvsLIS (sci ... oscar)\Sci program\mapfolder'; 29 %------------ 30 31 %For adding LIS FOV to LMA files: 32 ISSLIS datafile = '/Users/Icar/Desktop/transfer/EUMETSAT/1 DE/ISS LIS/ISS LIS.mat';... %file containing the mat with all LIS data 33 LMA data dir = '/Users/Icar/Desktop/transfer/EUMETSAT/1 DE/LMA/';%WHere the LMA txt ... files are and the corrected ones will be written 34 HDF file = ... '/Users/Icar/Desktop/transfer/EUMETSAT/1 DE/HDFprova/ISS LIS SC P0.2 20180809 NQC 09105.hdf'; 35 day of the pass = 180809; 36 37 %Other------- 38 write other info dir='C:\Users\icar\Desktop\LIS HDF files\Ebre\Ebre other info\';... %specify this to save the workspace and other info there 39 %------------ 40 41 %coordinates info 42 deltebre=[40.75 0.95]; 43 santamarta=[11.2403547 -74.2110227]; 44 barranca=[7.06878 -73.744418]; 45 46 %scanning area specification 47 centroid=deltebre; %LAT/LON (remember, on the plot, this would be y and x) 48 range=60*sqrt(2); %range in km. With the range provided (box of LAT: 40.1 to 41.4, ... LON: 0.4 to 1.5) this should be 46km approx. But tolerance cna be applied 49 %time interval 50 starttime=datetime(2018,6,1,0,0,0); 51 endtime=datetime(2018,7,3,8,0,0); 89
A.6. CODES FOR DATA PROCESSING 451 disp(['Printing file ' filename ' @ ' write dir]); 452 end 453 454 end 455 456 457 458 end 459 function w txtfiles(interestingfiles,write dir,centroid, ang range,timerange,n) 460 disp('Printing general events .txt files...'); 461 462 463 ninterestingfiles=length(interestingfiles); %added constant for % use 464 for k=1:length(interestingfiles) %We will go through all interestingfiles 465 %and search for interestingevents 466 clear area vdata flash vdata group vdata event vdata 467 468 filename=interestingfiles(k).Filename; 469 fileinfo=hdfinfo(filename); 470 471 event vdata=hdfread(fileinfo.Vgroup.Vgroup.Vgroup.Vdata(4)); 472 event.location=event vdata{3}; 473 event.tai time=event vdata{1}; 474 475 area vdata=hdfread(fileinfo.Vgroup.Vgroup.Vgroup.Vdata(1)); 476 flash vdata=hdfread(fileinfo.Vgroup.Vgroup.Vgroup.Vdata(2)); 477 group vdata=hdfread(fileinfo.Vgroup.Vgroup.Vgroup.Vdata(3)); 478 event vdata=hdfread(fileinfo.Vgroup.Vgroup.Vgroup.Vdata(4)); 479 480 event.location=event vdata{3}; 481 event.tai time=event vdata{1}; 482 event.observe time=event vdata{2}; 483 event.radiance=event vdata{4}; 484 event.address=event vdata{6}+1; %add +1 so it doesn't start w/ 0 adress 485 event.parent address=event vdata{7}+1; 486 event.bg radiance=event vdata{11}; 487 488 event.x pixel=event vdata{8}; 489 event.y pixel=event vdata{9}; 490 event.bg rad=event vdata{11}; 491 492 group.parent address=group vdata{7}+1; %no need to store group.address as ... they are ordered by number inside each file 493 group.location=group vdata{3}; 494 495 flash.parent address=flash vdata{8}+1; 496 flash.location=flash vdata{4}; 497 498 area.location=area vdata{4}; 499 area.observetime=area vdata{3}; 500 501 orbit vdata=hdfread(fileinfo.Vgroup.Vdata(1)); 502 orbit id=orbit vdata{1}; 503 504 505 506 %We could size this info for printing and plotting, but better not 507 %to change what works 508 time=datetime(1993,1,1)+seconds(event.tai time); 509 lats=event.location(1,:); 510 lons=event.location(2,:); 511 512 ang distances=sqrt((lats-centroid(1)).ˆ2+(lons-centroid(2)).ˆ2); 513 514 space ind=find(ang distances<ang range);%get indexes of space dominum 515 time ind=intersect(find(time>timerange(1)),find(time<timerange(2))); 516 %get indexes of time dominum. Intersect gives common values, so 517 %here its ok because finds gives indexes and we want to find 518 %common indexes 519 520 [view ind]=intersect(space ind,time ind); %relevant elements of each file 521 %intersect both indexes in order to find positios that meet 522 %space-time dominum 96
A.6. CODES FOR DATA PROCESSING 523 starttime=time(min(view ind)); 524 starttime=datestr(starttime,'HHMM'); 525 endtime=time(max(view ind)); 526 endtime=datestr(endtime,'HHMM'); 527 %----------------------------------------------------------- 528 529 %prepare for writing interestingevents 530 q=erase(filename,'.hdf'); 531 q=erase(q,' NQC'); 532 erasethis=[' ' num2str(orbit id,'%05i')]; 533 q=erase(q,erasethis); 534 q=erase(q,' SC P0.2'); 535 filetextname=[q,' ' starttime, ' ', endtime,' events','.txt']; 536 fullfilename=fullfile(write dir,filetextname); 537 fileID=fopen(fullfilename,'w'); 538 539 event properties{1,17}=[]; 540 if fileID==-1 541 disp('Could not open file!'); 542 543 else 544 545 %disp(['Printing file n' num2str(k) ', ' filetextname ' @ ' write dir]); 546 disp([num2str(k*100/ninterestingfiles) '% of .txt files printed']); 547 548 %Write Header of the file 549 550 fprintf(fileID, 'TAI93 time e lat e lon e radiance group g lat g lon flash ... f lat f lon area a lat a lon a observe time x pixel y pixel ... bg radiance\r\n'); 551 552 553 554 555 556 557 j=1; %n of intersting events in each file 558 for i=1:length(event.tai time) %HERE THE PROGRAM PRINTS 559 560 eventinsidetime=false; 561 eventinsiderange=false; 562 563 %Is this event interesting? 564 %---------------------------------------------------------------------- 565 coordinates=event.location(:,i); 566 c distance=sqrt((coordinates(1)-centroid(1))ˆ2+(coordinates(2)-centroid(2))ˆ2); 567 if c distance<ang range 568 eventinsiderange=true; 569 end 570 571 %check time domain 572 time=datetime(1993,1,1)+seconds(event.tai time(i)); 573 if time<timerange(2) && time>timerange(1) 574 eventinsidetime=true; 575 end 576 %---------------------------------------------------------------------- 577 578 if eventinsidetime==true && eventinsiderange==true 579 580 581 %Make new clear info with time event, its radiance, adress, group 582 %adress, flash adress and area adress 583 584 tai time=event.tai time(i); 585 event lat=event.location(1,i); 586 event lon=event.location(2,i); 587 radiance=event.radiance(i); 588 bg rad=event.bg rad(i); 589 x pixel=event.x pixel(i); 590 y pixel=event.y pixel(i); 591 592 egroup=event.parent address(i);%his group address 593 group lat=group.location(1,egroup); 97
A.6. CODES FOR DATA PROCESSING 594 group lon=group.location(2,egroup); 595 596 eflash=group.parent address(egroup);%his flash address 597 flash lat=flash.location(1,eflash); 598 flash lon=flash.location(2,eflash); 599 600 earea=flash.parent address(eflash);%his area address 601 area lat=area.location(1,earea); 602 area lon=area.location(2,earea); 603 area observetime=area.observetime(earea); 604 605 fprintf(fileID, '%-18.16E %7.3f %7.3f %4.1d %u %7.3f %7.3f %u %7.3f %7.3f ... %u %7.3f %7.3f %u %u %u ... %u\r\n',tai time,event lat,event lon,radiance,egroup,group lat,group lon,eflash,flash lat,flash lon,earea,area lat,area lon,area observetime,x pixel,y pixel,bg rad); 606 607 a={... tai time,event lat,event lon,radiance,egroup,group lat,group lon,eflash,flash lat,flash lon,earea,area lat,area lon,area observetime,x pixel,y pixel,bg rad}; 608 event properties(j,:)=a; 609 %the group, event flash and area address will not be ordered nor 610 %listed from 1 to X, maybe from 543 to 987. That's because the 611 %parent address of an event is reset in each file (1-to-end), but 612 %when we recheck if an event is inside the spacew-time dominum we 613 %will eliminate, for example, all events that were part of de 614 %groups 1 to 453. 615 616 if isempty(event properties{j,4}) 617 disp('Radiance empty. code line 598 aprox'); 618 end 619 620 j=j+1; 621 622 623 end 624 625 626 end 627 628 fclose(fileID); 629 630 631 end 632 633 634 if n==2 635 disp('Plotting events form the current file...'); 636 plot events(event properties,centroid,ang range); 637 638 639 end 640 end 641 642 end 643 644 function read hdf4 and plot(interestingfiles,centroid,ang range,timerange) 645 646 disp('Reading data from interestingfiles and plotting...'); 647 for k=1:length(interestingfiles) %We will go through all interestingfiles 648 %and search for interestingevents 649 650 eventinsiderange=false; 651 eventinsidetime=false; 652 653 654 filename=interestingfiles(k).Filename; 655 fileinfo=hdfinfo(filename); 656 657 event vdata=hdfread(fileinfo.Vgroup.Vgroup.Vgroup.Vdata(4)); 658 event.location=event vdata{3}; 659 event.tai time=event vdata{1}; 660 661 area vdata=hdfread(fileinfo.Vgroup.Vgroup.Vgroup.Vdata(1)); 662 flash vdata=hdfread(fileinfo.Vgroup.Vgroup.Vgroup.Vdata(2)); 663 group vdata=hdfread(fileinfo.Vgroup.Vgroup.Vgroup.Vdata(3)); 98
A.6. CODES FOR DATA PROCESSING 664 event vdata=hdfread(fileinfo.Vgroup.Vgroup.Vgroup.Vdata(4)); 665 666 event.location=event vdata{3}; 667 event.tai time=event vdata{1}; 668 event.tai time=event vdata{1}; 669 event.observe time=event vdata{2}; 670 event.radiance=event vdata{4}; 671 event.address=event vdata{6}+1; %add +1 so it doesn't start w/ 0 adress 672 event.parent address=event vdata{7}+1; 673 event.bg radiance=event vdata{11}; 674 675 676 group.parent address=group vdata{7}+1; %no need to store group.address as ... they are ordered by number inside each file 677 group.location=group vdata{3}; 678 679 flash.parent address=flash vdata{8}+1; 680 flash.location=flash vdata{4}; 681 682 area.location=area vdata{4}; 683 area.observetime=area vdata{3}; 684 685 %prepare for writing interestingevents 686 687 688 %Write Header of the file 689 690 691 event properties{1,14}=[]; 692 for i=1:length(event.tai time) 693 694 %Is this event interesting? 695 %---------------------------------------------------------------------- 696 coordinates=event.location(:,i); 697 c distance=sqrt((coordinates(1)-centroid(1))ˆ2+(coordinates(2)-centroid(2))ˆ2); 698 if c distance<ang range 699 eventinsiderange=true; 700 end 701 702 %check time domain 703 time=datetime(1993,1,1)+seconds(event.tai time(i)); 704 if time<timerange(2) && time>timerange(1) 705 eventinsidetime=true; 706 end 707 %---------------------------------------------------------------------- 708 if eventinsidetime==true && eventinsiderange==true 709 710 711 %Make new clear info with time event, its radiance, adress, group 712 %adress, flash adress and area adress 713 714 tai time=event.tai time(i); 715 event lat=event.location(1,i); 716 event lon=event.location(2,i); 717 radiance=event.radiance(i); 718 719 egroup=event.parent address(i);%his group address 720 group lat=group.location(1,egroup); 721 group lon=group.location(2,egroup); 722 723 eflash=group.parent address(egroup);%his flash address 724 flash lat=flash.location(1,eflash); 725 flash lon=flash.location(2,eflash); 726 727 earea=flash.parent address(eflash);%his area address 728 area lat=area.location(1,earea); 729 area lon=area.location(2,earea); 730 area observetime=area.observetime(earea); 731 732 a={... tai time,event lat,event lon,radiance,egroup,group lat,group lon,eflash,flash lat,flash lon,earea,area lat,area lon,area observetime}; 733 event properties(i,:)=a; 734 end 99
A.6. CODES FOR DATA PROCESSING 735 736 eventinsiderange=false; 737 eventinsidetime=false; 738 end 739 740 plot events(event properties,centroid,ang range); 741 end 742 743 744 745 end 746 747 function read txt and plot(read dir,centroid,ang range,timerange) 748 749 addpath(read dir); 750 folderinfoprev=dir(read dir); 751 folderinfo=folderinfoprev(¬ismember({folderinfoprev.name},{'.','..','.DS Store'})); ... %.DS Store is a metadata file created by iOS environment 752 753 for k=1:size(folderinfo,1) 754 filename=folderinfo(k).name; 755 756 data=import event LIS txtfiles(filename); 757 event properties=[]; 758 time=datetime(1993,1,1)+seconds(data(:,1)); 759 lats=data(:,2); 760 lons=data(:,3); 761 762 ang distances=sqrt((lats-centroid(1)).ˆ2+(lons-centroid(2)).ˆ2); 763 764 space ind=find(ang distances<ang range); 765 time ind=intersect(find(time>timerange(1)),find(time<timerange(2))); 766 767 view ind=intersect(space ind,time ind); %relevant elements of each file 768 769 event properties(:,:)=data(view ind,:); 770 771 plot events(event properties,centroid,ang range); 772 773 %{ 774 for i=1:size(data,1) 775 eventinsiderange=false; %for each element we initialise inside checking 776 eventinsidetime=false; 777 778 coordinates=[data(i,2),data(i,3)]; 779 c distance=sqrt((coordinates(1)-centroid(1))ˆ2+(coordinates(2)-centroid(2))ˆ2); 780 if c distance<ang range 781 eventinsiderange=true; 782 end 783 784 %check time domain 785 time=datetime(1993,1,1)+seconds(data(i,1)); 786 if time<timerange(2) && time>timerange(1) 787 eventinsidetime=true; 788 end 789 790 if eventinsiderange==true && eventinsidetime==true 791 event properties(i,:)=data(i,:); 792 end 793 794 795 end 796 797 798 %call this function for each file. If the .txt file contains 799 %interesting events or not doesn't matter. If event properties is 800 %empty the function plot events will simply do nothing. 801 %This doesn't happen with HDF4 files because in that case we 802 %first search for interesting files that they already contain 803 %interesting information. 804 %} 805 806 end 100
A.6. CODES FOR DATA PROCESSING 807 end 808 809 function correct hdf urls(urls file,relevant orbits file,corrected urls file dir) 810 811 disp('Reading urls and relevant orbits txt file...'); 812 relevant orbits=import relevant orbits(relevant orbits file); 813 allurls=import urls times(urls file); 814 urls=import GHRCURLs(urls file); 815 816 disp('Reading URLs timing...'); 817 for i=1:length(allurls) 818 819 date=num2str(allurls(i,1)); 820 year=str2double(date(1:4)); 821 month=str2double(date([5 6])); 822 day=str2double(date([7 8])); 823 orbit days from urls(i)=datetime(year,month,day); 824 825 826 end 827 828 days that LIS passes the zone=relevant orbits(:,1); 829 830 unique days LIS passes=table2cell(unique(days that LIS passes the zone)); 831 832 833 disp('Checking URLs coincidence with relevant orbits...'); 834 eliminate indexes=[]; 835 for i=1:size(orbit days from urls,2) 836 837 finding=false; 838 for j=1:size(unique days LIS passes,1) 839 if orbit days from urls(i) == unique days LIS passes{j,1} 840 841 finding=true; 842 843 end 844 end 845 846 if finding==false 847 try 848 849 eliminate indexes=[eliminate indexes; i]; 850 851 catch 852 853 disp('error line 673'); 854 end 855 end 856 857 disp([num2str(i*100/size(orbit days from urls,2)) '% checked.']); 858 859 end 860 861 disp(['Found ' num2str(length(eliminate indexes)) ' non-relevant URLs']); 862 urls(eliminate indexes,:)=[]; 863 864 fullfilename=fullfile(corrected urls file dir,'interesting URLs.txt'); 865 writetable(urls,fullfilename); 866 disp(['Urls .txt file written at: ' corrected urls file dir]); 867 868 869 end 870 %plotters 871 function plot coastline(read dir,centroid,ang range) 872 873 load coastlines; 874 addpath(read dir); 875 disp('Plotting coastline...'); 876 877 878 latlim=[ centroid(1)-ang range centroid(1)+ang range ]; 879 lonlim=[centroid(2)-ang range centroid(2)+ang range]; 101
A.6. CODES FOR DATA PROCESSING 880 881 tf=ingeoquad(coastlat,coastlon,latlim,lonlim); 882 lats=coastlat(tf); 883 lons=coastlon(tf); 884 885 hold on; 886 plot(lons,lats,'k-'); 887 888 889 end 890 function plot events(event properties,centroid,ang range) 891 892 hold on; 893 894 if ¬isempty(event properties) 895 896 if iscell(event properties) 897 898 size=cell2mat(event properties(:,4)); 899 color=cell2mat(event properties(:,5)); 900 lats=cell2mat(event properties(:,2)); 901 lons=cell2mat(event properties(:,3)); 902 903 904 905 else 906 907 size=event properties(:,4); 908 color=event properties(:,5); 909 lats=event properties(:,2); 910 lons=(event properties(:,3)); 911 end 912 913 plot interesting area(centroid,ang range); 914 scatter(lons,lats,size,color,'x'); 915 xlabel('Longitude [deg.]'); 916 ylabel('Latitude [deg.]'); 917 grid on; 918 title('Events detected. Size ->Radiance. Color ->group'); 919 axis equal; 920 921 922 else 923 disp('Event properties vector is empty!'); 924 end 925 926 927 end 928 function plot interesting area(centroid,r) 929 x=centroid(2); 930 y=centroid(1); 931 932 hold on; 933 grid on; 934 935 th = 0:pi/50:2*pi; 936 xunit = r *cos(th) + x; 937 yunit = r *sin(th) + y; 938 plot(xunit, yunit,'k--','LineWidth',0.01); 939 %plot(x,y,'kx','LineWIdth',0.01); 940 axis equal; 941 942 end 943 944 %LMA FOV correction. Made by Prof. Montanya, J. 945 %adapted, updated and assembled to the general code by Fontcuberta . 946 947 function add FOV to LMA function(day of the pass, HDF, LMA data dir, LIS data file) 948 % J. Montanya 949 % 950 % Adds a column (Nbr 8) to LMA data where 1 = that source was in the FOV 951 % 0 = that source was NOT in the FOV 952 % 102
A.6. CODES FOR DATA PROCESSING 953 % 954 % Input: HDF of the pass 955 % LMA file from the Batch program 956 % ISS LIS.mat (ISS LIS data file) 957 % 958 % Output: LMA file in 'mat' format ith the extra columns 959 % 960 % The program uses the function check ISS FOV() 961 format long g 962 963 % Troba HH:MM:SS dels llamps de ISS per tal de saber franja horaria 964 % (p.e. ho faig servir per el batch program del LMA (nomes precessar el 965 % fitxer de 10 minuts que toca) 966 967 % 20181018 968 % dia=20181018; 969 % HDF=('C:\Joan\EUMETSAT\1 DE\HDF\ISS LIS SC P0.2 20181018 NQC 10190.hdf'); 970 % LMA=load('C:\Joan\EUMETSAT\1 DE\LMA\181018.txt'); 971 972 % 20180918 973 %dia=20180918; 974 %HDF=('C:\Joan\EUMETSAT\1 DE\HDF\ISS LIS SC P0.2 20181018 NQC 10190.hdf'); 975 %LMA=load('C:\Joan\EUMETSAT\1 DE\LMA\181018.txt'); 976 977 dia = ['20', day of the pass]; 978 979 disp('Loading LMA and LIS data...'); 980 981 LMA = load([LMA data dir, num2str(day of the pass), '.txt']); 982 LIS data=load(LIS data file); 983 LIS raw=LIS data.data; 984 clear LIS data 985 disp('Done!'); 986 987 % IMPORTANT: Anar al final a posar el nom del fitxer de sortida 988 %UPDATE: Solucionat (Icar) 989 990 [X,Y]=find(LIS raw(:,1)==dia); 991 LIS=LIS raw(X,:); 992 993 clear X Y LIS raw 994 995 996 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 997 %%%% First check for all LMA sources is ISS-LIS was within FOV % 998 %%%% Sets a new column (Nbr 8) with 1=YES in FOV 0: NO in FOV % 999 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1000 1001 [fil,col]=size(LMA); 1002 1003 1004 1005 1006 % Read HDF file used to determine if each LMA is within FOV 1007 disp('Reading selected HDF file...'); 1008 1009 1010 ISS info=hdfinfo(HDF); % Read HDF file 1011 ISS data=hdfread(ISS info.Vgroup.Vgroup.Vdata(2)); % Read Viewtime ... location, start time, end time, duration 1012 1013 Location=cell2mat(ISS data(1,1,1)); % Read location LAT LON of the ... 0.5x0.5 center of grid cells 1014 Location=Location'; % 1015 Location=double(Location); % Change to double in order to ... use later 1016 1017 1018 1019 % Time start : obtains the Start Times of the FOV for each 0.5x0.5 grid 1020 Time start TAI= cell2mat(ISS data(2,1)); % TAI 93 start time 1021 time=seconds(Time start TAI); % Converts times to format seconds 103
A.6. CODES FOR DATA PROCESSING 1022 TAI UTC = seconds(10) ; % This is the delay that matches ... with the ISS LIS data of the GHRC website 1023 time = (time - TAI UTC); 1024 1025 date = time + datenum([1993,1,1]); 1026 date = datenum(date); 1027 myDateTime = datetime(date, 'ConvertFrom','datenum'); 1028 myDateTime.Format = 'dd-MMM-uuuu HH:mm:ss.SSSSS';% Format of the ... myDateTime 1029 1030 % Creates the Start Time in seconds of the day (UTC) 1031 MyTime start = hour(myDateTime)*3600 + ... minute(myDateTime)*60+second(myDateTime); % Time in seconds of the day UTC 1032 1033 % Get the day in numeric format 1034 Date dia= year(myDateTime(1,1))*10000+month(myDateTime(1,1))*100 ... +day(myDateTime(1,1)); 1035 1036 clear time date myDateTime checkDate 1037 1038 1039 % Time end : obtains the End Times of the FOV for each 0.5x0.5 grid 1040 Time end TAI= cell2mat(ISS data(3,1)); % TAI 93 end time 1041 time=seconds(Time end TAI); 1042 TAI UTC = seconds(10) ; 1043 time = (time - TAI UTC); 1044 1045 date = time + datenum([1993,1,1]); 1046 date = datenum(date); 1047 myDateTime = datetime(date, 'ConvertFrom','datenum'); 1048 myDateTime.Format = 'dd-MMM-uuuu HH:mm:ss.SSSSS'; 1049 1050 % Creates the End Time in seconds of the day (UTC) 1051 MyTime end = hour(myDateTime)*3600 + ... minute(myDateTime)*60+second(myDateTime); % Time in seconds of the day UTC 1052 1053 % Duration of observation when the ISS LIS is withtin the FOV of the ... 0.5x0.5 grid cell %%%% 1054 1055 Duration= cell2mat(ISS data(4,1)); % Duration in seconds 1056 1057 1058 % We obtained: Location MyTime end MyTime start 1059 1060 1061 1062 disp('Done!'); 1063 1064 disp('Checking which sources were inside LIS FOV...') 1065 for i=1:1:fil 1066 1067 time LMA=LMA(i,2); 1068 LATLMA=LMA(i,4); 1069 LONLMA=LMA(i,5); 1070 1071 FOV = check ISS FOV(time LMA,LATLMA,LONLMA,Location,MyTime start,MyTime end); 1072 1073 if FOV==1 1074 LMA(i,8)=1; 1075 end 1076 1077 if FOV==0 1078 LMA(i,8)=0; 1079 end 1080 1081 clear FOV LATLMA LONLMA time lma 1082 1083 %i 1084 1085 end 1086 disp('Done!'); 1087 1088 104
A.6. CODES FOR DATA PROCESSING 1089 1090 1091 1092 1093 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1094 % PART 2: Adds Column 9 with the ID (also for the sources classified as 1095 % noise) 1096 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1097 disp('Adding a 9th column to the LMA data with the ID...'); 1098 disp(' Start to relate LMA noisy sources with the flash ID') 1099 clear i X Y FOV fil col 1100 1101 1102 LMA(:,9)=LMA(:,1); % Firt copy the flash ID of column 1 to column 9 1103 % that is beacuse sources with ID not 0 will not be 1104 % analyzed 1105 1106 [fil,col]=size(LMA); 1107 1108 1109 1110 End ID flash=max(LMA(:,1)); 1111 1112 for i=1:1:End ID flash 1113 1114 ID=i; 1115 1116 [X,Y]=find(LMA(:,1)==ID); 1117 1118 TimeLMAstartID=min(LMA(X(:,1),2)); 1119 TimeLMAendID=max(LMA(X(:,1),2)); 1120 1121 clear X Y 1122 1123 [X,Y]=find(LMA(:,1)==0 & LMA(:,2)≥TimeLMAstartID & LMA(:,2)≤TimeLMAendID ); 1124 1125 LMA(X,9)=ID; 1126 1127 clear X Y ID 1128 1129 1130 1131 1132 end 1133 disp('Done!'); 1134 1135 disp(['Writing data at ' LMA data dir '... ']); 1136 1137 LMA txt filename = [LMA data dir, 'LMA FOV ', num2str(day of the pass), ' wFOV']; 1138 1139 save([LMA txt filename '.mat'],'LMA'); 1140 1141 save([LMA txt filename '.txt'],'LMA','-ascii'); 1142 disp('Done!'); 1143 1144 end 1145 1146 1147 function FOV = check ISS FOV(time LMA,LATLMA,LONLMA,Location,MyTime start,MyTime end) 1148 1149 % 1150 % Cheks if a source is within the fiel of view 1151 % 1152 % Returns: 1153 % 1154 % FOV =1 if LMA source is within fov 1155 % FOV = 0 if LMA source is not within fov 1156 % 1157 1158 1159 1160 1161 105
A.6. CODES FOR DATA PROCESSING 1576 % column1: text (%s) 1577 % column2: datetimes (%{MMM}D) 1578 % column3: datetimes (%{dd}D) 1579 % For more information, see the TEXTSCAN documentation. 1580 formatSpec = '%s%{MMM}D%{dd}D%*s%*s%*s%*s%*s%*s%[ˆ\n\r]'; 1581 1582 % Open the text file. 1583 fileID = fopen(filename,'r'); 1584 1585 % Read columns of data according to the format. 1586 % This call is based on the structure of the file used to generate this 1587 % code. If an error occurs for a different file, try regenerating the code 1588 % from the Import Tool. 1589 dataArray = textscan(fileID, formatSpec, endRow(1)-startRow(1)+1, 'Delimiter',... delimiter, 'MultipleDelimsAsOne', true, 'TextType','string','HeaderLines',... startRow(1)-1, 'ReturnOnError', false, 'EndOfLine','\r\n'); 1590 for block=2:length(startRow) 1591 frewind(fileID); 1592 dataArrayBlock = textscan(fileID, formatSpec, endRow(block)-startRow(block)+1, ... 'Delimiter', delimiter, 'MultipleDelimsAsOne', true, 'TextType','string',... 'HeaderLines', startRow(block)-1, 'ReturnOnError', false, 'EndOfLine','\r\n'); 1593 for col=1:length(dataArray) 1594 dataArray{col}= [dataArray{col};dataArrayBlock{col}]; 1595 end 1596 end 1597 1598 % Close the text file. 1599 fclose(fileID); 1600 1601 % Post processing for unimportable data. 1602 % No unimportable data rules were applied during the import, so no post 1603 % processing code is included. To generate code which works for 1604 % unimportable data, select unimportable cells in a file and regenerate the 1605 % script. 1606 1607 % Create output variable 1608 files = table(dataArray{1:end-1},'VariableNames',{'name','month','month day'}); 1609 1610 % For code requiring serial dates (datenum) instead of datetime, uncomment 1611 % the following line(s) below to return the imported dates as datenum(s). 1612 1613 % files.month=datenum(files.month); 1614 % files.month day=datenum(files.month day); 1615 1616 end 112
A.6. CODES FOR DATA PROCESSING A.6.2 LIS vs LMA comparator.m 1%//////////////////////////////////////////// 2%// LIS HDF4 data processor // 3%// MATLAB converter // 4%// Universitat Politecnica de Catalunya // 5%// [email protected] // 6%// // 7%// Requires: MATLAB R2017b at least // 8%// // 9%// // 10 %//////////////////////////////////////////// 11 12 13 14 %% Program Presets 15 clear all; 16 close all; 17 clc; 18 19 % To use the program: 20 %1. select LMA filename of the day you want to study the sources 21 %2. Set the "examined day" variable as the same day. 22 %3. Set timestep 23 %4. Set plotting options: histograms/plotting (sources in 24 %time)/sources+events in time 25 %5.RUN 26 27 disp("########## START OF EXECUTION ##########"); 28 %{ 29 sources data file = 'C:\Users\icar\Google ... Drive\LIGHTNING\10minPeriodData\1710181700\171018 1700.txt'; 30 events data file = ... 'C:\Users\icar\Desktop\LIS HDF files\Ebre\Ebre txt\ISS LIS 20171018 1701 1701 events.txt'; 31 read workspace dir = ... 'C:\Users\icar\Desktop\LIS HDF files\Ebre\Ebre other info\workspace 40.7 0.7 171018 180713.mat'; 32 corrected file='C:\Users\icar\Desktop\txt4database\corrected file.txt'; 33 %} 34 LIS total filename = '/Users/Icar/Google Drive/TFG/LMA LIS DATA/ISS LIS.txt'; 35 LMA filename = '/Users/Icar/Google Drive/TFG/LMA LIS DATA/LMA FOV 20171018 pass1.txt'; 36 examined day = datetime(2017,10,18); %used to get the LMA date correctly 37 38 39 disp("###Remember to set the examined day to the one from where the LMA is coming"); 40 41 timestep = 2e-3; %sec 42 43 %{ 44 % addpath('C:\Users\icar\Desktop\LIS HDF files\Ebre\Ebre txt\'); 45 % addpath('C:\Users\icar\Desktop\Data from scilab\'); 46 47 %starttimes/endtimes 48 %ebre 49 %[18-Oct-2017 10:32:46 18-Oct-2017 10:34:31] 50 %[18-Oct-2017 17:01:25 17:01:28] 51 52 %barranca 53 %[18-Jul-03 6:40:14 6:41:18] 54 %[18-Jun-08 3:56:04 3:56:38] 55 56 %lisdata = import lis(events data file); 57 correcting=0; %1 if the database is used to correct postion 58 toinput=0; %1 if the user wants to input the start time and endtime of fov. If 0 the ... program will take the min(listime) and max(endtime). THIS IS BAD 59 storedinput=1; %the same as above but previously saved in fovtime: 60 %} 61 62 %Plotting Options 63 histograms =0; 64 plotting=1; 65 sources and events = 0; %plot sources and events over time. COMPUTATION LOADING 113
A.6. CODES FOR DATA PROCESSING 66 comparing length section=0; 67 68 %Other Options 69 save workspace=0; %if you want to write workspace. SAFEMODE=0 70 savingcsvfile=0; 71 72 73 74 disp(' '); 75 disp('-----------------------------------'); 76 %% Get LMA data 77 ouput data = get LMA data(LMA filename); 78 79 disp("Reading LMA data..."); 80 81 82 sources data = get LMA data(LMA filename); 83 84 lma.flash = sources data(:,1); 85 lma.sources time = examined day + seconds(sources data(:,2)); 86 lma.slats = sources data(:,4); 87 lma.slons = sources data(:,5); 88 lma.salts = sources data(:,6); 89 lma.pwr = sources data(:,7);%power 90 lma.LIS FOV = sources data(:,8); 91 lma.noise fID = sources data(:,9); 92 93 %eliminate those sources that were not under LIS FOV: 94 indexes = find(lma.LIS FOV ==0); 95 96 firstsize = length(lma.sources time); 97 secondsize = length(indexes); 98 disp(['Eliminating ' 'sources outside the LIS FOV time...']); 99 disp([num2str(firstsize-secondsize) ' elements within the FOV']); 100 101 lma.flash(indexes) = []; 102 lma.sources time(indexes) = []; 103 lma.slats(indexes) = []; 104 lma.slons(indexes) = []; 105 lma.salts(indexes) = []; 106 lma.pwr(indexes) = []; 107 lma.LIS FOV(indexes) =[]; 108 lma.noise fID(indexes) = []; 109 110 %The starttime of that day will be the minum time of the sources that were 111 %inside the FOV, that day. 112 113 starttime = min(lma.sources time); 114 115 endtime = max(lma.sources time); 116 117 118 119 120 %% Get LIS data 121 122 lisdata=import LIS total data(LIS total filename); 123 124 disp('Reading LIS data...'); 125 126 %LIS data 127 day string = num2str(lisdata(:,1)); 128 date day = datetime(day string,'InputFormat','yyyyMMdd'); 129 [lis.events time,I] = sort(date day+seconds(lisdata(:,2))); 130 131 lis.elats = lisdata(I,3); 132 lis.elons = lisdata(I,4); 133 lis.erad = lisdata(I,5); 134 lis.group = lisdata(I,6); 135 lis.glat = lisdata(I,7); 136 lis.glon = lisdata(I,8); 137 lis.flash = lisdata(I,9); 138 lis.flat = lisdata(I,10); 114
A.6. CODES FOR DATA PROCESSING 139 lis.flon = lisdata(I,11); 140 lis.area = lisdata(I,12); 141 lis.alat = lisdata(I,13); 142 lis.alon = lisdata(I,14); 143 lis.aobstime = lisdata(I,15); 144 lis.xpixel = lisdata(I,16); 145 lis.ypixel = lisdata(I,17); 146 lis.bgrad = lisdata(I,18); 147 lis.dis2lma = lisdata(I,19); 148 149 150 151 lis starttime address=find(lis.events time≥starttime,1,'first'); %first source ... time where LIS had it under its FOV 152 lis endtime address=find(lis.events time≤endtime,1,'last'); 153 154 155 erase indexes = [find(lis.events time<starttime); find(lis.events time>endtime)]; 156 157 158 lis.events time(erase indexes)=[]; 159 %lma.seconds flash(erase indexes)=[]; 160 lis.elats(erase indexes)=[]; 161 lis.elons(erase indexes)=[]; 162 lis.erad(erase indexes)=[]; 163 lis.group(erase indexes)=[]; 164 lis.glat(erase indexes)=[]; 165 %lma.cstp(erase indexes)=[]; 166 lis.glon(erase indexes)=[]; 167 lis.flash(erase indexes)=[]; 168 lis.flat(erase indexes)=[]; 169 lis.flon(erase indexes)=[]; 170 lis.area(erase indexes)=[]; 171 lis.alat(erase indexes)=[]; 172 lis.alon(erase indexes)=[]; 173 lis.aobstime(erase indexes)=[]; 174 lis.xpixel(erase indexes)=[]; 175 lis.ypixel(erase indexes)=[]; 176 lis.bgrad(erase indexes)=[]; 177 lis.dis2lma(erase indexes)=[]; 178 179 disp(['Start time: ']); 180 disp(starttime); 181 disp(['End time: ']); 182 disp(endtime); 183 184 %% Make chunksinfo structures 185 186 %{ 187 %load(read workspace dir,'fovinfo'); 188 %[min starttime on area, max endtime on area]=lisboundaries(fovinfo,lma); 189 %} 190 191 %{ 192 %Let's set a function that leaves only the data that should be seen from 193 %both sensors simultaniously 194 195 %Analize all the LMA detections and eliminate those that are outside the 196 %field of view. Create new lma struct and do the following: 197 %} 198 199 200 timeperiod = seconds(endtime-starttime); 201 202 %ftimestep = timeperiod/(lma.total nflashes.*10); %an adimensional number. 203 %The flashes may not fit in the same chunk but 204 %but w/ this criteria we got a avalue 205 %that changes with no of flashes 206 207 208 %etimestep = timeperiod/(lis.nevents*2); %check if inside a eventstime chunk therees ... a LMA detection! 209 etimestep = timestep; 115
A.6. CODES FOR DATA PROCESSING 210 211 disp('Dividing time in chunks....'); 212 % Precence 213 [chunksinfo,chunksdata]=check detections(etimestep,starttime,endtime,lis,lma); 214 215 chunksinfo=e vs s presence(chunksdata,chunksinfo); 216 217 chunksinfo.annotation='.times gives the minutes coordinates, from the hour we are ... regarding, of all chunks that divide the detection period. Other .chunk guive ... the ADDRESS of the chunks.'; 218 219 220 %physical properties of chunks where events and/or sources where detected 221 [secproperties, scproperties]=chunk physical properties(chunksinfo,lis,lma); 222 disp('Info about the time chunks saved'); 223 224 225 chunksinfo 226 227 %% Statistics Mean,Medians... 228 %power 229 secmaxpwrarray=array max power(secproperties); 230 scmaxpwrarray=array max power(scproperties); 231 232 semeanmaxpower=mean(secmaxpwrarray); 233 semedianmaxpower=median(secmaxpwrarray); 234 235 smeanmaxpower=mean(scmaxpwrarray); 236 smedianmaxpower=median(scmaxpwrarray); 237 238 secpwrcentroid=array power centroid(secproperties); 239 scpwrcentroid=array power centroid(scproperties); 240 241 semeanpowercentroid=mean(secpwrcentroid); 242 semedianpowercentroid=median(secpwrcentroid); 243 244 smeanpowercentroid=mean(scpwrcentroid); 245 smedianpowercentroid=median(scpwrcentroid); 246 247 248 %height 249 %secmedian, secmen, ... are vectors taht contain the mean height of all the 250 %sources inside each chunk, one value per chunk. 251 secmedian=array h median(secproperties); 252 scmedian=array h median(scproperties); 253 254 secmean=array h mean(secproperties); 255 scmean=array h mean(scproperties); 256 257 sheightmedianmean=mean(scmedian); 258 seheightmedianmean=mean(secmedian); 259 260 seheightmeanmean=mean(secmean); 261 sheightmeanmean=mean(scmean); 262 263 seheightmeanmedian=median(secmean); 264 sheightmeanmedian=median(scmean); 265 266 %number of sources 267 %nsources/nesources is the vector that contain information about the number 268 %of sources in each chunk that contains only/with events sources 269 270 [nesources,nsources]=nsourcesxchunks(chunksinfo); 271 senumsourcesmean=mean(nesources); 272 senumsourcesmedian=median(nesources); 273 274 snumsourcesmean=mean(nsources); 275 snumsourcesmedian=median(nsources); 276 277 %Sum of powers in each chunk 278 %sesumpower/ssumpower are the vectors that contain the simple sum of all 279 %the powers in each chunk with events/only sources 280 [sesumpower,ssumpower]=sumpowersinchunks(chunksinfo,lma); 116
A.6. CODES FOR DATA PROCESSING 281 282 283 sesumpowermean=mean(sesumpower); 284 ssumpowermean=mean(ssumpower); 285 286 sesumpowermedian=median(sesumpower); 287 ssumpowermedian=median(ssumpower); 288 289 if save workspace==1 290 disp('saving workspace...'); 291 save([erase(sources data file,'.txt')'.mat']); 292 disp('done'); 293 294 end 295 296 297 298 299 300 %% Plotting representative values 301 n=1; 302 if plotting==1 303 disp('Plotting over time started...'); 304 close all; 305 %x axis limits: 306 [h,m,s]=hms([starttime-seconds(etimestep) endtime+seconds(etimestep)]); 307 lims=minutes(minutes(m)+seconds(s)); 308 309 % Power may be related to the detection from LIS 310 311 %maximum power 312 figure(n); 313 n=n+1; 314 315 scatter(minutes(chunksinfo.times(chunksinfo.chunks w both)),secmaxpwrarray,'*b'); 316 hold on; 317 scatter(minutes(chunksinfo.times(chunksinfo.chunks only sources)),scmaxpwrarray,'.k'); 318 grid on; 319 title('Maximum power from the sources registered in each time chunk'); 320 xlabel('Time from the current hour [min.]'); 321 ylabel('Max power [dW]'); 322 legend('Chunk w/ events+sources','Chunk w/ only sources'); 323 %xlim(lims); 324 colormap(jet); 325 326 figure(n); 327 n=n+1; 328 scatter(minutes(chunksinfo.times(chunksinfo.chunks w both)),secpwrcentroid,'*b'); 329 hold on; 330 scatter(minutes(chunksinfo.times(chunksinfo.chunks only sources)),scpwrcentroid,'.k'); 331 grid on; 332 333 title('Height of power centroid in the chunk'); 334 xlabel('Time from the current hour [min.]'); 335 ylabel('Height [m]'); 336 legend('Chunk w/ events+sources','Chunk w/ sources'); 337 %xlim(lims); 338 colormap(jet); 339 340 %.----------------------------------------------------------------- 341 figure(n); 342 n=n+1; 343 [or times, indexv]=sort(lis.events time,1); 344 or pixels(:,1)=lis.xpixel(indexv); 345 or pixels(:,2)=lis.ypixel(indexv); 346 347 or times=datenum(or times); 348 scatter(or pixels(:,1),or pixels(:,2),1,or times,'.'); 349 axis equal; 350 grid on; 351 axis([0 128 0 128]); 352 title('Pixels excited on the CCD. Colored by time'); 353 colormap(jet); 117
A.6. CODES FOR DATA PROCESSING 354 355 %------------------------------------------------------------------- 356 357 disp('Continuing plotting...'); 358 % Comparing Altitudes in detection 359 %Median of the height 360 figure(n); 361 n=n+1; 362 363 364 365 scatter(minutes(chunksinfo.times(chunksinfo.chunks w both)),secmedian,'*b'); 366 hold on; 367 scatter(minutes(chunksinfo.times(chunksinfo.chunks only sources)),scmedian,'.k'); 368 grid on; 369 title('Median of sources height in each time chunk'); 370 xlabel('Time from the current hour [min.]'); 371 ylabel('Height [m]'); 372 legend('Chunk w/ events+sources','Chunk w/ sources'); 373 xlim(lims); 374 colormap(jet); 375 376 377 378 figure(n); 379 n=n+1; 380 381 scatter(minutes(chunksinfo.times(chunksinfo.chunks w both)),secmean,'*b'); 382 hold on; 383 scatter(minutes(chunksinfo.times(chunksinfo.chunks only sources)),scmean,'.k'); 384 grid on; 385 title('Mean of sources height in each time chunk'); 386 xlabel('Time from the current hour [min.]'); 387 ylabel('Height [m]'); 388 legend('Chunk w/ events+sources','Chunk w/ sources'); 389 %xlim(lims); 390 391 colormap(jet); 392 393 end 394 395 if sources and events ==1 396 disp("Plotting events & sources over time..."); 397 % plot events with sources 398 figure(n); 399 n=n+1; 400 k=0; 401 grid on; 402 hold on; 403 404 [C,ia,ic]=unique(lis.flash); 405 colorvalues(1)=5; 406 for j=2:length(C) 407 408 colorvalues(1,j)=colorvalues(j-1)+30; 409 410 end 411 412 colorvector=colorvalues(ic)'; 413 414 415 for i=1:length(chunksinfo.events) 416 417 local addresses=chunksinfo.events{i}; 418 if ¬isempty(local addresses) 419 scatter(lis.events time(local addresses),linspace(500,500,length(local addresses)),lis.erad(local addresses),colorvector(local addresses),'x'); 420 k=k+length(local addresses); 421 422 end 423 424 425 end 426 118
A.6. CODES FOR DATA PROCESSING 427 [C,ia,ic]=unique(lma.flash); 428 colorvalues(1)=5; 429 for j=2:length(C) 430 431 colorvalues(1,j)=colorvalues(j-1)+30; 432 433 end 434 435 colorvector=colorvalues(ic)'; 436 437 for i=1:length(chunksinfo.sources) 438 439 local addresses=chunksinfo.sources{i}; 440 if ¬isempty(local addresses) 441 try 442 scatter(lma.sources time(local addresses),lma.salts(local addresses),abs(lma.pwr(local addresses)),colorvector(local addresses),'+'); 443 catch 444 warning("Unable to plot Events & Sources"); 445 end 446 end 447 448 449 end 450 ylabel('Height [m]'); 451 title('Events and sources printed over time'); 452 legend('x Events. Size==radiance. Color==flash.','+ Sources. Size==power. ... Color==flash.'); 453 454 %etimestep=seconds(etimestep); 455 %timev=(starttime-etimestep/2):etimestep:(endtime+etimestep/2); 456 %xticks(timev); 457 458 xlim([starttime-seconds(etimestep) endtime+seconds(etimestep)]); 459 460 end 461 cif savingcsvfile==1 462 % Write CSV file to open with excel 463 disp('Writing csv file...'); 464 M=[semeanmaxpower semedianmaxpower smeanmaxpower smedianmaxpower semeanpowercentroid ... semedianpowercentroid smeanpowercentroid smedianpowercentroid sheightmedianmean ... seheightmedianmean seheightmeanmean sheightmeanmean seheightmeanmedian ... sheightmeanmedian senumsourcesmean senumsourcesmedian snumsourcesmean ... snumsourcesmedian sesumpowermean ssumpowermean sesumpowermedian ssumpowermedian]; 465 csvwrite([erase(sources data file,'.txt')'.csv'],M); 466 disp('Done.'); 467 468 end 469 470 %% Plot histograms 471 if histograms == 1 472 disp('Plotting histograms...'); 473 474 475 476 %maximum power 477 figure(n); 478 n=n+1; 479 480 histogram(secmaxpwrarray,30); 481 hold on; 482 histogram(scmaxpwrarray,30); 483 title('Bins max power histogram'); 484 xlabel('Max power in the bin [dbW]'); 485 ylabel('Counts'); 486 legend('Sources + events bins','only sources bins'); 487 488 489 %Heights 490 491 figure(n); 492 n=n+1; 493 histogram(secmedian,30); 494 hold on; 119
A.6. CODES FOR DATA PROCESSING 495 histogram(scmedian,30); 496 497 title('Bins median height histogram'); 498 ylabel('Counts'); 499 xlabel('Height [m]'); 500 legend('sources+events bins','only sources bins'); 501 502 503 figure(n); 504 n=n+1; 505 506 histogram(secmean,30); 507 hold on; 508 509 histogram(scmean,30); 510 title('Bins height mean histogram'); 511 ylabel('Counts'); 512 xlabel('Height [m]'); 513 legend('sources+events bins','only sources bins'); 514 515 516 517 518 %sources densities per bin 519 figure(n); 520 n=n+1; 521 histogram(nesources); 522 hold on; 523 histogram(nsources); 524 title('Density of sources per bin'); 525 ylabel('Counts'); 526 xlabel('Number of sources per bin'); 527 legend('Bins with sources+events','Bins with only sources'); 528 529 530 figure(n); 531 n=n+1; 532 histogram(secpwrcentroid,30); 533 hold on; 534 histogram(scpwrcentroid,30); 535 title('Bins power centroid histogram'); 536 ylabel('Counts'); 537 xlabel('Height [m]'); 538 legend('sources+events bins','only sources bins'); 539 540 541 figure(n); 542 n=n+1; 543 histogram(sesumpower) 544 hold on; 545 histogram(ssumpower) 546 title('Distributions for the sum of power in each bin'); 547 xlabel('Power [dbW]'); 548 ylabel('Counts'); 549 legend('sources+events bins','only sources bins'); 550 551 %Power population distribution 552 figure(n); 553 n=n+1; 554 555 formatOut='yyyy-mm-dd HH:MM'; 556 histogram(lma.pwr) 557 title( ['Power Histogram for ' datestr(starttime,formatOut)]); 558 xlabel('Power [dbW]'); 559 ylabel('Counts'); 560 561 figure(n); 562 n=n+1; 563 564 histogram(lis.erad) 565 title(['Radiance histogram for ' datestr(starttime,formatOut)]); 566 ylabel('Counts'); 567 xlabel('Radiance [J/(mˆ2 *sr *m)]'); 120
A.6. CODES FOR DATA PROCESSING 568 569 570 571 572 figure(n); 573 n=n+1; 574 histogram(lma.salts); 575 title(['Heights histograph for ' datestr(starttime,formatOut)]); 576 ylabel('Counts'); 577 xlabel('Height [m]'); 578 disp('plotting ended'); 579 580 581 end 582 583 %% Power of events that had a source at 2500m associated 584 585 DISCHARGE ALT = 2500; %m 586 tolerance = 50; %m. Tolerance for the search 587 588 [rad dischargealt] = ... typical radiance at height(DISCHARGE ALT,tolerance,lma,lis,chunksinfo); 589 590 disp(['Radiances at discharge height: ' num2str(DISCHARGE ALT) ... 591 'm with tolerance: ' num2str(tolerance) 'm:']); 592 disp(['Mean: ' num2str(mean(rad dischargealt.averages)) '[\mu J/sr/mˆ2/\mu m]']); 593 disp(['Median: ' num2str(median(rad dischargealt.medians)) '[\mu J/sr/mˆ2/\mu m]']); 594 595 figure(n); 596 n= n+1; 597 598 histogram(rad dischargealt.medians,30); 599 hold on; 600 histogram(rad dischargealt.averages,30); 601 title(['Typical values for the radiance of events that had sources associated at ' ... num2str(DISCHARGE ALT) ... 602 'm with tolerance: ' num2str(tolerance) 'm:']); 603 604 xlabel('Radiance [\mu J/sr/mˆ2/\mu m] '); 605 ylabel('Time bin counts'); 606 607 text = {strcat(['Altitude: ' num2str(rad dischargealt.alt) 'm ']); ... 608 strcat(['Tolerance: ' num2str(rad dischargealt.tol) 'm ']);... 609 strcat(['Total n of bins: ' num2str(rad dischargealt.nbins)])}; 610 611 dim = [.2 .55 .3 .3]; 612 annotation('textbox',dim,'String',text,'FitBoxToText','on'); 613 614 legend('Radiance median in the bin','Radiance average in the bin'); 615 616 617 %% Comparing lightning length 618 if comparing length section == 1 619 disp('comparing flash lengthes...'); 620 % GET THE LENGTH OF THE FLASHES. WE HAVE TO ASSIGN TIMINGS TO FLASHES AND 621 % THEN ASSIGN FLASHES OF LMA TO FLASHES OF LIS, SO WE COMPARE THE SAME 622 % FLASH. The LIS and LAST 623 624 %get the LIS flashes 625 %start times (and positions) 626 [flashnst,ilistst]=unique(lis.flash,'first'); %"flash number starts / index list ... starts] 627 %end times (and positions) 628 [flashnend,ilistend]=unique(lis.flash,'last'); %"flash number ends / index list ... ends" 629 630 lisflashprops.nflash=flashnst; 631 lisflashprops.ilistst=ilistst; 632 lisflashprops.ilistend=ilistend; 633 lisflashprops.times(:,1)=lis.events time(ilistst); 634 lisflashprops.times(:,2)=lis.events time(ilistend); 635 636 121
A.6. CODES FOR DATA PROCESSING 1055 for i=1:length(global addresses) 1056 local addresses=global addresses{i}; 1057 1058 pwr sumable = 10.ˆ(lma.pwr(local addresses)/10); %from dB to W 1059 1060 sesumpower(i)=10*log10((sum(pwr sumable))); %from W to dB 1061 1062 end 1063 global addresses=chunksinfo.sources(chunksinfo.chunks only sources); 1064 1065 ssumpower=zeros(1,length(global addresses)); 1066 1067 for i=1:length(global addresses) 1068 local addresses=global addresses{i}; 1069 1070 pwr sumable = 10.ˆ(lma.pwr(local addresses)/10); %from dB to W 1071 1072 ssumpower(i)=10*log10((sum(pwr sumable))); %from W to dB 1073 1074 end 1075 end 1076 1077 1078 %Reading data 1079 function sources data = import sourcesdata(filename) 1080 %IMPORTFILE Import numeric data from a text file as a matrix. 1081 % SOURCES DATA = IMPORTFILE(FILENAME) Reads data from text file FILENAME 1082 % for the default selection. 1083 % 1084 % SOURCES DATA = IMPORTFILE(FILENAME, STARTROW, ENDROW) Reads data from 1085 % rows STARTROW through ENDROW of text file FILENAME. 1086 % 1087 % Example: 1088 % sources data = importfile('2017 10 06 10 30.txt', 2, 4856); 1089 % 1090 % See also TEXTSCAN. 1091 1092 % Auto-generated by MATLAB on 2018/07/27 12:20:39 1093 1094 % Initialize variables. 1095 delimiter = ' '; 1096 if nargin≤2 1097 startRow = 2; 1098 endRow = inf; 1099 end 1100 1101 % Format for each line of text: 1102 % column1: double (%f) 1103 % column2: double (%f) 1104 % column3: double (%f) 1105 % column4: double (%f) 1106 % column5: double (%f) 1107 % column6: double (%f) 1108 % column7: double (%f) 1109 % column8: double (%f) 1110 % For more information, see the TEXTSCAN documentation. 1111 formatSpec = '%f%f%f%f%f%f%f%f%*s%[ˆ\n\r]'; 1112 1113 % Open the text file. 1114 fileID = fopen(filename,'r'); 1115 1116 % Read columns of data according to the format. 1117 % This call is based on the structure of the file used to generate this 1118 % code. If an error occurs for a different file, try regenerating the code 1119 % from the Import Tool. 1120 dataArray = textscan(fileID, formatSpec, endRow(1)-startRow(1)+1, 'Delimiter',... delimiter, 'MultipleDelimsAsOne', true, 'TextType','string','EmptyValue', NaN, ... 'HeaderLines', startRow(1)-1, 'ReturnOnError', false, 'EndOfLine','\r\n'); 1121 for block = 2:length(startRow) 1122 frewind(fileID); 1123 dataArrayBlock = textscan(fileID, formatSpec, endRow(block)-startRow(block)+1, ... 'Delimiter', delimiter, 'MultipleDelimsAsOne', true, 'TextType','string',... 'EmptyValue', NaN, 'HeaderLines', startRow(block)-1, 'ReturnOnError', false, ... 128
A.6. CODES FOR DATA PROCESSING 'EndOfLine','\r\n'); 1124 for col = 1:length(dataArray) 1125 dataArray{col}= [dataArray{col};dataArrayBlock{col}]; 1126 end 1127 end 1128 1129 % Close the text file. 1130 fclose(fileID); 1131 1132 % Post processing for unimportable data. 1133 % No unimportable data rules were applied during the import, so no post 1134 % processing code is included. To generate code which works for 1135 % unimportable data, select unimportable cells in a file and regenerate the 1136 % script. 1137 1138 % Create output variable 1139 sources data = [dataArray{1:end-1}]; 1140 end 1141 function sources header = import sources header(filename) 1142 %IMPORTFILE Import numeric data from a text file as a matrix. 1143 % SOURCES DATA = IMPORTFILE(FILENAME) Reads data from text file FILENAME 1144 % for the default selection. 1145 % 1146 % SOURCES DATA = IMPORTFILE(FILENAME, STARTROW, ENDROW) Reads data from 1147 % rows STARTROW through ENDROW of text file FILENAME. 1148 % 1149 % Example: 1150 % sources data = importfile('2017 10 06 10 30.txt', 1, 1); 1151 % 1152 % See also TEXTSCAN. 1153 1154 % Auto-generated by MATLAB on 2018/07/27 10:45:10 1155 1156 % Initialize variables. 1157 delimiter = ' '; 1158 if nargin≤2 1159 startRow = 1; 1160 endRow = 1; 1161 end 1162 1163 % Format for each line of text: 1164 % column1: double (%f) 1165 % column2: double (%f) 1166 % column3: double (%f) 1167 % column4: double (%f) 1168 % column5: double (%f) 1169 % column6: double (%f) 1170 % column7: double (%f) 1171 % column8: double (%f) 1172 % column9: double (%f) 1173 % For more information, see the TEXTSCAN documentation. 1174 formatSpec = '%f%f%f%f%f%f%f%f%f%[ˆ\n\r]'; 1175 1176 % Open the text file. 1177 fileID = fopen(filename,'r'); 1178 1179 % Read columns of data according to the format. 1180 % This call is based on the structure of the file used to generate this 1181 % code. If an error occurs for a different file, try regenerating the code 1182 % from the Import Tool. 1183 dataArray = textscan(fileID, formatSpec, endRow(1)-startRow(1)+1, 'Delimiter',... delimiter, 'MultipleDelimsAsOne', true, 'TextType','string','HeaderLines',... startRow(1)-1, 'ReturnOnError', false, 'EndOfLine','\r\n'); 1184 for block = 2:length(startRow) 1185 frewind(fileID); 1186 dataArrayBlock = textscan(fileID, formatSpec, endRow(block)-startRow(block)+1, ... 'Delimiter', delimiter, 'MultipleDelimsAsOne', true, 'TextType','string',... 'HeaderLines', startRow(block)-1, 'ReturnOnError', false, 'EndOfLine','\r\n'); 1187 for col = 1:length(dataArray) 1188 dataArray{col}= [dataArray{col};dataArrayBlock{col}]; 1189 end 1190 end 1191 129
A.6. CODES FOR DATA PROCESSING 1192 % Close the text file. 1193 fclose(fileID); 1194 1195 % Post processing for unimportable data. 1196 % No unimportable data rules were applied during the import, so no post 1197 % processing code is included. To generate code which works for 1198 % unimportable data, select unimportable cells in a file and regenerate the 1199 % script. 1200 1201 % Create output variable 1202 sources header = [dataArray{1:end-1}]; 1203 end 1204 function lisdata = import lis(filename) 1205 %IMPORTFILE Import numeric data from a text file as a matrix. 1206 % ISSLIS2017101916101610EVENTS = IMPORTFILE(FILENAME) Reads data from 1207 % text file FILENAME for the default selection. 1208 % 1209 % ISSLIS2017101916101610EVENTS = IMPORTFILE(FILENAME, STARTROW, ENDROW) 1210 % Reads data from rows STARTROW through ENDROW of text file FILENAME. 1211 % 1212 % Example: 1213 % ISSLIS2017101916101610events = ... importfile('ISS LIS 20171019 1610 1610 events.txt', 2, 7); 1214 % 1215 % See also TEXTSCAN. 1216 1217 % Auto-generated by MATLAB on 2018/07/27 12:09:22 1218 1219 % Initialize variables. 1220 delimiter = ' '; 1221 if nargin≤2 1222 startRow = 2; 1223 endRow = inf; 1224 end 1225 1226 % Format for each line of text: 1227 % column1: double (%f) 1228 % column2: double (%f) 1229 % column3: double (%f) 1230 % column4: double (%f) 1231 % column5: double (%f) 1232 % column6: double (%f) 1233 % column7: double (%f) 1234 % column8: double (%f) 1235 % column9: double (%f) 1236 % column10: double (%f) 1237 % column11: double (%f) 1238 % column12: double (%f) 1239 % column13: double (%f) 1240 % column14: double (%f) 1241 % column15: double (%f) 1242 % column16: double (%f) 1243 % column17: double (%f) 1244 % For more information, see the TEXTSCAN documentation. 1245 formatSpec = '%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%f%[ˆ\n\r]'; 1246 1247 % Open the text file. 1248 fileID = fopen(filename,'r'); 1249 1250 % Read columns of data according to the format. 1251 % This call is based on the structure of the file used to generate this 1252 % code. If an error occurs for a different file, try regenerating the code 1253 % from the Import Tool. 1254 dataArray = textscan(fileID, formatSpec, endRow(1)-startRow(1)+1, 'Delimiter',... delimiter, 'MultipleDelimsAsOne', true, 'TextType','string','HeaderLines',... startRow(1)-1, 'ReturnOnError', false, 'EndOfLine','\r\n'); 1255 for block = 2:length(startRow) 1256 frewind(fileID); 1257 dataArrayBlock = textscan(fileID, formatSpec, endRow(block)-startRow(block)+1, ... 'Delimiter', delimiter, 'MultipleDelimsAsOne', true, 'TextType','string',... 'HeaderLines', startRow(block)-1, 'ReturnOnError', false, 'EndOfLine','\r\n'); 1258 for col = 1:length(dataArray) 1259 dataArray{col}= [dataArray{col};dataArrayBlock{col}]; 130
A.6. CODES FOR DATA PROCESSING 1260 end 1261 end 1262 1263 % Close the text file. 1264 fclose(fileID); 1265 1266 % Post processing for unimportable data. 1267 % No unimportable data rules were applied during the import, so no post 1268 % processing code is included. To generate code which works for 1269 % unimportable data, select unimportable cells in a file and regenerate the 1270 % script. 1271 1272 % Create output variable 1273 lisdata = [dataArray{1:end-1}]; 1274 end 1275 1276 1277 function [min starttime on area, max endtime on area]=lisboundaries(fovinfo,lma) 1278 1279 %which file are we lookin in? 1280 1281 k=1; %i know "manually that in this case is the first interesting file 1282 coordinates=fovinfo.fov coordinates; 1283 1284 fovlats=coordinates(1,:); 1285 fovlons=coordinates(2,:); 1286 1287 minlat=lma.minlat; 1288 minlon=lma.minlon; 1289 maxlat=lma.maxlat; 1290 maxlon=lma.maxlon; 1291 1292 disp('Checking which points are inside the interesting area...'); 1293 inside range index=intersect(intersect(find(fovlats>minlat),find(fovlats<maxlat)),intersect(find(fovlons>minlon),find(fovlons<maxlon))); 1294 1295 %check at when the lis will start to see and leave the area. This is not 1296 %exact due to the fact that maybe the centroids are outside but the fov 1297 %cell has some part inside or viceversa. 1298 1299 interestingfovtimes=[fovinfo(k).fovstart(inside range index); ... fovinfo(k).fovend(insiderange index)]; 1300 1301 min starttime on area=min(interestingfovtimes(1,:)); %minimum time where the lis ... is sensing the area 1302 max endtime on area=max(interestingfovtimes(2,:)); %last time when LIS saw info ... on the area 1303 1304 min starttime on area=datetime(1993,1,1)+seconds(min starttime on area); 1305 max endtime on area=datetime(1993,1,1)+seconds(max endtime on area); 1306 1307 %we are assuming that LIS will record until the last moment on the 1308 %interesting area. Truly, during this last time the LIS only would be 1309 %able to see the little, last, only "franja" of the area. 1310 end 1311 1312 1313 1314 1315 function [date,time,lat,lon] = importcorrected(filename) 1316 %IMPORTFILE Import numeric data from a text file as column vectors. 1317 % [DATE1,TIME,LAT,LON] = IMPORTFILE(FILENAME) Reads data from text file 1318 % FILENAME for the default selection. 1319 % 1320 % [DATE1,TIME,LAT,LON] = IMPORTFILE(FILENAME, STARTROW, ENDROW) Reads 1321 % data from rows STARTROW through ENDROW of text file FILENAME. 1322 % 1323 % Example: 1324 % [date1,time,lat,lon] = importfile('paulino.txt.txt',1, 1713); 1325 % 1326 % See also TEXTSCAN. 1327 1328 % Auto-generated by MATLAB on 2018/07/31 13:20:32 1329 131
A.6. CODES FOR DATA PROCESSING 1330 % Initialize variables. 1331 delimiter = ';'; 1332 if nargin≤2 1333 startRow = 1; 1334 endRow = inf; 1335 end 1336 1337 % Format for each line of text: 1338 % column1: datetimes (%{yyyy-MM-dd}D) 1339 % column2: datetimes (%{HH:mm:ss}D) 1340 % column3: double (%f) 1341 % column4: double (%f) 1342 % For more information, see the TEXTSCAN documentation. 1343 formatSpec = '%{yyyy-MM-dd}D%{HH:mm:ss}D%f%f%[ˆ\n\r]'; 1344 1345 % Open the text file. 1346 fileID = fopen(filename,'r'); 1347 1348 % Read columns of data according to the format. 1349 % This call is based on the structure of the file used to generate this 1350 % code. If an error occurs for a different file, try regenerating the code 1351 % from the Import Tool. 1352 dataArray = textscan(fileID, formatSpec, endRow(1)-startRow(1)+1, 'Delimiter',... delimiter, 'TextType','string','HeaderLines', startRow(1)-1, 'ReturnOnError',... false, 'EndOfLine','\r\n'); 1353 for block=2:length(startRow) 1354 frewind(fileID); 1355 dataArrayBlock = textscan(fileID, formatSpec, endRow(block)-startRow(block)+1, ... 'Delimiter', delimiter, 'TextType','string','HeaderLines',... startRow(block)-1, 'ReturnOnError', false, 'EndOfLine','\r\n'); 1356 for col=1:length(dataArray) 1357 dataArray{col}= [dataArray{col};dataArrayBlock{col}]; 1358 end 1359 end 1360 1361 % Close the text file. 1362 fclose(fileID); 1363 1364 % Post processing for unimportable data. 1365 % No unimportable data rules were applied during the import, so no post 1366 % processing code is included. To generate code which works for 1367 % unimportable data, select unimportable cells in a file and regenerate the 1368 % script. 1369 1370 % Allocate imported array to column variable names 1371 date = dataArray{:, 1}; 1372 time = dataArray{:, 2}; 1373 lat = dataArray{:, 3}; 1374 lon = dataArray{:, 4}; 1375 1376 % For code requiring serial dates (datenum) instead of datetime, uncomment 1377 % the following line(s) below to return the imported dates as datenum(s). 1378 1379 % date1=datenum(date1); 1380 % time=datenum(time); 1381 end 1382 1383 1384 function [dir list]=check for folders(read dir,dir list) 1385 disp('Looking for folders with HDF4 files inside'); 1386 addpath(read dir); %current reading directory 1387 folderinfoprev=dir(read dir); 1388 folderinfo=folderinfoprev(¬ismember({folderinfoprev.name},{'.','..','.DS Store'})); ... %.DS Store is a metadata file created by iOS environment 1389 1390 aretherefolders=cell2mat({folderinfo.isdir}); 1391 1392 if any(aretherefolders)==true %check if there are folders inside the folder 1393 1394 namesarray={folderinfo(aretherefolders).name}; 1395 %gives a cell array but in char 1396 1397 %foldernames=convertCharsToStrings(namesarray); %lets convert it to string 132
A.6. CODES FOR DATA PROCESSING 1398 1399 for i=1:length(namesarray) 1400 % new read dir(i)=fullfile(read dir,foldernames(i)); 1401 new read dir=(fullfile(read dir,(namesarray{i}))); 1402 1403 [dir list]=check for folders(new read dir,dir list); %if the folder contains ... more folders, re-check 1404 1405 end 1406 1407 else 1408 1409 dir list=[dir list; read dir]; %if the folder is a file folder save its ... directory and make it travel through the function 1410 1411 1412 end 1413 1414 disp('Read directories will be:'); 1415 for i=1:size(dir list,1) 1416 disp(dir list(i,:)); 1417 end 1418 disp(' '); 1419 end 1420 1421 1422 1423 1424 1425 1426 %NEW VERSION FUNCTIONS 1427 function ouput data = get LMA data(filename, startRow, endRow) 1428 %IMPORTFILE Import numeric data from a text file as a matrix. 1429 % OUPUT DATA = IMPORTFILE(FILENAME) Reads data from text file FILENAME 1430 % for the default selection. 1431 % 1432 % OUPUT DATA = IMPORTFILE(FILENAME, STARTROW, ENDROW) Reads data from 1433 % rows STARTROW through ENDROW of text file FILENAME. 1434 % 1435 % Example: 1436 % ouput data = importfile('total LMA file noNoise.txt', 1, 4178); 1437 % 1438 % See also TEXTSCAN. 1439 1440 % Auto-generated by MATLAB on 2019/03/16 10:47:29 1441 1442 %% Initialize variables. 1443 delimiter = '\t'; 1444 if nargin≤2 1445 startRow = 1; 1446 endRow = inf; 1447 end 1448 1449 %% Format for each line of text: 1450 % column1: double (%f) 1451 % column2: double (%f) 1452 % column3: double (%f) 1453 % column4: double (%f) 1454 % column5: double (%f) 1455 % column6: double (%f) 1456 % column7: double (%f) 1457 % column8: double (%f) 1458 % column9: double (%f) 1459 % For more information, see the TEXTSCAN documentation. 1460 formatSpec = '%f%f%f%f%f%f%f%f%f%[ˆ\n\r]'; 1461 1462 %% Open the text file. 1463 fileID = fopen(filename,'r'); 1464 1465 %% Read columns of data according to the format. 1466 % This call is based on the structure of the file used to generate this 1467 % code. If an error occurs for a different file, try regenerating the code 1468 % from the Import Tool. 133
A.6. CODES FOR DATA PROCESSING 1469 dataArray = textscan(fileID, formatSpec, endRow(1)-startRow(1)+1, 'Delimiter',... delimiter, 'TextType','string','EmptyValue', NaN, 'HeaderLines',... startRow(1)-1, 'ReturnOnError', false, 'EndOfLine','\r\n'); 1470 for block=2:length(startRow) 1471 frewind(fileID); 1472 dataArrayBlock = textscan(fileID, formatSpec, endRow(block)-startRow(block)+1, ... 'Delimiter', delimiter, 'TextType','string','EmptyValue', NaN, ... 'HeaderLines', startRow(block)-1, 'ReturnOnError', false, 'EndOfLine','\r\n'); 1473 for col=1:length(dataArray) 1474 dataArray{col}= [dataArray{col};dataArrayBlock{col}]; 1475 end 1476 end 1477 1478 %% Close the text file. 1479 fclose(fileID); 1480 1481 %% Post processing for unimportable data. 1482 % No unimportable data rules were applied during the import, so no post 1483 % processing code is included. To generate code which works for 1484 % unimportable data, select unimportable cells in a file and regenerate the 1485 % script. 1486 1487 %% Create output variable 1488 ouput data = [dataArray{1:end-1}]; 1489 end 1490 function output data = import LIS total data(filename, startRow, endRow) 1491 %IMPORTFILE Import numeric data from a text file as a matrix. 1492 % OUTPUT DATA = IMPORTFILE(FILENAME) Reads data from text file FILENAME 1493 % for the default selection. 1494 % 1495 % OUTPUT DATA = IMPORTFILE(FILENAME, STARTROW, ENDROW) Reads data from 1496 % rows STARTROW through ENDROW of text file FILENAME. 1497 % 1498 % Example: 1499 % output data = importfile('ISS LIS.txt', 1, 216837); 1500 % 1501 % See also TEXTSCAN. 1502 1503 % Auto-generated by MATLAB on 2019/03/16 11:00:14 1504 1505 %% Initialize variables. 1506 if nargin≤2 1507 startRow = 1; 1508 endRow = inf; 1509 end 1510 1511 %% Format for each line of text: 1512 % column1: double (%f) 1513 % column2: double (%f) 1514 % column3: double (%f) 1515 % column4: double (%f) 1516 % column5: double (%f) 1517 % column6: double (%f) 1518 % column7: double (%f) 1519 % column8: double (%f) 1520 % column9: double (%f) 1521 % column10: double (%f) 1522 % column11: double (%f) 1523 % column12: double (%f) 1524 % column13: double (%f) 1525 % column14: double (%f) 1526 % column15: double (%f) 1527 % column16: double (%f) 1528 % column17: double (%f) 1529 % column18: double (%f) 1530 % column19: double (%f) 1531 % For more information, see the TEXTSCAN documentation. 1532 formatSpec = ... '%16f%16f%16f%16f%16f%16f%16f%16f%16f%16f%16f%16f%16f%16f%16f%16f%16f%16f%f%[ˆ\n\r]'; 1533 1534 %% Open the text file. 1535 fileID = fopen(filename,'r'); 1536 134
A.6. CODES FOR DATA PROCESSING 1537 %% Read columns of data according to the format. 1538 % This call is based on the structure of the file used to generate this 1539 % code. If an error occurs for a different file, try regenerating the code 1540 % from the Import Tool. 1541 dataArray = textscan(fileID, formatSpec, endRow(1)-startRow(1)+1, 'Delimiter','',... 'WhiteSpace','','TextType','string','EmptyValue', NaN, 'HeaderLines',... startRow(1)-1, 'ReturnOnError', false, 'EndOfLine','\r\n'); 1542 for block=2:length(startRow) 1543 frewind(fileID); 1544 dataArrayBlock = textscan(fileID, formatSpec, endRow(block)-startRow(block)+1, ... 'Delimiter','','WhiteSpace','','TextType','string','EmptyValue', NaN, ... 'HeaderLines', startRow(block)-1, 'ReturnOnError', false, 'EndOfLine','\r\n'); 1545 for col=1:length(dataArray) 1546 dataArray{col}= [dataArray{col};dataArrayBlock{col}]; 1547 end 1548 end 1549 1550 %% Close the text file. 1551 fclose(fileID); 1552 1553 %% Post processing for unimportable data. 1554 % No unimportable data rules were applied during the import, so no post 1555 % processing code is included. To generate code which works for 1556 % unimportable data, select unimportable cells in a file and regenerate the 1557 % script. 1558 1559 %% Create output variable 1560 output data = [dataArray{1:end-1}]; 1561 end 1562 1563 function [rad dischargealt] = ... typical radiance at height(DISCHARGE ALT,tolerance,lma,lis,chunksinfo) 1564 1565 1566 rad dischargealt.medians = []; 1567 rad dischargealt.averages = []; 1568 counter = 0; 1569 for i = 1:length(chunksinfo.chunks w both) 1570 1571 chunk address = chunksinfo.chunks w both(i); 1572 1573 local sources = chunksinfo.sources{chunk address}; 1574 1575 1576 if any(lma.salts(local sources)>DISCHARGE ALT-tolerance) ... 1577 && any(lma.salts(local sources)<DISCHARGE ALT +tolerance) %The ... heights of the sources is comprised in the tolerance 1578 1579 counter = counter +1; 1580 1581 local events = chunksinfo.events{chunk address};%access the content of ... the cell at that address; 1582 1583 local rads = lis.erad(local events); 1584 1585 median rads = median(local rads); 1586 average rads = mean(local rads); 1587 1588 rad dischargealt.medians(counter) = median rads; 1589 rad dischargealt.averages(counter) = average rads; 1590 1591 end 1592 1593 rad dischargealt.alt = DISCHARGE ALT; 1594 rad dischargealt.tol = tolerance; 1595 rad dischargealt.nbins = counter; 1596 end 1597 1598 end 135
Bibliography [1] H. J. Kramer. ISS Utilization: LIS (Lightning Imaging Sensor) on STP-H5’s investigations of DoD. Retrieved from https://directory.eoportal.org/web/eoportal/satellite-missions/i/iss-lis. [2] NASA,CHRC. ISS Lightning Sensors Data Sets. Retrieved from https://lightning.nsstc.nasa.gov/data/data iss lis.html. [3] Hugh J. Christian, Richard J. Blakeslee, and Steven J. Goodman. Lightning Imaging Sensor (US) for the Earth Observing System. NASA, February 1992. [4] P. M. Bitzer, H. Christian. Timing Uncertainty of the Lightning Imaging Sensor. J. Atmos. Oceanic Technol., 32, 453–460, https://doi.org/10.1175/JTECH-D-13-00177.1 [5] Richard J. Blakeslee. Lightning Imaging Sensor (LIS) on ISS and Plans for Sustained Ground Measurements in Support of GLM Cal/Val. Joint MTG LI Mission Advisory Group and GOES-R GLM Science Team Workshop, Rome, Italy. May 2015 [6] E.P. Krider. Atmospheric Electricity On-line Course. University of Arizona, 2015. Retrieved from: http://www.atmo.arizona.edu/students/courselinks/spring15/atmo589/ATMO489 online/CONTENTS.html. [7] Blumenfeld, J. New Lightning Imaging Sensor to be Installed on the International Space Station Retrieved from https://earthdata.nasa.gov/lis-on-iss. [8] S. Goodman, D. Mach, W. Koshak, R. Backslee. Algorithm Theoretical Basis Document. Center for Satellites Applications and Research. September 24, 2010. [9] R. J. Blakslee, H. Christian et al. Lightning Imaging Sensor (LIS) for the International Space Station (ISS): Mission Description and Science Goals. XV International Conference on Atmospheric Electricity, 15-20 June 2014, Norman, Oklahoma, U.S.A. [10] S.J. Goodman et al. The GOES-R Geostationary Lightning Mapper (GLM). Atmospheric Research 125–126 (2013) 34–49. [11] C. Lennon, L. Maier. LIGHTNING MAPPING SYSTEM NASA, Florida. [12] R.J. Thomas et al. Observations of VHF Source Powers Radiated by Lightning. Geophysical Research Letters, Vol. 28, NO. 1, pages 143-146, January 1,2001. [13] F. Fabr´o, J. Montany`a et al. Analysis of energetic radiation associated with 1 thunderstorms in the Ebro delta region in Spain. Geophys. Res. Atmos., 121, doi:10.1002/2015JD024573. [14] The HDF Group. HDF User’s Guide. Retrieved from https://www.hdfgroup.org/solutions/hdf4/, June 2017. [15] NetCDF 4.6.1. An introduction to NetCDF Retrieved from: https://www.unidata.ucar.edu/software/netcdf/docs/netcdf introduction.html. [16] Blakeslee Richard J., Douglas M. Mach, Michael F. Stewart, Dennis Buechler and Hugh Christian. 2017. Non-Quality Controlled Lightning Imaging Sensor (LIS) on International Space Station (ISS) Provisional Science Data. Dataset available online from the NASA Global Hydrology Center DAAC, Huntsville, Alabama, U.S.A. DOI: http://dx.doi.org/10.5067/LIS/ISSLIS/DATA204 136
BIBLIOGRAPHY [17] H. Christian, M. McCook et al. A Lightning Primer. Retrieved from https://lightning.nsstc.nasa.gov/primer/index.html. September 2018. [18] R. J. Thomas, Paul L. Krehbiel et al. Comparison of Ground-based 3-dimensional lightning mapping observations with satellite-based LIS observations in Oklahoma Geophysical Research Letters, vol. 27 nº12, pages 1703-1706, June 15, 2000. [19] Liu Feng et al. The Study of Active Atoms in High-Voltage Pulsed Coronal Discharge by Optical Diagnostics. 2005 Plasma Sci. Technol. 7 2851 [20] Servei Meteorol`ogic de Catalunya. Normals Clim`atiques Recents. Retrieved from www.meteo.cat/wpweb/climatologia/serveis i dades climatiques/normals climatiques recents/. April 2019. [21] P Andr´e et al. The calculation of monatomic spectral lines’ intensities and composition in plasma out of thermal equilibrium; evaluation of thermal disequilibrium in ICP torches. J. Phys. D: Appl. Phys. 30 (1997) 2043–2055. [22] E.V. Koryukina. Calculation of the Emission Spectra of Atoms and Ions in the External Electric field. 2nd International Congress on radiation physics, high current electronics and modification of materials. Tomsk Polytechnic University (2006). [23] C. Trassy, A. Tazeem. Simulation of atomic and ionic absorption and emission spectra for thermal plasma diagnostics: application to a volatilisation study in a plasma jet. Spectrochimica Acta Part B: Atomic Spectroscopy 54 (1999) 581]602. 137