Impact assessment of AWS constellation using real AWS data

ESA Contract 4000136311 Performance Evaluation of Arctic Weather Satellite Data Deliverable D6 v1.0 24th October 2025 Impact assessment of AWS constellation using real AWS data Reima Eresmaa1, Per Dahlgren2, David Schoenach1, Adam Dybbroe3 1Finnish Meteorological Institute (FMI) 2Norwegian Meteorological Institute (MET Norway) 3Swedish Meteorological and Hydrological Institute (SMHI) Corresponding author email [email protected] Table of Contents Table of Contents................................................................................................................... 2 1. Introduction............................................................................................................3 2. Summary of the previous impact evaluation...................................................... 4 3. New experiment runs............................................................................................ 5 3.1 Choice of the experiment dates........................................................................6 3.2 Technical settings............................................................................................. 7 3.2.1 Settings applied in the assimilation of Metop-B sounder data................. 8 3.2.2 Settings applied in the assimilation of AWS sounder data.......................8 4. Results.................................................................................................................. 11 4.1 Results against the depleted baseline............................................................11 4.2 Results against the realistic baseline............................................................. 16 5. Conclusions......................................................................................................... 21 References............................................................................................................... 22 ESA Contract 4000136311: Performance evaluation of Arctic Weather Satellite data Deliverable D6: Technical note on impact assessment of AWS constellation using real AWS measurement data v1.0 24th October 2025 2 1. Introduction This report is compiled within the research project “Performance evaluation of Arctic Weather Satellite Data” currently being funded by the European Space Agency (ESA). Being led by the Swedish Meteorological and Hydrological Institute (SMHI), the project has been running since December 2021 and it is scheduled to end in November 2025. The project organisation includes dedicated tasks focusing on the evaluation of the impact that the future EPS-Sterna satellite constellation can be expected to make on Numerical Weather Prediction (NWP) -based applications. Already before the launch of the Arctic Weather Satellite (AWS), that is, before August 2024, the project devoted a substantial effort to evaluating the expected EPS-Sterna constellation impact. The effort was documented in earlier project deliverables D4: “Technical note on impact assessment methodology” and D5: “Technical note on impact assessment of the AWS constellation using already operating microwave sounders”, released in August 2023 and May 2024, respectively. This first evaluation of the constellation impact was based on (1) quantifying the NWP forecast impact that is obtained via the assimilation of microwave sounder radiances collected from the Metop, NOAA, and Fengyun polar-orbiting satellites, and (2) making justified assumptions to extrapolate from the quantified impact to the hypothetical future impact attributed to the EPS-Sterna constellation of six polar-orbiting satellites. The results suggest that one can expect a positive impact that corresponds to a reduction of ca. 5% in the root-mean-squared error in the forecast of near-surface humidity and cloud-related parameters during the boreal winter season. In summer, as well as in parameters such as temperature and wind all year round, the magnitude of the constellation impact is expected to be considerably smaller. Having the Arctic Weather Satellite (AWS) been providing data for almost a year by now, it has become possible to refine the constellation impact evaluation on the basis of real measurements collected from a sounder that is similar to those to be employed by the future EPS-Sterna constellation satellites. Scientists at the Finnish and Norwegian meteorological institutes (FMI and MET Norway, respectively) have recently conducted further data assimilation experiments for this specific purpose. These actions are parts of the aforementioned SMHI-led research project. The remainder of this report documents the recent effort, presents the most important findings and discusses the implications regarding the expected EPS-Sterna constellation impact. ESA Contract 4000136311: Performance evaluation of Arctic Weather Satellite data Deliverable D6: Technical note on impact assessment of AWS constellation using real AWS measurement data v1.0 24th October 2025 3 2. Summary of the previous impact evaluation In a previous phase of the project, the EPS-Sterna constellation impact was estimated in two Nordic NWP domains that correspond to specific operational implementations of the HARMONIE-AROME NWP system. The domains adopted are those used by the AROME-Arctic (Müller et al., 2017a) and MetCoOp (Müller et al., 2017b) operational groups. The impact evaluation relied on running extensive numerical experiments, where microwave sounder radiances collected from the polar-orbiting satellites were fed into the NWP model through a four-dimensional variational (4D-Var) assimilation system. The accumulation of the forecast impact was investigated by evaluating the system performance in a series of model runs. As a baseline, there was a run that only assimilated microwave sounder data from one satellite (Fengyun-3D). In the subsequent runs, the number of assimilated data was gradually increased. This was done separately in two alternative scenarios, such that the new satellite data in each run came either from a satellite placed in the same sun-synchronous orbital plane as the baseline satellite or from a satellite placed in a complementary orbital plane. In both scenarios, the existing constellation of polar orbiters allowed to increase the number of used satellites from one to three. The impact of satellite data was found to be far greater in the scenario where the satellites were placed in complementary (rather than a single) orbital planes, and it was also found to be greater during the boreal winter season than in the summer. The most robust indication of a positive impact was in the forecast of near-surface humidity and cloud-related parameters at lead times up to 24 hours. The magnitude of the full constellation impact was estimated to be up to 5–10%, when measured in terms of reduction in the root-mean-squared error in the forecast of humidity at the surface level. However, in parameters such as temperature and wind speed, the impact expectation is notably smaller. The extrapolation from the demonstrated satellite impact at present to the expected constellation impact in future made use of a series of assumptions. A critical assumption was that the impact attributable to a single AWS-like satellite (that is, any one satellite in the EPS-Sterna constellation) should be comparable to that attributable to any one of the present microwave-sounding satellites in the Metop, NOAA and Fengyun satellite series. With the access established to the measurement data from the AWS satellite mission, the validity of this critical assumption can now be assessed for the first time. ESA Contract 4000136311: Performance evaluation of Arctic Weather Satellite data Deliverable D6: Technical note on impact assessment of AWS constellation using real AWS measurement data v1.0 24th October 2025 4 3. New experiment runs The new experiment runs are conducted in the MetCoOp computing domain shown in figure 1. As previously, the horizontal grid spacing remains at 2.5 km and the vertical discretization remains in the 65-level setup that extends from the 12-meter height above the ground to the 10 hPa pressure in the stratosphere. Other main characteristics of the NWP system setup include the following aspects: ● The reference system architecture is provided by the HARMONIE-AROME version 43h2.2, i.e., the version that remains the basis for the operational system e.g. at MetCoOp for the time being. ● The analysis of upper-air fields relies on the 4D-Var assimilation method. ● The analysis of upper-air fields is complemented by an independent optimal interpolation -based analysis of surface parameters. ● The analyses are conducted at the synoptic hours (00, 03, 06, 09, 12, 15, 18, and 21 UTC) and the numerical forecast is integrated out to the 36-hour lead time from the 00, 06, 12, and 18 UTC analyses. At the intermediate analysis times, the forecast is truncated to three hours. ● Lateral boundary forcing is applied and taken from the ECMWF global forecast model data There are a considerable number of system modifications implemented on top of the reference system in order to allow processing AWS radiance data. These will be explained in detail in subsection 3.2 below. The verification methods applied are similar to those applied in the previous assessment. These include (1) the forecast verification against observations taken at ground and radiosonde stations and (2) offline computation of Observation minus Background (OmB) departure statistics on various observation types. Figure 1: The MetCoOp operational computing domain used in this study. ESA Contract 4000136311: Performance evaluation of Arctic Weather Satellite data Deliverable D6: Technical note on impact assessment of AWS constellation using real AWS measurement data v1.0 24th October 2025 5 The new approach to the impact evaluation is to quantify the impact obtained, on the one hand, via the assimilation of AWS data and, on the other hand, via the assimilation of microwave sounder data collected from a single reference satellite. For this purpose, a baseline run is produced such that the use of microwave sounders is relatively limited. Additionally, separate reference and AWS-test runs are produced. These differ from the baseline only in terms of additionally assimilating the microwave sounder data from either the reference satellite or from AWS. Taking account of orbit characteristics of AWS and other relevant Earth Observation satellites, Metop-B is chosen as the reference satellite in this study. 3.1 Choice of the experiment dates The experiment dates are selected based on the following criteria: ● Given the limited working time remaining in the project and the heavy computational load associated with the runs with the 4D-Var data assimilation, the overall length of the evaluation period is targeted at four weeks. ● There is a need to run the assimilation in passive mode for two weeks prior to the evaluation period to ensure realism of the variational bias correction. ● Given that the previous impact assessment indicated considerable seasonal variation in the magnitude of the satellite impact, a time period in the cold season is preferred. ● The processing of the AWS sounder data was subjected to a new side-lobe correction on 13th March 2025. It is desirable to run the evaluation through dates after this correction was implemented. ● AWS provided no measurement data between 17th March, 21 UTC, and 20th March, 09 UTC. To run through a period that is as representative of boreal winter as possible, while avoiding the dates preceding the side-lobe correction, the new experiments are started on 21st March 2025. Allowing two weeks for the passive monitoring, the four-week evaluation period is from 4th April to 1st May 2025. No active assimilation of AWS data was in place in the ECMWF global forecast model at the time. During the evaluation period, only a few low pressure systems made notable impact on weather inside the domain. During the first few days, the weather was dominated by a high pressure situated over the Norwegian sea and northerly flow in most of the domain. A relatively strong low pressure system passed on the Northern side of the domain around 9th April, causing a brief period of strong westerly flow in the Northern parts of the domain. A period of mostly westerly or southwesterly (though weak) flow occurred from 11th to 14th April. There was a series of low pressure systems passing through the domain and causing mild baroclinic activity in the period of 15th to 19th April, followed by a calm period with no dominant flow from 20th to 24th April. Towards the end of the evaluation period, the synoptic situation varied a lot including influence from both high pressure in Central Europe and a low pressure system passing through the domain. ESA Contract 4000136311: Performance evaluation of Arctic Weather Satellite data Deliverable D6: Technical note on impact assessment of AWS constellation using real AWS measurement data v1.0 24th October 2025 6 3.2 Technical settings The NWP system runs are conducted on the High Performance Computing platforms provided and maintained by ECMWF. The details of the model runs are summarized in Table 1. In essence, there are two baseline runs that differ in terms of the use of observations. In the “depleted” baseline run, i.e., in bas1, none of the available microwave sounders are included in the assimilation (while the use of other observations is relatively close to operational settings and includes e.g. the radiance data from the IASI instrument). In the “realistic” baseline, i.e., in bas2, microwave sounder radiances collected from the NOAA and Fengyun satellites are assimilated the same way as in operational setup at MetCoOp at the time. The reference and AWS-test runs, i.e., ref1, aws1, ref2, and aws2 add the assimilation of microwave sounder data from either Metop-B or AWS satellites on top of the respective baseline runs as shown in the table. Table 1: The use of microwave sounder radiance data in the experiment runs. All runs make use of the MetCoOp operational domain as shown in figure 1. Run ID Experiment dates Use of microwave sounders bas1 21st March to 1st May 2025 None ref1 4th April to 1st May 2025 Metop-B: AMSU-A, MHS aws1 4th April to 1st May 2025 AWS: AWS sounder bas2 4th April to 1st May 2025 NOAA-18: AMSU-A NOAA-19: AMSU-A, MHS NOAA-20: ATMS NOAA-21: ATMS FengYun-3D: MWHS-2 FengYun-3E: MWHS-2 ref2 4th April to 1st May 2025 All those included in bas2 + Metop-B: AMSU-A, MHS aws2 4th April to 1st May 2025 All those included in bas2 + AWS: AWS sounder ESA Contract 4000136311: Performance evaluation of Arctic Weather Satellite data Deliverable D6: Technical note on impact assessment of AWS constellation using real AWS measurement data v1.0 24th October 2025 7 3.2.1 Settings applied in the assimilation of Metop-B sounder data The assimilation of the Metop-B microwave sounder radiances applies the operational settings as they are implemented at MetCoOp. The most relevant aspects are listed as follows: ● The use of AMSU-A includes active assimilation of channels 5, 7, 8, and 9, and the use of MHS includes channels 3, 4, and 5. ● The use of the low-peaking channels benefits from the dynamic estimation of surface emissivity and, consequently, use of data is made over all surface types. ● The standard deviation of observation error (σo) is assigned in the range from 0.162 K to 0.225 K on the AMSU-A channels, while it is set to 1.8 K on the MHS channels. All observation errors are assumed to be mutually uncorrelated. ● The screening for cloud contamination relies on the OmB departure on window channels: ○ AMSU-A channels 5 and 7 are rejected if the absolute value of OmB in channel 4 exceeds 0.7 K. ○ AMSU-A channels 8 and 9 are not screened for cloud contamination. ○ MHS channels 3, 4, and 5 are rejected if the absolute value of OmB in channel 2 exceeds 5.0 K. ● The data is thinned horizontally such that each 60 km by 60 km square will (at most) include only one observation point. 3.2.2 Settings applied in the assimilation of AWS sounder data As far as meaningful, the use of AWS sounder radiance data is made similar to that of other microwave sounders in the HARMONIE-AROME reference version. A notable exception is each 3 by 3 AWS pixel array is condensed into one superobservation at the location of the centre pixel. This pixel averaging is applied on all channels. The purpose of the pixel averaging is to reduce excessive noise and make the measurements better comparable with other microwave sounders. Regarding the use of the AWS channels, the following rules are applied: ● From the 50 GHz spectral region, the active assimilation includes channels 4, 5, 6, and 7. Note that channel 8 is discarded from the assimilation because of excessive noise. ● From the 183 GHz spectral region, the active assimilation includes channels 11, 12, 13, 14, and 15. ● The screening for cloud contamination makes use of the window channels 2 and 10. ● The dynamic estimation of surface emissivity makes use of the window channels 1 and 9. The use of the AWS channels is summarized in Table 2. ESA Contract 4000136311: Performance evaluation of Arctic Weather Satellite data Deliverable D6: Technical note on impact assessment of AWS constellation using real AWS measurement data v1.0 24th October 2025 8 Table 2: The use of the AWS channels. Channel number Frequency (GHz) Assimilation mode σo specification Notes 1 50.3 passive ( 33.3K )1 Used in the dynamic estimation of emissivity in channels 4-7 2 52.8 passive ( 33.3K ) Used in the screening for cloud contamination in channels 1-8 3 53.246 passive ( 33.3K ) 4 53.596 active 0.36K Temperature-sounding 5 54.4 active 0.315K Temperature-sounding 6 54.94 active 0.36K Temperature-sounding 7 55.5 active 0.405K Temperature-sounding 8 57.290344 passive ( 33.3K ) Noise out of specification 9 89 passive ( 33.3K ) Used in the dynamic estimation of emissivity in channels 11-15 10 165.5 passive ( 33.3K ) Used in the screening for cloud contamination in channels 9-17 11 176.311 active 1.8K Humidity-sounding 12 178.811 active 1.8K Humidity-sounding 13 180.311 active 1.8K Humidity-sounding 14 181.511 active 1.8K Humidity-sounding 15 182.311 active 1.8K Humidity-sounding 16 325.15±1.2 passive ( 33.3K ) 17 325.15±2.4 passive ( 33.3K ) 18 325.15±4.1 passive ( 33.3K ) 19 325.15±6.6 passive ( 33.3K ) 1 σo is set to 33.3K on the channels that are monitored in passive mode. This way their weight in the analysis is negligible, while their bias correction coefficients are let to evolve over time. ESA Contract 4000136311: Performance evaluation of Arctic Weather Satellite data Deliverable D6: Technical note on impact assessment of AWS constellation using real AWS measurement data v1.0 24th October 2025 9 4.2 Results against the realistic baseline Scorecard representations of the forecast impact from the assimilation of Metop-B and AWS microwave sounder radiances on top of the realistic baseline system are shown in figures 7 and 8, respectively. These scorecards give a rather disappointing overall view of the impact. It is clear that the negative impacts outnumber the positive ones in both figures. Particularly against the surface-based observations, the general picture is more seriously negative in figure 8 than in figure 7. That is, the impact of AWS is, on balance, more strongly on the negative side than that of the Metop-B microwave sounders. The behaviour is the opposite in the verification against radiosonde-based data, but there, once again, the signals are isolated and do not support firm conclusions either way. The statistically significant negative impact shows most clearly against mean sea level pressure and dew point temperature at the surface level. In the case of aws2, there are significant negative impacts in the near-surface humidity too. The quantification of the forecast impact on selected forecast parameters, shown in figure 9, confirms the lack of any positive impact in either ref2 or aws2 runs. The differences between the scores in the two runs appear less serious than in the scorecard representations. The lines of the two runs follow each other and remain almost exclusively within the confidence intervals of each other, suggesting that the signals obtained against the baseline run are, again, reasonably consistent in the two runs. For most parts, the impact on these selected forecast parameters is within the confidence intervals and thus considered neutral. The negative impact against mean sea level pressure appears genuine though. Finally, results from the short-range verification based on the OmB departure statistics are shown in figure 10. Since the realistic baseline includes the assimilation of ATMS and MWHS-2 radiances, the evaluation of the data fits is extended to these data types in addition to those included also in figure 6. In comparison with figure 6, there are similarities in the structure of the impact. Radiosonde-based temperature data suggest a perfectly neutral impact, and the radiosonde-based humidity data suggest a slight positive impact, that however is stronger in ref2 than in aws2. The temperature-sounding channels of IASI again indicate an improvement in the data fit due to the assimilation of Metop-B microwave sounder data. In the humidity-sounding channels of IASI, there are no positive suggestions in either run. Basically, the only suggestions of a positive impact in the aws2 run appear against the humidity-sounding channels of the microwave sounders. On channels 18–22 of ATMS, aws2 shows a significant improvement in the data fit. Also against the MWHS-2 data, the data fit improvement is stronger in aws2 than in ref2, but these suggestions remain within the confidence intervals. ESA Contract 4000136311: Performance evaluation of Arctic Weather Satellite data Deliverable D6: Technical note on impact assessment of AWS constellation using real AWS measurement data v1.0 24th October 2025 16 Figure 7: Scorecard representation of the forecast impact obtained via assimilating the Metop-B microwave sounder radiances on top of the realistic baseline system. Blue (red) boxes indicate a positive (negative) impact, that is, a reduction (increase) in the root-mean-squared forecast error. Outlining indicates statistically significant impact at 95% confidence level. ESA Contract 4000136311: Performance evaluation of Arctic Weather Satellite data Deliverable D6: Technical note on impact assessment of AWS constellation using real AWS measurement data v1.0 24th October 2025 17 Figure 8: Scorecard representation of the forecast impact obtained via assimilating the AWS microwave sounder radiances on top of the realistic baseline system. Blue (red) boxes indicate a positive (negative) impact, that is, a reduction (increase) in the root-mean-squared forecast error. Outlining indicates statistically significant impact at 95% confidence level. ESA Contract 4000136311: Performance evaluation of Arctic Weather Satellite data Deliverable D6: Technical note on impact assessment of AWS constellation using real AWS measurement data v1.0 24th October 2025 18 Figure 9: Normalised root-mean-squared forecast error in ref2 (black) and aws2 (red) runs as a function of forecast lead time. The normalisation is against the baseline run bas2. Positive values indicate a reduction in the forecast RMSE, i.e., a positive impact from the assimilation of additional sounder data. The scores are shown for near-surface temperature (top left), near-surface specific humidity (top right), cloud cover (bottom left), and mean sea level pressure (bottom right). Bars indicate the 95% confidence intervals. The poor impact displayed in the verification against the realistic baseline system is concerning. While not being able to explain any negative impacts (such as the one on the forecast of mean sea level pressure), the general lack of positive suggestions may be attributable to the assimilation of microwave sounder data from NOAA-18 and NOAA-19 satellites in the baseline run. At the time of this impact evaluation, these old NOAA satellites were in orbital planes very similar to those of Metop-B and AWS satellites. It can be argued that the potential gains from additional use of Metop-B and AWS data were reduced in such circumstances. Another aspect that reduces the potential gains from satellite data assimilation is that the synoptic situation was relatively calm during most of the evaluation period. Consequently, the predictability in the short range was good and the numerical forecast did reasonably well already when including the NOAA and Fenyung satellite data only. However, even then one could expect at least a neutral impact from the use of the additional microwave sounders. ESA Contract 4000136311: Performance evaluation of Arctic Weather Satellite data Deliverable D6: Technical note on impact assessment of AWS constellation using real AWS measurement data v1.0 24th October 2025 19 Figure 10: Baseline-normalised standard deviation of Observation minus Background (OmB) departure for radiosonde-based temperature and humidity data (left panels), radiance data in temperatureand humidity-sounding channels of IASI (middle panels), and radiance data in the microwave sounders ATMS and MWHS-2 (right panels). Red and blue, respectively, indicate the data fit in ref2 and aws2 runs. Interpretation of the scores is as in figure 6. The negative impact seen in the verification against mean sea level pressure and near-surface temperature suggests some suboptimality in the setup of the data assimilation in the HARMONIE-AROME system. Determining the fundamental reason for such behaviour may be a difficult task and goes beyond the scope of this project. One possibility is that there is no sufficient interaction between the analyses of surface parameters and upper-air meteorological fields. This is a deficiency that is currently worked on and that may be particularly troublesome during the melting season in the spring. There is another possibility that the specification of observation and background errors with respect to each other is not quite right. While the continued operational practice (that is also applied in this study) is to apply fixed background error statistics at all times, some level of flow-dependency in the background error formulation would likely have potential to improve the data assimilation system performance considerably. ESA Contract 4000136311: Performance evaluation of Arctic Weather Satellite data Deliverable D6: Technical note on impact assessment of AWS constellation using real AWS measurement data v1.0 24th October 2025 20 Regardless of the poor overall verification scores against the realistic baseline, it can be concluded that the forecast impact at the surface level as obtained from AWS sounder data is comparable with that from the Metop-B microwave sounders. In the data fit comparisons, a consistent signal comes through that AWS fails to provide added value on the upperand mid-tropospheric temperature the same way as the microwave sounders of Metop-B do. This is, however, to some extent compensated by the better extraction of tropospheric humidity information from the AWS data. 5. Conclusions The impact of AWS sounder radiance assimilation is evaluated in the context of the HARMONIE-AROME 4D-Var system over a four-week evaluation period in Spring 2025. The evaluation is conducted on top of two alternative baseline systems, each representing a specific setup where the use of other microwave sounders is either switched off completely or reduced to contain the sounders of the NOAA and Fengyun satellites only. For reference, additional runs are produced where the microwave sounder data from Metop-B are assimilated instead of the AWS sounder data. The goal is to quantify the impact of AWS with respect to that of Metop-B and consequently refine the previous estimate of the EPS-Sterna constellation impact, that was presented in deliverable D5 of this project. The assimilation of AWS sounder data on top of the depleted baseline system is found to make a small positive impact on near-surface humidity. There is generally no significant impact on other forecast parameters. The magnitude of the humidity impact is by far smaller than the impact previously demonstrated from other microwave sounders in winter, but it is greater than that previously demonstrated in summer. In comparison with the Metop-B reference data, the impact of AWS sounder data is at a similar level and no major discrepancies are found in the verification of surface-level weather parameters. Against the realistic baseline system, neither AWS nor Metop-B microwave sounders are able to demonstrate a solid positive impact in this evaluation. There are notable negative impacts in particular in the forecast of mean sea level pressure and these call for further investigation beyond this project. Comparisons of OmB data fits indicate that AWS is not capable of extracting information on upperto mid-tropospheric temperature the same way as the Metop-B sounders do. In terms of extracting humidity information, AWS serves at least as well as the microwave sounders of Metop-B. Overall, the impact of AWS data is considered to match that of the Metop-B reference to such extent that the previously reported estimate of the EPS-Sterna constellation impact remains valid. ESA Contract 4000136311: Performance evaluation of Arctic Weather Satellite data Deliverable D6: Technical note on impact assessment of AWS constellation using real AWS measurement data v1.0 24th October 2025 21 References Müller M., Y. Batrak, J. Kristiansen, M. A. Ø. Køltzow, G. Noer, and A. Korosov, 2017a: Characteristics of a Convective-Scale Weather Forecasting System for the European Arctic. Mon. Wea. Rev., 145, 4771-4787, https://doi.org/10.1175/MWR-D-17-0194.1. Müller M., M. Homleid, K.-I. Ivarsson, M. A. Ø. Køltzow, M. Lindskog, K. H. Midtbø, U. Andrae, T. Aspelien, L. Berggren, D. Bjørge, P. Dahlgren, J. Kristiansen, R. Randriamampianina, M. Ridal, and O. Vignes, 2017b: AROME-MetCoOp: A Nordic Convective-Scale Operational Weather Prediction Model. Wea. Forecasting, 32, 609-627, https://doi.org/10.1175/WAF-D-16-0099.1. ESA Contract 4000136311: Performance evaluation of Arctic Weather Satellite data Deliverable D6: Technical note on impact assessment of AWS constellation using real AWS measurement data v1.0 24th October 2025 22