D3.5 Categorisation and evaluation of visualisation practices for communicating uncertain predictions in climate services

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 101037293 Deliverable D3.5 Categorisation and evaluation of visualisation practices for communicating uncertain predictions in climate services May 2025 This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 101037293 Innovating Climate services through Integrating Scientific and local Knowledge Deliverable Title: Categorisation and evaluation of visualisation practices for communicating uncertain predictions in climate services Author(s): Schalk Jan van Andel Contributing Authors(s): Maurizio Mazzoleni (VUA), Calum Baugh (ECMWF), Györgyi Bela (IDEAS), Ilias Pechlivanidis (SMHI), Micha Werner (IHE) Date 27 April 2025 Suggested citation: Van Andel et al., 2025: Categorisation and evaluation of visualisation practices for communicating uncertain predictions in climate services, I-CISK Deliverable 3.5 Availability: ☒ PU: This report is public [Please select] ☐ CO: Confidential, only for members of the consortium (including the Commission Services) Document Revisions: Author Revision Date Schalk Jan van Andel Outline 17 December 2024 Schalk Jan van Andel, WP3 team Revised outline, content specified, v01 18 February 2025 Schalk Jan van Andel, v02, for review coordinator 27 April 2025 Micha Werner, Schalk Jan van Andel v03, Final version 23 May 2025 D3.5 – Categorisation and evaluation of visualisation practices 1 Executive summary In this deliverable we report and reflect on the co-development process and resulting visualisation prototypes of climate service information in the I-CISK Living Lab pilot applications. Following an introduction based on literature and examples of state-of-the-art operational climate services, including examples from Copernicus and other continental and global services, a categorisation of visualisation methods is proposed that constitutes maps, graphs, alerts and aggregated information, uncertainty, and user-centred evaluation. The pilot applications for the seven Living Labs, in Lesotho, Greece, Netherlands, Spain, Italy, Georgia, and Hungary, are analysed for each of the visualisation categories, providing examples of commonly used methods, and of visualisation prototypes that stand-out as co-developed by a particular Living Lab. A reflection is provided on the I-CISK visualisation co-design process and on the resulting visualisation in the pilot climate services. A key step in the visualisation co-design process, was the preparation of multiple visualisation examples (mock-ups) for each information component for each living-lab (Section 2). These were then used in multiactor platform workshops and bilateral meetings for feedback, refinement, and selection for implementation. This worked well in the Living Labs, as partly evidenced by the high variability of the lookand-feel of the pilot climate service applications. This is strengthened by the local knowledge on natural hazards, alert levels, challenges, mitigation and adaptation measures, decision processes, user needs and user stories that have found their way in a large part of the display of information in the pilot applications. All the pilot applications that include hydroclimatic forecasts or climate change projections, also show the uncertainty of these predictions, mostly using forecast plumes and box-plots, in-line with state-of-the-art examples. We found the need for balancing and integrating best practices on the one hand (e.g., to not unintentionally over-state prediction accuracy), and user preferences on the other (e.g., including actual critical event thresholds and requesting localised information at high resolution). This balancing we see as a key success factor from and for human-centred climate services. For future research and further development of tailored climate services, we recommend to explore the challenges, and use or rather non-use of animated display of hydroclimatic information, which is featured in only a limited number of the I-CISK pilot applications. D3.5 – Categorisation and evaluation of visualisation practices 2 Table of Contents 1 Introduction ................................................................................................................ ................................. 6 1.1 Background .......................................................................................................................................... 6 1.2 Objectives ................................................................................................................ ............................. 6 1.3 Document structure ............................................................................................................................. 6 2 Co-creation process of climate service visualisation prototypes ................................................................ 7 3 Visualisation in literature and state-of-the-art operational climate services ............................................. 9 3.1 Literature on visualisation hydroclimatic information and uncertainty .............................................. 9 3.2 Visualisation in state-of-the-art operational services ........................................................................ 12 4 Co-developed visualisation in Living Lab pilot climate services ................................................................ 21 4.1 Maps ................................................................................................................................................... 21 4.2 Graphs ................................................................................................................................................ 25 4.3 Alert and aggregated information ..................................................................................................... 26 4.4 Display of information on uncertainty ............................................................................................... 29 4.5 User-centred evaluation .................................................................................................................... 31 4.6 Functionalities for users ..................................................................................................................... 32 5 Reflection ................................................................................................................................................... 35 5.1 Visualisation categories ..................................................................................................................... 35 5.2 Co-design process .............................................................................................................................. 36 5.3 Visualisation of uncertainty ............................................................................................................... 37 5.4 State-of-the-art .................................................................................................................................. 37 6 Conclusions ................................................................................................................. ............................... 38 References ......................................................................................................................................................... 39 Appendix 1 Glossary .......................................................................................................................................... 42 D3.5 – Categorisation and evaluation of visualisation practices 3 List of Figures Figure 1 The steps followed in WP3 to co-create visualisation prototypes for each Living Lab. Key element of this process is the co-design with users by feedback, refinement, and selection of mock-ups (step 3). Co-creation of the selected visualisation methods will continue in WP5 and WP1 where feedback is being collected from the Living Lab users on the pre-operational pilot applications................................................... 7 Figure 2 The Miro board with the visualisation examples. ................................................................................. 8 Figure 3 In the Climate Information explorer, users can display maps at gridded or sub-catchment scale with colours indicating the projected percentage change with regard to the present climate for a number of hydroclimatic variables and indicators, selecting from climate projection ensembles, emission scenarios, and future periods (2011-2040, 2041-2070, and 2071-2100). ................................................................................. 12 Figure 4 In the Climate Information explorer, users can select a grid cell on the map, for which a graph or table with the climate change projection data can be displayed and downloaded, as can be done with the underlying data. The graphs present box-plots with a varying level of detail; high for 30-year average (lower panel), and lower for monthly means (upper panel). Links to background, metadata and explanatory information are provided, as are links to key external websites. ..................................................................... 13 Figure 5 The SMHI Hypeweb climate service visualizes seasonal hydrometeorological forecasts with multiple icons showing where background information can be retrieved, including also an option to mask regions for which the forecasts (for the selected variable and lead time) have no skill. The colour legend shows that the seasonal forecasts of monthly means are presented as above, near or below normal conditions (model climatology), with high, medium or low probability (calculated from the forecast ensemble). ...................... 14 Figure 6 Seasonal hydrological forecasts based on SMHI’s Word-wide HYPE model with a map to select a region for which forecasts (box-plots for each lead month) of a hydrometeorological variable are visualized on a graph at the bottom. Forecast data can be downloaded if required. ....................................................... 15 Figure 7 SMHI’s service over Europe displays a map for historic mean simulated data at the sub-catchment resolution for a number of hydro-meteorological variables. These variables can be selected from a dropdown list. Opacity slides are used to visualise country borders and catchment outlines. ............................... 15 Figure 8 SMHI’s service over Europe visualizes the upstream area of the region of interest, while a map displays the historical simulations including past climatology for comparison. This is available for a number of hydro-meteorological variables selectable through the drop-down list. A download and an information button are provided. ......................................................................................................................................... 16 Figure 9 CEMS European Flood Awareness System (EFAS) map display providing rapid awareness of river locations in Europe where probabilistic forecasts exceed flood pre-alert thresholds. .................................... 17 Figure 10 CEMS European Flood Awareness System (EFAS) pop-up forecast graph when selecting a river location. The uncertainty information derived from ensemble hydrometeorological forecasts is displayed through box-plots. ............................................................................................................................................. 17 Figure 11 CEMS European Flood Awareness System (EFAS) visualises forecast quality information with coloured dots for each evaluated river location. .............................................................................................. 18 Figure 12 CEMS European Flood Awareness System (EFAS) pop-up graphs for forecast quality information for different lead times. .......................................................................................................................................... 18 Figure 13 CEMS European Drought Observatory (EDO) displays current drought situation over Europe, e.g. with the combined drought indicator expressing pre-alerts of three different levels from watch to alert, on a coloured gridded map in which detailed information can be derived from each grid-cell. ............................. 19 Figure 14 CEMS European Drought Observatory (EDO) enables users to show bar-charts of recent and present time series of the selected drought indicator ...................................................................................... 19 Figure 15 CEMS European Drought Observatory (EDO) seasonal drought forecasts over Europe, displaying a course resolution gridded colour map, indicating driest to wettest predicted conditions relative to D3.5 – Categorisation and evaluation of visualisation practices 4 climatology for a 3-month lead time. Using the drop-down menu, other hydroclimatic variables, drought indicators, and lead times can be selected. ...................................................................................................... 20 Figure 16 The pilot CS application from the Living Lab in Georgia uses a map to select river stations for display of seasonal streamflow predictions ...................................................................................................... 21 Figure 17 Pilot CS application from the Living Lab in Greece, with a high-resolution coloured map as the central visualisation tool. Multiple layers can be selected, and for any grid-cell on the map, pop-up information and time series graphs can be displayed. ..................................................................................... 22 Figure 18 Pilot CS application from the Living Lab in Spain, with visualisation of seasonal forecasts on isoline coloured maps with pre-alert information ........................................................................................................ 23 Figure 19 Pilot CS application from the Living Lab in Spain, with a display of climate change projected impact on drought indicators such as number of consecutive days without precipitation. ......................................... 23 Figure 20 The pilot CS application from the Living Lab in Hungary uses maps to display urban heat-island measurements and model predictions ............................................................................................................. 24 Figure 21 Detailed, high resolution, static background information based on local knowledge of the case study area as displayed for (parts of) the Living Lab in Spain. .......................................................................... 24 Figure 22 Pilot CS application from the Living Lab in Spain, with an innovative map with sliding mask to compare hydroclimatic information and derived indices for two different points in time. ............................. 25 Figure 23 Examples of commonly used types of time series graphs, from the pilot CS application of the Living Lab in Spain. ....................................................................................................................................................... 26 Figure 24 Pilot CS application from the Living Lab in Lesotho, aggregating hydroclimatic information on cold waves and droughts displayed with a colour ramp to indicate the population that is potentially exposed at district level. ...................................................................................................................................................... 26 Figure 25 Pilot CS application from the Living Lab in Greece, showing aggregated information of sectorspecific impact indicators, in this case on hydroclimate state for tourism. ...................................................... 27 Figure 26 Pilot CS application from the Living Lab in the Netherlands, with user-specific aggregated coloured and textual alert information for droughts. ...................................................................................................... 27 Figure 27 For younger audiences, the Living Lab in Hungary features both a puzzle visual where children piece together the thermal image of an elephant, and classroom-focused demonstrations (pictured) that let students experiment with heat cameras, sparking interest in citizen science from an early age. ................... 28 Figure 28 Pilot CS application from the Living Lab in Greece, displaying seasonal ensemble streamflow forecasts with box-plots on a time series graph. .............................................................................................. 29 Figure 29 Pilot CS application from the Living Lab in Italy, with seasonal forecast information displayed for selected stations as a plume detailed with colour-shades for multiple bandwidths based on percentiles. .... 30 Figure 30 The Pilot CS application from the Living Lab in the Netherlands displays 2-months lead time of a seasonal forecast with a plume containing two shades based on percentiles, in combination with userdefined thresholds. ............................................................................................................................................ 30 Figure 31 Example of a heat map showing the Threat score computed by applying different levels of forecast probability of yellow status low flow conditions for the Living Lab in Italy. ..................................................... 31 Figure 32 Example of the hits, misses and false alarm counts computed by applying a 10% predicted crossing probability threshold to predict the occurrence of yellow status low flow conditions for the Living Lab in Italy. ........................................................................................................................................................................... 32 Figure 33 Landing page of the pilot CS application from the Living Lab in Spain with project information and main objective ................................................................................................................................................... 33 Figure 34 Pop-up text boxes (upper panel) and navigation by drop-down menus of the pilot CS application from the Living Lab in Greece (lower panel) ..................................................................................................... 33 Figure 35 Visual representation of currently displayed time domain in the pilot CS application from the Living Lab in Greece. .................................................................................................................................................... 34 D3.5 – Categorisation and evaluation of visualisation practices 5 List of Tables Table 1 Summary table of visualisation methods used in each of the seven I-CISK Living Labs (Grey fill indicates 'not included, text in italics indicate examples of specific type of visualisation presented) ............. 36 D3.5 – Categorisation and evaluation of visualisation practices 6 1 Introduction 1.1 Background Climate services provide platforms for users to easily access and make use of climate information. How information is provided on a climate services platform varies depending on the ultimate target audience’s needs and proficiency in interpreting the information. Information is usually presented through a combination of visual aids which may include maps, diagrams, graphs, tables, etc. The choice depends on the specific attributes of the information to be communicated and on user preferences. Central to the I-CISK project, is the co-creation of climate services within the seven Living Labs (LLs) that have been established within the project (WP1). In each of these Living Labs, multi-actor platforms (MAP) have been established, including a wide range of sectors and stakeholders. These have been instrumental in the user-centred codesign of visualisation methods that have been included the (pre-)operational climate services (CSs) that have been piloted within the project. The locations of the seven Living Labs constitute the upper and middle Guadalquivir basin in Spain, Emilia Romagna in Italy, Rijnland in the Netherlands, Crete in Greece, Budapest in Hungary, Alazani river basin in Georgia, and Southern Lesotho and Senqu Valley in Lesotho. For information about these Living Labs, the users involved, and their climate information needs, we refer to I-CISK Deliverables 1.1. and 2.1. In the remainder of this document we will refer to the Living Labs by the name of the country they are located in. Developing the co-design process for visualisation of uncertain climate information in the LL pilot applications, has been the responsibility of Task 3.4, which forms a part of Work Package 3 (WP3) of the ICISK project. This has strong linkages to Task T3.2, which focused on the providing of tailored climate information in these pilot applications, and Task T3.3, which focused on the user-centred evaluation of the information provided. The work in this task progresses and tests part of the co-creation framework that has been developed in the project (in WP2), and has also been in close collaboration with WP5 where the pilot applications have been developed. 1.2 Objectives The aim of this deliverable is to reflect on the co-design process and the resulting visualisation prototypes that have been developed in the LL pilot applications of climate information, including the representation of uncertainty. Based on literature and state-of-art operational CS examples, mostly providing information at a global or continental scale, a categorisation of visualisation components is used to analyse the I-CISK pilot climate services that have been developed; identify challenges and possibly innovations; and recommend future steps in CS visualisation. 1.3 Document structure In Section 2 the visualisation co-design and selection process is described, after which Section 3 reports on literature and state-of-the-art examples of operational climate services. Section 4 is the central chapter of this deliverable, reporting on the visualisation of climate information as selected, co-designed, and implemented in the I-CISK LL pilot applications. These results are then reflected upon in Section 5 and the overall conclusions are provided in Section 6. D3.5 – Categorisation and evaluation of visualisation practices 7 2 Co-creation process of climate service visualisation prototypes The procedure to co-create the visualisation prototypes was based on a five-step approach that varies from developing a catalogue of state-of-the-art mock-ups to developing scripts that can allow the operationalisation of the visualisation. In this section we present the procedure (Figure 1). We note that the procedure is generic and is not limited to the type of predictions or projections, i.e. sub-seasonal to seasonal (S2S) predictions or decadal/centennial projections. The five steps are explained below. Figure 1 The steps followed in WP3 to co-create visualisation prototypes for each Living Lab. Key element of this process is the co-design with users by feedback, refinement, and selection of mock-ups (step 3). Co-creation of the selected visualisation methods will continue in WP5 and WP1 where feedback is being collected from the Living Lab users on the pre-operational pilot applications. Catalogue of visualisation practices In this preparatory step, we collected visualisation examples from literature and state-of-the-art climate services and early warning systems. These examples were archived in a Miro board (left-side Figure 2). The examples cover a broad range of visualisation types, including maps, graphs, and aggregated information including alerts. Setting up user stories We took note of the user needs (using information from WP1 and WP2) and transformed these needs into story-lines. This process allowed framing the problem, identifying the variables and periods of interest. The user stories served as an input to the developing of the first set of visualisation mock-ups (prototypes). Co-designing prototypes In this step we encouraged interactions between scientists, service providers, and local users (via WP1 workshops) in each Living Lab (LL), to co-design a preferred set of visualisation prototypes for the climate service (right-side Figure 2), recognising that the capacity of users differs between LLs as well as within the LL. Starting from the initial set of mock-ups based on the literature and state-of-the-art examples, feedback was collected from the users to inform the co-design process through preferences, selections, and refinements. D3.5 – Categorisation and evaluation of visualisation practices 14 and as a function of lead time, regions with no skill can be masked (not displayed) accordingly. The focus of such seasonal forecasting services is to provide information on above/near/below normal conditions (classes defined by the terciles) or above/below the high/low extreme conditions respectively (classes defined by the 90th and 10th percentiles). With the colour itself indicating a specific condition being forecast for a specific region, the density of the colour indicates the confidence of this condition being met. Here confidence is defined by the probability of the values in the forecast ensemble exceeding a defined threshold. Besides maps, graphs are also presented where the hydro-climatic conditions of a region are predicted for some months ahead and the probabilistic results are visualized with box-plots (which provide summary statistics of the large ensemble such as min-max, median and 25th and 75th percentiles). Moreover, monthly dependent thresholds are also depicted to represent the different conditions (above/near/below and extremes), with this assisting in decision-making. Figure 5 The SMHI Hypeweb climate service visualizes seasonal hydrometeorological forecasts with multiple icons showing where background information can be retrieved, including also an option to mask regions for which the forecasts (for the selected variable and lead time) have no skill. The colour legend shows that the seasonal forecasts of monthly means are presented as above, near or below normal conditions (model climatology), with high, medium or low probability (calculated from the forecast ensemble). D3.5 – Categorisation and evaluation of visualisation practices 15 Figure 6 Seasonal hydrological forecasts based on SMHI’s Word-wide HYPE model with a map to select a region for which forecasts (box-plots for each lead month) of a hydrometeorological variable are visualized on a graph at the bottom. Forecast data can be downloaded if required. Apart from future projections and seasonal forecasts, climate services can also provide information and data on the historical conditions and monitoring statuses of the regions of interest. The SMHI Hypeweb service visualizes as maps the long-term means of multiple hydro-climatic variables, while graphs are presented for a specific region and every historical year (Figure 7 and Figure 8). The latter graphs also present as a background the past climatology so that users can have an improved understanding of the conditions observed for the year of interest with respect to past records. Figure 7 SMHI’s service over Europe displays a map for historic mean simulated data at the sub-catchment resolution for a number of hydro-meteorological variables. These variables can be selected from a drop-down list. Opacity slides are used to visualise country borders and catchment outlines. D3.5 – Categorisation and evaluation of visualisation practices 16 Figure 8 SMHI’s service over Europe visualizes the upstream area of the region of interest, while a map displays the historical simulations including past climatology for comparison. This is available for a number of hydro-meteorological variables selectable through the drop-down list. A download and an information button are provided. Copernicus Emergency Management Services As a part of the Copernicus Emergency Management Services (CEMS, emergency.copernicus.eu), the European Flood Awareness System (EFAS) and European Drought Observatory (EDO) provide information on hydroclimatic variables and natural hazard indicators that are closely related to the I-CISK Living Labs that incorporate seasonal forecasts for early preparedness. EFAS uses an inter-active map as the central display of its services (Figure 9). River stations for which there is a probability of high flows in the observations or forecasts are indicated with a colour-coded square, with yellow, red, and purple representing forecasts exceeding the 2, 5, or 20-year return period flow. Symbols are used to make the user aware of other information to be displayed, such as latest observations for both meteorology and hydrology, climatology, and forecast quality. By clicking a river section, station, or reporting point on the map, more detailed information for that location is displayed in multiple graphs and tables, such as the streamflow time series graph of Figure 10. The time series graph uses the same alert colours in the back ground, includes recent observations, deterministic streamflow forecasts and ensemble based probabilistic forecasts. Forecast uncertainty is represented with box-plots derived from the ensemble forecast. D3.5 – Categorisation and evaluation of visualisation practices 17 Figure 9 CEMS European Flood Awareness System (EFAS) map display providing rapid awareness of river locations in Europe where probabilistic forecasts exceed flood pre-alert thresholds. Figure 10 CEMS European Flood Awareness System (EFAS) pop-up forecast graph when selecting a river location. The uncertainty information derived from ensemble hydrometeorological forecasts is displayed through box-plots. Forecast quality can be visualised by colouring river stations depending on the evaluated skill (Figure 11). An intensity colour pallet is used, with light intensity showing low forecast skill and high intensity for high performance. Clicking on a station displays graphs of forecast skill against lead time (Figure 12). D3.5 – Categorisation and evaluation of visualisation practices 18 Figure 11 CEMS European Flood Awareness System (EFAS) visualises forecast quality information with coloured dots for each evaluated river location. Figure 12 CEMS European Flood Awareness System (EFAS) pop-up graphs for forecast quality information for different lead times. The European Drought Observatory (EDO) displays current drought conditions on a map (Figure 13), with a limited set of three colours yellow, orange, and red, to indicate the level of pre-alert from low to high (Watch, Warning, Alert). A range of drought indicators and hydroclimatic variables, observed and forecast, can be selected for display on the map. By clicking on a location on the map, a bar-chart displays the time series of the selected information with a 10-day interval (Figure 14). D3.5 – Categorisation and evaluation of visualisation practices 19 Figure 13 CEMS European Drought Observatory (EDO) displays current drought situation over Europe, e.g. with the combined drought indicator expressing pre-alerts of three different levels from watch to alert, on a coloured gridded map in which detailed information can be derived from each grid-cell. Figure 14 CEMS European Drought Observatory (EDO) enables users to show bar-charts of recent and present time series of the selected drought indicator Seasonal drought forecasts can be displayed on the map as well, though at a courser resolution, which reflects the atmospheric models used. An classification of the anomaly (drier or wetter compared to climatology) rather than pre-alerts, is used to indicate the higher level of uncertainty associated with the forecasts (Figure 15). D3.5 – Categorisation and evaluation of visualisation practices 20 Figure 15 CEMS European Drought Observatory (EDO) seasonal drought forecasts over Europe, displaying a course resolution gridded colour map, indicating driest to wettest predicted conditions relative to climatology for a 3-month lead time. Using the drop-down menu, other hydroclimatic variables, drought indicators, and lead times can be selected. D3.5 – Categorisation and evaluation of visualisation practices 21 4 Co-developed visualisation in Living Lab pilot climate services In this section we report on the climate service (CS) visualisation prototypes selected and co-developed with the Living Lab multi-actor platforms as implemented in the I-CISK pilot CS applications. The screenshots and analysis presented reflect the status of March and April 2025. Further updates to the applications may occur following user feedback up to the end of the I-CISK project, or beyond where services are continued after October 2025. Using the literature and operational CS examples described in the previous section, we analyse the co-developed visualisation methods that have been implemented in the I-CISK pilot applications. This analysis uses the following categorisation: maps, graphs, aggregated information (including alerts), uncertainty, and quality (forecast performance). The final sub-section presents methods used to navigate the application and select layers to be displayed. The aim of this section is to analyse and discuss the range of visualisation practices that are used, and not to provide an exhaustive replication of the on-line pilot applications (i-cisk.dev.52north.org/living-labs/). For each visualisation category we identify the methods that are used most, and provide representative examples with screenshots, as well as examples of visualisation practices that are unique from individual Living Labs. 4.1 Maps All Living Lab pilot CS applications use maps for visualisation, and the majority has a map as the central means of display. The level of detail of information displayed on the map, and level of inter-activity differs among the applications. As an example, the Living Lab in Georgia displays a map of the area to indicate location and allow selection of hydrological stations at which seasonal forecasts can then be displayed. Subcatchments are also displayed but are not selectable and are for information only. There are no hydrometeorological variables or alerts displayed on the map. The format and colours of the map are chosen such that the stations stand out and are easy to select. For the selected station seasonal streamflow predictions can be displayed (Figure 16). Figure 16 The pilot CS application from the Living Lab in Georgia uses a map to select river stations for display of seasonal streamflow predictions D3.5 – Categorisation and evaluation of visualisation practices 22 The Living Lab in Greece displays a map with layers of historic, forecast, or projected hydroclimatic variables and derived impact indicators. Clicking on any point on the map brings up the current climate information and a graph with projected climate with an uncertainty band. The map combines a standard topographical look with high resolution coloured grid overlay (Figure 17). The diverging colour palette with blue, green/yellow, and red can be further optimised for colour-blind users. Figure 17 Pilot CS application from the Living Lab in Greece, with a high-resolution coloured map as the central visualisation tool. Multiple layers can be selected, and for any grid-cell on the map, pop-up information and time series graphs can be displayed. The Living Lab in Spain displays seasonal predictions of precipitation and temperature on gridded maps, with the option to display a graph for the selected 230 m grid-cell (Figure 18). The information is displayed at a monthly time step. This is in-line with the best practice recommendation from literature to display forecasts at a lower time resolution when the lead time increases, e.g. a monthly rather than a daily time step for seasonal lead times to match the lower forecast skill at longer lead times. D3.5 – Categorisation and evaluation of visualisation practices 23 Figure 18 Pilot CS application from the Living Lab in Spain, with visualisation of seasonal forecasts on isoline coloured maps with pre-alert information Climate change projections are mostly displayed at a courser resolution than the observations and seasonal forecasts, which is in-line with literature. The Living Lab in Spain, for example, displays climate projection information on course gridded maps (rather than the higher resolution smooth isoline maps used for seasonal forecasts) with selection button on top to select which information to display. The time resolution presented, with a slide-bar for selecting the year and season in the future to be displayed (Figure 19), is higher than in the example state-of-the-art operational CS (Section 3) where projections are presented averaged for 30-year periods. This component of the pilot application is still under development. Figure 19 Pilot CS application from the Living Lab in Spain, with a display of climate change projected impact on drought indicators such as number of consecutive days without precipitation. D3.5 – Categorisation and evaluation of visualisation practices 30 Figure 29 Pilot CS application from the Living Lab in Italy, with seasonal forecast information displayed for selected stations as a plume detailed with colour-shades for multiple bandwidths based on percentiles. Figure 30 The Pilot CS application from the Living Lab in the Netherlands displays 2-months lead time of a seasonal forecast with a plume containing two shades based on percentiles, in combination with user-defined thresholds. The CS application from the Living Lab in the Netherlands, shows the seasonal ensemble forecasts with a plume of five percentiles (Figure 30). Selected user pre-alert thresholds are indicated. The data is displayed continuously at the original daily step of the underlying data (e.g. not aggregated per week or month) and as absolute values. Displaying user-defined alert levels against forecast values, rather than, for example, forecast deviations from model climatology (anomalies), may (inadvertently) suggest more accuracy than the seasonal forecasts can provide, even though the uncertainty band of the ensemble forecast is shown. D3.5 – Categorisation and evaluation of visualisation practices 31 The pilot applications of the Living Labs in Georgia and Italy have multiple options for display of prediction uncertainty from which the user can select by drop-down menu, as presented in the beginning of this chapter. These are shown as pop-ups from its central map (Figure 16) and discussed in Section 4.2 on graphs. 4.5 User-centred evaluation Results for the evaluation of the quality of hydroclimatic information (forecasts and projections) and derived impact indicators for the Living Lab CSs are available as off-line information, putting evaluation results into user-centred thresholds. The Living Lab in Spain, for example, has analysed and found variability in skill for different regions within the case study area. Showing evaluation results on-line of user-centred performance assessment methods and metrics is a next step that can be introduced in the CS. Visualising uncertainty information in a climate service, as described above, conveys the level of confidence that a user can assign to its predictions, but a user may still be unclear about how to use the information when making decisions. Decisions are typically made using thresholds, for example the river discharge associated with out of bank flow i.e. flooding, and identifying when the climate service predicts the threshold to be crossed. Therefore, it is useful to evaluate the ability of a climate service to predict the crossing of these decision-making thresholds. In I-CISK this evaluation has been done using a contingencytable based approach, where the predicted crossing of a threshold is compared against the observed crossing of that threshold. This allows the number of hits (when a threshold is correctly predicted), misses and false alarms to be recorded. For probabilistic predictions, the evaluation was done for different levels of forecast probability of the threshold being exceeded. The recorded hits, misses and false alarms are then used to compute a skill score, such as the Threat score, which can show the accuracy of the climate service in predicting threshold crossing events. Results from this type of user driven evaluation were presented as heat maps, which showed the Threat score obtained at different lead times and different levels of probability of exceeding the threshold. These allow a user to quickly identify the optimum forecast probability for triggering alerts of exceeding the threshold, at each lead time. In the example below (Figure 31), a user would obtain the optimal performance when using a forecast probability of at least 10% for triggering alerts at all lead times. Figure 31 Example of a heat map showing the Threat score computed by applying different levels of forecast probability of yellow status low flow conditions for the Living Lab in Italy. Whilst the Threat score summarises the hits, misses and false alarms into a single score of accuracy, many users expressed a desire to also visualise these three components to provide context to the score. This context is required because users need to balance the risk of false alarms against the risk of missed events D3.5 – Categorisation and evaluation of visualisation practices 32 when making a decision, and the Threat score alone does not provide this context. The example above (Figure 31) identified requiring a 10% forecast probability as the optimum to trigger alerts, but plotting the associated hits, misses and false alarms (Figure 32) shows a large occurrence of missed events. This provides information to the user that there is still a high risk of missed events if they were to use this 10% forecast probability requirement within their decision making. A higher probability threshold would reduce the number of missed events, but would equally increase the number of false alarms. Figure 32 Example of the hits, misses and false alarm counts computed by applying a 10% predicted crossing probability threshold to predict the occurrence of yellow status low flow conditions for the Living Lab in Italy. The results of user driven evaluation are often presented to users through reports and meetings, and are accompanied with clear guidance on how to use the climate services within their decision making. However, the results are often not integrated directly into the visualisation of the climate service itself. For example, if a user-based evaluation identifies an optimum forecast threshold crossing probability associated with making a particular decision, the visualisation of the climate service could be adjusted to highlight when this threshold is predicted to be crossed. This would convey to the user that the climate service is forecasting conditions associated with the making a particular decision. Future work could investigate adapting the visualisation of climate services to integrate the findings from user-based evaluation more explicitly. 4.6 Functionalities for users Navigation between different CS components within the pilot application is an important aspect for user experience, and starts with the landing page. The landing pages follow three different approaches among the Living Labs, starting with 1) a map with stations (hydrometeorogical data focus), 2) Status indicator (alert focus), and 3) Information overview (context focus). The map and alert status examples (from the Living Labs in Italy and the Netherlands) have been presented above, but the landing-page with focus on project information, of the Living Lab in Spain, we include here. The information overview of the landing page of the pilot application, Figure 33, highlights that it has been developed as part of an EU research project product, and its main purpose is to provide climate information on droughts for different time scales. The available service components are displayed on top, ranging from historic, to seasonal forecast, to climate projection impact information. D3.5 – Categorisation and evaluation of visualisation practices 33 Figure 33 Landing page of the pilot CS application from the Living Lab in Spain with project information and main objective Most LL pilot applications have chosen for the selection of information to be displayed to be done through drop-down menus. Information icons are provided with a pop-up to allow meta-data and potentially static uncertainty or performance information to be added, as well as guidance on the selection for display options (Figure 34). Figure 34 Pop-up text boxes (upper panel) and navigation by drop-down menus of the pilot CS application from the Living Lab in Greece (lower panel) Feedback received on the visualisation preferences sometimes differed among members of the multi-actor platform in one Living Lab. In these cases majority choice often had to be made, although for some elements a user-customised functionality was developed. Users of the LL in the Netherlands, for example, can select a sectoral user group and through doing so de-select some of the hydroclimatic information displayed, and receive sector-specific pre-alert information (Figure 26). D3.5 – Categorisation and evaluation of visualisation practices 34 Lastly, we provide an example of visualisation practices that have been deigned to support navigation. A good example is implemented in the LL in Greece. This application indicates the time domain presently onscreen with a graphical overview as in Figure 35. Although this does not present any information on uncertainty, it elegantly triggers the user's awareness of the time-frame one is currently looking at, which helps to manage expectations, including expectations on the uncertainty of the information provided. Figure 35 Visual representation of currently displayed time domain in the pilot CS application from the Living Lab in Greece. D3.5 – Categorisation and evaluation of visualisation practices 35 5 Reflection From the overview of visualisation methods implemented in the pilot applications, we first draw general conclusions per visualisation category. In subsequent sections, we further reflect on the co-design process, the visualisation of uncertain predictions, and the position of I-CISK visualisation with respect to state-of-theart. 5.1 Visualisation categories From the overview presented in Section 4 of the visualisation methods used in the I-CISK Living Lab pilot applications, the 'map' stands out as the most widely used visualisation method (Table 1). Maps have been implemented in each of the seven Living Lab pilot applications (i-cisk.dev.52north.org). We can conclude that in today's user-centred climate services, an interactive map with at least the functionality for a sectoral user to zoom-in to and recognise her or his area of interest, e.g. catchment, district, agricultural field, etc., is a minimum requirement. This is in tune with the state-of-the-art operational climate service examples presented in Section 3.2. With respect to colour palettes, some of the I-CISK Living Lab maps (Section 4) show palettes that match existing local information services, but can be further optimised for colour-blind accessibility. This points at an interesting balance or choice to be made in the climate service co-creation process: balancing between on the one hand fitting-in with existing systems for uniformity and recognisability by the users, to ease the uptake of the new climate service, and on the other hand bringing in changes of visualisation intended to improve the climate service usability in the long run. Next to maps, graphs are displayed in almost all Living Lab pilot applications. Only in the pilot application for Lesotho graphs are not used (grey-filled cell in Graphs column of Table 1 ), which is explained by the strong focus of the CS in Lesotho on disaster risk alerts. Aggregated and alert information is used in some, but not all Living Lab pilot applications (Table 1). This may well align with the intended use and user groups of the climate service. For user groups with technical background in fields related to hydroclimatic information, such as water resource management, focus on the predicted hydroclimatic variables or anomalies from climatic mean of such variables, may be the main requirement. For sectoral users on the other hand (e.g. tourism, agriculture, etc.) main interest may be in derived indices or alerts for their sector. An additional reason for not displaying user-centred aggregated information such as alerts is that in some Living Labs, alert levels for early awareness of upcoming impactful events (e.g. droughts or floods) have not yet been established by the local user groups. A positive finding is that information on uncertainty of hydroclimatic predictions is visualised for each of the Living Lab climate services that display hydroclimatic seasonal forecasts or projections (five of the seven Living Labs, Table 1). Time series plumes composed of percentile uncertainty bandwidth (e.g. 10-90th percentile) are used most, followed by box-plots to display prediction uncertainty for monthly, yearly, or 30year average values. In addition to the visualisation categories discussed here, the Living Lab in Greece included in their pilot application the option to play an animation of climate model output maps for a sequence of years, and the Living Lab in Hungary developed videos and games focussed on awareness raising. D3.5 – Categorisation and evaluation of visualisation practices 36 Table 1 Summary table of visualisation methods used in each of the seven I-CISK Living Labs (Grey fill indicates 'not included, text in italics indicate examples of specific type of visualisation presented) Living Lab country and sectors involved Main theme of the climate service pilot application Maps Graphs Aggregated information and Alerts Uncertainty information Other Georgia Water resource management Hydropower, Agriculture Water Resource Management Service Topographic map to select points of interest Time series line graphs Box-plots and plumes of percentile bands Greece Tourism Water resource management, Transport, Infrastructure, Accommodation Cross-Sector Planning Service for Tourism Climatic variable and indicator values Time series line graphs Sectoral hydroclimatic suitability indicators Box-plots on line graph Animated maps Hungary Urban Planning Health, Tourism Urban Heat Planning Service Climatic variable values Time series line graphs Awareness raising images, videos, and games Italy Water management Agriculture, Industry, Energy, Environmental management, Utilities Water Resource Management Service Topographic map to select points of interest Time series line graphs Box-plots and plumes of percentile bands Lesotho Disaster Risk Reduction Anticipatory Action Humanitarian aid Government, Agriculture Impact Based Forecasting Service Alert status Alert status (on/off) for natural hazards Netherlands Water management, water tourism, Agriculture Drought Awareness Service Climatic indicator values Time series line graphs Alert level for natural hazards Plumes of percentile bands Spain Agriculture, Livestock Forestry Climate Planning Service Climatic variable and indicator values Bar charts Sectoral hydroclimatic indicators Plumes of percentile bands 5.2 Co-design process From the visualisation examples presented in Section 4, it can be seen that there is strong variability among the pilot climate service applications of the seven I-CISK Living Labs, which indicates a strong influence of each of the Living Lab multi-actor platforms with a unique composition of sector representatives and their user requirements. In cases where there are similarities , this concerns Living Labs with partly common types of users and objectives, such as water resources managers in the Living Labs of Georgia and Italy that both have a clear interest in seasonal availability of water . Overall, the many differences between the pilot applications, rather than uniformity, can be seen as an indication that the co-design process for visualisation has been successful. The approach to prepare mock-up presentations with a range of visualisation options for the multi-actor platform (MAP) members to indicate their preferences, has successfully contributed to these highly customised hence different applications. This is strengthened by the local knowledge on natural hazards, alert levels, challenges, mitigation and adaptation measures and decision processes, and the user needs (user stories) derived from those by the MAP members, that have found their way in a large part of the selection and visualisation of information in the pilot applications. D3.5 – Categorisation and evaluation of visualisation practices 37 5.3 Visualisation of uncertainty From the hydroclimatic prediction examples presented in Section 4, we see as a positive result that uncertainty is being presented in all pilot applications that present seasonal predictions and climate change projections. When reflecting on the details of how these represent uncertainty and best-practice recommendations from operational systems and literature (Section 3), we like to highlight some differences, both for presenting climate change (impact) information and for seasonal forecasts. For presenting climate change and impact information, best-practice recommendations from climate sciences do not always match with needs expressed by sectoral users in I-CISK Living Labs (e.g. water tourism and agriculture). State-of-the-art climate projection information is often presented as 30-year period average to account for uncertainty, whereas sectoral users in the Living Labs (e.g. in the Netherlands and Spain) expressed the need for information over the next five to ten years. This time scale is important for strategic investment planning by these sectoral users. For the seasonal forecasts, in some of the Living Lab pilot applications, we see time series graphs with a daily time step uncertainty bands of actual forecast levels and volumes of the hydroclimatic variables, along with user-defined alert thresholds. Best practices for seasonal hydroclimatic forecasts, however, suggest only displaying monthly box-plots of anomalies from model climatology (not from thresholds based on actual observations) because of the high degree of uncertainty at seasonal lead times. Both examples indicate the need for balancing and integrating best practices from hydroclimatic sciences on the one hand, and sectoral user preferences and interpretability on the other. This balancing we see as a key success factor of the user driven design of human-centred climate services. 5.4 State-of-the-art When comparing the visualisation examples of I-CISK pilot applications in Section 4 with operational climate services of Section 3, we conclude that the visualisation methods used in the Living Labs generally match state-of-the-art standard. However, the co-design in the individual Living Labs has also indicated directions for innovation. These include intuitive visualisation of which time horizon is presently on screen in the pilot application of the Living Lab in Greece, and a dynamic map (slider) to interactively display hydroclimatic differences and similarities from one year to another in the pilot application of the Living Lab in Spain. A third innovation concerns the approach to more clearly visualise the availability within the same application of climate change information next to the seasonal predictions, e.g. in the Living Lab in the Netherlands, aiming to attract users to the long-term information as well, and foster awareness and discussion on adaptation strategies. Multi-panel plots, used in literature for displaying multiple climate change information, e.g. multiple projections from different scenarios and climate models, have not been included in the pilot CS applications. This may well be because these are considered too technical by most sectoral end users, and too small on (mobile phone) displays without the option to zoom-in an area of interest as with single maps. Only one of the pilot CS applications includes animations displaying a time sequence of maps (LL in Greece), whereas these have been identified in literature as suitable means of illustrating trends in historic hydroclimatic data and expected changes in climate change projections. Animated maps thus remain a potential direction for further innovation of visualisation practices in co-designed climate services. D3.5 – Categorisation and evaluation of visualisation practices 38 6 Conclusions In this document we have categorised and evaluated the visualisation methods as implemented in the pilot climate service applications of each of the seven I-CISK Living Labs; in Lesotho, Greece, Netherlands, Spain, Italy, Georgia, and Hungary. All of the pilot applications that display hydroclimatic forecasts or projections, adopted state-of-the-art methods to visualise uncertainty, mostly concerning forecast plumes of uncertainty bands and box-plots. When evaluating the co-design process, the strong variability among the pilot CS applications developed shows that the process was successful, resulting in user-centred visualisation tailored to the needs of the diverse users that were engaged in the process. This reflection on climate service visualisation also showed that the user-centred approach, aiming to integrate user preferences and local knowledge in the visualisation of hydroclimatic data as much as possible, sometimes needs to balance requirements and preferences of users, with best-practice recommendations from climate and forecasting sciences. Promising innovations concern an attractive and interactive comparison of hydroclimatology between different years, e.g. for visualising climate change impact, and combining short-term seasonal forecasting for event management with long term impact projections for climate change adaptation in one application. An intuitive visualisation of which time horizon is presently on screen supports such multi-time domain climate services. For future climate service development in general, we recommend exploring the challenges, and the use, or rather non-use of animated display of hydroclimatic information, as these currently feature only limited in the I-CISK pilot applications. D3.5 – Categorisation and evaluation of visualisation practices 39 References Bauer, P., Thorpe, A., & Brunet, G. (2015). The quiet revolution of numerical weather prediction. Nature, 525(7567), 47–55. Blake, E. S., & Zelinsky, D. A. (2018). National Hurricane Center Tropical Cyclone Report: Hurricane Harvey (AL092017). NOAA/National Weather Service, 76 pp. Blunden, J., & Arndt, D. S. (2019). State of the Climate in 2018. Bulletin of the American Meteorological Society, 100(9), Si–S305. Bremer, P., Yi, S., Pascucci, V., Binyahib, R., & Mascarenhas, A. (2019). Data analysis and visualisation for climate research. Computers & Geosciences, 127, 86–95. Brewer, C. A., Hatchard, G. W., & Harrower, M. A. (2003). ColourBrewer in print: A catalog of colour schemes for maps. Cartography and Geographic Information Science, 30(1), 5–32. C3S (2021). Climate Data Store. Retrieved from https://cds.climate.copernicus.eu/ (Accessed date in 2021 for reference). Corner, A., Webster, R., Teriete, C., et al. (2018). Climate visuals: A mixed methods investigation of public perceptions of climate images in three countries. Global Environmental Change, 53, 156–167. Corsi, S. C., Long, D. T., & Wein, G. (2021). GIS Tools for environmental data visualisation. Environmental Science & Technology, 55(1), 7–20. Doblas-Reyes, F. J., García-Serrano, J., Lienert, F., Pintó, B., & Rodrigues, L. R. (2013). Seasonal climate predictability and forecasting: Status and prospects. Wiley Interdisciplinary Reviews: Climate Change, 4(4), 245–268. Esri (2019). ArcGIS StoryMaps. https://storymaps.arcgis.com/ Few, R., Brown, K., & Tompkins, E. L. (2013). Public participation and climate change adaptation: Avoiding the illusion of inclusion. Climate Policy, 7(1), 46–59. Funk, C., Dettinger, M. D., Michaelsen, J. C., Verdin, J. P., Brown, M. E., Barlow, M., & Hoell, A. (2008). Warming of the Indian Ocean threatens eastern and southern African food security but could be mitigated by agricultural development. Proceedings of the National Academy of Sciences, 105(32), 11081–11086. Goddard, L., et al. (2014). A verification framework for interannual-to-decadal predictions experiments. Climate Dynamics, 40(1–2), 245–272. Grell, G. A., Freitas, S. R., Stuefer, M., & Fast, J. (2005). Inclusion of biomass burning in WRF-Chem: Impact of wildfires on weather forecasts. Atmospheric Chemistry and Physics, 11(11), 5289–5303. Hansen, J., Ruedy, R., Sato, M., & Lo, K. (2010). Global surface temperature change. Reviews of Geophysics, 48(4), RG4004. Harrower, M., & Brewer, C. A. (2003). ColorBrewer.org: An online tool for selecting colour schemes for maps. The Cartographic Journal, 40(1), 27–37. Hawkins, E., & Sutton, R. (2009). The potential to narrow uncertainty in regional climate predictions. Bulletin of the American Meteorological Society, 90(8), 1095–1107.