Comparing the structural uncertainty and uncertainty management in four common Land Use Cover Change (LUCC) model software packages

Environmental Modelling and Software 153 (2022) 105411 Available online 17 May 2022 1364-8152/© 2022 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/bync-nd/4.0/). Comparing the structural uncertainty and uncertainty management in four common Land Use Cover Change (LUCC) model software packages David García-´ Alvarez a , * , María Teresa Camacho Olmedo b , Hedwig Van Delden c , Jean-François Mas d , Martin Paegelow e a Departarmento de Geología, Geografía y Medio Ambiente, Universidad de Alcal´ a, Facultad de Filosofía y Letras, Alcal´ a de Henares, Spain b Departarmento de An´ alisis Geogr´ afico Regional y Geografía Física, Universidad de Granada., Granada, Spain c Research Institute for Knowledge Systems, Maastricht, the Netherlands d Centro de Investigaciones en Geografía Ambiental, Universidad Nacional Aut´ onoma de M´ exico (UNAM), Morelia, Michoac´ an, Mexico e D´ epartement de G´ eographie, Am´ enagement, Environnement, Universit´ e Toulouse Jean Jaur` es, Toulouse, Garonne (Haute), France ARTICLE INFO Keywords: Uncertainty LUCC Modeling CA_Markov Dinamica EGO Land change modeler Metronamica ABSTRACT Research on the uncertainty of Land Use Cover Change (LUCC) models is still limited. Through this paper, we aim to globally characterize the structural uncertainty of four common software packages (CA_Markov, Dinamica EGO, Land Change Modeler, Metronamica) and analyse the options that they offer for uncertainty management. The models have been compared qualitatively, based on their structures and tools, and quantitatively, through a study case for the city of Cape Town. Results proved how each model conceptualised the modelled system in a different way, which led to different outputs. Statistical or automatic approaches did not provide higher repeatability or validation scores than user-driven approaches. The available options for uncertainty management vary depending on the model. Communication of uncertainties is poor across all models. 1. Introduction Uncertainty is inherent in spatial analysis because of the need of abstraction to represent any of the earth’s characteristics or processes through a map or a GIS procedure. It is also inherent to any analysis that involves human understanding of any real-world process. We understand uncertainty as an indicator of the degree of distrust of the images and concepts of the real world that we are using (Castilla and Hay 2007). This uncertainty must be carefully examined to be aware about the limitations of our analysis and studies. Land Use Cover Change (LUCC) models have many sources of uncertainty, which are difficult to disentangle (Uusitalo et al., 2015). Altogether, they are known as model output uncertainty (Refsgaard et al., 2007; Klein Goldewijk and Verburg 2013) or the uncertainty cascade (Refsgaard et al., 2013). Notwithstanding, several authors have tried to classify them in different groups (Van Asselt 2000; Walker et al., 2003; Refsgaard et al., 2007, 2013; Matott et al., 2009; Klein Goldewijk and Verburg 2013; García-´ Alvarez et al., 2019), mainly differentiating the following types of uncertainty: ⁃ Epistemological uncertainty. The uncertainty that comes from the delimitation and conceptualization of the problem to be modelled. When strictly referring to the uncertainty of the way a problem is conceptualized in a model, several authors talk about “structural uncertainty” (Ferchichi et al., 2017). Brown et al. (2021) specifically refer to model paradigms when analysing model structures that lie in a very different conceptualization of the systems to be modelled. ⁃Model technical uncertainty. The uncertainty arising from the computer implementation of the model, concerning not only the model algorithm, but also the data formats, resolution and other issues. It is related to epistemological uncertainty. ⁃ Process variability uncertainty. The uncertainty that comes from the different ways a system can evolve in the future. ⁃ Input uncertainty. The uncertainty that comes from the data used in the model and its ability to represent the earth surface and/or its characteristics. ⁃ Parameter uncertainty. The uncertainty associated to the values at which the different model parameters are calibrated. * Corresponding author. E-mail addresses: [email protected] (D. García-´ Alvarez), [email protected] (M.T. Camacho Olmedo), [email protected] (H. Van Delden), [email protected] (J.-F. Mas), [email protected] (M. Paegelow). Contents lists available at ScienceDirect Environmental Modelling and Software journal homepage: www.elsevier.com/locate/envsoft https://doi.org/10.1016/j.envsoft.2022.105411 Received 25 March 2022; Accepted 26 April 2022 Environmental Modelling and Software 153 (2022) 105411 2 ⁃ Model operation uncertainty. The uncertainty that arises from the accumulation and interaction of uncertainties propagated through the model. These sources of uncertainty in LUCC modelling exercises have been addressed widely in literature. Many papers assess the output uncertainty of specific exercises (Memarian et al., 2012; Ligmann-Zielinska 2013), whereas others focus on specific sources of uncertainty, like parameter uncertainty (Dietzel and Clarke 2004; Conway 2009; García et al., 2011; Van Vliet et al., 2013; Houet et al., 2015; Confalonieri et al., 2016; Grinblat et al., 2016; Liao et al., 2016) or process variability uncertainty (Kok and Van Delden 2009; Verburg et al., 2013; Alexander et al., 2015; Maier et al., 2016). Less common is the research about epistemological uncertainties of LUCC models and, specifically, about the models’ structural uncertainty (Elsawah et al., 2020). Some studies address specific topics related to this issue, like the different procedures to calculate change potential and change allocation (Riveira and Maseda 2006; Lin et al., 2011; P´ erez-Vega et al., 2012; Camacho Olmedo et al., 2013; Shafizadeh-Moghadam et al., 2015). Ferchichi et al. (2017b) propose a framework to quantify structural uncertainty of LUCC models based on probabilistic theory and sensitivity analysis. However, there is a lack of studies that characterize the overall structural uncertainty of available LUCC model software packages and analyse the tools and options that each model offers for uncertainty management and communication. Today, there is a large availability of standard model software packages to simulate different spatial dynamics (Camacho Olmedo et al., 2018b). Although they are considered too simple by some users to model complex phenomena, their use is ever-increasing (Wickramasuriya et al., 2009; Chaudhuri and Clarke 2013; Leija et al., 2021) and they are tools used in practice for real policy cases (Barredo et al., 2003; Van Delden et al., 2011; Eastman and Toledano 2018b; Guzman et al., 2020). Information about the uncertainty associated to the use of these software packages is not usually widespread and no paper analysing their structural uncertainty and their approaches to uncertainty management has been found in the literature. However, knowledge about these aspects is required to improve their understanding and characterization. In addition, it will help to engage planning agents and spread their use in real-world problems solving (Yeh and Li 2006; Batisani and Yarnal 2009; Sohl et al., 2016). Through this paper, we aim to fill the previous research gap by characterizing and comparing four standard LUCC model software packages. Model comparison has been proposed by several authors as a way to assess the structural uncertainty (Kelly et al., 2013; Uusitalo et al., 2015; Brown et al., 2021) and has been usually employed as a useful approach to better characterize and understand the available software packages (García et al., 2012; Toro Balbotín 2014; Mas et al., 2014; Aguejdad et al., 2016; Camacho Olmedo et al., 2018b). Through the comparison, we will answer the following research questions: ⁃ Which are the sources of uncertainty that come from the different model structures? ⁃ How does each model manage and communicates uncertainty? We will analyse the model structure of each software and the options that they offer for uncertainty management and communication through a qualitative comparative analysis of the models. Additionally, we will assess the potential uncertainty associated to the model structure by applying the four compared models to the same study case. In the following section, we explain the methodological approach of this paper in detail. 2. Materials and methods 2.1. Model software packages We compared four standard pattern-based LUCC model software packages: CA_Markov, Dinamica EGO, Land Change Modeler, as included in the TerrSet 2018 version, and Metronamica. Below, we provide a short description of each model. A graphic representation of each one is provided in the Annex 1. Annex 2 includes a comparative table to evaluate the differences among models. The models have been selected based on the authors’ deep experience with them (Mas et al., 2010, 2011, 2014; Camacho Olmedo et al., 2018a; García-´ Alvarez, 2018) and wide use among the LUCC modelling community (Sant´ e et al., 2010; Kamusoko 2012; Eastman and Toledano 2018b; Ferreira et al., 2019). Practical experience with the models is essential to fully understand the model conceptualizations and structures and their limitations in real-case applications. The wide use of these models among the LUCC community guarantees the utility of the results here delivered, as they help to characterize standard tools used by many users for different purposes, either as part of scientific studies or real case applications. In addition, as they rely on common LUCC modelling theories, we can draw general lessons from their analysis and comparison, which can be applied to any LUCC model. CA_Markov (Eastman and Toledano 2018a) is a modelling tool which makes use of several procedures integrated in TerrSet (previously IDRISI), a software of geospatial analysis and modelling. The quantity of changed pixels is determined by a Markov matrix, whereas the location of those pixels is performed through the combination of a series of suitability layers, a contiguity filter and a multi-objective allocation procedure. Dinamica EGO (Soares-Filho et al., 2002, 2009; Rodrigues and Soares-Filho 2018) is a free environmental modelling platform that includes LUCC modelling methods. Due to the flexibility that it offers, there is a wide variety of ways to set up a LUCC model. As common practise, Markov chains and Weights of Evidence (WoE) are used for the estimation of the quantities and change potential. Change allocation is performed through a couple of stochastic cellular automata functions: patcher, which produces new patches, and expander, which simulates the growth as expansion of previous patches. Land Change Modeler (LCM) (Eastman 2015a; Eastman and Toledano 2018b) is a constrained LUCC model which is also integrated in TerrSet. The change potential calculation is empirically obtained through three possible methods: neural networks, logistic regression and a machine learning algorithm (SimWeight). The change allocation is performed through a multi-objective allocation procedure, whereas the quantity of change is estimated by means of a Markov matrix. Metronamica (RIKS 2012; Van Delden and Vanhout 2018) is a constrained cellular automata model based on the theory developed by White and Engelen in the 90’s (White and Engelen 1993; White et al., 1997). Land use is allocated according to a “competition for space” principle based on the following inputs: interaction rules between land uses (human behaviour), accessibility, land suitability (environmental conditions) and zoning (planning). Demands can be defined externally or by means of a regional model simulating job and population dynamics. 2.2. Model comparison Model structures were compared according to the following aspects: change potential calculation (including the explanatory factors considered by each model), quantity of changes estimation, allocation of changes, pattern simulation, and validation and outputs. Although only the last one specifically relates with uncertainty management and communication, these questions have been reviewed for all other aspects as well, regarding the extent to which the models include tools or allow user intervention for uncertainty management. D. García-´ Alvarez et al. Environmental Modelling and Software 153 (2022) 105411 3 The comparison followed a both qualitative and quantitative analysis of each software (Fig. 1). The qualitative analysis complements the limitations of the quantitative analysis to assess sources of uncertainty that are not usually addressed in the literature (Elsawah et al., 2020). Through the qualitative approach (2.2.1) we compared the way in which each model conceptualizes the modelled system and the available options they offer for uncertainty management. Through the quantitative approach (2.2.2), we compared model outputs for the same case study (city of Cape Town) to assess the differences that come from the model structure. The case study is part of the urban modelling practice. Accordingly, results from our analysis may be affected by the specificities of urban dynamics: urban growth is usually simulated through a common set of driving forces (accessibility, physical suitability, zoning) and neighbour interactions. Nonetheless, general understanding of the models’ options for uncertainty management and system conceptualization can be also applied to other types of application. The study case is explained in detail in annex 3. It consisted of a model set up with a spatial resolution of 100 m grid cells for the city of Cape Town and the period 1990–2013. 2.2.1. Qualitative model characterization We reviewed and characterized the structures and features of the different software packages. This includes, among other items, system conceptualization, available methods for quantity of changes estimation, change potential calculation, change allocation, uncertainty management/validation and communication of results as well as the theory behind those methods. This information was obtained through the model’s documentation and based on the deep experience of authors with the analysed software (Mas et al. 2014, 2018; Paegelow et al. 2014, 2018; Camacho Olmedo et al., 2018b; García-´ Alvarez, 2018). 2.2.2. Model outputs assessment The four models were calibrated for the same case study following the approach described in annex 3. Outputs generated by each model were then analysed and compared. This allowed to analyse which differences between simulations came from the use of different model structures. According to this criterion, the higher the consensus among model outputs, the more certain is the simulation. Simulation success was also measured by comparing each simulation with reference data through common LUCC validation indices and metrics, Kappa Simulation (Van Vliet et al., 2011) and Spatial metrics (Mcgarigal 2018) (see below). The higher the agreement between reference and simulated data, the more successful the simulation is considered. For the comparison of output maps, we differentiated between softclassified and hard-classified maps (Camacho Olmedo et al., 2013). The first ones, which we will also refer to as Change Potential (CP) maps, show the probabilities of change to a specific category. Hard-classified maps, which are the final land use maps simulated by the models and we refer to as simulation results, assign every pixel to a specific category and, therefore, show states instead of probabilities. Agreement between CP maps obtained through different methods of change potential calculation was measured through the Spearman correlation coefficient incorporated in the R package “ENMTools” (Warren et al., 2021). To this end, Transition Potential (TP) maps to the same category in Dinamica and LCM were aggregated and compared to the Land Use Potential (LUP) maps for that category in CA_Markov and Metronamica. LUP maps show the probability of change to a specific category (e.g. B), whereas TP maps show the probability of a specific transition (e.g. A to B) happening. To make both types of maps comparable, areas of CA_Markov and Metronamica LUP maps not considered in the transitions of Dinamica and LCM TP maps were masked. Hard-classified simulated outputs from different models were compared by means of standard cross-tabulation techniques, Kappa Simulation (Ksim) and a set of spatial metrics calculated at the class level: number of patches, patch mean size and standard deviation and proportion of like adjacencies. Ksim evaluates the agreement between the changes simulated by each model compared to the agreement that is expected by chance (Van Vliet et al., 2011). Spatial metrics characterize the shape and size of patches and the way they are allocated on the map, that is, the maps’ patterns. A patch is a group of neighbour pixels with the same value (Botequilha Leitao et al., 2006). The proportion of like adjacencies inform about the aggregation or cohesion between patches of the same class (Mcgarigal 2018). That is, how aggregated or fragmented are the patches that make up a class. Each model was executed 20 times under the same parameters and conditions to assess the intra-model output variability. Only outputs from the first executions were employed for the previous assessments, whereas outputs from the remaining 19 executions were only employed to assess the agreement between model executions. Outputs from different model executions show low variability and do not significantly alter the pattern logic of the simulated CP areas or LUC changes. Thus, single outputs are enough to compare the output uncertainty caused by different model structures. Agreement between change potential maps obtained through the same production method for each model was assessed by calculating the average standard deviation of the pixel values across the 20 outputs. Intra-model agreement between simulations was assessed through KSim and cross-tabulation, as in the inter-model comparison. 3. Results 3.1. Change potential calculation Change potential is calculated in the four compared models based on the relation defined or found between a set of factors or drivers of change and the LUC changes. That relation can be defined by the user in the case of expert-driven models (CA_Markov, Metronamica) or calculated through automatic or statistical approaches, as defined by Van Vliet et al. (2016), in the case of data-driven models (Dianmica EGO, LCM). In the second case, models are trained until the best statistical relation between explanatory factors and LUC changes is obtained. In the first case, models are trained based on user criteria, which is usually driven by validation metrics, such as Kappa or Quantity and Allocation (dis)agreement indices. Although we can obtain the same relation between factors and LUC changes through any of the methods implemented in the four software packages, each method entails a specific workflow for the combination of factors and produces a specific type of change potential map. Depending on the method, there are also some restrictions regarding the number or type of factors that can be taken into account. Accordingly, the selected methods and the way they have been implemented are closely connected with the model conceptualization. 3.1.1. LUC and LUCC explanatory factors Only Metronamica puts restrictions regarding the number and type of factors considered. It only comprises four factors (neighbourhood interactions, accessibility, suitability and zoning), although the user can choose how many base maps he or she likes to include in the different factors and even rule out some of them by modifying the transition potential formula that guides the change potential map creation. The Metronamica factors may be dynamic. The model also includes a random component in the change potential map creation, which introduces variability between different CP maps produced by the model (yellow cell in Table 1). It is possible to run the model in a deterministic way as well (random factor =0), although the developers do not recommend this due to the inherent uncertainty in land use dynamics. CA_Markov is not able to work with dynamic factors, which makes the model more uncertain when simulating processes that are explained by different or variable factors along time. On the other hand, Dinamica EGO and LCM admit any type and number of factors, including the dynamic ones. Notwithstanding, for LCM, when using Logistic D. García-´ Alvarez et al. Environmental Modelling and Software 153 (2022) 105411 4 Fig. 1. Conceptual chart of the methods followed to compare the four model software packages. D. García-´ Alvarez et al. Environmental Modelling and Software 153 (2022) 105411 5 Regression as the change potential calculation method, the model requires factors that are linearly related to the potential for transition (Eastman 2015b). To this end, LCM includes tools for factors transformation. However, they transform the factors based on LUC data, assuming a temporal stationarity, i.e. the continuation of past trends to the future, which may not be real. 3.1.2. Two different types of change potential maps Dinamica EGO and LCM produce Transition Potential (TP) maps (Camacho Olmedo et al., 2013), which indicate the potential of a set of defined land uses to transition to another set of land uses. Although transitions can take place from any class to every other class on TP maps, they are usually restricted to the more meaningful, as it is difficult to find statistically significant relationships between a few LUC changes and a set of factors. In these cases, the found relationships may be significantly affected by errors in data or the presence of one-off events. Through expert-driven approaches, models produce Land Use Potential (LUP) or suitability maps (Camacho Olmedo et al., 2013; 2018a). They indicate the preference of each land use class to occupy any location of a study area based on a set of drivers defined by the user, which in practice allows any transition to happen at any point in time. They do not necessarily require of historical data to be obtained (Aguejdad 2021), although Camacho Olmedo et al. (2013) point out how users create these maps based on the understanding of distribution of the considered land uses in time, which implicitly includes the understanding of previous past changes. For our study case, correlation between Change Potential (CP) maps is independent of the type of maps compared. The CA_Markov and Metronamica LUP mas show the highest correlation with the TP maps of LCM, whereas TP maps automatically produced by Dinamica EGO show the highest correlalation with Metronamica LUP maps (Table 1). 3.1.3. Methods for change potential calculation Each model calculates the change potential though different procedures (Fig. 2), which are closely related to the way each model has been conceptualized. LCM offers three methods for transition potential map creation: Neural Networks and SimWeight, based on machine learning techniques, and Logistic Regression. The three methods trust automatic or statistical procedures as defined by Van Vliet et al. (2016) to find out the relation between changes and drivers of change. To this end, they use a sample of pixels as training and then, in the case of Neural Networks and SimWeight, the inferred relations are compared to a set of validation pixels. As far as the analysis sample varies with each model run, the inferred relations change with the sample as well (Kim 2010), although this variation is low (Table 1). The logistic regression procedure allows the user to employ all the pixels in the analysis and, therefore, avoid this possible uncertainty. Dinamica EGO makes use of the Weights of Evidence (WoE) to calculate the change potential maps, although the model also admits external maps produced through other methods to bypass the incorporated methods. The WoE is a Bayesian method that relates the presence of a given set of factors with the probability of land use change (Eastman et al., 2005). Soares-Filho et al. (2013) developed a Genetic Algorithm that allows the user to refine the change potential calculated through the WoE method. The software also allows the user to manually edit the obtained weights to account for some of the uncertainties that the data, calibrations periods, etc can entail. This manual adjustment may have a great impact on the obtained maps. In our modelling exercise, change potential maps obtained with automatic and adjusted weights showed big differences (Table 1). CA_Markov does not integrate a specific method for change potential calculation, although the model help advises to employ the Multicriteria Evaluation (MCE) implemented in TerrSet as the standard tool for this purpose. When using this method, the model will rely on user or expert knowledge, becoming very dependent on the uncertainty of that knowledge. In this regard, in this method he decides which factors to use and how they should be transformed and combined. He even assigns a weight to every factor. In Metronamica, the change potential map is calculated through a formula that combines a series of input data manually adjusted by the user. The user can also edit the formula, which gives him the chance to account for the model structure uncertainty. However, they can also introduce new sources of uncertainty by doing so. With the exception of Dinamica EGO, CP maps produced through the different methods implemented by each model show very high correlation among them and lower correlation with CP maps produced by other models (Table 1). In addition, there is not a clear correlation between CP maps based on their production method: manual vs automatic/statistical approaches. Accordingly, the Dinamica EGO CP maps automatically produced through the WoE show lower correlations with CP maps of other models than the CP maps obtained after the manual Table 1 In diagonal and grey, average standard deviation of change potential maps produced through the same method after 20 model executions for the transitions to residential areas. Off-diagonal cells show the Spearman correlation coefficient between change potential maps produced through different methods for the transitions to residential areas. CAM: CA_Markov; Metro 0: Metronamica with random factor =0; Metro 0.5: Metronamica with random factor =0.5; LG SY10: Logistic Regression in LCM with a Systematic Sampling =10%; LG SY100: Logistic Regression in LCM with a Systematic Sampling =100%; LG ST10: Logistic Regression in LCM with a Stratified Sampling =10%; LG ST100: Logistic Regression in LCM with a Stratified Sampling = 100%; NN: LCM with Neuronal Networks; SM: LCM with SimWeight; Manual WoE: Dinamica EGO with manual adjustment of Weights of Evidence; Auto WoE: Dinamica EGO with Weights of Evidence automatically calculated. D. García-´ Alvarez et al. Environmental Modelling and Software 153 (2022) 105411 6 Fig. 2. Methods for change potential maps production offered by each model. D. García-´ Alvarez et al. Environmental Modelling and Software 153 (2022) 105411 7 modification of the obtained weights. 3.2. Quantity of changes estimation CA_Markov, Dinamica EGO and LCM calculate the simulated quantities from Markov chains. They indicate the probability of every category to transition to every other class and to remain the same (Camacho Olmedo and Mas 2018a; Aguejdad 2021). Thence, they make the modelling process to focus on transitions instead of land use states. Through this approach, it is difficult to model systems where the land use dynamics change frequently and that do not follow historical patterns of growth (Mas et al., 2018; Paegelow 2018; Aguejdad 2021). That is, systems where the transitions between land uses are not always the same and at similar intensities. In addition, Markov chains usually calculate transition probabilities from past changes, extracted by comparing LUC maps at two different time points (Camacho Olmedo and Mas 2018a). In these cases, the uncertainty of input maps will be transferred to the obtained probabilities. Nonetheless, the three models allow the user to manually modify the obtained Markov probabilities from input data, and, therefore, to account for some of the uncertainties associated to input data. However, this step may not be easy for some users. The Markov probabilities tool implemented in TerrSet and used in the context of CA_Markov allows to consider in the uncertainty of input maps in the quantity of change estimation. However, this introduces important modifications in the calculated quantities of change (Mas et al., 2014). Accordingly, this method can introduce more uncertainty than the one for which it finds an answer. Metronamica does not include any method for quantity of changes estimation. The user enters the total number of cells (persistence + changes) that will make up each function class. The tool the user employs to decide the total number of cells will determine the uncertainty of this data. In addition, the transitions modelled for every category will rely on the settings of the interaction rules in the CA component. If the user does not introduce high values of inertia for the existent pixels of the function classes, some incongruent transitions can take place. Moreover, since the user cannot enter any information about the quantities of the vacant classes, their final size will also rely on the user calibration. Metronamica is therefore the most flexible model for modelling different types and speeds of change. However, because of this high level of flexibility, the ability of the user to replicate the dynamics of change is critical when calibrating the model and simulating the correct quantities of change. All four models calculated different quantities of change, despite being based on the same reference data (Table 2). Small differences in the way Markov probabilities are calculated and used to calculate the number of pixels explain the disagreements between CA_Markov, Dinamica EGO and LCM. Metronamica’s differences are explained by its specific method and the user calibration. 3.3. Allocation of changes LCM, CA_Markov and Metronamica follow a deterministic procedure for change or land use allocation. On the contrary. Dinamica EGO includes a stochastic algorithm of change allocation that simulates different changes each time that the model is run (Table 3 and Fig. 3). LCM and CA_Markov make use of a Multi Objective Land Allocation (MOLA) mechanism. It selects those pixels with the highest potential to change, solving conflicts between different objectives based on the minimum-distance-to-ideal-point-rule (Eastman et al., 1995). Metronamica follows a similar procedure. It selects the pixels with the highest potential to change for every category, allocating first the demands of the function classes and, then, the pixels of the vacant classes. Although the process is deterministic, the random factor added to the change potential maps allows transitions to take place in areas less likely to change. CA_Markov includes a contiguity filter as part of the allocation process, forcing pixels with lower potential values to be simulated as change if located next to previous pixels of the simulated category. In LCM, a zoning layer can be included in the allocation of changes step, multiplying the values of the change potential maps. The allocation functions of Dinamica EGO (patcher and expander) include a Cellular Automata component, favouring the simulation of pixels adjacent to land use classes of the same category. They also include a stochastic component to account for the unpredictability of human decision-making, which however is not easy to control. It is associated to a specific prune factor, which can be managed by the user, but also to a Monte Carlo approach of land use allocation and the parametrization of the expander and patcher functions (García-´ Alvarez, 2018). To reduce this stochasticity to a minimum, the user must choose a prune factor of 1, define clear and different transition potential values for the candidate cells and parametrize the expander and patcher functions according to the pixel size. In our case study, the stochasticity was relatively high: in 20 model executions, only 7.6% of the pixels simulated as change were allocated in the same place, with 34% of the changing pixels allocated in the same place less than 10 times (Table 4 and Fig. 3). Simulated changes from each model usually show more agreement with the outputs from other models than with reference maps (Table 4). CA_Markov and Metronamica are the models that simulate the most similar changes. On the contrary, CA_Markov and Dinamica EGO are the models simulating more different changes. These differences cannot be explained by differences between change potential maps, as there is not a direct relation between correlation of change potential maps and the agreement of simulated changes (Tables 1 and 4). 3.4. Pattern simulation CA_Markov, Dinamica EGO and Metronamica include a Cellular Automata component to replicate the real LUC pattern. This is lacking in LCM, which however allows to include a dynamic factor of distance to any of the map categories. LCM can infer from this variable the relation between LUC changes and the distance to cells of the other categories. However, this is calculated automatically by the model and therefore dependent on input data uncertainty. Because there is no user intervention possible, there is no direct control of the modelled pattern. This CA component or attraction factor is especially relevant to simulate urban dynamics, such as the ones of the City of Cape Town, as new urban areas usually grow on the urban edges, usually in the search Table 2 Changes in pixels simulated by every model (2002–2013) for every transition compared to the changes measured in reference maps for the same period. Modelled transitions Residential areas Vegegation areas to… Other cultivated areas to… Cultivated vine ´ areas to… Rural residential to… Reference 7376 658 232 331 CA_Markov 7139 649 231 330 Dinamica EGO (Manual) 6853 594 NA 297 LCM (Neuronal) 9361 121 NA 497 Metronamica (0.5) 8137 252 17 570 Table 3 Number of times that each pixel is allocated in the same place across 20 model executions in Dinamica EGO. 1-5 6–10 11–15 16–19 20 (Deterministic) 15.5% 18.3% 27.1% 31.4% 7.6% D. García-´ Alvarez et al. Environmental Modelling and Software 153 (2022) 105411 8 for economies of agglomeration (White et al., 1997). For our modelling exercise, although LCM simulated a general pattern very similar to the reference landscape, the simulation includes many small and scattered patches that do not fit the common pattern associated with land uses like urban residential. Even if we can check a visual coherent pattern in Fig. 4, the spatial metrics reveal how LCM was the model that simulated the most new patches of urban residential: 120 opposite to 7 new patches of change between the reference maps of 2002 and 2013 (Table 5). In CA_Markov the user controls the compactness of the simulated pattern through a user-defined contiguity filter. It up-weights the land use potential values of pixels close to pixels of the considered class and down-weights those which are far from this (Camacho Olmedo and Mas 2018b). It applies the same compactness logic to all modelled classes. In our study case, urban residential and urban informal changes were simulated according to the same pattern: patches of both classes became more compact, with a reduction in the total number of patches and a bigger mean path size (Table 5). However, urban informal pattern is more scattered (large increment of the number of patches and a lower mean patch size) than the urban residential one. Metronamica allows to define neighbourhood interactions between all classes of the map and each function class, making it possible to get a specific pattern for each class, solving the previous limitation of CA_Markov. The user can also play with the weight of self-attraction rules and the random factor to facilitate the production of new patches in Metronamica. However, both CA_Markov and Metronamica faced difficulties when trying to simulate changes as new patches. In the two models, all changes were simulated as infill of existing patches or as an organic halo from them (Fig. 4). Dinamica EGO simulates the desired pattern through two different functions: expander and patcher, which can be used together or independently. The expander function simulates changes as expansion of previous patches of the same use, whereas the patcher function simulates changes as new patches, disconnected from previous pixels of the same use. The user must indicate to the model the size and shape of the new patches for each modelled transition through three parameters: mean, variance and isometry. Accordingly, Dinamica EGO is the model that gives more control to the user regarding pattern simulation. For our study, the simulated landscape of Dinamica resembled quite well the reference landscape (Table 5), being maybe the best model when it comes to this point. 3.5. Tools for validation, uncertainty management and communication Directly or indirectly, all four models offer the user a wide range of tools to test the accuracy and uncertainty of the simulation results. This is possible in Metronamica through the complementary Map Comparison Kit software and in CA_Markov and LCM through TerrSet, the software where these models are included. The Dinamica EGO platform is directly able to calculate a wide range of validation measures. Metronamica is the only model that explicitly includes a tool for scenario management. It also allows to inform about the stochasticity associated to single run simulations. Dinamica EGO, because of its flexibility, can be designed to produce similar results. CA_Markov and LCM are more constrained to this end. LCM is only able to simulate business-as-usual scenarios (Eastman and Toledano 2018b). In CA_Markov, different model applications must be set up to account for different scenarios. All four models provide manuals and tutorials that describe the models’ methods and explain how to use them properly (Soares-Filho et al., 2009; RIKS 2012; Eastman 2015b). However, information about how to validate or assess the uncertainty of the modelling exercises is usually lacking. In addition, none of the four models is open source, which limits the user development and understanding of the software. Nonetheless, Metronamica foundations are deeply addressed in the literature (White and Engelen 1993; White et al., 1997) and described in detail in the model documentation (RIKS 2012), facilitating the replication of the model by other users. 4. Discussion Each of the four model software packages compared conceptualized the systems and processes to be modelled in a different way, which resulted in different outputs. These sources of structural uncertainty are discussed in detail in section 4.1. In addition, all models provide different methods or tools to deal, manage and communicate the possible uncertainties of the modelling exercise. They are discussed in section 4.2. Part of the results here discussed are limited by the study case selected for the model comparison. Model outputs were only compared for one specific case and one historic period. Analyses making use of different historic periods and study areas could provide complementary results. In addition, we have only judged the models’ success based on Fig. 3. Map showing the number of times that each pixel is allocated in the same place across 20 model executions in Dinamica EGO. Table 4 Off-diagonal cells show Kappa simulation scores between simulated changes by the compared models. The higher the KSim (0–1), the higher the agreement between changes simulated by the two simulations diagonal cells show. Diagonal cells show Kappa simulation scores between simulated and observed changes for each model. The higher the KSim (0–1), the higher the agreement between observed and simulated changes. CA_Markov Dinamica EGO (Manual) LCM (Neuronal networks) Metronamica (0.5) CA_Markov 0.23 – – – Dinamica EGO (Manual) 0.274 0.33 – – LCM (Neuronal networks) 0.303 0.415 0.41 – Metronamica (0.5) 0.566 0.374 0.515 0.33 D. García-´ Alvarez et al. Environmental Modelling and Software 153 (2022) 105411 9 their quantitative performance with respect to a historic period of reference. Assessing the plausibility of model parameters and results, based on expert judgment or other strategies, could also provide complementary conclusions. 4.1. Structural uncertainty How a system is conceptualized in a model comes with an important source of epistemological uncertainty, which may depend to a great extent on the purpose or objective for which the model was initially developed. Models developed for a specific purpose and application can include a structure that suit the simulated processes best. For standard models, like the ones assessed in our application, the sources of structural uncertainty can be bigger. The models assessed in our study have all developed towards generic modelling frameworks. Although they might have been developed for a specific purpose originally, over time they have been adapted to be able to simulate a wider range of dynamics. LCM and Dinamica EGO were initially developed to simulate deforestation dynamics (Soares-Filho et al., 2002; Eastman 2015a), but successfully simulated urban processes in our study case and have been applied with success in other domains (Eastman and Toledano 2018b; Rodrigues and Soares-Filho 2018). In this regard, they do not limit the number or nature of the factors considered, which makes them very flexible tools. That is also the case of CA_Markov, that was not specifically developed for any application, but has been successfully applied to Fig. 4. Changes simulated by every model (2002–2013) for an example area of Cape Town compared to the changes that the reference map show. The last map in the lower right corner shows the coincidences between the simulated changes by the four models. Table 5 Spatial metrics difference between 2002 and 2013 for the reference map and the four simulations for the categories urban residential (R) and urban informal (I). NP: Number of patches; Mean; Patch mean size; SD: Patch standard deviation; PLAJD: Proportion of like adjacencies. *E.g. 7 means that the reference map of 2013 has 7 more patches than the land use map of 2002. NP MEAN SD PLAJD R I R I R I R I Reference map 7* 25 4.97 2.09 53.92 5.86 0.17 4.02 CA_Markov −35 −8 19.44 5.23 88.03 13.62 1.67 8.39 Dinamica EGO 7 9 4.08 3.08 62.57 6.57 0.23 −0.69 LCM (Neuronal networks) 120 47 −20.18 −0.10 −1.87 7.65 0.25 −0.10 Metronamica 22 65 0.55 −0.25 87.88 5.35 0.63 3.16 D. García-´ Alvarez et al.