Optimal discretization for decision analysis

Woodruff, Joshua; Dimitrov, Nedialko B. Article Optimal discretization for decision analysis Operations Research Perspectives Provided in Cooperation with: Elsevier Suggested Citation: Woodruff, Joshua; Dimitrov, Nedialko B. (2018) : Optimal discretization for decision analysis, Operations Research Perspectives, ISSN 2214-7160, Elsevier, Amsterdam, Vol. 5, pp. 288-305, https://doi.org/10.1016/j.orp.2018.09.002 This Version is available at: https://hdl.handle.net/10419/246356 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. https://creativecommons.org/licenses/by/4.0/ Contents lists available at ScienceDirect Operations Research Perspectives journal homepage: www.elsevier.com/locate/orp Optimal discretization for decision analysis Joshua Woodruff ⁎ , Nedialko B. Dimitrov The University of Texas at Austin, Austin, TX, United States ARTICLE INFO Keywords: Optimization Discretization Joint discretization ABSTRACT The use of discretization in decision analysis allows practitioners to use only a few assessments to estimate the certain equivalent (CE) or expected value of a decision without knowing the functional form of the distribution of each uncertainty. The discretization shortcuts are fast, but are created with a specific distribution, or families of distributions in mind. The discretizations are not formulated with the decision problem in mind. Each discretization is specific to one uncertainty distribution, or is even more generalized. In this article, we introduce a novel mathematical formulation for selecting an optimal discretization for a specific problem. With optimal discretization, a decision analyst can use the newly-created shortcuts in repeated decisions and improve the expected accuracy of the CE calculations. 1. Introduction A common task in Decision Analysis is discretization, which involves reducing probability distributions of uncertainties to just a few point masses [13]. For many uncertainties in a Decision Analysis model, determining the value of multiple points in a distribution, let alone, the true distribution may be impossible or costly. Discretization is intended to reduce the cost of calculating certain equivalents (CEs) and simplify both communication and computation because it substitutes otherwise complex and computationally intensive integrations to the evaluations of just a handful of utilities. Discretizations significantly improve a decision analyst’s ability to communicate with clients [16]. Even with an increase in computing power, discretizations allow for human-un- derstandable assessment and evaluation of decisions. The discretization process can be thought of as having two distinct components. The first component is to select the points that will be used for discretization. Often, these points are chosen to be percentiles of the original distribution. The second component of discretization is to assign a probability mass to each point. An example of a discretization applied to two common distributions is shown in Fig. 1. Because of the usefulness of discretization, many methods for discretization exist [13]. One discretizations method divides the probability distribution into intervals based on values or cumulative distributions. Each interval is given a percentile equal to it median or its mean. Finally, each interval is assigned a probability based on the size of the interval. Another way of discretizing is to choose percentiles and probabilities that match the moments of the original distribution. Typically, the underlying distributions of the uncertainties are unknown. As a result, these methods have given way to shortcut methods that are easy to implement and work across a broad range of distributions. Decision analysts seek discretization methods that produce nearcorrect certain equivalents across the entire class of decision problems of interest. Typically the number of percentiles that a decision analyst uses in a discretization is three, which provide a low, high, and mostlikely value. Though shortcut methods are easy to use, they are not created with the decision problem in mind. This can lead to reduced accuracy and situations where the results of different strategies are within a few percent of each other, sub-optimal decisions. Our method for choosing a discretization differs from these methods in that we seek a set of discretizations that provide the lowest certain equivalent error over a large set of potential distributions for a specific problem. The error particularly matters when the certain equivalent of a decision is close to zero. In a go, no-go decision, an error in the calculated certain equivalent would result in the wrong decision. We generate problem-specific discretizations by solving an optimization problem that chooses the percentiles and the corresponding probabilities that minimize an error metric over a large set of potential versions of a problem. The main contributions of this article are: 1) We mathematically formulate the problem of selecting an optimal discretization for a set of decision problems. 2) We are able to derive tractable instances of this optimization problem. These tractable instances allow us to compute optimal discretizations for a specified set of decision problems. 3) Prior discretization methods produce discretizations for each uncertainty in https://doi.org/10.1016/j.orp.2018.09.002 Received 9 May 2017; Received in revised form 19 September 2018; Accepted 19 September 2018 ⁎ Corresponding author. E-mail addresses: [email protected] (J. Woodruff), [email protected] (N.B. Dimitrov). URLS: http://www.optimizedfinancialsystems.com/ (J. Woodruff), http://neddimitrov.org/ (N.B. Dimitrov). Operations Research Perspectives 5 (2018) 288–305 Available online 20 September 2018 2214-7160/ © 2018 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/BY/4.0/). T the decision problem. We introduce a joint discretization where the probabilities of the percentiles of a discretization are not independent from each other. The mathematical formulation for computing optimized discretizations opens a new area of computing joint discretizations. Finally, we show that our methodology for computing independent and joint discretizations outperforms prior methods across a set of computational examples. This article is structured as follows. We begin with related work in Section 2, describing some popular and novel methods of discretization. Next we provide a general formulation for optimal discretization in Section 3. The general formulation is followed by the a tractable instance in Section 4.1 and modifications required to create a joint discretization in Section 4.2. In Section 5, we analyze two examples from the literature to determine the benefits of optimal and joint discretization, and the effects of sampling and pre-determining percentiles.We conclude with a discussion and future work in Section 6. 2. Related work Discretizations are typically divided into distribution-specific methods and shortcuts. Distribution-specific methods require knowledge of the probability density function (PDF) of an uncertainty’s distribution prior to discretization. These methods choose the discretization based on some criteria of the original distribution that the decision analyst is trying to match. Discretization shortcuts require experts to assess a few (usually three) percentiles of the uncertainty’s distribution. They do not require knowledge of the shape or moments and are easily applied. A third type of method is a hybrid approach. This method assumes limited knowledge of the underlying distribution and provides a discretization based on this knowledge. Prior methods focus on computing discretizations independently for each uncertainty. In areas such as stochastic optimization there are more examples of multivariate discretizations [20]. We will review a few of these discretization methods and compare them to our approach. In a decision analysis project, if the client, decision analyst, or some other expert knows the true distribution of each uncertainty, then the decision analyst can determine the CE of even the largest problem using Monte Carlo sampling or some other technique. With the true CE, the decision analyst can recommend the strategy with the highest CE with the certainty that this is the best recommendation. In reality, the form of each uncertainty is unknown. The decision analyst will elicit assessments for the uncertainties and apply a discretization to these uncertainties to create an estimate of the CE. Two common distribution-specific discretization methods are bracket mean and Gaussian quadrature. Bracket mean, described by McNamee and Celona [15], is also known as the equal areas discretization. In bracket mean discretization, the PDF is partitioned into three sections, with probabilities 0.25, 0.50, and 0.25. The percentiles assigned to each region represent the mean value within the probability region. This method matches bounded means of the uncertainty’s PDF. Though the method matches these bounded means for each uncertainty, there is no guarantee that the discretization with match the certain equivalent (CE) in the decision problem. Miller III and Rice [16] and later Smith [19] proposed techniques based on Gaussian quadrature (GQ). With a discretization of Npoints, it is possible to match the first N2 1 moments of an uncertainty’s distribution. The idea behind matching the moments comes from considering expectations of low degree polynomials. A discretization that matches an uncertainty in the first N2 1 moments, will produce the same expectation for any N2 1 order polynomial. Smith [19] improved the method for GQ by making it more efficient and provided examples of how GQ matched the moments of the input distributions ([19], Table 2, P.345). GQ requires knowledge of the distributions being discretized – at least its first N2 1 moments. If these are not known in the literature, one may require complex numerical integrations. An application of GQ also requires solving a multivariate system of polynomial equations, which is easily computed with matrix manipulation software. The polynomial matching argument misses crossterms of the uncertainties when the utility depends on several uncertainties. From a client perspective, this discretization may ask for percentiles that are not easily assessed, such as the value of the uncertainty at the 99.50th percentile. In contrast, our method is able to limit the discretization’s percentiles to easily assessed values, directly targets computing CEs, and the discretization is dependent on a set of decision problems, as opposed to a single specific distribution. Shortcuts are easy to use and generally perform well. Shortcut discretizations do not require knowledge of the distributions of the uncertainties and apply the same discretization percentiles and probabilities to all uncertainties. Two common shortcut methods are the McNamee–Celona Shortcut (MCS) and the Extended Swanson–Megill (ESM) method. Both of these discretization methods use the 10th, 50th, and 90th percentiles. The Extended Swanson–Megill (ESM) method for discretization is described by Hurst et al. [10] and is commonly used in the oil and gas industries [2]. MCS assigns probabilities of (0.25,0.50,0.25) while ESM assigns probabilities of (0.3,0.4,0.3) to each of the respective percentiles. In examining several discretization methods, Keefer and Bodily [13] proposed the Extended Pearson–Tukey method. This method proposed using the 5th, 50th, and 95th percentile with probabilities of (0.185,0.630,0.185). Keefer and Bodily [13] found EPT outperformed several other methods. Additionally Hammond and Bickel [8] found EPT to have the best performance among MCS, EPT, and ESM when calculating average absolute error, average absolute percent error, maximum error, and maximum percent error. While easy to apply, these methods do not take into consideration specific knowledge about the decision problem. In contrast, our method computes an optimized discretization for a specific set of decision problems. It is often known if the distributions of the uncertainties have specific shapes or are bounded. For example, when drilling for oil it can be assumed that the percentage of oil to be recovered will be between 0 and 100%. To that extent, there are newer shortcuts that are based on shape-specific assumptions. Hammond and Bickel [9] and Hammond and Bickel [8] offer new shortcuts based on the Pearson and Johnson families of distributions. Based on the specific zone of the assumed distribution of the uncertainty, Hammond and Bickel [8] provide a specific discretization to use. For example, a normal distribution is symmetric and unbounded on either side. A log-normal distribution is bounded from below, might be right-skewed, and unbounded from above. These methods are similar to the shortcut methods, but address a specific family of distributions. There are shortcuts for bounded, semibounded, and unbounded distributions. The purpose is to provide a better discretization that uses potentially available information while still allowing for an unknown distribution. Distribution-specific Fig. 1. These are two examples of the same three-point discretization applied to a standard normal and a log-normal distribution. The placement of the points on the independent axis are determined by the percentiles of the discretization, and the height is based on the probabilities assigned. J. Woodruff, N.B. Dimitrov Operations Research Perspectives 5 (2018) 288–305 289 shortcuts are similar to the other discretization methods in that these methods only take into account individual uncertainties as opposed to the decision problem as a whole. In contrast, our method considers a set of similar decision problems and discretizes all the uncertainties simultaneously. Wallace and Hoyland [20] developed a technique based on nonlinear programming that can be used to generate a finite number of discrete events that satisfy specific statistical properties. The main difference between Wallace and Hoyland [20] and our method is that they consider a single distribution, and attempt to match potentially multiple statistical properties of that distribution using a small discrete set of scenarios. On the other hand, we have potentially many distributions and decision problems and we are looking for a single set of discrete percentiles that return approximately correct CEs for all the decision problems. Most research has focused on matching aspects of a single uncertainty’s distribution – for example, its moments or bounded means. Brockett and Kahane [3] found that moment matching does not produce accurate CEs, partly because it does not take into account other aspects of the decision problem such as the value function and the utility function. Keefer [12] also tested the accuracy of three point discretizations and found “substantial” (P.763) errors in the CEs they produce. Three point discretizations can match the first five moments, which may have contributed to the failure of moment matching in this instance. While the uncertainty’s moments may match, the uncertainties combine and are passed through a value function, a utility function, and a certain equivalent calculation. In certain problems this may result in CE errors because the distribution of the CE may be completely different than any of the uncertainty distributions. Keefer [12] further tested six shortcut methods to determine their CE errors, with EPT identified as the best performer overall. Smith [19] found that bracket median and bracket mean had higher errors when calculating the CE than did EPT and GQ. In contrast to past methodologies, our method focuses on matching the CEs of a set of decision problems. In this way, the methodology can both capture lack of knowledge – because the discretization is based on a set of decision problems – and specific knowledge about the decision problem structure – which is incorporated through similarities in the set of decision problems. 3. General formulation for discretization Optimal discretization finds a discretization for the uncertainties of a problem that is going to be revisited frequently. In the problem there are uncertainties whose distributions will change over time. For example, each oil field will have different potential reservoir and recovery ratio. In the consumer packaged goods industry every new product will have a different potential market share and market size. The realizations of the uncertainties are fed into a value function, which computes the net present value. Finally, there may be a utility function to convert project values to certain equivalents. The general discretization problem assumes independence among the percentile values of the uncertainties. In the independent discretization, the percentiles chosen from one uncertainty are independent from one another. We still allow for dependence between the values of uncertainties. In the examples we use in this article, the values of one or more uncertainties depend on one or more other uncertainties. We use various methods to determine those values. In an assessment framework, the decision analyst would still elicit dependent (correlated) assessments from the independent percentiles. In our decision problem, the functional form of the uncertainties is known. What we do not know is which decision problem we are facing and consequently are unsure of the functional forms we will face when needing to make a decision. The combination of all the potential uncertainty distributions when applied to a value and utility function come to define our set of decision problems, . To determine the CE of one instance of , it is possible to compute the CE by means of Monte Carlo sampling. When interacting with a client, a decision analyst does not know the exact problem in they are addressing. We seek a discretization that works over all cases of . Let define a prior probability distribution over the potential decision problems . This is a probability assignment on each problem d . This allows the decision analyst to specify that some decision problems are more likely than others. For a specific decision problem d, let CE d denote the certain equivalent of decision problem d. Let be the set of allowable probability mass functions of the uncertainties. Let CE d (p) for p denote the certain equivalent of decision problem dwhen the distribution pis used for the uncertainties instead of the true distribution. In other words, CE d (p) is the certain equivalent when we use the discretized distribution instead of the true distribution of the uncertainties. We want to find a p that matches CE d (p) ≈ CE d for all decision problems d . A discretization with a perfect fit will have =CE CE p( ) d d for all d . We begin by formulating discretization as an optimization problem. This is a novel contribution to the area of discretization, and leads us to the results in the rest of the paper. The discretization problem can be formulated as an optimization as follows: +Err d p E Err d pargmin max ( , ) (1 )[ ( , )] , pdd (1) where =Err d p CE CE p CE ( , ) ( ) or d d d (2) =Err d p CE CE p( , ) ( ) . d d (3) Given a parameter λ∈[0, 1], optimization (1) defines the discretization problem. The problem seeks to find a distribution p that yields the minimum convex combination of worst case absolute error and expected absolute error, E Err d p( , ), d with respect to the distribution . We use as the distribution of decision problems to indicate there is a probability associated with each decision problem. Eq. (2) defines the absolute percentage error in the CE when using the discretized distribution of uncertainties instead of the true distribution. We also include (3) as an alternative formulation to the error function when CE d has values that are orders of magnitude different, such as when CE d may be positive or negative. When λis one, we seek a discretized distribution that yields the minimum worst case error. When λis zero, we seek a discretized distribution that yields the minimum average error over the distribution of decision problems . 4. A tractable discretization instance In order to determine an optimal discretization, we need to formulate a model that is solvable by available software in a reasonable amount of time. In this section, we give a specific and tractable instance of the more general discretization problem described in the previous section. We do this by defining a specific set of discretized probability distributions , a specific set of problems , and a probability distribution over the problems. The optimal choice of p defines the optimal discretization These definitions allow us to formulate the discretization problem as a tractable, though non-linear, integer program (NLIP). In this section we discuss our process for creating a tractable model. In our process we have a challenge that prevents us from formulating the model as we envision in Section 3. We discuss how we solve this challenge. Finally, we find there are some assumptions we can relax and provide a second tractable formulation. Solving a tractable discretization instance requires defining the objective value, decisions, and constraints with data such that the engines that solve the model are able to solve it within a given time. J. Woodruff, N.B. Dimitrov Operations Research Perspectives 5 (2018) 288–305 290 4.1. NLIP formulation of discretization problem The successful implementation of our NLIP requires we pre-calcu- late CE d for each d . We estimate the CE d values through simulation. Specifically, we sample the uncertainties from the distribution combinations defined by d . From the simulation we compute uncertainty values, a project value distribution, expected utility, and a certain equivalent. To formulate optimization (1) as a tractable NLIP, one key obstacle to overcome is that the objective Err(d, p) is a non-linear function in pas shown in (2). In this formula, CE d is already a constant we obtained from our formulation, but CE d (p) is our calculated CE. The formula for CE d (p) is given by =CE p ln prob u pCertainEquivalent: ( ) · · ( ) , dd p p d (4) =u x exp xwithexponentialutility: ( ) ( / ), dd (5) =P X x pand ( ) (6) where ρ d is the risk tolerance, prob p is the probability assigned to percentile combination pand u d (p) is calculated expected utility for decision problem dwith percentile combination p. We use the exponential utility function in this example. Different utility functions will change the formulations for both u d (p) and CE d . Normally utility is expressed in terms of the value of the project/decision, x. We are searching for the optimal percentiles, so we used (6), the definition of the CDF, to relate p to x. When we formulate a non-linear IP by using formula for CE given by (4) directly, our non-linear solver, Bonmin 1.8.4, we found it to be intractable. To create a tractable formulation we linearize the objective function. We choose to linearize Err(d, p) around the expected utility, E [u d (x)]. We first substitute the equation for certain equivalent, (17), into the definition of Err(d, p) to get =Err d p E u x CE ( , ) 1 ln( [ ( )]) , dp d d (7) where E p [u d (x)] is the expected utility of the decision problem under the new discretized distribution p . Given a d, the only variable in the above formula is E p [u d (x)], and everything else is a constant. We choose to linearize this equation, viewed as an equation in the variable E p [u d (x)] around the true expected utility E[u d (x)]. We choose to linearize because this is easier for most solvers to solve. An extra Taylor expansion term can be added to improve the accuracy of the approximation at a cost of additional solve time. For brevity let T d , our target utility, denote E[u d (x)]. To compute a linearization, we first drop the absolute value sign, assuming the second summand of (7) is smaller than one. This gives =f w w CE ( ) 1 ln( ) , d d which is now a continuous function of w. We create was a shorthand for the variable E p [u d (x)]. We can now do a first order Taylor expansion of this function around T d to get + = f w f T w T CE T w T ( ) 0 ( )( ) ·( ). d d d d d d This approximation is valid when the second summand of (7) is smaller than one. This happens when wis greater than or equal to T d . A similar argument, assuming that the second summand is greater than one, yields the approximation w T( ), CE T d · d d d which is valid when wis smaller than or equal to T d . Together, these two linearizations can be summarized as = CE T· d d d d (8) Err d p E u x T( , ) · [ ( )] , d p d d (9) which is a linearization of Eq. (7).Fig. 2 plots an example of the true error function and corresponding linearization. In the case where the decision maker is risk neutral, we can skip the calculation of δ d (8) and just use = dCE 1 d . With this linearized objective function, we can now write an IP for computing an optimal discretization as follows. Indices and sets i I :the set of uncertainties vi Vi :the set of percentile discretization of uncertainty i . These are candidate percentiles for the uncertainty, such as {5, 10, 15, 45, 50, 55, 90, 95} . Vvi :a percentile combination for all uncertainties. v is a vector of length I| | . d :a finite, discrete set of decision problems j J :the indexes for each of the J| | incompatible sets discretizations Parameters :used to compute a convex combination of average and maximum error T d :the true expected utility for decision problem d d :a shorthand for , CE T· d d d a constant used in linearization Ni :the maximum number of percentiles per uncertainty for the output discretization Uv( ) d :the utility of the project value for decision problem d and at percentile combination v j :the incompatible discretizations, v, in set j P d :the probability assigned to decision problem, d Decision variables pv :the probability assigned to a combination of percentiles v . od :the over-estimation in approximating T d with a discretized probability distribution ud :the under-estimation in approximating T d with a discretized probability distribution z :the estimated max Err d p( , ) d Fig. 2. This is a sample of a linearization of the absolute percentage error and the true absolute percentage error as a function of the expected utility. If the results of the discretization are close enough to the true value, the linearization of the error function is sufficient. J. Woodruff, N.B. Dimitrov Operations Research Perspectives 5 (2018) 288–305 291 xvi :the probability assigned to candidate percentile vi for uncertainty i yvi :1 if percentile vi is used for uncertainty i and 0 otherwise + +z P P u o Formulation min · (1 ) · ·( ) d d d d d d (10a) + =U p o u T dvs.t. ( ) V d d d d v v i (10b) +z o u d·( ) d d d (10c) =x i I1.0 v V v i i i (10d) y N i I v V vi i i i (10e) y i I j J1 , v v ij i (10f) x y i I v V; v v i i i i (10g) =p x Vv v vi v v i i (10h) x i I v V0 1 ; v i i i (10i) y i I v V{0, 1} ; v i i i (10j) o u d, 0 d d (10k) Objective (10a) of the optimization model is to minimize a convex combination of the largest error zand the average error. In this formulation we show the generalized distribution on . The second term of the objective function is the average error. In order to compute the linearized error (9), we should compute the absolute value of the difference between T d and E p [u d (x)]. The formula for T d is given for formula (18) in Appendix A.1. Constraint (10b) computes the difference between the target and expected utility. Constraint (10c) computes the maximum error, z. Constraint (10d) forces the sum of the probabilities for each uncertainty to sum to one. Constraint (10e) limits the number of percentiles allowed for each uncertainty. Constraint (10f) forces only a single low, a single medium, and a single high percentile in our discretizations. This helps the optimization engine find a solution faster. For example, our low-percentile candidates are P5 and P10. Only one may be selected for the deiscretization. Constraint (10g) forces the assigned probability to zero if the percentile is not used in the discretization. Constraint (10h) computes the probability assigned to a percentile combination as a function of the probabilities of each of the uncertainties. This is the only non-linear constraint in the formulation and it enforces that the output distribution p is independent over the uncertainties. The remaining constraints bound the probability values between 0 and 1, make the indicator variables binary, and make the underage and overage non-negative. 4.2. Joint discretization problem A joint discretization differs from an independent discretization in that the probability of a percentile combination is assigned individually, and is not the product of the probabilitiy assigned to each percentile in the percentile combination. For a joint distribution, we assign the probability to each percentile combination separately. This relaxation increases the flexibility of the values assigned to a percentile combination and it linearizes the model formulation. Previous discretization techniques only considered uncertainty discretizations independently. Because we consider a set of decision problems and compute the best discretization for that set of problems, it is possible to compute this joint discretization. The feasible solutions for a joint discretization include the feasible solutions for independent discretizations. As a result, computing an optimal joint discretization is guaranteed to produce at least as good a result as computing an optimal independent discretization. Furthermore, computing an optimal joint discretization has the potential to decrease solution times, as it removes the non-linearity in Model (10). By removing the non-linearity, we could also switch solvers from Bonmin to CPLEX. In this section, we compute such optimal joint discretizations. We alter Model (10) as follows to compute optimal joint discretizations The formulation drops variables xv i and any constraints where they appear. These are Constraints (10d) and (10h). We also add the following constraints: =p1 Vv v i (11a) p y V vv v, vii vi (11b) p0. v (11c) Constraints (11a) and (11c) ensure the variables p v compute a joint probability. Constraint (11b) ensures the support of that joint probability is limited to the N i percentiles for each uncertainty i. The result of Constraint (11b) is that experts make the same number of assessments as before. 5. Analysis In this section we solve Model (10) for a sample problem given by Smith [19]. We briefly describe the example here and more in depth in Appendix A.1. We also apply the methodology to a second problem originally given by Clemen [4] and expanded by Reilly [18] and further described in Section A.2. We begin with the Smith [19] wildcatter problem. A wildcatter is a person that drills for oil in a previously undeveloped field, and will not know exactly what he or she will find. In this simple model there are four uncertainties that impact the value of the well: the oil price at which the wildcatter can sell, the amount of oil in the well, the amount of oil the wildcatter can recover, and the cost of the recovery. The exact distribution of the each uncertainty is unknown. Rather than solving the problem with the functional forms used by Smith [19], we use several candidate distributions as shown in Fig. 3. Any one of the candidate distributions could be the true distribution. We use nine distributions per uncertainty resulting in a total of 6561 decision problems, any of which is equally likely to be the true problem. In this formulation we assume the risk tolerance is known at the time of the problem definition. The objective of optimal discretization is to find the discretization that minimizes that error over all the problems. For each of the 6651 decision problems we use Latin hypercube sampling as originally described by McKay et al. [14]. We generate 4,000,000 values for each uncertainty to generate a set of present values, . For each x X( ) we generate a utility and determine CE d using (17) with a risk tolerance value, =$16, 000, 000 . The distribution of the CE d is found in Fig. 4. From each CE d we are also able to obtain a target utility,T d , using (18). For each decision problem we also calculate δ d using (8). The set of percentile combinations is drawn from a Cartesian product of the candidate percentile for each uncertainty. We define ⊗V i as the set of potential percentile combinations. We allow each of the four uncertainty percentiles to be in the set {5, 10, 45, 50, 55, 90, 95}. This represents the set of potential assessment values we might ask an expert to give for each uncertainty. This set is neither inclusive, nor exhaustive. These are illustrative choices, and a decision analyst may add J. Woodruff, N.B. Dimitrov Operations Research Perspectives 5 (2018) 288–305 292 or remove percentiles. An increase in candidate percentiles may improve accuracy, and it will increase computational complexity. For each decision problem we have 2401 potential percentile combinations. we will choose =3 81 4 of those percentile combinations and assign them probabilities to create our optimal discretization. For this discretization instance, we are defining the distributions in ⊗V i as independent over the uncertainties. For each of the p2401 we calculate the utility for each decision problem d using (16) to calculate U d (v) for each v∈ ⊗V i . This provides the data we need to populate our optimization model. We begin our comparison of optimal discretization to four incumbent distributions. These are the MCS, ESM, EPT, and HB methods. These discretization each use three percentiles and assign probabilities. With four uncertainties in the problems, this yields 81 potential outcomes for each decision problem. We use the percentile from each discretization to get a value from the decision problem’s uncertainty distributions inverse CDF. We compute the project value and utility based on the samples. Finally, we compute the CE using the probabilities assigned to each percentile. This gives us an estimated CE for each decision problem. We compare the estimated CE using the discretization to the CE we obtained by using the simulation for the same problem using the equation CE CE p CE 100* ( ) . d d d (12) We create a distribution of errors for each discretization method and present them in Fig. 5. The HB and EPT methods use more extreme percentiles like the 5th and 95th percentiles. The MCS and EPT discretizations use the 10th and 90th percentiles. The accuracy of the discretizations with more extreme values is visible in Fig. 5. We use two measures of accuracy. The first is the worst-case error. This is the largest absolute value of a percent error from the true CE across all decision problems. The other error metric is the average of the absolute errors. When we measure the worst case error, HB has a worst-case of 1.75% and EPT has a worst case of 1.47%. ESM has an absolute worst case error of 2.02% and MCS has a worst case of 6.34%. The mean absolute errors of HB and EPT are both 0.25%. HB has a slightly better performance in terms of absolute error, but when rounded to the nearest hundredth of a percent, they are the same. MCS and ESM have average absolute errors of 1.30 and 0.41%, respectively. In the wildcatter example, the standard deviation of Err(d, p) is larger and the mean CE is further away from 0 when the discretizations using the extreme ( 5th and 95th ) percentiles is used. In this section, we solve Model (10) using two parameters for λ. When we set =1, we minimize the worst case error, when we set =0, we minimize the average error. 5.1. Independent discretization In creating optimal discretizations we have two goals in mind. The Fig. 3. Each uncertainty has nine candidate distributions. The reservoir, price, and cost distributions are bounded from below at zero. The recovery distribution is bounded by 0 and 100%. Though most distributions are similar in shape, we also included a uniform distribution in as potential distribution for the fraction of the reserves that may be recovered. Fig. 4. The distribution of CE values for the decision problems created from all the combinations of potential distributions for each uncertainty and a single risk tolerance. Fig. 5. The distribution of percent errors for four shortcut methods. J. Woodruff, N.B. Dimitrov Operations Research Perspectives 5 (2018) 288–305 293 first goal is to find discretizations that minimize (10a). The second goal is to find this solution quickly. We define quickly rather loosely. If this is being done for an ongoing project, we want to be able to generate an optimal discretization for the client between sessions; Otherwise, we want to have the discretizations computed for the next time a decision problem comes up. We solved the complete model, with all the candidate percentiles for each decision problem. We also solved different versions of problem (1) using subsets of the candidate discretizations or using a subset of the decision problems. By limiting the candidate discretizations, we are able to reduce the number of variables. Specifically, when we reduce the number of candidate percentiles to three, we are able to solve the model as a continuous problem instead of as a non-linear mixed-integer problem. Our candidate percentiles for the full problem are P5, P10, P45, P50, P55, P90, P95. Some of the most popular discretization methods use either P10, P50, P90 or P5, P50, P95. We try these two sets of three plus the full set. Our second way of reducing the computational time is to reduce the number of decision problems by sampling them.We test how the results differ when we chose to minimize the worst-case discretization error and when we try to minimize the average discretization error. 5.2. How much benefit do we get from optimizing an independent discretization? The calculation of the certain equivalent is given by multiplying the probability of each percentile of each uncertainty to determine the probability of an outcome. There may be a covariance among the resulting values, but the percentiles are treated as independent. In the case of the four uncertainties in our sample problem, the probability of any one outcome is the product of the probability of each of the individual uncertainties. The drawback of the nonlinear approach is the that there are few available solvers, and large problems generally take too long to solve. For example, in our test problem, solving the full problem with =0.0 using Bonmin 1.8.4 using an Intel 6-core I7 processor running at 2.6 GHz, the average time to generate the model and solve the problem was 129,937.72 s (1.5 days). This problem has 6651 decision problems and over 15,000,000 nonzeros. The results from of the optimization are shown in Fig. 6. When comparing to HB, which has the best results in Fig. 5, independent discretization improves the worst case mean error and the standard deviation of error. 5.3. How much do we lose by solving a smaller sample of decision problems? Given the intractability of solving for 6651 decision problems with 20,736 possible combinations of percentiles for the uncertainties, we tested the effects of sampling the decision problems. In sampling the decision problems, we randomly select a subset of the decision problems and then applied the optimal discretization for that problem to the entire set of problems. The results are displayed in Fig. 7. We test the sampling with 1, 5, 10, and 20% of the 6651 decision problems. We Fig. 6. The histogram of results compares HB, which has the best average error of the incumbent methods with the optimized discretizations using both the optimal average and optimal worst-case preferences. The optimized results show a reduction in average absolute error of 56% and a reduction of worst-case error of 74%. Fig. 7. As the number of samples increases in percentage, the overall accuracy of the discretization improves. After 10% of the samples, the average error is 1% worse than the optimum using the entire set of decision problems. J. Woodruff, N.B. Dimitrov Operations Research Perspectives 5 (2018) 288–305 294 test this method using the worst case and the best average objectives. The solve time increases roughly linearly with the number of decision problems that we sample. Sampling with 1% took 24 min, sampling with 10% took just under 3 h, and sampling with 20% took just under 6 h. Solution quality, as measured as the increase in objective value from the optimal value with 100% sampling improves with the number of samples, but not linearly. Sampling with 1% results in a 34% increase in the average error, sampling with 10% results in an increase of 1% in the average error, and sampling with 20% results in and increase of 1.4% in the average error. The increase in average error in the sampling is likely due to the randomness of the sampling. When we looked at the solution quality, there is a noticeable difference between choosing =1 and =0 . For the smaller samples ( < 20%), minimizing the worst case led to varying degrees of over fitting, with increases in worst case error of 25, 15.7, and 6% for the 1, 10, and 20% samples respectively. Sampling the decision problems results in roughly linear speedups in performance with a small loss of accuracy. 5.4. How much do we lose by restricting the candidate percentiles? Some of the most common discretizations use either P10, P50, P90 or P5, P50, P95. In comparing both the shortcut methods and the discretization results, it seems the most accurate discretizations come from using the more extreme percentiles. If P5, P50, P95 are more accurate, it may be better to only consider these percentiles and optimize the probabilities. Given the P10, P50, P90 discretizations are also popular, we also want to know what improvement in accuracy we can expect when considering the more extreme percentiles. We solve the problem using either the maximum error or the average error objectives. The first improvement is the rapid speedup in solution time. The range of reduction is from 99.5 to 99.9% reduction in the time required to generate a discretization. The solution times were in the 100–300 s range. When limiting the candidate discretizations to P10, P50, P90, the mean absolute error is 4.8% lower than the mean absolute error using the HB shortcut. It should be pointed out this slight improvement comes using less-extreme values than those required by HB. In comparison to using all the candidate percentiles from a full optimization, the mean absolute error is still 117% worse when using P10, P50, P90. These results are shown in Fig. 8. Using the P5, P50, P95 percentiles improves the accuracy of the optimal discretization while solving quickly. The optimized discretization increases worst case error by just 0.25% over the optimal results obtained from considering all the percentiles. This result is shown in Fig. 9. 5.5. What benefit do we derive when we remove the independence of uncertainties? The results in solving the problem with a nonlinear solver have mixed results. Though the improvement in accuracy is substantial, some instances take days to solve. In a large business problem with 12 or 15 uncertainties, the size of the problem could become intractable. Previous discretization methods focused on individual uncertainties, which were combined to create a distribution of the decision problem values. We propose a new approach which relaxes the independence of uncertainty percentiles and creates a joint distribution. Joint discretization improves both performance time and the Fig. 8. Limiting the candidate percentiles to 10-50-90 reduces the error in comparison to the shortcut methods, and is worse that when considering a larger assortment of candidate percentiles. Fig. 9. Limiting the candidate percentiles to 5-50-95 reduces the error in comparison to the shortcut methods, and is worse that when considering a larger assortment of candidate percentiles. J. Woodruff, N.B. Dimitrov Operations Research Perspectives 5 (2018) 288–305 295 tolerances penalize losses more. An infinite risk tolerance makes the CE to be equal to the expected value of the project. Given a d, we can compute a CE numerically by sampling project values based on the uncertainty distributions, converting values to expected utility, and expected utility to a CE. A2. Eagle Airlines Eagle Airlines is a problem described by Clemen [4] and further refined by Clemen and Reilly [5] and Reilly [18] in the area of fleet expansion. Here we used the information from [18] and the distributions given by Montiel and Bickel [17]. In this problem the owner of Eagle Airlines must decide whether or not to expand his fleet with the addition of one plane. The alternative is to invest the money in a money market earning a certain return. The problem has several uncertainties, and the ones determined to be of significance are price (P), hours flown (H), capacity (C), and operational cost (O). The owner is risk neutral and will make the decision based on comparing the expected profit to the risk-free return of the money market. In addition to the uncertainties, the owner uses the following parameters in the profit calculation: Parameter Value Description CR 0.5 Charter ratio PF 40% Percentage financed I 11.5% Risk-free interest rate PU $87, 500 Purchase price IN 20,000 Insurance cost CP P3.25· Charter price N 5 Number of seats The true distributions for the uncertainties are: Uncertainty Distribution Parameters Range P Beta = =9, 15 [$81.94, $133.96] H Beta = =9, 15 [66.91, 1, 136.26] C Beta = =9, 15 [0, 1] O Normal = =µ245, 11.72 ( , ) The formulas for revenue, costs and profits are: = + +Cost H O IN PU PF I· · · (19) = +Revenue CR H CP CR H C N P· · (1 )· · · · (20) =Profit Revenue Cost (21) Furthermore, the uncertainties are have a Spearman rank correlation with each other: Spearman correlation Uncertainty P H C O P 1 H 0.5 1 C 0.25 0.5 1 O 0 0 0.25 1 In order to calculate the expected mean for this problem we use the simulation methods descibed in [11] to generate values for the uncertainties. For the discretizations, we generate the correlated uniform values from the percentile discretizations which we then use to generate the Pearson rank correlated uncertainty values. When using rank correlation, the values we get for each percentile for an uncertainty, say H, will be different when we use different percentiles for P. A3. Discretization values We present selected joint and independent discretization values for the Wildcatter problem and then Eagle Airlines. The shortcut names are the same as used in the article. The optimized discretization names are coded. The first code is either “NLP” or “MIP”. Independent discretizations are solved with a non-linear programming solver and hence have the code “NLP”. Joint discretizations are solved with a mixed-integer programming solver and hence have the code “MIP”. The next code is either a zero or a one. Zero indicates the discretization is solved to minimize the average error. A one indicates the discretization is solved to minimize the worst case error. The next three numbers are optional. These numbers indicate whether specific percentiles are used. For example, “5_50_95” indicates the 5th, 50th, and 95th were the only percentiles allowed in the discretization. The final code indicates how many sample decision problems are used to create the discretization. Wildcatter Independent Discretizations Percentiles Probabilities Discretization Uncertainty Q1 Q2 Q3 P1 P2 P3 MCS Reservoir 0.10 0.50 0.90 0.25 0.50 0.25 MCS Recovery 0.10 0.50 0.90 0.25 0.50 0.25 MCS Price 0.10 0.50 0.90 0.25 0.50 0.25 MCS Cost 0.10 0.50 0.90 0.25 0.50 0.25 J. Woodruff, N.B. Dimitrov Operations Research Perspectives 5 (2018) 288–305 302 ESM Reservoir 0.10 0.50 0.90 0.30 0.40 0.30 ESM Recovery 0.10 0.50 0.90 0.30 0.40 0.30 ESM Price 0.10 0.50 0.90 0.30 0.40 0.30 ESM Cost 0.10 0.50 0.90 0.30 0.40 0.30 HB Reservoir 0.04 0.50 0.96 0.16 0.67 0.16 HB Recovery 0.05 0.50 0.95 0.18 0.63 0.18 HB Price 0.04 0.50 0.96 0.16 0.67 0.16 HB Cost 0.04 0.50 0.96 0.16 0.67 0.16 EPT Reservoir 0.05 0.50 0.95 0.18 0.63 0.18 EPT Recovery 0.05 0.50 0.95 0.18 0.63 0.18 EPT Price 0.05 0.50 0.95 0.18 0.63 0.18 EPT Cost 0.05 0.50 0.95 0.18 0.63 0.18 NLP_1.0_5_50_95_all Reservoir 0.05 0.50 0.95 0.25 0.61 0.14 NLP_1.0_5_50_95_all Recovery 0.05 0.50 0.95 0.27 0.50 0.23 NLP_1.0_5_50_95_all Price 0.05 0.50 0.95 0.03 0.86 0.11 NLP_1.0_5_50_95_all Cost 0.05 0.50 0.95 0.21 0.63 0.16 Wildcatter Independent Discretizations Percentiles Probabilities Discretization Uncertainty Q1 Q2 Q3 P1 P2 P3 NLP_0.0_5_50_95_all Reservoir 0.05 0.50 0.95 0.15 0.66 0.18 NLP_0.0_5_50_95_all Recovery 0.05 0.50 0.95 0.20 0.60 0.20 NLP_0.0_5_50_95_all Price 0.05 0.50 0.95 0.19 0.63 0.18 NLP_0.0_5_50_95_all Cost 0.05 0.50 0.95 0.19 0.62 0.19 NLP_0.0_10_50_90_all Reservoir 0.10 0.50 0.90 0.35 0.34 0.31 NLP_0.0_10_50_90_all Recovery 0.10 0.50 0.90 0.27 0.46 0.27 NLP_1.0_10_50_90_all Price 0.10 0.50 0.90 0.16 0.62 0.22 NLP_1.0_10_50_90_all Cost 0.10 0.50 0.90 0.35 0.41 0.24 NLP_0.0_all Reservoir 0.05 0.50 0.95 0.20 0.61 0.19 NLP_0.0_all Recovery 0.10 0.55 0.95 0.26 0.57 0.17 NLP_0.0_all Price 0.05 0.50 0.95 0.19 0.63 0.18 NLP_0.0_all Cost 0.10 0.55 0.95 0.29 0.53 0.18 NLP_1.0_all Reservoir 0.05 0.55 0.90 0.22 0.52 0.27 NLP_1.0_all Recovery 0.05 0.55 0.90 0.32 0.33 0.36 NLP_1.0_all Price 0.05 0.50 0.90 0.25 0.44 0.31 NLP_1.0_all Cost 0.10 0.55 0.95 0.28 0.52 0.19 Wildcatter Independent Discretizations Percentiles Probabilities Discretization Uncertainty Q1 Q2 Q3 P1 P2 P3 NLP_0.0_66 Reservoir 0.05 0.45 0.95 0.13 0.67 0.20 NLP_0.0_66 Recovery 0.05 0.50 0.95 0.20 0.61 0.19 NLP_0.0_66 Price 0.05 0.50 0.95 0.18 0.64 0.18 NLP_0.0_66 Cost 0.05 0.50 0.95 0.19 0.62 0.19 NLP_0.0_328 Reservoir 0.10 0.50 0.95 0.18 0.64 0.19 NLP_0.0_328 Recovery 0.05 0.50 0.95 0.20 0.60 0.20 NLP_0.0_328 Price 0.05 0.50 0.95 0.18 0.64 0.18 NLP_0.0_328 Cost 0.05 0.45 0.95 0.15 0.64 0.20 NLP_0.0_656 Reservoir 0.10 0.50 0.95 0.20 0.61 0.19 NLP_0.0_656 Recovery 0.05 0.50 0.95 0.20 0.61 0.19 NLP_0.0_656 Price 0.05 0.50 0.95 0.18 0.64 0.18 NLP_0.0_656 Cost 0.10 0.55 0.95 0.29 0.52 0.19 NLP_0.0_1312 Reservoir 0.10 0.55 0.95 0.24 0.59 0.18 NLP_0.0_1312 Recovery 0.05 0.50 0.95 0.20 0.60 0.20 NLP_0.0_1312 Price 0.05 0.50 0.95 0.19 0.63 0.18 NLP_0.0_1312 Cost 0.10 0.55 0.95 0.29 0.53 0.18 NLP_0.0_all Reservoir 0.05 0.50 0.95 0.20 0.61 0.19 NLP_0.0_all Recovery 0.10 0.55 0.95 0.26 0.57 0.17 NLP_0.0_all Price 0.05 0.50 0.95 0.19 0.63 0.18 NLP_0.0_all Cost 0.10 0.55 0.95 0.29 0.53 0.18 Wildcatter Joint Discretizations Reservoir Quantile Recovery Quantile Price Quantile Cost Quantile MIP_1.0_10_50_90_all MIP_0.0_10_50_90_all 0.10 0.10 0.90 0.90 0.088 0.10 0.50 0.10 0.10 0.086 J. Woodruff, N.B. Dimitrov Operations Research Perspectives 5 (2018) 288–305 303 0.10 0.50 0.50 0.10 0.133 0.026 0.10 0.50 0.50 0.90 0.047 0.10 0.50 0.90 0.10 0.156 0.10 0.90 0.50 0.50 0.065 0.10 0.90 0.90 0.10 0.049 0.10 0.90 0.90 0.50 0.023 0.50 0.10 0.10 0.90 0.135 0.50 0.10 0.50 0.90 0.254 0.109 0.50 0.50 0.50 0.10 0.048 0.50 0.50 0.50 0.50 0.052 0.50 0.50 0.90 0.50 0.051 0.50 0.90 0.50 0.90 0.030 0.50 0.90 0.90 0.50 0.045 0.022 0.90 0.10 0.10 0.50 0.031 0.90 0.10 0.10 0.90 0.001 0.90 0.10 0.50 0.50 0.057 0.90 0.50 0.10 0.10 0.057 0.102 0.90 0.90 0.10 0.50 0.061 0.90 0.90 0.50 0.50 0.172 0.90 0.90 0.90 0.50 0.100 Wildcatter Joint Discretizations Reservoir Quantile Recovery Quantile Price Quantile Cost Quantile MIP_1.0_all 0.10 0.05 0.55 0.90 0.033 0.10 0.50 0.55 0.45 0.064 0.10 0.50 0.55 0.90 0.176 0.10 0.95 0.55 0.45 0.060 0.45 0.05 0.55 0.45 0.040 0.45 0.05 0.55 0.90 0.100 0.45 0.50 0.05 0.10 0.147 0.45 0.50 0.55 0.10 0.035 0.45 0.50 0.55 0.45 0.158 0.45 0.95 0.55 0.45 0.006 0.90 0.05 0.55 0.10 0.008 0.90 0.05 0.55 0.45 0.047 0.90 0.95 0.95 0.10 0.111 0.90 0.95 0.95 0.45 0.016 The next joint discretization is for the average error minimization. The number of non-zero percentile combinations is much larger here, which gives a more robust answer when computing out-of-sample percent errors. Wildcatter Joint Discretizations Reservoir Quantile Recovery Quantile Price Quantile Cost Quantile MIP_0.0_all 0.05 0.10 0.05 0.10 0.045 0.05 0.10 0.50 0.10 0.010 0.05 0.10 0.50 0.55 0.018 0.05 0.55 0.05 0.10 0.001 0.05 0.55 0.05 0.95 0.013 0.05 0.55 0.50 0.10 0.049 0.05 0.55 0.50 0.95 0.035 0.05 0.55 0.95 0.10 0.004 0.05 0.90 0.05 0.55 0.013 0.05 0.90 0.05 0.95 0.001 0.05 0.90 0.95 0.10 0.026 0.50 0.10 0.05 0.95 0.020 0.50 0.10 0.50 0.55 0.111 0.50 0.55 0.50 0.10 0.046 0.50 0.55 0.50 0.55 0.210 0.50 0.55 0.50 0.95 0.060 0.50 0.55 0.95 0.55 0.007 0.50 0.90 0.50 0.10 0.060 0.50 0.90 0.95 0.55 0.047 0.50 0.90 0.95 0.95 0.028 0.95 0.10 0.05 0.95 0.007 0.95 0.10 0.50 0.55 0.028 0.95 0.10 0.95 0.10 0.060 J. Woodruff, N.B. Dimitrov Operations Research Perspectives 5 (2018) 288–305 304 0.95 0.55 0.50 0.55 0.002 0.95 0.55 0.50 0.95 0.015 0.95 0.55 0.95 0.95 0.003 0.95 0.90 0.05 0.55 0.074 0.95 0.90 0.50 0.10 0.000 0.95 0.90 0.50 0.55 0.001 0.95 0.90 0.50 0.95 0.004 Eagle Airlines Independent Discretizations Percentiles Probabilities Discretization Uncertainty Q1 Q2 Q3 P1 P2 P3 NLP_0.0_5_50_95_all Price 0.05 0.50 0.95 0.186 0.629 0.185 NLP_0.0_5_50_95_all Hours 0.05 0.50 0.95 0.184 0.632 0.184 NLP_0.0_5_50_95_all Capacity 0.05 0.50 0.95 0.190 0.620 0.190 NLP_0.0_5_50_95_all Operating Cost 0.05 0.50 0.95 0.185 0.630 0.185 NLP_0.0_10_50_90_all Price 0.10 0.50 0.90 0.308 0.386 0.306 NLP_0.0_10_50_90_all Hours 0.10 0.50 0.90 0.293 0.415 0.292 NLP_0.0_10_50_90_all Capacity 0.10 0.50 0.90 0.270 0.461 0.269 NLP_0.0_10_50_90_all Operating Cost 0.10 0.50 0.90 0.318 0.373 0.309 Supplementary material Supplementary material associated with this article can be found, in the online version, at 10.1016/j.orp.2018.09.002. References [1] Alpert A, Raiffa H. Judgement under uncertainty: heuristics and biases. A progress report on the training of probability assessors. Cambridge, England: Cambridge University Press; 1982. p. 294–305. [2] Bickel JE, Lake LW, Lehman J, et al. Discretization, simulation, and Swanson’s (inaccurate) mean. Soc Petrol Eng Econ Manage 2011;3(03):128–40. [3] Brockett PL, Kahane Y. Risk, return, skewness and preference. Manage Sci 1992;38(6):851–66. [4] Clemen RT. Making hard decisions: an introduction to decision analysis. 2nd ed. Duxbury Press; 1996. [5] Clemen RT, Reilly T. Correlations and copulas for decision and risk analysis. Manage Sci 1999;45(2):208–24. [6] Dantzig GB, Wolfe P. Decomposition principle for linear programs. Oper Res 1960;8(1):101–11. [7] Dupačová J, Gröwe-Kuska N, Römisch W. Scenario reduction in stochastic programming. Math Program 2003;95(3):493–511. [8] Hammond RK, Bickel JE. Reexamining discrete approximations to continuous distributions. Decis Anal 2013;10(1):6–25. [9] Hammond RK, Bickel JE. Approximating continuous probability distributions using the 10th, 50th, and 90th percentiles. Eng Econ 2013;58(3):189–208. [10] Hurst A, Brown G, Swanson R. Swanson’s 30-40-30 rule. Am Assoc Pet Geol Bull 2000;84(12):1883–91. [11] Iman RL, Conover W-J. A distribution-free approach to inducing rank correlation among input variables. Commun Stat-Simul Comput 1982;11(3):311–34. [12] Keefer DL. Certainty equivalents for three-point discrete-distribution approximations. Manage Sci 1994;40(6):760–73. [13] Keefer DL, Bodily SE. Three-point approximations for continuous random variables. Manage Sci 1983;29(5):595–609. [14] McKay M, Beckman R, Conover W. A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 1979;21(2):239–45. [15] McNamee P, Celona J. Decision analysis with supertree. San Francisco, CA: Scientific Press; 1990. [16] Miller III AC, Rice TR. Discrete approximations of probability distributions. Manage Sci 1983;29(3):352–62. [17] Montiel LV, Bickel JE. A simulation-based approach to decision making with partial information. Decis Anal 2012;9(4):329–47. [18] Reilly T. Sensitivity analysis for dependent variables. Decis Sci 2000;31(3):551–72. [19] Smith JE. Moment methods for decision analysis. Manage Sci 1993;39(3):340–58. [20] Wallace SW, Hoyland K. Generating scenario trees for multistage decision problems. Manage Sci 2001;47(2):295–307. J. Woodruff, N.B. Dimitrov Operations Research Perspectives 5 (2018) 288–305 305