## Services on Demand

## Article

## Indicators

## Related links

- Cited by Google
- Similars in Google

## Share

## Water SA

##
*On-line version* ISSN 0378-4738

### Water SA vol.41 n.1 Pretoria Jan. 2015

#### http://dx.doi.org/10.4314/wsa.v41i1.13

**Reducing uncertainty based on model fitness: Application to a reservoir model**

**Anna C Linhoss ^{I, *}; Rafael Muñoz-Carpena^{II}; Gregory Kiker^{II}; Piotr Wolski^{III}**

^{I}Agricultural and Biological Engineering Department, Mississippi State University, 130 Creelman Drive, Starkville, MS 39762, USA

^{II}Hydrology and Water Quality, Agricultural and Biological Engineering Department, University of Florida, 287 Frazier Rogers Hall, Gainesville, FL 32611-0570, USA

^{III}Okavango Research Institute, University of Botswana, Private Bag 285, Maun, Botswana

**ABSTRACT**

Sensitivity and uncertainty analysis are important tools in the modelling process: they assign confidence to model results, can aid in focusing monitoring and preservation efforts, and can be used in model simplification. A weakness of global sensitivity and uncertainty analysis methodologies is the often subjective definition of prior parameter probability distributions, especially in data-poor areas. We apply Monte Carlo filtering in conjunction with quantitative variance-based global sensitivity and uncertainty analysis techniques to address this weakness and define parameter probability distributions in the absence of measured data. This general methodology is applied to a reservoir model of the Okavango Delta, Botswana. In addition to providing a methodology for setting prior parameter distributions, results show that the use of Monte Carlo filtering reduces model uncertainty and produces simulations that better represent the calibrated ranges. Thus, Monte Carlo filtering increases the accuracy and precision of parametric model uncertainty. Results also show that the most important parameters in our model are the volume thresholds, the reservoir area/volume coefficient, floodplain porosity, and the island extinction coefficient. The reservoir representing the central part of the wetland, where flood waters separate into several independent distributaries, is a keystone area within the model. These results identify critical areas and parameters for monitoring and managing, refine and reduce input/output uncertainty, and present a transferable methodology for developing parameter probability distribution functions, especially when using empirical models in data-scarce areas.

**Keywords:** sensitivity analysis, uncertainty analysis, Monte Carlo filtering, reservoir model, Okavango Delta

**INTRODUCTION**

Global sensitivity and uncertainty analysis (GSA/UA) systematically and quantitatively investigates input/output uncertainties to assess a model's reliability (Scott, 1996; Saltelli et al., 2008) and can be used to assign confidence to model results. Global uncertainty analysis (GUA) quantifies total model uncertainty and global sensitivity analysis (GSA) apportions that uncertainty to each of the parameters. While these are two separate methods with different objectives, the overlap between them is important. GUA is useful for quantifying model reliability based on the uncertainty of the input parameters. GSA is useful for understanding which of those parameters are responsible for driving that uncertainty.

According to Beven (2006), ignoring model uncertainty undermines the value of a model for its use in the decision-making process. Understanding the uncertainty and sensitivity associated with a model is useful in many contexts for both modellers and managers. It allows decision makers to acknowledge the reliability of models when assessing forecasts and weighing risks between decisions (Saltelli et al., 2008). Identifying important model parameters facilitates strategic data collection for parameter refinement and reduced model uncertainty. Additionally, identifying important areas, dynam ics, or parameters in a model of natural systems allows managers to focus preservation efforts on those aspects. Recognising unimportant parameters can allow model simplification which reduces over-parameterisation. Furthermore, GSA can provide an understanding of the importance of parameter interactions In a complex model, these interactions may not be obvious and may have significant impacts on the model output due to their non-additive nature (Saltelli et al., 2008). For all of these reasons, GSA/UA is a crucial step in the modelling process.

There is a large body of work regarding various techniques for assessing model sensitivity and uncertainty. Reviews of these methods include (but are not limited to) Cacuci and Ionescu-Bujor (2004), Matott et al. (2009), Saltelli et al. (2005), and Saltelli et al. (2008). SA methods can be categorised as local or global and global methods can be based on regression, correlation, parameter bounding, and variance decomposition (Matott et al., 2009). Methods based on correlation and regression use graphics or statistics to assign sensitivity. Parameter bounding is an inverse method that maps the parameter value: producing behavioural results. In variance-based techniques, the output variance is defined as the sum of the variances assigned to each input and the interactions between the inputs.

Values for model parameters are inherently uncertain because of the limitations in data collection, our understanding of a system, and simplifications inherent to the modelling process. In a GSA/UA, this uncertainty is accounted for by using probability distribution functions (PDFs) which represent the range and frequency of possible values for a parameter. PDFs are generally set according to experimental data, literature values, expert knowledge, or some fixed interval. One of the biggest criticisms of Monte Carlo based GSA/UA is the subjective selection of parameter PDFs (Ivanovi and Freer, 2009; Pappenberger and Beven, 2006). For example, setting all parameter PDFs in a model to ±20% of their calibrated value ignores the physical range of those parameters. However, setting parameter PDFs based on just a few data points may not reflect the true range of the parameter either. This is especially problematic in data-poor areas where experimental data, literature values, and expert knowledge regarding the ranges for parameters are lacking. It is also an issue in empirical models where parameters cannot be measured directly.

To address this problem of the subjective selection of parameter PDFs when conducting a GSA/UA, we employ Monte Carlo filtering (MCF) in conjunction with GSA/UA to refine prior PDFs. In MCF, model simulations are accepted or rejected based on predefined criteria for acceptable model performance (Rose et al., 1991; Salteli et al., 2008). MCF has been shown to reduce the variance in model outputs and to be useful in calibration (Rose et al., 1991). Here, we use MCF to redefine prior PDFs based on the goodness of the model fit (Linhoss et al., 2012). Redefining prior PDF's based on the MCF results represents a novel method for reducing model uncertainty in a reservoir model. This process accomplishes 3 goals:

- it presents an objective method for defining parameter PDFs,
- it reduces parameter based uncertainty, and
- it forces the model to represent more realistic results in the uncertainty analysis (UA).

We applied our MCF-GSA/UA method framework to a reservoir model of the Okavango Delta (Wolski et al., 2006), located in north-western Botswana (Fig. 1). The Okavango Delta ('the Delta') is a large, biologically diverse, and economically important wetland. Threats to the hydrology and ecology of the Delta include climate change and development. Hydrologic modelling of the Delta is characterised by large uncertainties because of its size, remote location, flat topography, and complex ecohydrology. The Okavango Research Institute (ORI) model is a reservoir model developed specifically for the Delta's hydrology, which has been applied in a number of recent studies (Wolski et al, 2006; Wolski and Murray-Hudson, 2008; Wolski, 2009). The objectives of this research were to:

- identify important parameters within the model by applying GSA,
- reduce model complexity by setting unimportant parameters to constants, and
- objectively define parameter probability distributions in the absence of measured data using MCF-GUA.

Parameters are defined as non-time series inputs in the model. The GSA/UA used a framework in which the modified Morris GSA (Morris, 1991; Campolongo et al., 2007) was first used to qualitatively screen the least important parameters. Based on the results from the modified Morris method, the least important parameters were set to constants and the most important parameters were further examined using the extended Fourier Amplitude Sensitivity Test (FAST) GSA/UA technique (Cukier et al, 1978; Koda et al, 1979; Saltelli et al, 1999), which quantifies and apportions variance-based uncertainty. MCF (Saltelli et al., 2008) was then used to objectively redefine PDFs and refine model uncertainty. Through this methodology the uncertainty of the model was quantified, the accuracy and precision of the model was improved, and a deeper understanding of the model's internal dynamics and reliability was gained.

**METHODS**

**Study site**

The Okavango Basin is a large (530 000 km^{2}) transbound-ary watershed located in southern Africa and shared between Angola, Namibia, and Botswana (Fig. 1). Within the basin, the Okavango River feeds the Okavango Delta, which is a large inland delta whose alluvial fan spreads out on the edge of the Kalahari Desert. This inland delta and its hydrology are particularly important and unique because it supports a diversity of wildlife and people who otherwise exist in an extremely arid environment (Kgathi et al., 2006).

Modelling the hydrology of the Okavango Delta is especially problematic because of its large size, remote location, lack of data, complex hydrology, and shallow topography. The system is large. The entire geologic alluvial fan encompasses 40 000 km^{2} (Gumbricht et al., 2005) and flooding extents range from 6 000-12 000 km^{2} (Wolski et al., 2006). No roads traverse the area and the field collection of data is extremely cumbersome. The area also is extremely smooth and flat. Maximum local relief is generally less than 2 or 3 m (McCarthy et al., 2003; Gumbricht et al., 2001). As a result, like many wetlands, flow direction and inundation area is determined by very slight differences in elevation. Furthermore, variations in flow are caused by poorly understood and hard to model processes such as hippopotami *(Hippopotamus amphibius) *movements, sedimentation, vegetation, peat accumulation, and fire. Overall, data scarcity is a significant issue in this area.

**The Okavango Research Institute (ORI) model**

Because of these data limitations, a reservoir modelling approach has often been used to simulate flows in the Delta (Dincer et al., 1987; SMEC, 1990; Scudder et al., 1993; Gieske, 1997; WTC, 1997; Wolski et al., 2006). A review of these models can be found in Kiker et al. (2008) and Wolski et al. (2006). The ORI model is the newest reservoir model of the Okavango Delta (Wolski et al., 2006) (Figs 2 and 3). The ORI model has been calibrated and tested (Wolski et al., 2006), and subsequently used by policy makers in the area (DEA, 2006). The model has also been used to predict the impact of climate change scenarios (Wolski, 2009). However, no formal uncertainty nor sensitivity analysis has been conducted on the model.

The ORI model represents flood duration, flood frequency, and flooding extents in the Okavango Delta, and outflow from it via the Boro River at Maun. A complete description of the ORI model, its parameters and calibration can be found in Wolski et al. (2006). The model operates on a monthly time step. Flow from the river is input at the top of the Delta at Mohembo from measured or modelled data (Hughes et al., 2006, Wolski 2009). The volume of water in each reservoir is then computed by the continuity equation (Eq. (1)) where *V *is the volume of water, *I *is inflow, *ET *is evapotranspiration, *P *is precipitation, *Q *is outflow, and *Q _{jnf}*is infiltration to ground- water (Wolski et al., 2006).

The inundated area in each reservoir is calculated from water volume through a power relationship (Eq. (2)) where *A *is area, *V *is volume and *n *and *b *are coefficients.

There are 16 linked reservoirs in the model but 6 of these reservoirs act only to receive waters. Model parameters for each reservoir include area, topography, evaporation, and flow parameters (Table 1 and Fig. 2). Some of these parameters are constant throughout the model (extinction depths, porosities, and *n *in the area/volume relationship), some vary between the reservoirs *(b *in the area/volume relationship, delay, rainfall ratio parameter, and groundwater reservoir areas), and some vary between the reservoir connections (flow resistance and volume threshold). Outputs from the reservoir model include monthly inundation area and outflow from the Boro distributary.

The area/volume relationship parameters for each of the reservoirs *(n *and *b _{i}) *were calibrated based on a 30-m digital elevation map (DEM) developed from a sparse network of gravimetric measurements and land-cover map (Gumbricht et al., 2005). Flows between reservoirs are expressed though volume thresholds (v

_{a}), where

*a*is the upstream reservoir and

*z*is the downstream reservoir. Upstream reservoirs may have several outlets that can feed more than one downstream reservoir. The volume threshold parameters (v

_{az}), were empirically derived and used to calibrate the model (Wolski et al., 2006). The reservoir time constant coefficient

*(k*) is equivalent to surface flow resistance. In addition to the 16 reservoirs, parameters for

*k*and

_{az}*v*are given for 2 double links (Fig. 3) and 2 dummy reservoirs. The delay parameters

_{az}*(delay)*are empirically-derived discrete on/off switches that also slow the water flow.

Each reservoir is divided into floodplain area *(fa _{i}) *and island area

*(ia*so that groundwater flow between the two areas can be represented under floods of varying size. Soil porosity is represented as 2 parameters in the model: island (ipor) and floodplain (spor) soil porosity. Groundwater flow and infiltration processes are represented by a series of sub-reservoirs including 5 floodplain groundwater reservoirs, and 5 island groundwater reservoirs within each surface reservoir computational unit. Water in the surface reservoir infiltrates into the floodplain groundwater reservoirs which then flows to the island groundwater reservoirs.

_{i})Actual evaporation is based on potential evapotranspiration. The island and floodplain extinction depths *(idet *and *fdet) *empirically represent rooting depth and simulate a linear decrease in the rate of evapotranspiration with depth from the surface. The model uses the Penman-Monteith equation (Allen et al., 1998) to calculate the reference crop transpiration. These calculations are adjusted based on weather station measurements made inside the Delta and also with measurements from an eddy covariance system located at Maun and Nxaraga (Wolski et al., 2006). Rainfall is input over the inundated areas of the Delta based on an inverse distance-weighted relationship between 2 weather stations in the area. The rainfall ratio parameter *(m _{i}) *is used to vary the rainfall in the reservoirs based on this relationship.

The ORI model links the concept of the reservoir model, as described above, to a GIS model to simulate the spatial distribution of the flood (Wolski et al., 2006). The GIS model is based on satellite imagery of flooding extents. A Gaussian PDF was assigned to each pixel describing the likelihood of the pixel being inundated given the total extent of inundation. Then, given a volume of water in the reservoir model each pixel is assigned an inundation/non-inundation status based on the more likely probability.

The hydrological model was run from 1967 to 2003 with a calibration period between 1968 and 1986. It was calibrated manually through trial-and-error adjustments of the parameters (Wolski et al., 2006). The objective functions for calibration were the inundation extents and outflow from the Boro River at Maun. The estimation of inundation extents was derived from satellite imagery available from McCarthy et al. (2003). Outflow from the Boro River was gauged in the river.

Results from the ORI calibration show good correlation with observed data. The model produced a monthly inundation area that compares to observed data with a root mean squared error (RMSE) of 528 km^{2} and a correlation coefficient of 0.90 for the entire Delta (Wolski et al., 2006). Additionally, the outflow at Maun showed a RMSE of 11.8 M m^{3}/month and a correlation coefficient of 0.91 when compared to observed flows (Wolski et al., 2006).

The objective function for our study was defined as the Nash-Sutcliffe coefficient of efficiency (NSE) (Nash and Sutcliffe, 1970), which was applied to quantify the degree of matching between the monthly inundated area of the entire Delta that was obtained in the original calibration model runs (Wolski et al., 2006) and our uncertainty analysis runs. The inundation area is used as the objective function because, in the absence of good topographic data, it serves as a proxy for volume, and is an important social and ecological factor with regards to floods and droughts.

**Global sensitivity and uncertainty analysis**

There are a number of different approaches available for conducting a sensitivity analysis on models ranging from simpler one-at-a time (OAT) methods to advanced global techniques (Saltelli et al., 2008, Cacuci et al., 2003). With OAT techniques the variation of the model output is investigated by changing a single model parameter at a time, hence the name 'local' sensitivity. This traditional sensitivity analysis method is limited since it usually explores a limited parametric range (central value +/- some perturbation percentage) and does not account for interactions between parameters (Saltelli et al., 2008). When model responses to parameter perturbations are non-additive and non-linear, as with many complex models, simple OAT techniques are not appropriate. Instead, global methods are preferred because they simultaneously vary the parameters all at once, and explore the entire parametric space (as described by the parameter PDGs), and are thus more appropriate. Our study uses an evaluation framework based on two such modern global techniques: a qualitative screening method (Morris, 1991) and a quantitative variance-based method (Saltelli et al., 1999). This 2-step process has been used in the examination of input/output relationships in several environmental investigations (e.g. Munoz-Carpena et al., 2007; Chu-Agor et al., 2011; Linhoss et al., 2013). The screening method allows for an initial reduction in the number of parameters to use in the more computationally intensive quantitative FAST GSA/UA.

*Modified Morris method*

The GSA/UA of the ORI model began with a qualitative, less computationally intensive, screening assessment using the modified Morris method (Morris, 1991; Campolongo et al., 2007). When performing the modified Morris method, the model is run for *p(k+1) *iterations, where *k *is the number of parameters and *p *is the number of levels within the parameter PDFs (i.e. resolution). Thus, the region of experimentation, , is described by a hypergrid of parameter sets with *k *dimensions and *p *levels. Equation (3) describes the Morris method 'elementary effect' *(EE _{i}) *on the output of interest (y) from varying the

*ith*parameter

*(x*by , for a given set of parameter values (vector X), where

_{i})*is a value in {0,*

*1/(p-1)...,1-1/(p-1),...,1}*and

*X=(x*is a sample from

_{1}, x_{2},..., x_{k})Morris proposed 2 sensitivity indices, which are plotted together to provide a visual indication of relative parameter importance:

•is the mean of theEEand represents the direct (or first-order) effect of the parameter on the output. This mean magnitude of the effect also indicates the degree to which uncertainty associated with the parameter will propagate through the model and onto the output. This information therefore proves valuable for uncertainty reduction._{i}

•σ is the standard deviation of EEi, which estimates the variation in the elementary effects, and therefore indicates the extent to which parameter perturbation effects depend on the values of other parameters (interaction effects) or produce nonlinear output effects. The modified Morris method (Campolongo et al., 2007) has a number of improvements over the original method. It allows for an analysis of models with multiple outputs, allows factors to be grouped, and has a more effective sampling strategy at no additional computation cost. Furthermore, the enhanced sensitivity index μ* calculates the mean of the absolute value of the elementary effects and is approximately as good as indices which are based on variance methods (Campolongo et al., 2007).

*Extended FAST*

Once the most important parameters were identified using the modified Morris method, a quantitative FAST analysis (Cukier et al., 1978; Koda et al., 1979) was conducted on the subset of these most important parameters. FAST uses Fourier analysis to decompose the variance of model outputs into first-order variances for each parameter. In the method, *N *design points are chosen within a search curve of the *k *parameters' input space, thus creating *N *input sets. The curves are constructed to explore each parameter with a different sampling frequency The model is run at each design point. The outputs are then reordered so that the design points are sorted in increasing order according to the magnitude of parameter *x _{i}*.The Fourier spectrum is computed on the output which gives the importance of the parameter

*x*

_{i}.FAST defines the first-order sensitivity index, *S _{i}, *as the fraction of the model output variance that can be attributed to the th parameter. In a model that is perfectly additive (i.e. no parameter interactions) the summation of the variances for each of the parameters will be unity; S

_{Ti}= 1. In models where there are interactions this sum will be greater than one. In this work, we apply an enhanced version of FAST, the extended FAST technique (Saltelli et al., 1999), which allows for the additional quantification of higher order variance (interactions) for a given parameter (

*i*) using the total sensitivity index, S

_{Ti}. For example, the total sensitivity for parameter

*1*in a model with

*n*parameters is defined as:

From Eq. (4) interactions can be computed from *S _{T1}-S_{V}*One limitation of this variance-based technique is that it cannot be used on parameters that are interdependent (Crosetto and Tarantola, 2001). When performing the extended FAST GSA/UA, a model is run for M(2k+2) iterations, where

*M*is a number between 100's and 1 000's and

*k*is the number of parameters (Saltelli et al., 1999).

*Prior parameter PDFs*

Both the modified Morris method and extended FAST method rely upon creating model input sets by sampling values from PDFs for each parameter and running the model iteratively using these input sets. A careful selection of the prior PDFs for each parameter is especially important because these ranges directly affect the magnitude of uncertainty and the importance of the parameters. According to Munoz-Carpena et al. (2007), when the data for the parameters show no apparent distribution such as normal or triangular, a uniform PDF allows for equal probability of selection across the defined range. Thus, because of the lack of data available in the Okavango Delta, conservative uniform distributions of ±20% of the calibrated value were used as the default prior distribution type throughout. While ±20% is an arbitrary cut-off, we show in the results that this range of values is sufficient because the output frequency histograms that were used to develop the refined PDFs generally display normal distributions whose tails can be inferred (see Fig. 5). SimLab v3.2.6 (SimLab, 2011) software was used to sample parameter PDFs, and post-process the model results according to the modified Morris and FAST methods.

*Monte Carlo filtering*

Using the results from the FAST GSA, Monte Carlo filtering (MCF) (Saltelli et al., 2008; Rose et al., 1991) was conducted to objectively reduce and refine input/output model uncertainty. For MCF, a threshold in the model results was used to designate the FAST model results as either behavioural or non-behavioural. Outputs that exceed the threshold are considered 'good' or 'desirable' and are classified as behavioural (B), and outputs that do not meet the threshold are considered 'undesirable' and are classified as non-behavioural (). For this work, the threshold was chosen based on the NSE between the FAST model runs and the calibrated model's total monthly inundated area. Behavioural simulations were defined as simulations where the NSE was __>__ 0. Non-behavioural simulations were defined as simulations that performed below 0. Zero was used as a threshold because NSE values above 0 indicate that the model performs better than the mean of the observed data.

Two subsets of the parameter sets, x_{i}, which were used to produce each output, are then defined as *x. _{i-B}*and

*x*. This assigns 2 new PDFs for each parameter, one PDF based on the behavioural outputs (m),

_{i-B}*f*and one based on the non-behavioural outputs (n),

_{m}(x_{i-B})*f*To determine if

_{n}(x_{jt}).*f*is statistically different from

_{m}(x_{i-B})*f*2-sided Smirnov test was used on each parameter independently. The null hypothesis (H) states that the distribution of the values for parameter

_{n}(x_{i-b}) the*x.*producing

*B*is equal to the distribution of parameter values producing . If these two parameter distributions are shown to be significantly different, then

*x.*is considered an important parameter in defining the behaviour of the model (Saltelli et al., 2008).

In this work, if the two parameter distributions *f _{m}(x_{i-B}) *and

*f*shown to be significantly different, then the prior distribution for

_{n}(x_{i-B}) are*x.*is redefined based on

*f*Normal distributions were curve fit to the behavioural parameter distributions that were significantly different from the non-behavioural distributions and set as the new posterior PDF. The posterior PDFs were not changed for the parameters where the behavioural and non-behavioural distributions were not significantly different. The FAST GUA was then rerun based on the posterior parameter distributions.

_{m}(x_{i-B}).This method is designed to reduce the output uncertainty but does not directly prescribe model uncertainty. This is because the parameter sets that lead to undesirable model outputs are not discarded. Rather, the PDFs for each parameter are redefined based on significant differences between behavioural versus non-behavioural distributions.

**RESULTS AND DISCUSSION**

**Modified Morris method**

To screen the initial 114 parameters (Table 1), the model was run 1 150 times using the modified Morris method. The results show the Morris scatter plot (Fig. 4) where the *x*-axis represents the direct effects (µ*) and therefore ranks the direct importance of each parameter, and the y-axis represents the indirect effects (σ), and therefore ranks the role of higher-order interactions of the parameters. These higher-order interactions represent the range of the importance of a parameter, which depends on the values for the other parameters. A threshold of 0.1 on the *x*-axis was set to distinguish important from unimportant parameters. Though this threshold is arbitrary, results from the FAST analysis showed low sensitivity of the model to parameters that were included using this threshold, indicating that it is sufficiently conservative.

Because the parameters are scattered on both the a and µ* axes (Fig. 4), both direct (first-order) and indirect (higher-order) effects are important. The modified Morris results gave 9 parameters that stand out as the most important (Fig. 4):

- Island extinction coefficient (idet)
- Floodplain soil porosity
*(fpor)* - Volume thresholds for the linkages between the Nqoga1 reservoir and the reservoirs: Nqoga2(v
_{23})Thaoge (v_{24}), Xudum (v_{25}), Boro(v_{26})Manuachira1 (v_{27}), and Selinda(v_{29}) - Power area/volume coefficient for the Xudum reservoir (b
_{5})

From the strong model sensitivity to the 6 volume thresholds, the importance of this type of model parameter is apparent. Furthermore, all 6 of the important volume thresholds flow out of a single reservoir, Nqoga1 (Fig. 3). There are a total of 7 volume thresholds that flow out of Nqoga1. The one volume threshold that originates from Nqoga1 but is not highly important is linked to the Boro reservoir where there is a double link to the Nqoga1 reservoir (Fig. 3). This double link in the Boro, which splits the flow between Nqoga1 and the Boro, is unique in the Nqoga1 reservoir and may explain why this volume threshold did not demonstrate the same level of importance.

This analysis shows that Nqoga1 is a keystone reservoir and is responsible for much of the sensitivity in the model, having an important influence in determining how the predicted flood pulse is routed between the reservoirs. The identification of Nqoga1 as a keystone reservoir and the volume thresholds as important parameters in general has valuable management implications. Monitoring and management efforts should focus on the topography and hydrology of the Nqoga1 area because it is likely that physical changes here could have important impacts on how the water is apportioned downstream throughout the Delta.

**FAST and Monte Carlo filtering**

Following the modified Morris screening method the FAST GSA/UA was used to quantify the uncertainty and sensitivity associated with the most important parameters. The model was run 14 985 times, while varying the 9 most important parameters throughout their parametric space as defined by the probability distributions. All other parameters were set to constants. As with the Morris analysis, the objective function was defined as the NSE between the original calibration model runs (Wolski et al., 2006) and the uncertainty analysis runs of the monthly inundation total area of the Delta. This method was performed twice; first based on the prior PDFs and then again based on the MCF PDFs.

MCF resulted in 5 207 out of 14 985 simulations meeting the behavioural criteria. The 2-sided Smirnov test showed that the behavioural and non-behavioural distributions for *idet, fpor,v _{23} V_{24}, *V

_{25}, V

_{26}, V27, and were significantly different. Histograms show the skew of these results (Fig. 5). The behavioural distribution for

*b*was not significantly different from the non-behavioural distribution and so its posterior distribution remained unchanged. Posterior normal PDFs were assigned to the parameters that demonstrated a significant difference between the behavioural and non-behavioural distributions (Fig. 5). The FAST methodology was then rerun using the posterior MCF distributions. The results from this second MCF FAST analysis are referred to as the posterior GUA.

_{5}The comparison between the prior and posterior UA FAST runs shows that MCF reduced model uncertainty and increased the model's accuracy and precision. Within the context of this study, a confidence interval represents the distance between output values and represents the model uncertainty. According to the 95% confidence intervals, MCF reduced input/output uncertainty in all of the reservoirs as well as in the overall Delta (Table 2). Reduction of uncertainty varied between 12 and 171%. Reducing uncertainty increased the precision of the model results (i.e. the closeness of the outputs to each other). Additionally, MCF produced results that more accurately represented the calibrated ranges (i.e. the closeness of the outputs to the calibrated value). The average NSEs in all of the reservoirs, except the most upstream reservoir (i.e. the Panhandle) were pushed closer to 1 in the posterior GUA (Table 3). Improvements between the prior and posterior average NSE range from 0.0 to 12.8. Thus, MCF was shown to be a useful method for objectively constraining the prior PDFs, reducing the input/output model uncertainty, and improving the accuracy and precision of the results.

These results also reveal a range of uncertainty between the reservoirs. The reservoirs with the least amount of uncertainty are the Panhandle and Nqoga1, which is likely a result of their locations upstream from the complex linkages (Fig. 3). Thaoge and Selinda have the highest levels of uncertainty with Xudum and Nqoga2 following. These reservoirs retained relatively high levels of uncertainty even after the PDFs for their volume thresholds were constrained in the MCF.

The confidence intervals shown in Table 2 indicate that the uncertainty in the total Delta is not the summation of the uncertainty in the individual reservoirs. In fact, the width of the Delta's confidence interval ranks as the third narrowest when compared to all of the reservoirs in the prior GUA and the second narrowest in the posterior GUA. This is likely a result of the underlying mechanics of the model. The parameters in the ORI model are more focused on varying which reservoir the water is routed to, versus the total volume of water in the system. This is in part due to the fact that the vast majority of water flowing into the Delta comes from upstream inputs. This structure leads to a higher level of confidence in the model's mass balance (and the total flooding extents in the entire Delta) than in the distribution of the flood between the various reservoirs.

Like the Morris results, the FAST GSA showed that both first-order effects and higher-order effects are important in determining the model behaviour (Figs 6 and 7). The higher order effects account for 72% of the total model output variance. Also, as with the Morris analysis, the importance of the volume thresholds flowing out of the Nqoga1 reservoir continued to dominate the model's sensitivity, with v_{26} showing the highest level of importance. This reiterates that, for management purposes, the entire area represented by the Nqoga1 reservoir should be monitored for changes and protected from disturbance.

**CONCLUSIONS**

This GSA/UA of the ORI reservoir model presents a novel demonstration of a previously described generic method for objectively reducing and refining uncertainty (Linhoss et al., 2012) which is especially relevant for data-scarce areas. It also provides important insights into the hydrology of the Okavango Delta, the sensitivity and uncertainty of the ORI model, and reservoir models in general. These insights are useful for both modellers and managers in the Okavango Delta as well as for water resource researchers and practitioners in a more general sense.

Ivanovi and Freer (2009) state that it is especially difficult to objectively test the uncertainty of parameters for empirical models and that most methods proposed thus far have been subjective (e.g. Beven, 2006; Beven, 2007; Goldstein and Rougier, 2009). Pappenberger and Beven (2006) also made the point that uncertainty analyses may be too subjective to be of practical use because of the arbitrary selection of prior parameter distributions. The subjectivity of traditional uncertainty analysis techniques was directly addressed through this work. MCF was used in conjunction with GUA to objectively refine parameter PDFs based on the goodness of fit of model results. Additionally, through this method, the model outputs more closely resembled the calibrated values in both precision and accuracy. This is especially useful in data-poor areas where there is a lack of information available for defining PDFs.

This work also provides insights into the hydrology of the Okavango Delta and reservoir models in general. The GUA showed that the flooding extents in some of the individual reservoirs are more uncertain than the flooding extent of the Delta as a whole. This is because most of the dynamics within the model relate to where water is apportioned rather than how much water is in the system. The GSA highlighted the importance of volume thresholds in the ORI model and showed that the thresholds within a highly-linked reservoir were particularly important. The importance of these volume thresholds, particularly in highly-linked reservoirs, can be applied to reservoir models in general.

This work identified the most and least important model parameters, which is useful in focusing future calibration and model adjustments. Unimportant parameters, which have little impact on the model outputs, include: k_{az}, ia_{i}, and *m.. *In future modelling efforts these parameters may be ignored or set to constants in order to simplify modelling and calibration without compromising accuracy. Conversely, the parameters: *idet, fpor, v *, *v *, *v *, *v *, *v *, *v *, and *b *were found to be particularly important. Future calibration efforts should begin by adjusting these parameters within their likely bounds. In particular, Nqoga1 was identified as a keystone reservoir as volume thresholds that represent water flowing out of Nqoga1 were found to be especially important. Future efforts may focus on obtaining physically-based values for these parameters. Doing so may not only improve the model but will also result in a more physically representative model.

Managers can also use the results of this GSA/UA to develop strategic monitoring plans based on the important parameters. The Nqoga1 reservoir feeds 6 reservoirs directly, and understanding these linkages was shown to be important for understanding and modelling how water is apportioned within the Delta. The importance of the volume thresholds shows that the quality of the topographic data is an important factor in understanding the hydrology of the Delta. Acquiring topographic data that is accurate enough for physically routing water in a hydrologic model is a significant challenge because of the large, flat, and remote nature of this site. Furthermore, there is a lack of understanding what even defines topography within the Delta in the context of hydrologic modelling (i.e. mineral sediment, top of peat layer, etc.). Future monitoring efforts, which are aimed at improving model performance, should be focused on understanding the topography of the Nqoga1 area and how water is routed from this area to the downstream distributaries. Managers should also be aware that any changes in the Nqoga1 area may have important downstream consequences. Interestingly, channel aggradation through sedimentation is taking place in the Nqoga area (Wolski and Murray Hudson, 2008). Because of this study, we are now aware that these processes may have an important impact on the partitioning of water between distributaries.

In conclusion, this work has technical and management implications by presenting an objective method for refining prior parameter PDFs in a data-scarce area, reducing model uncertainty, and identifying important areas for study. As the ORI model is used and refined in the future, these insights can optimise how the model is improved. The MCF method for objectively refining prior PDFs and reducing uncertainty presented here is a transferable tool that can be used to improve the accuracy and precision of any model. This is especially useful when using empirical models in data-poor areas.

**ACKNOWLEDGEMENTS**

The authors would like to thank the AM-W3 NSF IGERT, Mark Brown, NASA (project 08-LCLUC08-2-0025), and the U.S. Department of Agriculture (Project No. 160000-010-30002700) for their financial and institutional support of this work. The authors also acknowledge the University of Florida High-Performance Computing Center for providing computational resources and support. Finally, thanks to the University of Botswana's Okavango Research Institute for all of their onsite assistance.

**REFERENCES**

ALLEN RG (1998) Crop Evapotranspiration: Guidelines for Computing Crop Water Requirements. Food and Agriculture Organization of the United Nations, Rome. [ Links ]

BEVEN K (2006) On undermining the science? *Hydrol. Process. ***20 **(14) 3141-3146. [ Links ]

BEVEN K (2007) Towards integrated environmental models of every where: uncertainty, data and modelling as a learning process. *Hydrol. Earth Sys. Sci. 11 *(1) 460-467. [ Links ]

BEVEN K and BINLEY, A (1992) The future of distributed models: model calibration and uncertainty prediction *Hydrol. Process. 6 *279-298. [ Links ]

CACUCI DG, IONESCU-BUJOR M and IONEL MN (2003) *Sensitivity and Uncertainty Analysis. *Chapman & Hall/CRC Press, Boca Raton. 285 pp. [ Links ]

CACUCI DG and IONESCU-BUJOR M (2004) A comparative review of sensitivity and uncertainty analysis of large-scale systems - II: Statistical methods. *Nucl. Sci. Eng. ***147 **(3) 204-217. [ Links ]

CAMPOLONGO F, CARIBONI J and SALTELLI A (2007) An effective screening design for sensitivity analysis of large models. *Environ. Model. Softw. 22 *(10) 1509-1518. [ Links ]

CHU-AGOR ML, MUNOZ- CARPENA R, KIKER G, EMANUEL-SSON A and LINKOV I (2011) Exploring vulnerability of coastal habitats to sea level rise through global sensitivity and uncertainty analyses. *Environ. Model. Softw. 26 *(5) 593-604. [ Links ]

CROSETTO M and TARANTOLA S (2001) Uncertainty and sensitivity analysis: tools for GIS-based model implementation. *Int. J. Geogr. Inf. Sci. 15 *(5) 415-437. [ Links ]

CUKIER RI, LEVINE HB and SHULER KE (1978) Nonlinear sensitivity analysis of multiparameter model systems. *J. Comput. Phys. 26 *(1) 1-42. [ Links ]

DEA (DEPARTMENT OF ENVIRONMENTAL AFFAIRS, BOTSWANA) (2006) Okavango Delta management plan. Okavango Delta management plan project. Department of Environmental Affairs, Gaborone, Botswana. [ Links ]

DINCER T, CHILD S and KHUPE B (1987) A simple mathematical model of a complex hydrologic system-Okavango Swamp, Botswana. *J. Hydrol. ***93 **(1-2) 41-65. [ Links ]

GIESKE A (1997) Modelling outflow from the Jao/Boro River system in the Okavango Delta, Botswana. *J. Hydrol. 193 *(1-4) 214-239. [ Links ]

GOLDSTEIN M and ROUGIER J (2009) Refined Bayesian modelling and inference for physical systems. *J. Stat. Plann. Inference ***139 **(3) 1221-1239. [ Links ]

GUMBRICHT T, McCARTHY TS and MERRY CL (2001) The topography of the Okavango Delta, Botswana, and its tectonic and sedimentological implications. *S. Afr. J. Geol. ***104 **(3) 243-264. [ Links ]

GUMBRICHT T, McCARTHY TS and BAUER P (2005) The micro-topography of the wetlands of the Okavango Delta, Botswana. *Earth Surf. Process. Landforms 30 *(1) 27-39. [ Links ]

HUGHES DA, ANDERSSON L, WILK J and SAVENIJE HHG (2006) Regional calibration of the Pitman model for the Okavango River. *J. Hydrol. ***331 **(1-2) 30-42. [ Links ]

IVANOVI RF and FREER JE (2009) Science versus politics: truth and uncertainty in predictive modelling. *Hydrol. Process. ***23 **(17) 2549-2554. [ Links ]

KGATHI DL, KNIVETON D, RINGROSE S, TURTON AR, VANDER-POST CHM, LUNDQVIST J and SEELY M (2006) The Okavango: A river supporting its people, environment and economic development. *J. Hydrol. 331 *(1-2) 3-17. [ Links ]

KIKER GA, MUNOZ-CARPENA R, WOLSKI P, CATHEY A, GAUGHAN A and KIM J (2008) Incorporating uncertainty into adaptive, transboundary water challenges: a conceptual design for the Okavango River basin. *Int. J. Risk Assess. Manage. 10 *(4) 312-338. [ Links ]

KODA M, MCRAE GJ and SEINFELD JH (1979) Automatic sensitivity analysis of kinetic mechanisms. *Int. J. Chem. Kinet. 11 *(4) 427-444. [ Links ]

LINHOSS AC, MUNOZ-CARPENA R, ALLEN M, KIKER G and MOSEPELE K (2012) A flood pulse driven fish population model for the Okavango Delta, Botswana. *Ecol. Model. ***228 **27-38. [ Links ]

LINHOSS AC, MUNOZ-CARPENA R, KIKER G and HUGHES D (2013) Uncertainty, sensitivity, and impacts for scenario modeling using the Pitman hydrologic model in the Okavango Basin. *J. Hydrol. Eng. 18 *(12) 1767-1778. [ Links ]

MATOTT LS, BABENDREIER JE and PURUCKER ST (2009) Evaluating uncertainty in integrated environmental models: A review of concepts and tools. *Water Resour. Res. 45 *(6) W06421. [ Links ]

MCCARTHY JM, GUMBRICHT T, MCCARTHY T, FROST P, WESSELS K and SEIDEL F (2003) Flooding patterns of the Okavango wetland in Botswana between 1972 and 2000. *Ambio 32 *(7) 453-457. [ Links ]

MORRIS MD (1991) Factorial sampling plans for preliminary computational experiments. *Technometrics 33 *(2) 161-174. [ Links ]

MUNOZ-CARPENA R, ZAJAC Z and KUO YM (2007) Global sensitivity and uncertainty analyses of the water quality model VFSMOD-W. *Trans. ASABE 50 *(5) 1719-1732. [ Links ]

NASH JE and SUTCLIFFE JV (1970) River flow forecasting through conceptual models part I - A discussion of principles, *J. Hydrol. 10 *(3) 282-290. [ Links ]

PAPPENBERGER F and BEVEN KJ (2006) Ignorance is bliss: Or seven reasons not to use uncertainty analysis. *Water Resour. Res. 42 *(5) W05302. [ Links ]

ROSE KA, SMITH EP, GARDNER RH, BRENKERT AL and BARTELL SM (1991) Parameter sensitivities, Monte Carlo filtering, and model forecasting under uncertainty. *J. Forecast. 10 *(1-2) 117-133. [ Links ]

SALTELLI A, TARANTOLA S and CHAN KP (1999) A quantitative model-independent method for global sensitivity analysis of model output. *Technometrics 41 *(1) 39-56. [ Links ]

SALTELLI A, RATTO M, TARANTOLA S and CAMPOLONGO F (2005) Sensitivity analysis for chemical models. *Chem. Rev. 105 *(7) 2811-2828. [ Links ]

SALTELLI A, RATTO M, ANDRES T, CAMPOLONGO F, CARIBONI J, GATELLI D, SAISANA M and TARANTOLA S (2008) Global Sensitivity Analysis: The Primer. John Wiley & Sons, Ltd., Sussex, England. 292 pp. [ Links ]

SCOTT EM (1996) Uncertainty and sensitivity studies of models of environmental systems. In: Charnes JM, Morrice DJ, Brunner DT, Swain JJ (eds) *Winter Simulation Conference: Proceedings of the 28th conference on Winter Simulation, Coronado, California. *255-259. [ Links ]

SCUDDER T, MANLEY R, COLEY R, DAVIS R, GREEN J, HOWARD G, LAWRY SW, MARTZ D, ROGERS PP, TAYLOR ARD and co-authors (1993) The IUCN review of the Southern Okavango Integrated Water Development Project. IUCN, Gland, Switzerland. [ Links ]

SIMLAB (2011) Software package for uncertainty and sensitivity analysis. Joint Research Centre of the European Commission. URL: http://simlab.jrc.ec.europa.eu. (Accessed 27 March 2014). [ Links ]

SMEC (SNOWY MOUNTAINS ENGINEERING CORPORATION) (1990) Southern Okavango Integrated Water Development Project. Snowy Mountains Engineering Corporation, Gaborone, Botswana. [ Links ]

WOLSKI P, SAVENIJE HHG, MURRAY-HUDSON M and GUM-BRICHT T (2006) Modelling of the flooding in the Okavango Delta, Botswana, using a hybrid reservoir-GIS model. *J. Hydrol. 331 *(1-2) 58-72. [ Links ]

WOLSKI P and MURRAY-HUDSON M (2008) Alternative futures of the Okavango Delta simulated by a suite of global climate and hydro-ecological models. *Water SA 34 *(5) 605-610. [ Links ]

WOLSKI P (2009) Assessment of hydrological effects of climate change in the Okavango Basin. EPSMO, Cape Town, South Africa. [ Links ]

WTC (WATER TRANSFER CONSULTANTS) (1997) Feasibility study on the Okavango River to Grootfontein link of Eastern National Water Carrier. Water Transfer Consultants, Windhoek, Namibia. [ Links ]

Received 27 March 2014

Accepted in revised form 7 January 2015

* To whom all correspondence should be addressed. (662) 325-1983, Fax: (662) 325-3853; e-mail: alinhoss@abe.msstate.edu