Global Sensitivity Analysis using Meta-Models

Models are used for both prediction and attribution in ecological and conservation research. In both cases we need to rigorously test the behavior of our models over a wide range of parameter combinations. This process is called global sensitivity analysis. There a several tools to carry out global sensitivity analysis. In our paper Coutts and Yokomizo (2014) we focus on a set of tools called meta-models (also called emulators).

Firstly, what do I mean by prediction and attribution? When models are used for prediction, a model of the system is built. Then, when different parts of the model are changed (for example the mortality rate might be raised), the behavior of the model before and after the change is then compared. Fisheries often have well developed predictive models that are used to predict how a given catch will affect the fish population. When models are used to attribute a mechanism to an observed phenomenon, again a model of the system is built and different parts of the model are changed. For example, density dependence might be added or removed. If the observed behavior of the real population can be replicated by the model with density dependence, but not when density dependence is removed, it is taken as evidence that density dependence is important for population regulation in the real world.

This sort of testing, where just one or two parts of the model changed at a time, is called scenario testing. Scenario testing is important as it shows us the behavior of the model over the parameter values that are considered to be most realistic or relevant to the question at hand. The model is tested with a few carefully selected parameters changed between a few carefully selected few values, with the rest of the parameters in the model held constant. But what if we are wrong about the value of some of those parameters? Or we are right, but only for the time and place we measured them, i.e. if measured in other places and at other times those parameters could have different values. If the conclusions or predictions of a model don’t change very much when we change a parameter, we say the model is robust to changes in that parameter. We can test the robustness of a model to each parameter by slightly changing that parameter and recording how much the model output changes (this is called local sensitivity analysis).

However, in complicated models there will often be interactions between parameters. In other words, the amount that the output of a model changes when one parameter is changed depends on the value of a second parameter (or a second and third etc.). For a detailed explanation of interaction effects in sensitivity analysis, see Coutts and Yokomizo (2014). In this case we need to check that the conclusions or predictions hold across lots of different parameter combinations. This much harder task is called global sensitivity analysis.

Global sensitivity analysis using meta-models involves re-purposing statistical and machine learning tools such as regression and CART methods (e.g. boosted regression trees) to explore the behavior of the model. These tools are used in a similar manner as when applied to data sets collected in the field. The difference is that in a global sensitivity analysis the data set being analyzed is generated by randomly changing all the parameters of the model and recording the model output. That output is then used as the response and the model parameters that were randomly changed are used as the predictors. For a step by step guide on how to do this, with examples see our paper (Coutts and Yokomizo 2014).

Global sensitivity analysis is under used in ecological and conservation research. This is especially problematic considering the widespread use of complex simulation models in these disciplines, the types of models for which global sensitivity analysis is most useful. Perhaps most troubling is that the situation has not improved over time (see Table).

Table: Overview of suitable simulation based PVA papers found in the Web of Science search, classified by the type of sensitivity analysis they used. Papers are grouped into three year time periods (except for the first period which is only two years). The proportion of papers in each time period that used each type of sensitivity analysis, scenario testing, local sensitivity analysis, global sensitivity analysis and building an analytical model is given below the counts (reproduced from Coutts and Yokomizo 2014 supplementary material).

This is despite increases in computational power and improved tools for carrying out global sensitivity analysis. It seems that most researchers in ecology and conservation research have used increasing computer power to build models that include more processes and interactions, rather than less detailed models that are tested more thoroughly. Both approaches are needed to push ecology and conservation research forward, but at the moment it seems that not enough emphasis is placed on testing and understanding the models we build.

Example code to carry out global sensitivity analysis using meta-models can be found here.