Sensitivity_analysis_StrathE2E {StrathE2E2} | R Documentation |
Performs a one-at-a-time parameter sensitivity analysis on the StrathE2E model using the Morris Method for factorial sampling of the physical configuration parameters, ecology model parameters, the fishing fleet model parameters, and the environmental forcings.
Sensitivity_analysis_StrathE2E(model, nyears = 50, n_iter = 16, trajsd = 0.0075, n_setoflevels = 4, v_setoflevels = 0.1, coldstart = TRUE, quiet = TRUE, postprocess = TRUE)
model |
R-list object generated by the read_model() function which defined the model configuration |
nyears |
Number of years to run the model in each iteration (default=50) |
n_iter |
Number of trajectories of parameter sets (default=15) |
trajsd |
Coefficient of variation used to set the standard deviation for the gaussian distribution from which new paramater values are drawn to create each trajectoy baseline from the initial parent values (default=0.0075) |
n_setoflevels |
Number of fixed levels of coefficient of variation used to generate the individual parameter values in each level-run. Must be an even number (default=4) |
v_setoflevels |
Maximum coefficient of variation for the set of levels (default=0.1, i.e. -10 percent to +10 percent) |
coldstart |
(TRUE or FALSE, default=TRUE) Set to TRUE if the run is starting from cold, set to FALSE if the run is a continuation of a previous run. This is to enable the task to be divided up across multiple processors and the results concatenated afterwards - set coldstart=TRUE for only one of the runs, and coldstart=FALSE for all the others. |
quiet |
(TRUE or FALSE, default=TRUE) Suppress informational messages at the start of each iteration |
postprocess |
(TRUE or FALSE, default=TRUE) Process the results through to a final sorted list of parameter sensitivies for plotting, or just produce the raw results. The reason for NOT processing would be if the job has been shared across multiple machines/processors and several raw result files need to be concatenated before processing. |
WARNING - This function will take several days to run to completion on a single processor with even a modest number of iteration. However, it is possible to spread the load over multiple processor/machines and concatenate the results at the end into a single results file for post-processing.
The basis for the method is a scheme for sampling the model parameters, one at a time, from distributions around baseline sets, and testing the effects on the performance of the model against some criterion. In this case, the criterion is the likelihood of the observational data on the state of the ecosystem that are used as the target for parameter optimisation in the various simulated annealing functions supplied with the package.
The process requires an initial set of parameters for the model. We refer to this as the 'parent' parameter set. It is recommended that this should be the parameters producing the maximum likelihood of the observational target data (as estimated by e.g. the annealing_find_ecology_parameters() function). The MODEL_SETUP.csv file in the folder /Models/Modelname/Modelvariant/ should be configured to point to the relevant files, and then loaded with the read_model() function.
From this parent set, a series of 'child' parameter sets are generated by applying a separate random increment to each parameter drawn from a gaussian distribution of mean 0 and standard deviation given by a fixed coefficient of variation applied to the parent-set value of each parameter.
For each of the child-sets of parameters, the likelihood of the observed target data is calculated following runs of StrathE2E to stationary state. We refer to these as trajectory baselines.
Then, for each trajectory, the parameters are varied in turn, one at a time, by adding a fixed proportionality increment, the model re-run, and the likelihood computed. We refer to these as 'level runs'. The proportionality increment for all of the level runs within a given trajectory is drawn at random from a set of fixed levels distributed symetrically around 0 (e.g. -10, -5, +5, +10 percent, i.e. proportions of the trajectory values = 0.9, 0.95, 1.05, 1.10).
The total number of model runs required to support the analysis is r*(n+1) where r is the number of trajectories and n is the number of parameters. The function incorporates all of the physical configuration parameters, ecology model parameters, the fishing fleet model parameters, and the environmental forcings into the analysis, and r = 453. The mininimum recommended number of trajectories is 15, and each model run needs to be sufficiently long to achieve a stationary state. Hence run-times for this function are several days.
For each level run, the 'Elementary Effect (EE)' of the given parameter is calculated from the difference between the level run likelihood and the corresponding trajectory baseline likelihood.
On completion of all the trajectories, the raw results are (optionally) post-processed to generate the mean and standard deviations of all the EE values for each parameter is calculated. EE_mean is an index of the magnitude of the sensitivity, and EE_sd is an index of the extent of interaction with other parameters.
During the run process the function produces a real-time plot for each trajectory, in which the x-axis represents the sequence of parameters, and the y-axis is the likelihood of the target data. A horizontal red line indicates the likelihood fof the parent parameter set, horizontal grey line indicates the likelihood for each trajectory baseline and each level-run likelihood is shown by a symbol. The y-axis range can be changed in real-time by editing the setup file "/Models/Modelname/Modelvariant/Parameters/Parameter_SD_Control/OATsensitivity_SD.csv"
The outputs from the function are directed to CSV files in the current "results" folder. The outputs are: a) Table of parameter values applied in each run of the model (OAT_parameter_values-*.csv, where * = model.ident as defined by read_model()) b) Table of the likelihood and EE value for each trajectory/level run (OAT_results-*.csv) c) Table of Mean EE and standard deviatiion of EE for each parameter, sorted by the absolute value of EE_mean (sorted_parameter_elemental_effects-*.csv)
Since Sensitivity_analysis_StrathE2E() is extremely time consuming, it makes sense to share the load across multiple processors and combine the results afterwards. Arguments availabele here in Sensitivity_analysis_StrathE2E() allow for management of this sharing of the load. Afterwards, the raw results files can be combined into a single data set using the concatenate_raw_sensitivity_analysis_results() function.
A separate function plot_sensitivity_analysis_results() produces a graphical representation of the EE_mean and EE_sd results.
CSV files of raw parameter vectors, likelihoods and Elementary Effects for each run, and optionally dataframe and csv file parameter list sorted by EE_mean
Morris, M.D. (1991). Factorial sampling plans for preliminary computational experiments. Technometrics, 33, 161-174.
read_model
, StrathE2E
, plot_sensitivity_analysis_results
, concatenate_raw_sensitivity_analysis_results
, process_sensitivity_analysis_results_offline
# Load the 2003-2013 version of the North Sea model supplied with the package: model <- read_model("North_Sea", "1970-1999") # # Run the sensitivity analysis process: # WARNING - Running a full sensitivity analysis takes days of computer time on a single machine/processor because it involves a huge number of model runs. # The example below is just a (relatively) quick minimalist demonstration and should NOT be taken as the basis for any analysis or conclusions. # Even so, this minimalist demo could take 45 min to run to completion because it involves 1359 model runs. sens_results <- Sensitivity_analysis_StrathE2E(model, nyears=1, n_iter=3) head(sens_results) # # A more realistic sensitivity analysis would be something like: sens_results <- Sensitivity_analysis_StrathE2E(model, nyears=50, n_iter=16, postprocess=TRUE) # DO NOT launch this configuration unless you are prepared to wait many days for the results # # To share the computational load across two (or more) machines/processors, launch separately on each processor... model1 <- read_model("North_Sea", "1970-1999", model.ident="TEST1") Sensitivity_analysis_StrathE2E(model1, nyears=50, n_iter=10, coldstart=TRUE, postprocess=FALSE) # On processor 1 # model2 <- read_model("North_Sea", "1970-1999", model.ident="TEST2") Sensitivity_analysis_StrathE2E(model2, nyears=50, n_iter=10, coldstart=FALSE, postprocess=FALSE) # On processor 2 # In these cases the function returns a NULL object since postprocess=FALSE but the raw data are still output to CSV # # Then concatenate the two raw results files with text-tags TEST1 and TEST2 afterwards, and post process # the combined file, using the function concatenate_raw_sensitivity_analysis_results() #