In an article titled ‘The impact of uncertainty on predictions of the CovidSim epidemiological code’ and published by Nature Computational Science on 22 February 2021, VECMA reports on an analysis of CovidSim, the Imperial College model upon which the UK government relied in part to model various non-pharmaceutical intervention strategies, and guide its government policy in seeking to deal with the rapid spread of the COVID-19 pandemic during March and April 2020.
CovidSim, albeit a sophisticated model, contains several sources of uncertainty that affect the quality of its predictions: parametric uncertainty, model structure uncertainty and scenario uncertainty. A result of these different types of uncertainty combined was the serious underestimation of the first wave: predictions of COVID-19 deaths in the influential Report 9 issued in March were about half the number of deaths that actually occurred in the UK, when simulating the most comparable scenarios -albeit with some differences, such as lockdown occurring later than simulated.
In this Nature Computational Science article, VECMA reported on the results of applying its Toolkit on CovidSim to perform a parametric sensitivity analysis and uncertainty quantification. From the 940 parameters used as input into CovidSim, a subset of 19 was identified which involved the highest sensitivity. Imperfect knowledge of those parameters could lead to a large variance in the code’s output. This variance in output -coronavirus deaths, in this case- can be seen through the example in the figure below, reported in the study, where it can be seen that the predicted cumulative deaths may vary by as much as 100% (i.e. double), while still remaining within one standard deviation in terms of confidence.
The model displays substantial bias with respect to observed data, failing to describe validation data well. Quantifying parametric input uncertainty is therefore not sufficient: the effect of model structure and scenario uncertainty must also be properly understood.
Through the above study, VECMA calls for a better public understanding of the inherent uncertainty of models predicting COVID-19 mortality rates, saying they should be regarded as “probabilistic” rather than being relied upon to produce a particular and specific outcome.