Dear Label_Name,

Greetings! I hope you are well.

I am writing on behalf of an international publishing house, Science and Education Publishing, USA (http://www.sciepub.com).

Science and Education Publishing (SciEP) is a multidisciplinary open access publisher of Journals covering the fields of Science, Technology and Medicine. All SciEP publications are subject to high-quality peer review, editorial and production processes and are freely available online.

Welcome to the April, 2013 edition of SciEP newsletter, providing monthly summaries of articles on science & education related topics of interest to scientist and professionals. The mission of our journals is to provide an open access forum for publishing new findings on issues pertinent to scientific and academic field. We also invite you to submit original research papers, review articles, short reports, or case studies are considered for publication in our Multidisciplinary Journals.

American Journal of Public Health Research
DOI: 10.12691/ajphr-1-1-3

The Implementation of a Tele-Homecare System with Service Oriented Architecture and HL7 Message Transmission Standard
Ching-Sung Wang

American Journal of Medical and Biological Research
DOI: 10.12691/ajmbr-1-1-2

The Influence of Ethanol on Pyruvate Kinases Activity in Vivo, in Vitro, in Silico
Lelevich Sergey Vladimirovich, Khrustalev Vladislav Victorovich, Barkovsky Eugene Victorovich, Shedogubova Tatyana Aleksandrovna

American Journal of Medical Sciences and Medicine
DOI: 10.12691/ajmsm-1-1-1

Clinical and Histological Features in Wegener Granulomatosis
Li Cavoli Gioacchino, Passantino Rita, Bono Luisa, Tortorici Calogera, Ferrantelli Angelo, Giammarresi Carlo, Rotolo Ugo

American Journal of Infectious Diseases and Microbiology
DOI: 10.12691/ajidm-1-1-2

The Curse of Lymphatic Filariasis: Would the Continual Use of Diethylcarbamazine Eliminate this Scourge in Papua New Guinea?
Anne Lanham, Lillian Mwanri

We have released issues of some journals on February 2013.
American Journal of Public Health Research
American Journal of Medical Sciences and Medicine
American Journal of Medical and Biological Research
American Journal of Pharmacological Sciences
American Journal of Clinical Medicine Research
American Journal of Nursing Research
American Journal of Cancer Prevention
American Journal of Medicine Studies
American Journal of Infectious Diseases and Microbiology
American Journal of Cardiovascular Disease Research
American Journal of Biomedical Research
American Journal of Mechanical Engineering
American Journal of Microbiological Research
Journal of Preventive Medicine
American Journal of Food and Nutrition
American Journal of Hypertension Research
American Journal of Sports Science and Medicine
Journal of Cancer Research and Treatment
American Journal of Zoological Research
Journal of Biomedical Engineering and Technology
American Journal of Microbiological Research
Biomedical Science and Engineering

We are pleased to announce that submissions are now being accepted for peer review for all the journals across all our subject areas in SciEP.

Please submit your manuscript via our online submission system:
http://www.sciepub.com/author/signin

To unsubscribe from our mailing list, please click on the following link: unsubscribe

I am looking forward to hearing from you.

Kind regards,

David Rain
Email:join@sciepub.com
Science and Education Publishing (http://www.sciepub.com)

Probabilistic Programming Method for Time-Series Forecasting of COVID-19 Cases Based on Empirical Data

Matti Pärssinen*, Ilkka Sillanpää, Mikko Kotila

Aalto University, Department of Communications and Networks, Espoo, FINLAND

Abstract: During a pandemic, leaders and decision-makers are compelled to urgently make decisions due to public health concerns, public and expert opinions, and other factors. It is typical, particularly in the early phases of a pandemic, for decisions to depend on partial or missing information. A current trend in scientific publication is to present repeatable results, accompanied by data sources and artefacts. This results in complete transparency and auditability of the results, as well as a platform for future research. CoronaCaster is a tool based on the probabilistic programming method, built for transparent forecasting of pandemic cases, hospital capacity, and mortality rate. Probabilistic programming, i.e. Bayesian inference, has been shown to perform well with time series prediction challenges with small sample size and great uncertainty. CoronaCaster uses an advanced Bayesian method to obtain model parameters and their confidence intervals for the user-selected shape function (polynomial, exponential or sigmoid). They are obtained by sampling parameter space using the training period data.

Keywords: Bayesian analysis, forecasting, pandemic

Received June 29, 2021; Revised August 01, 2021; Accepted August 10, 2021

Citation: Matti Pärssinen, Ilkka Sillanpää, Mikko Kotila, Probabilistic Programming Method for Time-Series Forecasting of COVID-19 Cases Based on Empirical Data. American Journal of Epidemiology and Infectious Disease, 9 (1). 18-23. January 2021. doi:10.12691/ajeid-9-1-4

1. Introduction

Many pandemic forecasting models which attempt to account for epidemiological complexity and population structures, have been developed. Results from such models are often challenging for decision makers to interpret, and for other researchers to reproduce. According to the editorial policy of Nature Research Journal on repeatability, the ability for others to replicate and build upon claims is an inherent principle of publication [1]. Nature considers it best practice to release custom computer code in a way that allows readers to repeat the published results, and strongly encourages authors to 1) deposit and manage source code with modern version control system, 2) use a DOI-minting repository to uniquely identify the code, 3) provide details of its license of use, and 4) use a license which is approved by the open-source initiative. Put another way, if data sources or model artefacts are not publicly available as an easily accessible package, the results are not easily reproducible, resulting in the work lacking in credibility. [2]

During a pandemic, leaders and decision-makers are compelled to urgently make decisions due to public health concerns, public and expert opinions, and other factors. It is typical, particularly in the early phases of a pandemic, for decisions to depend on partial or missing information. Dealing with radical uncertainty is a known challenge in all areas of research and decision making. The case of restrictive actions during a pandemic is just a special case of this underlying generic challenge.

This paper provides a readily available, extendable, and well-documented open-source platform for time-series forecasting of cases, hospital capacity, and rate of mortality for the current and future pandemics. CoronaCaster reduces the previously complex probabilistic programming prediction pipeline into a single line of code and does not require expert knowledge to operate. The intent is to minimize the barrier of entry to 1) review the prediction results presented in Section IV, 2) apply CoronaCaster to pandemic prediction, 3) extend current CoronaCaster capability, and 4) to use CoronaCaster for other time-series predictions.

The paper has been structured in the following way: Section II provides a review of the most common modeling methods used in pandemic forecasting. In addition, the methods used by CoronaCaster are introduced. Section III describes the CoronaCaster solution. Section IV presents results from an experiment of using CoronaCaster to predict the number of national cases three weeks into the future, with ten weeks of data from 174 countries. Finally, in Section V, the results and further research are discussed.

2. Materials and Methods

While implementing a custom probabilistic programming pipeline for time-series forecasting continues to be a non-trivial engineering task, python libraries such as PyMC3 [3] used by CoronaCaster, make the Bayesian method more accessible to researchers. The Bayesian method was selected due to being applicable for time-series forecasting with limited number of previous observations available, and the Bayesian theorem being more intuitive to understand than its frequent counterparts such as differential equations.

The dataset for the experiments detailed in this paper is from the European Center for Disease Prevention and Control dataset COVID-19 Geographic Distribution Worldwide [4]. CoronaCaster supports any similar datasets where daily case, hospitalization, or death observations are available from a given area. The area can be any locale, for example a town or a region. Next, the Bayesian method is introduced with a simple example:

A scientist is interested in finding out the probability that a US farmer lives in Texas, given that 1.46% of Texans are farmers (2017 Census of Agriculture, USDA).

Statement A = A person from the US is a farmer. According to data from 2017 by the Census of Agriculture, there were 3.4 million farmers, which is 1.05% of all people living in the US. The probability of A being true is P(A) = 0.0105.

Statement B = A person from the US is a Texan. There were 28.3 million Texans in 2017. P(B) = 0.0871.

Among Texans, the probability of being a farmer is the Figure 1.46% from above. This is the A|B, or A when B is true. P(A|B)=0.0146.

Applying Bayes’ theorem yields:

In other words, if a person from the US is a farmer, there is a 12.1% chance of the person also being a Texan – a chance much higher than that of being a farmer among Texans.

The Bayesian method is optimally suited for pandemic or epidemic cases where data is gathered gradually and more detailed estimates can be made over time. Moreover, the results can be given as confidence intervals for the parameters of the model and not simply as point estimates.

Our approach uses shape functions as a priori information. These are functions that are expected to be able to capture the overall features of the cumulative case counts during the studied time period. Currently, we use simple exponential, polynomial and logistic functions. A logistic function

is a fairly simple sigmoid or “S-curve” shape function with three parameters (L, k and x0) that make it versatile while also keeping it computationally relatively light. The logistic growth model is common in various population studies [5,6,7], as well as those directly related to epidemiology [8]. The sigmoid shape can capture the key phases of an epidemic very well, especially in cases of limited geographical area or, say, a small to midsize country: the slow onset, the exponential growth that turns to linear growth and finally plateaus as the epidemic has run its course or effective mitigation efforts are in place.

More recently, there have also been many other studies that have applied Bayesian methods to Covid-19 studies ranging from the pathogen’s evolution [9,10] and propagation mitigation [11] to testing and treatment options [12].

The most appropriate approach for each situation must be chosen carefully. [13] A real-world phenomenon can often have multiple models attempting to describe it. This is understandable, as no single model can provide fully accurate results. Also, the complexity of models varies, and simple and complex models are of interest to different user groups.

In the beginning of a pandemic, information is scarce. Forecasting confidence can potentially be improved with multiple independent models. According to M.J. Keeling et al., a good model is fit for its purpose and should be parameterizable by available data. [2]

In C.E. Walters et al. article, purely statistical models were identified. Additionally, agent-based or population-wide models were identified. An example of an agent-based model is FluTE. [14] FluTE is an agent-based model for simulating influenza spread across major metropolitan areas. The FluTE model uses travel data from within the US. In population-wide models, each individual is not traced throughout the model. [13]

The second type is the metapopulation model. Examples of such models include the Global Epidemic and Mobility project (GLEaM). GLEaM is a computational model that can be used to represent global infectious disease spread, which is based on a metapopulation approach [15]. In the model, local and global mobility patterns are represented by distinct spatial regions that are connected by a network.

The third model type includes cellular automaton models (CA). CA models involve a grid lattice made up of cells and, at each discrete time step, the state of an individual cell is affected by the states of its neighbors according to a predefined mathematical rule [16]. Infectious disease CA models could involve each cell representing an individual who, at each time step, will be either susceptible, infected, or recovered. A more detailed description can be found in [2].

The fourth model type includes gravity models, which are used to represent the flow of commodities, people, or information from one region to another, allowing them to be adapted to model infectious disease spread from a source location to a destination location [17]. The measure of disease spread from one region to another is proportional to the sizes of the populations and the distance between them.

The fifth model type includes dimensionless models, which are commonly used in physics to understand the role of parameters in the dynamics of the solution. The compartmental models have a dimensionless form. There are two timescales dictated by β and γ, so if time is rescaled by γ, τ = γt, and s = S/N, i = I/N, and r = R/N represent fractions of the population in each compartment, then we retain only one dimensionless parameter R0, that in conjunction with the initial conditions completely determines the resulting behavior. There are three timescales in Susceptible-Exposed-Infectious-Removed (SEIR), thus resulting in a dimensionless equation with two dimensionless parameters. [18] An example of such online SEIR model can be found openly available on the Internet [19].

3. Solution

CoronaCaster provides an effective method for forecasting cases, hospitalizations, and deaths based on empirical data. CoronaCaster is based on the Bayesian inference method that allows forecasting future values based on historical time-series data.

When using CoronaCaster, the user makes a choice among the model shape functions that include exponential, polynomial (any order), and sigmoid. These shape functions allow detailed modelling of different phases of a pandemic, within any demographic sample.

The exponential function is useful for the outbreak stage of a pandemic. Growth in the number of new cases, or any other modelled quantity, is not obstructed by social distancing or other mitigations, and unrestricted growth results in an exponential curve. When mitigations have been effectively implemented or pandemic growth has been limited by other factors, growth can be described well with the polynomial shape function. The sigmoid function is useful in the final phase of growth, when the number of new cases reduces greatly. It can also be useful if the whole pandemic cycle covering the outbreak, growth, disruption and the recovery stage is to be modelled.

The code for CoronaCaster includes optimized initial parameters for the shape functions. They greatly increase the likelihood of reaching optimal model parameters when sampling the models, and can also greatly reduce the number of samples needed. The user can, however, choose to use their own initial parameters; these are given as a pair of the mean and standard deviation for each initial parameter. Other inputs are dated observations of new cases or demises, the shape function (in case of a polynomial function, also its order), the number of samples used for sampling the model and, optionally, the start and end date to use for the fit. The needed number of samples with those determined by CoronaCaster is around 5000. For poorer initial conditions, ill-fitting shape function, or using a high-order polynomial, some 20,000 may be needed.

CoronaCaster creates a model for the data by providing a pair of values for each shape function parameter: an optimal mean value and its standard deviation signifying a confidence interval for the mean value. Number metrics for accuracy of the model are calculated. In addition, the tool can provide a graph depicting data, optimal fit, and effects of the confidence intervals of each parameter on the resulting model.

There is one optional CoronaCaster input parameter not yet mentioned; it is the prediction date, which is a date after the last date of the input data. The created model is evaluated for that date and the result is returned as a prediction value with confidence intervals. It should be noted that the modeling method takes the shape function as a priori information for the Bayesian inference problem and cannot be directly used to estimate the suitability of the shape function for the modeling task.

Figure 1 summarizes the main functionalities of CoronaCaster. The input data set is shown in green. Blue shows the processing with Bayesian modeling, as well as the shape functions and the use of calculated initial parameters. The outputs of CoronaCaster are visualizations of the original data or the Bayesian fits, the fit parameters for the shape functions, and the predictions for the cumulative number of cases for a chosen date.

Figure 1. A schematic on the function of the CoronaCaster tool

CoronaCaster is a powerful tool making use of the established PyMC3 library. The code for CoronaCaster is openly accessible via the Autonomio project on GitHub [20] and an installable package is available on PyPi. The code is written in Python3 using only standard libraries and the PyMC3 library.

Output from CoronaCaster has the following attributes:

Table 1. Output from CoronaCaster

Intercept and a1 here are the parameters for a first-order polynomial shape function; other shape functions have additional or different parameters.

4. Forecasting Results with CoronaCaster

An experiment with CoronaCaster was conducted to demonstrate its forecasting capabilities. In the experiment, five different shape functions were used: 1) exponential, 2-4) 1st, 2nd, and 3rd order polynomial, and 5) sigmoid. As input data, daily Covid-19 case data by country from European Centre for Disease Prevention and Control was used.

The prediction task was to forecast the sum of all cases by country on the 14th of November 2020, using data from the period between the 24th of July 2020 and the 24th of October 2020, empirical data ending three weeks prior to the prediction date. The predictions were made for 174 countries, each with a minimum of 500 Covid-19 cases reported by the 24th of October 2020.

The default CoronaCaster configuration was used in the experiment. The results of the experiment are described next.

The predicted increase of cumulative cases over the observed increase in the cumulative cases during the 3-week prediction period was chosen as the primary metric, with perfect prediction giving 1.

In Figure 2, the results in this metric are presented on a logarithmic scale with each shape function identified. The results for the polynomial functions center around 1, whereas the predictions by the exponential function exceed the observed by a large factor. The results by the sigmoid function on average underestimate the increase in cumulative cases and never exceed the observed increase by more than a factor of 3.

A smaller absolute difference to 1 is taken to be better in the primary metric. Then, picking only the best shape function for each country, the results have a median of 0.9312, and the 25th and the 75th percentiles were 0.6653 and 1.154, respectively. The best fit functions were poly3 (i.e. 3rd-degree polynomial) in 42% of countries, sigmoid in 24%, poly2 at 23%, poly1 in 10%, and exponential in one of the 174 countries.

Since the best shape function cannot be known beforehand, a method that can improve the prediction over a randomly selected shape function is desirable. The optimal objective choice was found to be the shape function which resulted in a model with the best correlation, using the training period data as measured by the Pedersen correlation coefficient. The distribution of the predictions by these shape functions is shown in Figure 3, with shape function types identified. The median of the resulting primary metric of the results by best correlation shape functions for each country is 0.7789, the 25th percentile 0.4816, and the 75th percentile 1.317, respectively.

Figure 2. Prediction results by shape function type

Figure 3. Prediction results for each country, with the shape functions that resulted in models with the best correlation, using the training data

The shape functions with the best correlation coefficients were poly3 in 69% of the countries, logistic in 30%, and poly2 in two of the 174 countries. There were ten countries whose prediction results of the highest correlation shape function exceeded four by primary metric; Aruba, Australia, China, Equatorial Guinea, Gabon, Lesotho, Malawi, Madagascar, Singapore, and South Sudan. All these countries experienced a period of rapid increase in cumulative cases during the 3-month data period, which did not continue in the prediction period. Nine of these countries had third-degree polynomial shape functions as their best correlation models.

There were six underestimated countries with the primary metric below 0.1 (15 countries with <0.2) with the best correlation shape function. These six countries were Cuba, Palestine, Sudan, Syria, Trinidad and Tobago, and the Turks and Caicos Islands. Three of these countries (Sudan, Trinidad and Tobago, and Turks and Caicos) had very few, if any new COVID-19 cases at the end of the three-month training data period, but during the three weeks predicted, there was an abrupt increase of new cases. The other three countries experienced strong and somewhat steady growth throughout the training data and prediction periods, and third-order polynomials gave good correlations for the training data period but predicted a slow increase for the prediction period.

Many of the overestimated and underestimated results by the highest correlation functions point to the false promise of higher-order polynomials, which can result in high correlations using the training data. The danger comes from the higher-order terms that can be near insignificant in the training data period, but the extrapolation used for the prediction can over or undershoot the most reasonable targets. It is recommended to visually check the predictions made, especially so that the prediction model captures the trend of the training period also during the prediction period.

5. Discussion

CoronaCaster is a solution for robust modeling and prediction methodology. It answers many current needs of the various approaches in analyzing epidemic or pandemic data in the context of hospital capacity and medical care resources needed. As a purely numeric exercise, it ignores the underlying population dynamics that more detailed epidemiological approaches, such as SIR and SEIR models, aim to tackle. However, there are many benefits to the CoronaCaster approach including ease of use without expert-level knowledge in epidemiology, population, network dynamics, or data analytics. With this advanced Bayesian model tool written in open-source code, a reliable and fast-to-utilize tool is provided for the interested public to visualize, analyze, predict, and track the epidemic data.

Predictions generated using CoronaCaster have the caveats of the original empirical data used as input. The often sporadically updated data sets and limited data collection facilities in many countries and regions add to uncertainties in the data. Further, the method works best to capture the identified phases of epidemic spreading and may have issues with long training or analysis periods with two or more epidemic waves. In addition, it is worth noting that in the test conducted for this study, the prediction period was close to the maximum period recommended. Increasing the training period much beyond the three months used in the experiment may prove ineffective in improving prediction accuracy.

Despite data on COVID-19 cases being easily available for forecasting, this data cannot be readily validated, since measuring people exposed to the virus is extremely difficult. Individuals with mild symptoms generally do not seek medical care, and there are many symptomatic individuals who do not suffer from complications. Moreover, behavior varies significantly between countries and cultures. These uncertainties also affect the quality of higher-order products based on the available data, and they become naturally very difficult or impossible to estimate. Serological analysis can provide high-quality estimates of the people who have been exposed to the virus, but it is costly and generally uses a very limited sample of the population. From a numerical standpoint, the estimated infection rate also depends on the level of detail at which the transmission dynamic is modeled within the population: for instance, it is well known that homogeneous models generally overestimate the infection rate due to lack of clustering and structure in the population. In order to have more accurate estimates of infection rates, different patterns of contact between individuals of different ages, and country-specific features of households, schools, and workplaces, must be considered in the modeling approach. [21]

The Bayesian method has been used in several studies related to COVID-19. For example, R. Noriega and M.H. Samore (2020) conducted a Bayesian inference analysis in which hierarchical testing stages can have different sensitivities implemented and benchmarked against early COVID-19 testing data. Optimal pool size and increases in throughput and case detection were calculated as a function of disease prevalence. Even for moderate losses in test sensitivity upon pooling, substantial increases in testing throughput and detection efficiency were predicted, suggesting that sample pooling is a viable avenue to circumvent current testing bottlenecks for COVID-19. [12]

M. F Boni et al. (2020) used Bayesian evolutionary rate and divergence date estimates that were consistent for all three recombination-free alignments and robust to two different prior specifications based on HCoV-OC43 and MERS-CoV evolutionary rates. Divergence dates between SARS-CoV-2 and the bat sarbecovirus reservoir were estimated as 1948, 1969, and 1982. Despite the intensified characterization of sarbecoviruses since SARS, the lineage giving rise to SARS-CoV-2 has been circulating unnoticed for decades in bats, and has been transmitted to other hosts such as pangolins. The occurrence of a third significant coronavirus emergence in 17 years, together with the high prevalence and virus diversity in bats, implies that these viruses are likely to cross species boundaries again. [10]

Numerical epidemiological models and analysis tools will slowly evolve towards more automatically optimized model selection approaches, but increased reliability of forecasts is not a directly expected outcome of this process. A major issue that remains to be addressed is the reliability of the underlying data, and how representative sampled data factually is. The future outlook in purely numerical analysis and prediction, automatic shape, and epidemic phase-detection, could enable more complete and automated data and analysis pipelines. Such a pipeline could include automated training period selection based on phase detection.

References

[1]   | https://www.nature.com/nature-research/editorial-policies/reporting-standards [Accessed 14.4.2021].
[2]   | Keeling, Matt J., and Pejman Rohani. Modeling infectious diseases in humans and animals. Princeton university press, 2011.
[3]   | https://docs.pymc.io/ (2018) [Accessed 14.4.2021].
[4]   | https://www.ecdc.europa.eu/sites/default/files/documents/COVID-19-geographic-disbtribution-worldwide.xlsx [14.4.2021].
[5]   | Cantrell, Robert Stephen, and Chris Cosner. “Diffusive logistic equations with indefinite weights: population models in disrupted environments.” Proc. Roy. Soc. Edinburgh Sect. A 112.3-4 (1989): 293-318.
[6]   | Taira, Kazuaki. “Introduction to diffusive logistic equations in population dynamics.” Korean Journal of Computational & Applied Mathematics 9.2 (2002): 289-347.
[7]   | Afrouzi, G. A., and K. J. Brown. “On a diffusive logistic equation.” Journal of mathematical analysis and applications 225.1 (1998): 326-339.
[8]   | Li, Jinhui, et al. “Stability and bifurcation analysis of an SIR epidemic model with logistic growth and saturated treatment.” Chaos, Solitons & Fractals 99 (2017): 63-71.
[9]   | Velazquez-Salinas, Lauro, et al. “Positive selection of ORF3a and ORF8 genes drives the evolution of SARS-CoV-2 during the 2020 COVID-19 pandemic.” BioRxiv (2020).
[10]   | Boni, Maciej F., et al. “Evolutionary origins of the SARS-CoV-2 sarbecovirus lineage responsible for the COVID-19 pandemic.” Nature Microbiology 5.11 (2020): 1408-1417.
[11]   | Bayes, Cristian, and Luis Valdivieso. “Modelling death rates due to COVID-19: A Bayesian approach.” arXiv preprint arXiv:2004.02386 (2020). [2004.02386] Modelling death rates due to COVID-19: A Bayesian approach (arxiv.org) [Accessed 2.3.2021]
[12]   | Noriega, Rodrigo, and Matthew Samore. “Increasing testing throughput and case detection with a pooled-sample Bayesian approach in the context of COVID-19.” bioRxiv (2020).
[13]   | Walters, Caroline E., Margaux MI Meslé, and Ian M. Hall. “Modelling the global spread of diseases: A review of current practice and capability.” Epidemics 25 (2018): 1-8.
[14]   | Chao, Dennis L., et al. “FluTE, a publicly available stochastic influenza epidemic simulation model.” PLoS Comput Biol 6.1 (2010): e1000656.
[15]   | Balcan, Duygu, et al. “Modeling the spatial spread of infectious diseases: The GLobal Epidemic and Mobility computational model.” Journal of computational science 1.3 (2010): 132-145.
[16]   | Weisstein, Eric W. “Cellular Automaton.” MathWorld., A Wolfram Web Resource. https://mathworld.wolfram.com/CellularAutomaton.html [Accessed 2.3.2021].
[17]   | Rodrigue, Jean-Paul. The geography of transport systems. Routledge, 2020.
[18]   | Bertozzi, Andrea L., et al. “The challenges of modeling and forecasting the spread of COVID-19.” Proceedings of the National Academy of Sciences 117.29 (2020): 16732-16738.
[19]   | The ICU Burden Calculator is an adaptation of the original Epidemic Calculator by Gabe Goh et al. https://www.icucalculator.com/. Source code: https://github.com/cavai-research/epcalc. [Accessed 2.3.2021]
[20]   | CoronaCaster., https://github.com/autonomio/coronacaster [Accessed 2.3.2021].
[21]   | y Piontti, Ana Pastore, et al. “THE NUMERICAL FORECAST OF PANDEMIC SPREADING: THE CASE STUDY OF THE 2009 A/H1N1 PDM.” Charting the Next Pandemic. Springer, Cham, 2019. 55-65.