Third Paradigm

In this chapter we discuss the general role of modelling and simulation in science and when and how it should be applied.

Objectives:

• What are the three (four) paradigms of science and where does modelling and simulation fit into these?

• What is the difference between deduction and induction?

• What is a first-principle model?

Deduction and Inducation¶

Traditionally, scientific methods belog to one of two paradigms: experimentation and observation, in which solutions to a problem are induced from experiments with the system, and theory and knowledge, where solutions are deduced from existing knowledge about the system. In this context, the term induction describes drawing general conclusions from specific statements (e.g. 10 of 1000 individuals have a feature $\Rightarrow$ probability is $1\%$). Vice versa, the term deduction describes drawing specific conclusions from (more) general statements (e.g. probablity is $1\%$ $\Rightarrow$ we expect 10 of 1000 individuals to have the feature). The main difference between the two is that reasoning of the specific from the general is logically consistent and therefore considered to be always as valid as the premises, whereas reasoning of the general from the specific is only valid with uncertaintany.

Modelling and Simulation as Third Paradigm¶

So-called computational science, also known as scientific computing, is a division of science which uses computing capabilities to understand and solve complex problems and modelling and simulation and is often (e.g. CSE Education (2001)) referred to as the third paradigm or third mode of science next to the other two. Modelling and Simulation (M&S) is one of the core methods in this field and can be regarded as a synthesis of paradigm one and two: a model of the real system is deduced from theory, and the derived model is then used for experimentation to induce a solution (see Figure 1).

Since experiments using a model of the system are less informative than those carried out on the real system, it is clear that this third paradigm of science is only reasonable if the other two modes are not available or are only available to an insufficient extent. However, since research questions are becoming increasingly complex this is often the case:

1. Experiments with the real system can be unethical, too cumbersome, too expensive or simply infeasible (e.g. performing experiments to forecast a system’s behaviour).

2. Deductive reasoning from theory is only possible if there are sufficient valid underlying premises and if the induction process is analytically possible (i.e. expressions can be evaluated, equations can be solved,...).

Modelling and Simulation and the Fourth Paradigm¶

Beginning in 2007, computer pioneer Jim Gray started to introduce the concept of a fourth scientific paradigm: Data-Driven, to be precise data-intensive scientific discovery, with the idea of extracting knowledge directly from data. This concept is famously published in the book “The Fourth Paradigm” Hey et al. (2009), collected by some of his former employees in 2009 after Gray’s tragic disappearance. Even the idea was not without its critics particularly at the time, it can surely be said beyond any doubt that a given real system can be described not only through theory and knowledge but also through corresponding system data, and that data analysis methods can indeed be used to generate knowledge and solve problems - any recent advances in AI and machine learning support this.

In the general interpretation, M&S does not belong to the fourth paradigm, because it is causality-driven and not data-driven. However, what can be said without a doubt is that not only data-driven but also M&S workflows depend heavily on the availability of data which they must integrate and combine with available theory and knowledge to create valid models and produce valid results (see Figure 2). I.e. while gaining insights via performing simulations is a clear inductive strategy to answer problems, the model itself can be deduced (from theory), induced (from data or observations), or both.

Black-Box or White-Box?¶

Transparency is one of the most important aspects of a scientific method. In scientific computing and data-scienc this often breaks down to two fundemental questions: (a) do I understand the origins of the model, i.e. how does the model relate to the real system, and (b) do I understand what the model does, e.i. how does input map to output? The less well one can answer these questions, the more the method becomes a black-box which, whilst it allows results to be calculated, makes it difficult to understand and communicate the underlying process.

We discuss a few examples:

• On a fictional “black-box-scale”, traditional machine learning (e.g. neural networks) is likely close to the lower end: It is unclear how the chosen network structure relates to the real-world system, and likewise how the trained model maps inputs to outputs due its complexity.

• Due to the more elegant mathematical models, the situation is better for classic statistics (e.g. regression analysis). The model structure is usually well understood and the impact of changed input onto the output can be well analysed. Still the parameter values are fitted in a process which is difficult to comprehend, and the method is selected primarily on the basis of the given data structure (and has little to do with the real system).

Having a reasonable answer to (a) is one of the core features of M&S since the model is generated tailored to the real system (compare, modelling process). Nevertheless, even within differend modelling and simulation techniques there is a clear gradient in terms of how well the question can be answered. This is heavily linked to how data and observations are integrated into the modelling process (compare Figure 2).

• One of the most extreme black-box examples for this is the strategy of system identification Söderström & Stoica (1988). This technique is usually counted to M&S and became prominent in the 1970s mainly in the field of control engineering. The concept allowed to extract structure and parameter values of a model (so-called behaviour models) solely from given system data which has many parallels with machine learning. Since extraction of causalities from data is clearly inductive reasoning, the resulting model is highly intransparent.

• At the very other end of the spectrum lies what is known as first-principles modelling. A first-principles model is one that is derived purely through a deductive reasoning from so-called first principles – axiomatic statements that are true without any need for further proof. Due to its deductive nature, the entire process is highly transparent. However, it requires a set of axioms which is typically only found in certain fields of the hard sciences (physics, electrical engineering, etc.).

• Most applications for M&S lie in between these two extreme cases. On the one hand, most systems do not have a set of axioms from which the model can be derived completely. This means that, at least in part, a modeller must rely on observations or data provided by domain experts. This is, of course, related to the application domain: for application in hard-sciences like physics or electrical engineering, chances for a first principle model are high, and decreases, the “softer” the science gets, such as chemsitry, environmental sciences, logistics, medicine, economy, sociology. On the other hand, pure first-principles models are often not parameterisable (i.e. not all parameter values can be directly taken from the real system but must be fitted/calibrated) or computable (e.g. describing a car engine via the forces between its atoms).

Ultimately, in the field of M&S, one finds differing answers to question (b) as well.

• Simulation models that are sufficiently compact – for example, smaller differential equation models – can be easily analysed for their properties (e.g. steady-state solutions, long-term behaviour, sensitivity, etc.) and therefore offer a high degree of transparency in terms of input-output behaviour.

• However, the larger and more complex the model itself becomes, the more complex and less transparent the input-output relationship becomes. Examples of this include huge coupled equation models, or large stochastic systems such as discrete-event or agent-based models. In such cases, it makes sense to use so-called meta-models or a simpler model-twin for improved analysis.

Just as deductive reasoning is preferable to inductive reasoning, a white-box method is always preferable to a black-box method (provided the level of accuracy is the same). The greater transparency offers advantages, for example, in validation, verification, reproducibility and communication. Being at-least partially white-box is one of the most significant features of M&S studies and distinguishes them from sole data-based approaches, which cannot answer question (a) properly.