![[External Link]](https://webarchive.library.unt.edu/eot2008/20090114122342im_/http://institutes.lanl.gov/_images/icons/external_site.gif){kind=icon}
The scientific method of formulating a hypothesis, developing an experiment and collecting information, and reaching a conclusion either in favor of the hypothesis or that contradicts it, has traditionally been based on physical experiments and measurements. Compared to the 4,000+ years of scientific history, science-based predictive Modeling and Simulation (M&S) is recent. Only since the late 1980’s, or for the past twenty years or so, have scientists started to be confident in their ability to understand and model moderately complex phenomena or applications.
The consequence is that physical experiments that have traditionally supported the decision-making process in physics and engineering are progressively being replaced by numerical simulations. The availability of increasingly faster, cheaper, and massively parallel computing resources are feeding this trend, together with the development of more powerful computing languages and user-friendly visualization tools. It also goes hand-in-hand with the economic imperative of reducing time-to-market cycles and research and development costs, a promise that many industries claim the increased reliance on M&S can fulfill.
One dramatic example where M&S replaced testing is the certification by Boeing Aerospace of their 777 aircraft, which was to a great extent based on simulations in the late 1990’s even though the U.S. Federal Aviation Administration still required full-scale testing for its own accreditation. Accrediting the aircraft involved performing large-scale computational fluid dynamics simulations of its flutter characteristics. Another example is the Nuclear Test Ban Treaty (NTBS) enforced in the United States since 1992, which makes nuclear testing impossible. Mathematical models, computer codes, and numerical simulations have been developed at national laboratories of the U.S. Department of Energy, such as LANL, to study the performance, reliability, and provide the annual certification of these very complex systems.
Because the scientific method in physics and engineering has embraced M&S and problem solving increasingly depends on the development of a predictive capability, as opposed to testing alone, the credibility of numerical simulations must be established. This is accomplished through various activities collectively referred to as model verification and validation. V&V is a rigorous and scientifically sound methodology to assess the prediction accuracy of models and numerical simulations. Confidence in predictions directly results from the breath and outcome of V&V activities.
As previously alluded to, LANL and the U.S. Department of Energy complex have been asked since 1992 to provide an annual certification of the U.S. nuclear weapons stockpile without full-scale testing. Prior to the NTBS, certification used to rely to a great extent on full-physics, integral testing. LANL, Lawrence Livermore National Laboratory (LLNL), and Sandia National Laboratories (SNL) were therefore confronted with the difficult task of developing their simulation programs to “replace” testing. After an initial focus on developing the platforms and science-based predictive computer codes, the emphasis is increasingly being shifted to the quantification of prediction accuracy, the assessment of uncertainty, and the formulation of a framework for decision-making that integrates these new tools. Much of this work is still in progress today.
It is emphasized that many industries have indicated a strong interest for, or are currently developing, the M&S approach. Nevertheless, few are deprived of the testing most critical to answer their accreditation questions. In contrast, the design agencies of the U.S. Department of Energy were placed in the absolute necessity to rely on simulations, which explains why Los Alamos has been a driver for the development of V&V and simulation-based certification methodologies.
A second reason has to do with the multi-disciplinary nature of V&V. While it is true that much of the research in predictive sciences and V&V is carried out at Universities and other academic institutions, few of them, if any, possess enough subject-matter expertise in all of the disciplines needed to tackle a complicated validation problem. Such disciplines typically include material sciences, structural and thermal dynamics, engineering, manufacturing, chemistry, physics, statistical sciences, information theory, etc. Remedying this short-coming by increasing cross-departmental or cross-university collaboration is made difficult, at least in the United States, by the current funding mechanisms for academia that reward competition between individuals, departments, and Universities instead of promoting the integration of resources towards a common goal.
Los Alamos, on the other hand, combines the advantages of a clear mission, well-defined applications, world-class scientists in most of the aforementioned disciplines, and over a decade of institutional investment in science-based M&S and V&V. This makes LANL a leader in the development of the theory as well as practical implementation of V&V.
V&V activities include, but are not restricted to, the design of validation experiments; code verification; calculation verification; the design of computer experiments; response feature extraction; statistical screening and classification; surrogate modeling; uncertainty modeling, propagation, and quantification; test-analysis correlation; model calibration and updating; and the assessment of prediction accuracy.
Uncertainty quantification (UQ) plays a central role in V&V because the assessments of experiments, codes, calculations, models, and numerical simulations are generally made in terms of quantifications of errors and uncertainties. UQ also provides the missing link between the assessment of prediction accuracy, provided by the activities of V&V, and methods for decision-making under uncertainty, such as reliability and robustness analyses. Decision-making under uncertainty is another area that currently undergoes significant interest and research in engineering and physics.
Wrong! While test-analysis correlation is an essential step of model validation, the later cannot be reduced to comparing predictions to measurements. The main outcome of a model validation study is to assess the prediction accuracy of a family of numerical simulations, especially away from configurations or settings that can be tested experimentally. This typically includes extrapolating the prediction accuracy away from the available test data, together with the quantification of its sources of uncertainty. Hence, validation cannot be reduced to comparing predictions to measurements.
Wrong! Calibration, also known as model updating or fitting, is a useful tool in model development. It nevertheless does not equate validation because a calibrated model provides no assessment of prediction accuracy away from those configurations or settings that have been tested experimentally. It has been recognized by many authors, for example, in the field of finite element model updating for structural dynamics simulations, that calibration can provide non-physical solutions. Calibration also leads to over-confidence. Both are detrimental to the ability to make predictions.
Wrong! There is no validation without data because model validation must assess prediction accuracy relative to a physical reality. While code verification and calculation verification are concerned with the accuracy of the numerical implementation and convergence, respectively, validation activities focus on the adequacy of numerical simulations when applied to the description of reality, which requires experimental observations. We nevertheless recognize that the lack of test data can pose serious problems to model validation. Rigorously controlled expert elicitation techniques can provide information that is substituted to experimental testing in cases of severe lack of data and uncertainty.
Not necessarily! Prior to engaging into a validation exercise, analysts should carefully define the scope and requirements of the study. This is achieved given the schedule and budget constraints, and depending on which decisions must be supported by the numerical simulation. For predictions to be credible, code and solution verification activities do not need to provide complete coverage of all aspects of the simulation. Likewise, validation testing can involve inexpensive, small-scale experiments that validate unit-level aspects of the physics. The bottom-line is that the goals of validation should be agreed upon by all stake-holders according to the level of credibility required for a specific application.
Wrong! The aim of Uncertainty Quantification (UQ) is not necessarily to characterize the probability density functions for all predictions of the numerical simulation. While it is true that a fully probabilistic UQ can be nearly impossible, if not prohibitively expensive, other techniques can be brought to bear to investigate the effect of uncertainty on predictions and even reduce it. They include sampling-based methods for the inverse propagation of uncertainty, statistical effect screening, and variance reduction techniques. The quantification of sources of experimental and modeling uncertainty can involve activities of various nature, such as the elicitation of expert opinion, unit-level physical experiments, or the analysis of phenomenological models. In cases of severe lack-of-knowledge, non-probabilistic models can be developed that indicate the effect of uncertainty on predictions, which may be sufficient to demonstrate the credibility of numerical simulations and aid the decision-making process.
Wrong! Analysts are not the sole stake-holders when it comes to V&V. Validation is a multi-disciplinary exercise that involves understanding the code and its capabilities; the hardware platform and its interactions with the software; the physical data and diagnostics available to validate various models; the statistical aspects of data analysis; and the purpose of the simulation in the decision-making process. Validation should therefore involve the code developers, computer scientists, experimentalists, statisticians, analysts, and application owners. Because of their expertise with testing, calibration, and the quantification of measurement error, test engineers can play a central role in the definition of validation experiments. Owners of the application, while soliciting the involvement of others, should play the central role because they are eventually responsible for making decisions based on simulation results. Guaranteeing a certain level of independence between V&V agents and other stake-holders is desirable because independence promotes credibility.
