Upload
easter-bates
View
220
Download
1
Tags:
Embed Size (px)
Citation preview
Verification and Validation as Applied Epistemology
SAND 2007-2628
Supported by Sandia National Laboratories LDRD Program 08-1158
Laura A. McNamara, George BackusExploratory Simulation Technologies
Timothy G. TrucanoOptimization and Uncertainty Quantification
Sandia National Laboratories
Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy’s National Nuclear Security Administration
under contract DE-AC04-94AL85000.
Here’s the plot:• Since 1998, the Department of Energy/NNSA
National Laboratories have invested millions in strategies for assessing the credibility of computational science and engineering (CSE) models used in high consequence decision making.
• The answer? There is no answer. There’s a process - and a lot of politics.
• The importance of model evaluation (verification, validation, uncertainty quantification, and assessment) increases in direct proportion to the significance of the model as input to a decision.
• Other fields, including computational social science, can learn from the experience of the national laboratories.
• Some implications for evaluating ‘low cognition agents’
Why “Applied Epistemology?”• Epistemology considers the question, How do we
know what we [think we] know? – Science: What makes Western science special in producing
reliable, predictive knowledge about the world?• V&V takes epistemology out of the realm of thought
and puts it into practice.– What is the role of modeling and simulation in the production
of reliable, credible scientific knowledge about the world?– What steps, investments, practices do I pursue to convince
myself that the model I have developed is producing credible knowledge?
• In large organizations, getting stakeholders to sign up for V&V can be… umm, a challenge.– Too hard, too slow, too expensive…
Rom Harre on the pitfalls of models:
“If we regard theories as descriptions… of reality produced by the human imagination, it is clear that there must be some account of
the constraints upon that imagination, for the human imaginative faculty is well-known for its capacity to generate mere fantasy: and yet, it is plain that the conceptions of reality which scientists have drawn upon from time to time are not fantasies, though at the end some have been abandoned as unrealistic.”
V&V is a methodology of constraint
Why care about V&V for computational social science?• Too much skepticism about models
– “paradigm-busting” role of modeling and simulation in the social sciences
– V&V can strengthen claims by tying model back to real world
• Too little skepticism about models– Search in intelligence, defense, policymaking
communities for predictive technologies– V&V can give ‘reality check’ about where model is
(and isn’t) valid
La prévision, c’est très chic!
• NSTC: “Network models and analyses … will allow for the development of more effective predictive strategies.”
• IEEE: “The US intelligence community… is looking for better ways to identify terrorists… and predict where and when they will strike.”
• DNI: “Predictive network-centric intelligence assessment offers the ability to foresee or warn of threats, be these strategic or tactical and conventional or asymmetric.”
• The purpose of computing is to provide “high-performance, full-system, high-fidelity-physics predictive codes to support weapon assessments, renewal process analyses, accident analyses, and certification.” (DOE/DP-99-000010592)
The purpose of computing is not insight.
Drivers for V&V and UQ in CS&E
• Codes have historically played a critical role in certifying high-consequence systems that can’t be repeatedly tested in a fully representative environment– Nuclear power plant operations– Nuclear waste storage and containment– Aerospace applications
• As computational power increases, so expands the demand for high-fidelity, predictive computational simulations– Nuclear weapons performance, safety, reliability– Should drive demands for higher credibility
V&V Definitions• VERIFICATION:
– The process of determining that a computational software implementation correctly represents a model of a physical process… or
– The process of determining that the equations are solved correctly
• VALIDATION: – The process of determining the degree to which a computer
model is an accurate representation of the real world from the perspective of the intended model applications…or
– The process of determining that equations are correct
V&V is a methodology.
CodeVerification
CodeVerification
DPApplication
DPApplication
PlanningPlanning
ExperimentDesign, Execution
& Analysis
ExperimentDesign, Execution
& Analysis
MetricsMetrics
AssessmentAssessment
Prediction & CredibilityPrediction
& Credibility
DocumentDocument
CalculationVerificationCalculationVerification
1
7
6
5
2
4
3
3
8
•Requirements and planning
Verification Validation Metrics
Credibility
Permanence
Validation Experiments
Let’s simplify and also bring this up to
date:
• Application(s)• Requirements• Application(s)• Requirements
PLANNING
Credibility/BE+U
M&SM&S
V&V
• Metrics• Assessment• Metrics• Assessment ReferentsReferents
• V&V results mean what?
• Use within decisionenvironment?
• V&V results mean what?
• Use within decisionenvironment?
Risk-Informed Decision Analysis (RIDA)PCMM
Simulation as a third corner of science‘Reality’
Conceptual Model
Experimental design,Simulation predictions
Computer Simulation
Model Development
Simulation Validation
Code Verification
Observation and Analysis; hypothesis formulation
Model development and code predictions
Ang, Trucano, Luginbuhl 1998
Aredata
valid?
“Too hard, too slow, too expensive…”
• CS&E – Computational science and engineering is mature, capable, and
reliable for the responsibilities being demanded of it. – I’ve used this code for such a long time that I trust it. I turn the
knobs, I calibrate, and I get the right answer. – V&V is… (see above)
• CSS– Models don’t make decisions; they provide insights to decision
makers– The code isn’t ready for V&V. When it’s more mature, we’ll do
V&V. – V&V for this code? We don’t have the money. Besides, it’s up to
the analyst to use the code right. – If V&V reveals problems with the code we’ve invested so much
money, time, energy… (finish the sentence).
Busting the Rational, Utility Maximizing Decision Maker
• Individuals gather, weigh, process information to maximize utility (rational behavior)– Problem: this assumes a lot of cognitive power in
very high dimensional environments– Problem: classical economic theory doesn’t predict
business cycles (or other cycles) very well
• What if agents are more efficient in processing information, making decisions, than traditionally assumed?
LOW COGNITION AGENTS• DATA: Aggregate data seem to follow an exponential
or power law distribution– This implies something about the relationship between agent
and structure • MODELING THEORY: ‘Degree of cognition that
needs to be assigned to agents decreases as system matures’
• IMPLICATIONS:– People have minimal cognitive ability– OR - People develop heuristics that help them simplify
complex information environments– OR - ‘Structure’ establishes boundaries on quality, quantity
of information required for agents to make a ‘good’ decision– OR - None of the above – ‘low cognition agents’ don’t
actually say much about agents themselves.
Low cognition agents
‘Reality’
Conceptual Model
Experimental design,Simulation predictions
Computer Simulation
Model Development
Simulation Validation
Code Verification
Observation and Analysis; hypothesis formulation
Implementation of conceptual model in codeMathematics works right?
Ang, Trucano, Luginbuhl 1998
Aredata
valid?
REFERENCES• Ang, J., Trucano, T. 1998. Confidence in ASCI Scientific Simulations.
Albuquerque, NM: Sandia National Laboratories. SAND 98-1525c.• Axelrod, R. 2003. Advancing the Art of Simulation in the Social Sciences.
Japanese Journal for Information Management Systems.12(3)• Goldstein, H. 2006.Modeling terrorists: New simulators could help intelligence
analysts think like the enemy. IEEE Spectrum September: 34-43.• Harre, H.R. 2003. Modeling: Gateway to the Unknown. Amsterdam, NL:
Elsevier Press. • Marks, Robert E. 2003. ‘Coffee, Segregation, Energy and the Law: Validating
Simulation Models.’ GET FULL CITATION• Trucano, T., Garasi, C., Mehlhorn, T. 2005. ALEGRA-HEDP Validation Strategy.
Albuquerque, NM: Sandia National Laboratories (SAND 2005-6890). • Oberkampf, W.L., Trucano, T. 2007. Verification and Validation Benchmarks.
Albuquerque, New Mexico: Sandia National Laboratories (SAND 2007-0853). • Pilch, M., Trucano, T., Moya, J., Groehlich, G., Hodges, A., Peercy, D. 2000.
“Guidelines for Sandia ASCI Verification and Validation Plans – Content and Format: Version 2.0.” Albuquerque, NM: Sandia National Laboratories, SAND 2000-3101.
• Smith, T. J.2007. “Predictive Network Centric Intelligence: Toward a Total Systems Transformation of Analysis and Assessment.” Washington, DC: Director of National Intelligence.
Agency and structure• Where does society reside, if not in the individual?• How is the social reproduced across generations?• Is the social world comprised by parts whose
dynamics are structured in terms of an externally imposed cause (structuralism)?
• To what extent are individual actions constrained by society?
• To what extent do the actions of individuals change society?
• How much of a rebel am I, really?• Nature or nurture? What is free will? • Do low cognition agents have anything to say about
these questions?
CSS vs CSE
• High-consensus ‘laws,’ rules, theories exist
• Implemented mathematically with (relative) ease
• Theory is explanatory and predictive
• Multiple theories explain the same set of phenomena
• Theories expressed in narrative form
• Theories are explanatory and descriptive
So where is the computational social science community?
• Axelrod: Does the program correctly implement the model? (internal validity)
• Carley: processes and techniques for addressing comparability between simulated world and ‘real’ world … (external validation)
• Marks: How successfully the model’s output exhibits the historical behaviors of the real world target system (‘output validation’, cf Manson 2002)
Q: Why do we create models?
• Kinds of models– To highlight features of a phenomenon we have observed (to
describe, explain, predict)– As our observations mature, so can our conceptual models
• The Ptolemaic universe• The Copernican universe
– To represent a conception of a phenomenon we have not yet observed
• Superstrings in cosmological physics
• Roles that models play– Models fix a mental representation, collective or otherwise, of a
phenomenon occurring in the world around us – Models are frameworks for organizing inquiry– Models enable knowledge to evolve
A: Because we can’t do science without them.
Uses of agent-based simulations
• Explain a phenomenon, explore a phenomenon, understand interactions that produce a phenomenon (Marks 2003)
• Insight into system control, make predictions, derive general principles (Haefner)
• Prediction, performance, training (flight simulators), entertainment, education (SimCity), existence proofs, discovery, gedankenexperiment (Axelrod 2003)