1

From Validating Models to Validating Systems

Peter Denno2013-02-25

University of Maryland ISR Colloquium

2

Outline

• Introduction / Scoping• Requirements for MBSE• Exchange Form Validation• NIST Work

3

Goals

• Describe a “design philosophy” for systems that assist in systems engineering– Framework for linking multiple viewpoints– Framework for research

• Link the design philosophy to NIST workin exchange form validation, requirements engineering, supply chain logistics simulation

4

What is special about V&V ? (1)

• IBM Watson- New techniques – exceed human capability in

knowledge-intensive tasks

- “Machine understanding is not human understanding.”

- “Knowledge is not the destination.”

5

What is special about V&V ? (2)

• Validation & Verification- Knowledge is the destination. – knowledge, or at least

credible rationale.

• Requirement:- Minimally: be able to explain how the design space was

characterized and demonstrate that requirements are being met.

- Ideally: Provide deductive arguments where appropriate- Show how certain alternatives are indeed incompatible- Reference principles of operation, functions

6

Outline

• Introduction / Scoping• Requirements for MBSE• Exchange Form Validation• NIST Work

7

Basis for SE decision making

• SE decision making macro level:– Trade studies, simulations, risk assessment, etc.

• SE decision making micro level:– A web (conceptual schema) of information• Uncertain • Conflicting• Isolated

• Uncertainty is quantified• Conflicts resolved• Inter-relation revealed

8

Strategy for the micro-level information

• Characterize elements of rationale for SE decision making.

• Each research project touches on only a few of these elements– No single overarching system design intended

9

9 Elements of Rationale (1)• Measurement Conditions

– Confidence in the process or environment under which it was measured– “Capacitance was measured using the AC impedance technique.”

• Logical Consistency– Confidence due to consistency with theory. Type consistent.– “P = .05, as we’d expect from the law on conservation of energy.”

• Associativity Across Views– Individuals: knowledge that two references made from different viewpoints refer to the

same thing– “The region P on the CAD model corresponds to these elements

in the FEA mesh.” (Individuals)– Concepts: knowledge that two conceptualizations can be used for the same purpose.– “What the supplier is calling ‘rated maximum pressure’ is what we call ‘rated pressure.”

(Concepts)

10

9 Elements of Rationale (2)• Change process

– Knowledge of precursors and the history of properties that distinguish them.

– “The value of P that we calculated for this design is close to what we found in earlier models.”

• Authority– The power that information has due to an approval that is granted or

an estimate of its maturity– “Supplier-provided data also suggest P=.05 is obtainable.”

• Origin in Requirements*– Belief that a requirement is sensitive to it– “Our ability to achieve requirement x diminishes as P exceeds 0.07.”

11

9 Elements of Rationale (3)• Origin in organization infrastructure

– Belief because you obtained it in ways consistent with the organization’s best practices.

– “P was obtained from the aero model in the preliminary design library.”

• Consistency with other belief– Belief due to consistency with prevailing contingent facts– “P=0.5 is reasonable in products using component y.”

• V&V Process– Belief that the system in place to manage the other 8 elements is sound

and comprehensive.– “The value of P is confirmed through simulation that is routinely

performed in validation of this product line.”

12

9 Elements : Observations

• Coupling and overlap– Authority / Origin in Organizational Infrastructure– Associativity across views / measurement conditions– etc.

• Though these are found in models, they can be expressed from a more comprehensive viewpoint where – Contradictions can be exposed– Cohesion across views can be noted– Trace to requirements is more evident– (These are all parts of V&V)

13

MBSE Concepts / Logical View

14

Sentence detail / rationale

15

Example Usage Patterns

• V & V– Origin in requirements – Automated generation of test cases

• Requirements Engineering– Origin in other belief, emphasis on tracking

contingent facts and engineering change– Refinement

16

Outline

• Introduction / Scoping• Requirements for MBSE• Exchange Form Validation• NIST Work

17

Exchange Form Validation : Two Methods

1. Axiomatic: How: Map the exchanged content to sentences Identify errors: ex falso quodlibet with a reasoner Advantage: Ontology explains intent Disadvantage: Proofs hard to interpret

2. Metamodel: How: Map the exchanged content to objects Identify errors: Direct structural, with OCL, etc. Advantage: Constraints relate to exchange form Disadvantage: Constraints look like code

18

Example use of metamodel

View / Viewpoint: Can be both consistent with a form (a view), and the form by which otherconceptualization are stated (a viewpoint.)

19

Example from the UML Metamodel

20

Example Specification Constraints

21

In MBSE, metamodels play a key role

• Metamodel =(1) a specification of the form a model can take. (well-formedness conditions)

(2) a formalization of the viewpoint that models will express

22

Metamodels also play a key role in model exchange

• Metamodel =(1) a specification of the form a model can take. (well-formedness conditions)

Definition of structure serialization

(2) a formalization of the viewpoint that models will express

Illuminate what program structures the elementsof exchange content map to/from.

23

Communication with Exchange Standards

24

Outline

• Introduction / Scoping• Requirements for MBSE• Exchange Form Validation• NIST Work– Model Interchange Working Group– Supply Chain Logistics Simulation– Collaborative Requirements Engineering

25

Outline

• Introduction / Scoping• Requirements for MBSE• Exchange Form Validation• NIST Work– Model Interchange Working Group– Supply Chain Logistics Simulation– Collaborative Requirements Engineering

26

OMG Model Interchange Working Group

• Goal: Improve the ability of OMG MOF-based tools (UML, SysML) to exchange information – XMI Serialization – common to MOF-based tools.

• Process– Group: Produce Test Case diagram and reference file– Tool developers: create diagram in their tool, serialize as XMI– Use NIST tool to identify errors (in files and metamodels)– Correct tools and specifications

27

NIST UML / SysML Validator

Enter below a file to upload:

28

NIST UML / SysML Validator

29

NIST UML / SysML Validator

30

NIST UML / SysML Validator

31

MIWG Results• Stakeholders witness significant improvement

in interoperability

Elaasar & Labiche, 2012

32

Outline

• Introduction / Scoping• Requirements for MBSE• Exchange Form Validation• NIST Work– Model Interchange Working Group– Supply Chain Logistics Simulation– Collaborative Requirements Engineering

33

Supply Chain Logistics Simulation

• Goal: Demonstrate integrated use of models toward enterprise goals

• Design: Map models, guided by metamodels into sentences that guide compilation of a discrete event simulation.

• Models– UML of ordering / logistics objects– QVT-r mapping of messages to orders– BPMN “stereotyped” + OCL of business decisions– Discrete Event Simulation

34

Round Trip Engineering of Supply Chain Logistics

35

Logistics Processes (1)

36

Logistics Processes (2)

37

Business Rule

38

Simulation Results

39

Outline

• Introduction / Scoping• Requirements for MBSE• Exchange Form Validation• NIST Work– Model Interchange Working Group– Supply Chain Logistics Simulation– Collaborative Requirements Engineering

40

Collaborative Requirements Engineering

• Goal: Demonstrate Engineering from Product Data Sheets

• Design: Map product data sheets in to sentences about requirements. Use these to guide engineering simulation and reasoning about alternative designs

41

Conclusions

• Continuing roles for deductive reasoning in the automation of SE processes – The nature of V&V, requirements engineering, the

way we think when we engineer, require it.

• Preparing and interpreting macro-level SE decision processes is aided by the integration of multi-viewpoint, micro-level information.

• Metamodels facilitate this integration.

42

References• Welty, C; Inside the mind of Watson, 2nd ESWC Summer School, Kalamaki,

2012, http://videolectures.net/eswc2012_welty_watson

• Denno, P; Thurman, T, Mettenburg, J; Hardy, D; On enabling a model-based systems engineering discipline – 18th INCOSE International Symposium (2008)

• Denno, P; Harrison, T; Using Legacy Modeling Artifacts in Supply Chain Logistics Simulation (in draft, 2013)

• ISO 15288 (2008) – Systems and software engineering – System life cycle processes. (2008)