Slide 1

Risk and Uncertainty in Design Trade StudiesDouglas Van BossuytPhD QualifierJune 11, 2009My BackgroundOSU Triple PlayResearch InterestsCollaborative designComplex system designDesign for culturesPsychologyBusiness management

OutlineOverview of Design Trade StudiesMethods, Tools, Groups Who Perform, ExamplesOverview of Risk and UncertaintyRisk quantification and mitigation toolsUncertainty assessment methodsUncertainty mitigation toolsPossible ways to incorporate risk and uncertainty into Trade StudiesFuture areas of researchExpected contributions of overall researchResearch planIntroductionComplex systems are here to stayEvery complex system design tries to maximize system utilitySystem utility metrics: ROI, system integrity, public perception of project, etcSubsystems optimized to achieve high overall system utility

Introduction (Cont)Design parameters (mass, power, volume, cost, heat dissipation, etc.) used to define subsystem parameters that determine system-level utilityParameters are traded between subsystems to optimize design in Trade StudiesRisk and uncertainty of systems is another important factor in complex system designIntroduction (Cont)DefinitionsRisk: probability of event X impact of eventSometimes more narrowly means probability of catastrophic event X impact of eventUncertainty: caused by variability and doubt in the status of an output that is either predictable or unpredictable, or caused by an unknown process or deviceReliability: Ability of a device to perform as intended over a given period of timeIntroduction (Cont)Definitions (Cont)Robustness: Ability of a device to continue to properly function under changes in input variables.Design Margins: Quantify the influence of uncertainties in the design process. Often a high and low bounding around a central parameter quantification.

MotivationMethods developed from this research will improve system utility and integrityImproved utility and integrity uses resources more efficiently and produces more desirable resultsOutlineOverview of Design Trade StudiesMethods, Tools, Groups Who Perform, ExamplesOverview of Risk and UncertaintyRisk quantification and mitigation toolsUncertainty assessment methodsUncertainty mitigation toolsPossible ways to incorporate risk and uncertainty into Trade StudiesFuture areas of researchExpected contributions of overall researchResearch planDesign Trade StudiesTrade Studies in complex systems designBoth design and decision toolTrade Studies attempt to find maximum system utilitySystem utility defined by many metrics:CostReturn on InvestmentSystem Reliability

Trade Studies MethodsMulti-step process to perform a Trade StudyUnderstand system goals, objectives and constraints (Eg: Functional requirements)Develop alternative conceptual design solutionsEvaluate alternatives based on system utilityUse mathematical models where appropriate to determine system utilityRank alternatives according to their system utilityRemove less desirable alternativesEither refine and continue to eliminate alternatives or select most desirable alternativeTrade Studies Methods (Cont)

Image Source: NASA Systems Engineering HandbookTrade Studies ToolsTrade Studies search for max system utility

Many mathematical ways to find max utilityModern software packages available to find optimum design pointsICEMaker: Used by many Collaborative Design Centers to find optimum designsAdvanced Trade Space Visualization (ATSV): Used to graphically view and explore optimum design points

Trade Studies Tools (Cont)Image Source: https://webhosting.its.psu.edu/atsv/webfiles/glyphscatter/WebStart_files/image003.jpg

ATSV ScreenshotsTrade Studies Tools (Cont)Software (Cont)ModelCenter: Integrates capabilities of ATSV with ability to link together many different types of programsMany other types of software available that help perform Trade StudiesTrade Studies Tools (Cont)

Image Source: Jensen, et. al.: ME 519 Class ProjectGroups Who Perform Trade StudiesMany CDCs exist in government organizations, academia, and industryOriginal is Team-X housed at NASA JPLHelped NASA reduce time to finish Trade Studies from 3-9 months to 2-3 daysReduced cost by a factor of fiveOther NASA facilities with CDCs: Langley Research Center, Goddard, Johnson Space CenterGroups Who Perform Trade Studies (Cont)Image Source: http://jplteamx.jpl.nasa.gov/images/teamx/team.jpg

Groups Who Perform Trade Studies (Cont)European Space Agency uses CDCs and Trade StudiesBoeing, Aerospace Corporation, TRW, and other aerospace companies use Trade StudiesSeveral academic institutions also use trade studiesExamples of Trade StudiesMany examples in literature of Trade StudiesMost come from Team-X but some also from academic institutionsVery few from private industryDue to proprietary information, etcFor those interested, long list of Trade Studies is availableOutlineOverview of Design Trade StudiesMethods, Tools, Groups Who Perform, ExamplesOverview of Risk and UncertaintyRisk quantification and mitigation toolsUncertainty assessment methodsUncertainty mitigation toolsPossible ways to incorporate risk and uncertainty into Trade StudiesFuture areas of researchExpected contributions of overall researchResearch planRisk and Uncertainty in Collaborative Design and Model Based EngineeringOverview of methods to account for riskOverview of uncertainty and how to account for it in design processNote: many methods not reviewed here due to space and time constraintsRisk Methods and ToolsMany methods and tools availableSome used in practice, some only in academiaPracticeRBD, Databases, FMEA/FMECA , ETA, FTA, PRA, QRATheory FFDM, FFIP, RED, HiPHOPS, RUBIC, FFAReliability Block DiagramsUsed for understanding fault toleranceEnergy, information, or material flow through block diagram

Image source: http://www.itemsoft.com/rbd.shtmlDatabasesContain failure and reliability data on systems, subsystems, components, and processesProprietary and industry-specificHigh amount of front-end work to have worthwhile databaseOften used in oil, automotive, and aerospace industriesFailure Modes and Effects (Criticality) AnalysisFMEA used to examine:Potential failures modesEffects of failuresSeverity of the effectsPotential causes of the failuresProbability or potential probability of failureCurrent detection methods of failureDetectability of failureRecommendations to mitigate cause or effects of failureFMEA/FMECA (Cont)FMEA also can be used to assign a Risk Priority NumberRPN = Severity x Occurrence x DetectionSeverity of each failure is ratedLikelihood of each occurrence is ratedLikelihood of prior detection is ratedFMECA is an extension of FMEA. Adds criticality analysis to FMEA.Mode Criticality = Expected Failures X Mode Ratio of Unreliability X Probability of LossItem Criticality = SUM of Mode CriticalitieSFMEA/FMECA (Cont)

Image Source: http://www.weibull.com/basics/fmea_fig1.htmEvent Tree AnalysisETA is visual representation of failure events and mitigating events in a systemUsed in safety system analysisStarting point is failure event Subsequent levels show additional failures and mitigations

Image Source: http://www.event-tree.com/images/et_example.JPGFault Tree AnalysisFTA starts with failure at top-level and proceeds down to analyze all possible causes of failureBoolean operators and logic gates used

Image Source: http://www.isograph-software.com/ftpoverdgc.htm

Probabilistic Risk AssessmentPRA is used to quantify the risk of failure in a systemEmploys FTA, ETA, and other techniques as desiredPRA quantifies risk by magnitude and likelihood of each possible failurePRA is essentially an umbrella for several other risk methodsQualitative Risk AssessmentUsed when quantitative assessment is not possibleNot enough time, money, expertiseRelies on expert opinionsUsually performed by interviewing key designers to determine their belief in the level of risk of a designFunction Failure Design MethodFFDM used to investigate potential failure modes during conceptual designUses failure databases to find failure rates of generic componentsImproves on FMEA and related techniquesFFDM (Cont)

Image Sources: Stone, Tumer, Van Wie: The Function-Failure Design Method Function Failure Identification PropagationFFIP estimates potential failures and their propagation paths through systemsThree components to FFIP:Graphical system modelBehavioral simulationReasoning scheme called Function Failure Logic

FFIP (Cont)

Image Sources: Kurtoglu and Tumer: A Graph-Based Fault Identification and Propagation Framework for Functional Design of Complex SystemsFFIP (Cont)

Risk in Early DesignAn extension of FFDMQuantifies risks identified in FFDMAutomated process for combining historical risk data with new system architecturesUses fever charts to show risksDisplays riskiest failure states RED (Cont)

Image Source: Lough, Stone, Tumer: Implementation Procedures for the Risk in Early Design (RED) MethodHierarchically Performed Hazard Origin and Propagation StudiesHiPHOPS uses elements of FMEA, FTA, and others to assess risk in systemsModel of system is annotated with formalized logical component failure descriptions and expected effectsThis method is too complex to ever gain widespread adoptionRisk and Uncertainty Based Integrated and Concurrent Design MethodologyA continuous risk management toolUsed to identify risk elements during conceptual designRUBIC continuously optimizes budgetary resources to mitigate risksGraphical tool helps find Pareto optimal sets of resource allocationsRUBIC (Cont)

Image Source: Mehr, Tumer: Risk-Based Decision-Making for Managing Resources During the Design of Complex Space Exploration SystemsFunctional Fault AnalysisFFA captures physical system architecture including connections of energy, material, and data flows in a functional modelModel contains sensor information, failure modes of each component, propagation effects of failure modes, and propagation timingApproach requires high level of detail in system before it is usefulOther Risk MethodsState Event Fault Tree AnalysisComponent Fault Tree AnalysisSimulation-Based Probabilistic Risk AnalysisComponent Stress and Conceptual Strength Interference TheoryVarious Bayesian Network Analysis toolsMany othersHow Risk Methods RelateAll try to identify and quantify riskAll good for identifying riskiest points in designsIn practice, lists of failures versus failure paths methodsMost theoretical tools trying to find subsystem and component interaction risksOutlineOverview of Design Trade StudiesMethods, Tools, Groups Who Perform, ExamplesOverview of Risk and UncertaintyRisk quantification and mitigation toolsUncertainty assessment methodsUncertainty mitigation toolsPossible ways to incorporate risk and uncertainty into Trade StudiesFuture areas of researchExpected contributions of overall researchResearch planUncertaintyDefinitions of uncertaintyAssessing System UncertaintyMitigating UncertaintyImage Source: http://www.martin-koser.de/images/enjoy%20uncertainty.jpg

Uncertainty (Cont)Many different ways to define uncertainty and many different places for it to be foundEasiest to think of uncertainty as being made of many different types and falling into two categoriesCategories:Intrinsic: Caused by randomness in natureEpistemic: Caused by lack of knowledge or dataUncertainty (Cont)

Assessing System UncertaintiesSeveral ways to assess uncertainties:Probabilistic MethodsBayesian Techniques1st, 2nd, 3rd level Bayesian AnalysisBayesian Team SupportStimulation MethodsMonte Carlo MethodsProbabilistic MethodsThese methods quantify the behavior of model uncertainties as a result of random model design input variablesThis allows engineers to find variables that are the most sensitive to changeEngineers then concentrate on these variablesBayesian TechniquesBased on Bayesian statistics and probabilityProbability interpreted as a state of knowledgeBayesian probability assumes that posterior probability is proportional to prior probabilityAll Bayesian analysis based on Bayes TheoremBayesian Techniques (Cont)

Bayesian Techniques (Cont)First Level Bayesian Analysis:Used for creating system success rate probabilities based on past success and failure dataSecond Level Bayesian Analysis:Used for systems with no prior data but that are similar to existing systems. Existing system data usedThird Level Bayesian Analysis:Same as Second Level but with normalizing available comparable systems data. Makes best estimate of future system success ratesBayesian Techniques (Cont)Bayesian Team Support helps to solve Arrows ParadoxArrows Paradox is not being able to rank order a groups voting choices (eg: A-B-C-A)BTS assumes all information is Uncertain, incomplete, inconsistent, and evolvingBTS implemented in Accord software packageBTS helps groups make decisions robustly by showing where information needs to be improved, etcBayesian Techniques (Cont)

Image Source: Ullman. Making Robust DecisionsSimulation MethodsMany techniques to simulate systemsMost work by simulating system design or input variablesUse random or semi-random numbersSimulation Methods useful for when direct analytic solution is not availableMonte Carlo MethodsMonte CarloClass of computational algorithmsModels with a high degree of uncertainty in input variables use MCMModel outputs calculated using random or well-chosen semi-random input variables with many repetitions Large dataset is created to adequately explore design spaceMany different algorithms availableOutlineOverview of Design Trade StudiesMethods, Tools, Groups Who Perform, ExamplesOverview of Risk and UncertaintyRisk quantification and mitigation toolsUncertainty assessment methodsUncertainty mitigation toolsPossible ways to incorporate risk and uncertainty into Trade StudiesFuture areas of researchExpected contributions of overall researchResearch planMitigating UncertaintyNatural Uncertainties:Hard to address without changing natural environmentEg: dikes to hold back rivers, oceans; bridges over bodies of water; snow sheds for rail linesModel Structure and Parameter Uncertainties:Usually addressed by refining model and parameters Mitigating Uncertainty (Cont)Data UncertaintiesHard to discover and diagnose. If found then:Inaccurate measurements: Better measurement toolsData gauging network problems: Improve networksData handling and transcription errors: Fix processAlternative approach: Relax requirements so uncertainties are acceptableMitigating Uncertainty (Cont)Computational UncertaintiesFaulty hardware: Very hard to find and diagnose. If found, replace bad hardware.Faulty software: Also hard to find. Fix software as needed. (Truncation and rounding errors can be fixed with longer integer registers, etc)Operational UncertaintiesUse Total Quality Management, Human Factors Engineering, etc to reduce this uncertaintyBehavioral and Ambiguity UncertaintiesVarious corrective behavioral techniques available

OutlineOverview of Design Trade StudiesMethods, Tools, Groups Who Perform, ExamplesOverview of Risk and UncertaintyRisk quantification and mitigation toolsUncertainty assessment methodsUncertainty mitigation toolsPossible ways to incorporate risk and uncertainty into Trade StudiesFuture areas of researchExpected contributions of overall researchResearch planIncorporating Risk and Uncertainty into Trade StudiesNo examples beyond Thunnissen of risk and uncertainty being traded in Trade StudiesThunnissen proposed a method to do thisThis method is rudimentary and needs to be expandedThunnissen also proposed a method of design margins for Trade Studies. Also needs to be implemented.Incorporating (Cont)Thunnissen was influenced by Antonsson and othersGrayscale System Reliability: Quantifies influence of partial failure states on system integrity and reveals tradeoffs between system reliability and costMethod of Imprecision: Represents uncertainty and preference on 0-1 scale. System to trade attribute levels without allowing any attribute to go to zero performance.Incorporating (Cont)Some interest at Team-X to implementThunnissons methodsBayesian Team SupportRisk methodsWork conducted at JPL this summer to integrate some methods into Trade Studies using ModelCenter (replacing ICEMaker)OutlineOverview of Design Trade StudiesMethods, Tools, Groups Who Perform, ExamplesOverview of Risk and UncertaintyRisk quantification and mitigation toolsUncertainty assessment methodsUncertainty mitigation toolsPossible ways to incorporate risk and uncertainty into Trade StudiesFuture areas of researchExpected contributions of overall researchResearch planFuture Areas of ResearchThe valuation and perception of risk and uncertainty is not well accounted for in the complex system design processIt is researched in insurance industry, stock market, and elsewhere. Need to move this knowledge into a workable form for engineers.Cultural factors that affect risk and uncertainty need to be better understood by engineersSeveral methods in business to quantify culture. Need to move and adapt into engineering.Expected Contributions of Overall ResearchMethod to maximize system utility and integrity via expanding Trade StudiesRisk methodsUncertainty methodsDesign MarginsRisk and uncertainty perception and valuationCultural methodsIntegration of method with ModelCenterResearch PlanMay 09Complete PhD qualifier. Research design Trade Studies andrisk and uncertainty. Prepare for summer internship at JPL.June-August 09Work with Steve Wall and Team-X at JPL. Prepare first example of design margins in Trade Studies using ModelCenter for Team-X. September 09Submit paper to Journal of Engineering Design detailing work at JPL.Sept-Dec 09Revise design margins in Trade Studies tool. Start work on integrating risk and uncertainty into Trade Studies.January 10Submit conference papers to IDETC and others.Jan-March 10Take final required coursework. Continue work on integrating risk and uncertainty into Trade Studies.April-May 10PhD Prelim. Publish journal paper on integrating risk and uncertainty into Trade Studies.Research Plan (Cont)June-Aug 10Work either at JPL, overseas, or another company to continue work. Test risk and uncertainty methods in an industrial setting. Begin investigating perception and valuation of risk, and cultural constructs of risk.Sept-Dec 10Finish work on integrating risk and uncertainty into TradeStudies. Implement valuation and perception metrics, andcultural standards.January 11Submit conference papers to IDETC, ICED and others.Jan-March 11Submit journal paper.April-June 11Prepare and defend dissertation. Take long vacation.Questions?Why so focused on aerospace?Why worthy of a PhD?