Upload
leanleadersorg
View
2.342
Download
5
Tags:
Embed Size (px)
Citation preview
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
National GuardBlack Belt Training
Module 47
Basic Design of Experiments (DOE)
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
2
CPI Roadmap – Improve
Note: Activities and tools vary by project. Lists provided here are not necessarily all-inclusive.
TOOLS
•Brainstorming
•Replenishment Pull/Kanban
•Stocking Strategy
•Process Flow Improvement
•Process Balancing
•Standard Work
•Quick Change Over
•Design of Experiments (DOE)
•Solution Selection Matrix
• ‘To-Be’ Process Mapping
•Poka-Yoke
•6S Visual Mgt
•RIE
ACTIVITIES• Develop Potential Solutions
• Develop Evaluation Criteria
• Select Best Solutions
• Develop Future State Process Map(s)
• Develop Pilot Plan
• Pilot Solution
• Develop Full Scale Action/
Implementation Plan
• Complete Improve Gate
1.Validate the
Problem
4. Determine Root
Cause
3. Set Improvement
Targets
5. Develop Counter-
Measures
6. See Counter-MeasuresThrough
2. IdentifyPerformance
Gaps
7. Confirm Results
& Process
8. StandardizeSuccessfulProcesses
Define Measure Analyze ControlImprove
8-STEP PROCESS
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
33Basic Design of Experiments
Learning Objectives
Learn benefits of DOE methodology
Discuss differences between DOE and trial and error (one-factor-at-a-time) approaches to experimentation
Learn basic DOE terminology
Distinguish between the concepts of full and fractional factorial designs
Use Minitab to run and analyze a DOE
Use results of DOE to drive statisticallysignificant improvements
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
4
Helicopter SimulationPhase One
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
5
Exercise: Helicopter Simulation
Customers at CHI (Cellulose Helicopters Inc.) have been complaining about the limited flight time of CHI helicopters
Management wants to increase flight time to improve customer satisfaction
You are put in charge of this improvement project
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
6
Exercise: Constraints
Project Mission: Find the combination of factors that maximize flight time
Project Constraints:
Budget for testing = $1.5 M
Cost to build one prototype = $100,000
Cost per flight test = $10,000
Prototype once tested can not be altered
See allowable flight test factors and parameters on the next page
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
7
Exercise: Test Factors and Parameters
Paper Type Regular Card stock
Paper Clip No Yes
Taped Body No 3 in of tape
Taped Wing Joint No Yes
Body Width 1.42 in 2.00 in
Body Length 3.00 in 4.75 in
Wing Length 3.00 in 4.75 in
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
8
Exercise: Roles & Responsibilities
Lead Engineer – Leads the team and makes final decision on which prototypes to build and test
Test Engineer – Leads the team in conducting the test and has final say on how test are conducted
Assembly Engineer – Leads the team in building prototypes and has final say on building issues
Finance Engineer – Leads the team in tracking expenses and keeping the team on budget
Recorder – Leads the team in recording data from the trials
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
9
Exercise: Phase One Deliverables
Prepare a Phase One Report showing:
Recommendation for optimal design
Predicted flight time at optimal setting
How much money was spent
Description of experimental strategy used
Description of analysis techniques used
Recommendations for future tests
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
10
Introduction to DOE
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
1111Basic Design of Experiments
1. Significant Event 2. Somebody Sees It
3. Research How can we learn more efficiently?
How Do We Learn?
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
1212Basic Design of Experiments
How Do We Learn? (Cont.)
Products and processes are continually providing data that could lead to their improvement - so what has been missing? There are several possibilities:
We are not collecting and analyzing the data provided
We are not proactive in data collection
We are unable to translate the data into information
A significant event has not occurred
“In order to learn, two things must occur simultaneously: something must happen (informative event) and someone must see it happen (perceptive observer).” – George Box
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
1313Basic Design of Experiments
How Do We Improve?
By creating significant events and observing them, we can obtain knowledge faster
That is basically what occurs in a designed experiment
Let‟s look at an example of these two things occurring (significant event and perceptive observer) simultaneously
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
1414Basic Design of Experiments
Champagne Example
Wine – The fermented juice of fresh grapes used as a beverage. Wine has been in existence since the beginning of recorded history
Champagne – A clear, sparkling liquid made by way of the second fermentation of wine. First discovered by a French monk in the late 1600s
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
1515Basic Design of Experiments
SPC tools and techniques improve observation, but we must wait for an event to happen in order to observe it
Need Improved Observation
Need to make sure that naturally occurring informative events are brought to the attention of the perceptive observer!
Improved observation increases the probability of observing naturally occurring informative events so appropriate action can be taken.
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
1616Basic Design of Experiments
Passive Observation Is Not Enough
We need to induce occurrence of informative events.
An experiment is set-up so that an informative event will occur!
By manipulating inputs to see how the output changes, we can understand and model Y (a dependent variable) as a function of X (an independent variable).
Designed Experimentation – The manipulation of controllable factors (independent variables) at different levels
to see their effect on some response (dependent variable)
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
1717Basic Design of Experiments
Experimental Process:
A controlled blending
of inputs which
generates corresponding
measurable outputs.
People
Material
Equipment
Policies
Procedures
Methods
Environment
Responses related toperforming a service
Responses related toproducing a product
Responses related tocompleting a task
Inputs (Factors) Outputs (Responses)
From Understanding Industrial Designed Experiments, Schmidt & Launsby
What Is Experimental Design?
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
1818Basic Design of Experiments
Example of a Recruiting Process
Process:
Recruiting
Job Description
Marketing
Candidate Pool
Hire Quickly
Inputs(Factors)
Outputs (Responses)
Economic Environment
Type of Job
Hire Best Candidate
Location of Job
Job Application Process
EEO Requirements Hire at Competitive Pay
DOE was originally used for manufacturing quality applications - it has now expanded to many other areas where performance characteristics are of interest
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
1919Basic Design of Experiments
Methods of Experimentation
Experimentation has been used for a long time. Some experiments have been good, some not so good
Our early experiments can be grouped into the following general categories:
1. Trial and Error
2. One-Factor-at-a-Time (OFAT)
3. Full Factorial
4. Fractional Factorial
5. Others
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
2020Basic Design of Experiments
Trial and Error - Increase Gas Mileage
Problem: Gas mileage for car is 20 mpg. Would like to get > 30 mpg.
Factors:
Change brand of gas
Change octane rating
Drive slower
Tune-up car
Wash and wax car
New tires
Change tire pressure
Remove hood ornament and external radio antenna
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
2121Basic Design of Experiments
Problem: Gas mileage for vehicle is 20 mpg. Would like to get > 30 mpg
How many more runs would you need to figure out the best configuration of variables?
How can you explain the above results?
If there were more variables, how long would it take to get a good solution?
What if there‟s a specific combination of two or more variables that leads to the best mileage (the optimum)?
MPG Results
One-Factor-at-a-Time (OFAT)- Gas Mileage
Speed Octane Tire Pressure Miles per Gallon
55 85 30 25
65 85 30 23
55 91 30 27
55 85 35 27
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
2222Basic Design of Experiments
Results
How would we find this optimum with OFAT testing?
How would we know that we‟d found it?
Miles per Gallon as a Function of Speed and Tire Pressure
26 28 30 32 34 36 38
75
70
65
60
55
50
45
40
35
30
18 23 26 26 20
17
26
26
18
Tire Pressure (lbs.)
Sp
ee
d (
mp
h)
35Optimum MPG
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
2323Basic Design of Experiments
Is there a better way?
One-Factor-at-a-Time (OFAT)
While OFAT is simple, it is inefficient in determining optimal results:
Unnecessary experiments may be run
Time to find causal factors is significant
Don‟t know the effects of changing one factor while other factors are also changing (no model)
Inability to detect or learn about how factors work together to drive the response
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
2424Basic Design of Experiments
Cake Example - Interactions
An Interaction occurs when the effect of one factor, X1, on the response, Y, depends on the setting (level) of another factor, X2:
Y = f(x)
For example, when baking a cake, the temperature that you set the oven at is dependent on the time that the cake will be in the oven.
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
2525Basic Design of Experiments
Cake Example - Interactions
Where would you set Time to get a good cake?
How would you experiment on this process to learn about this interaction?
Tim
e
Temp = 500 degreesTime = 20 minutes
Temp = 500 degreesTime = 45 minutes
Temp = 100 degreesTime = 45 minutes
Temp = 100 degreesTime = 20 minutes
Temp100 500
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
2626Basic Design of Experiments
Cake Example - Interactions
Duncan Hines used designed experiments in the 50‟s on their cake mixes.
Their goal was a robust design for the most consistent product.
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
2727Basic Design of Experiments
Why Use DOE?
The structured methodology provides a directed approach to avoid time wasted with “hunt and peck” - don‟t need 30 years of experience to design the tests
The designed experiment gives a mathematical model relating the variables and responses - no more experiments where you can‟t draw conclusions
The model is easily optimized, so you know when you‟re done
The statistical significance of the results is known, so there is much greater confidence in the results
Can determine how multiple input variables interact to affect results
“Often we have used a trial and error approach to testing, or just changed one variable at a time. Why is a statistically
designed experiment better?”
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
2828Basic Design of Experiments
Full Factorial DOE
Full Factorial examines every possible combination of factors at the levels tested. The full factorial design is an experimental strategy that allows us to answer most questions completely.
Full factorial enables us to:
Determine the Main Effects that the factors being manipulated have on the response variable(s)
Determine the effects of factor interactions on the response variables
Estimate levels at which to set factors for best results
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
2929Basic Design of Experiments
What can we do when resources are limited?
Minimum number of tests for a full factorial experiment: Xk
X = # of levels, k = # of factors
Adding another level significantly increases the number of tests!
Level 2 3 4
2 4 8 16
3 9 27 81
Factors
# of Tests
Full Factorial
Full Factorial Advantages
Information about all effects
Information about all interactions
Quantify Y=f(x)
Limitations
Amount of resources needed
Amount of time needed
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
3030Basic Design of Experiments
Full Factorial Notation
2 level designs are the most common because they provide a lot of information, but require the fewest tests.
The general notation for a full factorial design of 2 levels is:
2 is the number of levels for each factor (Range = High and Low)
k is the number of factors to be investigated
This is the minimum number of test runs required for a full factorial
2k = # Runs
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
3131Basic Design of Experiments
OFAT Runs
Problem: Gas Mileage is 20 mpg
What conclusion do you make now?
How many runs?
How many runs at each level?
Speed Octane Tire Pressure Miles per Gallon
55 85 30 25
65 85 30 23
55 91 30 27
65 91 30 23
55 85 35 27
65 85 35 24
55 91 35 32
65 91 35 25
MPG = f(Speed, Octane, Tire Pressure)
Full Factorial Experiment
Do we think 32 is best?
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
3232Basic Design of Experiments
Fractional Factorial
Looks at only a fraction of all the possible combinations contained in a full factorial.
If many factors are being investigated, information can be obtained with smaller investment.
Resources necessary to complete a fractional factorial are manageable.
Limitations - give up some interactions
Benefits
Economy
Speed
Fewer runs
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
3333Basic Design of Experiments
Fractional Factorial Notation
The general notation to designate a fractional factorial design is:
2 is the number of levels for each factor
k is the number of factors to be investigated
2-p is the size of the fraction (p = 1 1/2 fraction, p = 2 1/4 fraction, etc.)
2k-p is the number of runs
R is the resolution
pkR2 = # Runs
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
3434Basic Design of Experiments
Fractional Factorial Notation – Resolution
When we go to a fractional factorial design, we are not able to estimate all of the interactions
The amount that we are able to estimate is indicated by the resolution of an experiment
The higher the resolution, the more interactions we can measure Example: The designation below means fifteen factors will be
investigated in 16 runs. This design is a resolution III:
11152
III Note: A deeper discussion of design resolution is beyond the scope of the lesson. The content, above, is intended to only provide a brief explanation of the design resolution term.
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
3535Basic Design of Experiments
Speed
(A)
Octane
(B)
Tire Pressure
(C)
Mileage
(Y)
55 85 35 27
65 85 30 23
55 91 30 27
65 91 35 25
Gas Mileage Example
Problem: Gas mileage for vehicle is 20 mpg
Compare with previous full factorial:
How many runs?
How much information?
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
3636Basic Design of Experiments
A1 A2
Speed Octane Tire Pressure
55 65 85 91 30 35
23.8
24.8
25.8
26.8
27.8
Mil
ea
ge
Main Effects Plot (data means) for Mileage
In the gas mileage example, Speed,Octane, and Tire
Pressure all look to have an effect on average mileage.
DOE Will Help Us Identify Factors
Factors which shift the average
Longer line = greater effect
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
3737Basic Design of Experiments
B2
B1
Speed Octane Tire Pressure
55 65 85 91 30 35
1.0
1.5
2.0
2.5
3.0
Sta
nd
ard
De
v
Main Effects Plot for Standard Deviation
Only Tire Pressureis affecting standard
deviation
DOE Will Help Us Identify Factors
Factors which affect variation
Flat line = no effect
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
3838Basic Design of Experiments
C1C2
Speed Octane Tire Pressure
55 65 85 91 30 35
1.0
1.5
2.0
2.5
3.0
Sta
nd
ard
De
v
Main Effects Plot for Standard Dev
Speed Octane Tire Pressure
55 65 85 91 30 35
23.8
24.8
25.8
26.8
27.8
Mil
ea
ge
Main Effects Plot (data means) for Mileage
Only Tire Pressure affects both the average mileage and also the variability
DOE Will Help Us Identify Factors
Factors which shift the average and affect variation
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
3939Basic Design of Experiments
D1 = D2
Driver Radio
-1 1 -1 1
24
25
26
27
28
Mil
ea
ge
Main Effects Plot (data means) for Mileage
Driver Radio
-1 1 -1 1
1.0
1.5
2.0
2.5
3.0
Sta
nd
ard
De
v
Main Effects Plot for Standard Dev
An expanded study investigated the effect of driver and radio on mileage. These factors show no effect. This is also valuable information, because these factors can be set at their most economical (least cost) or
most convenient levels.
DOE Will Help Us Identify Factors
Factors which have no effect
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
4040Basic Design of Experiments
Benefits of DOE
Determine input settings which optimize results and minimize costs
Quick screening for significant effects
Obtain a mathematical model relating inputs and results
Reduction in the number of tests required
Verification of the statistical significance of results
Identification of low-impact areas allows for increased flexibility/tolerances
Standardized methodology provides a directed approach
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
4141Basic Design of Experiments
When Could I Use Design of Experiments?
Identification of critical factors to improve performance
Identification of unimportant factors to reduce costs
Reduction in cycle time
Reduction of scrap/rework
Scientific method for setting tolerances
Whenever you see repetitive testing
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
4242Basic Design of Experiments
DOE Review
What does DOE offer us that trial and error experimentation and OFAT do not?
What are the differences between full and fractional factorial DOE‟s?
What is the minimum number of runs required for a 2-level, 3-factor full factorial experiment?
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
4343Basic Design of Experiments
Minitab: Airline DOE Example
A contract airline is interested in reducing overall late take-off time in order to improve Soldier satisfaction
Previous Black Belt work has identified 4 key process input variables (KPIVs) that affect late time:
Dollars spent on training
Number of jets
Number of employees
% Overbooked
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
4444Basic Design of Experiments
Using the Airline DOE Data.mpj file and Minitab, the instructor will walk the class through the following activities:
A DOE to identify which factors affect “minutes late” in
terms of both the mean and standard deviation
Use the DOE results to determine new process settings
Hypothesis test to prove statistical significance of change.
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
4545Basic Design of Experiments
Current settings for these factors are as follows:
Dollars spent on training 200
Number of jets 52
Number of employees 850
% Overbooked 15
The target is zero minutes late, with a specification of +/- 10 minutes.
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
4646Basic Design of Experiments
Our goal in this experiment is to reduce late take-off times - we will measure late time in minutes
Here are the factors and their levels that we are going to investigate:
Factors Levels
Dollars spent on training 100 300
Number of Jets 50 55
Number of Employees 800 900
% Overbooked 0 25
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
4747Basic Design of Experiments
First, we need to set up the test matrix
Select Stat>DOE>Factorial>Create Factorial Design
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
4848Basic Design of Experiments
We left the default at: 2-level factorial design. That means we will test each factor at 2 different levels
We also selected 4 factors, since there are 4 variables that we want to test in this experiment. Select Display Available Designs to display possible experiments we can run…
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
4949Basic Design of Experiments
For 4 factors, we can either do an 8 run half-fraction or a 16 run full factorial. We will go with the 16 run full factorial experiment.
Click OK to return to the previous dialog box.
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
5050Basic Design of Experiments
Click the Designs button and highlight the 16-run Full Factorial design.
Leave the other settings at their defaults, click on OK.
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
5151Basic Design of Experiments
Next, click the Factors button
Enter the names of the Factors – Change -1 and 1 to actual levels per chart below
Minitab: Airline DOE Example
FactorsLevelsDollars spent on training 100
300Number of Jets 50 55Number of Employees 800
900% Overbooked 0 25
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
5252Basic Design of Experiments
Click the Options button and uncheck Randomize runs.
We do want to randomize our tests when we actually run an experiment. However, for this in-class demo, it will be easiest if everyone‟s screen is the same.
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
5353Basic Design of Experiments
Here are the tests that we need to run. Example, row 1 indicates that we first need to collect data at the low level for all four factors.(Tip: First check that you have the same test matrix. If you don‟t, it‟s likely that you did not uncheck “Randomize Runs.”)
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
5454Basic Design of Experiments
Next, we collect the data.
To allow us to measure variation, we need to run 3 repetitions at each set of settings.
Copy the data from the DOE data worksheet and paste into the design as shown below.
Min Late 1 = C9
Min Late 2 = C10
Min Late 3 = C11
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
5555Basic Design of Experiments
Run #1, with settings of $100 spent on training, 50 jets, 800 employees, and 0% overbooked, was 46.35 minutes late on the first repetition, 61.92 minutes late on the second repetition, and 75.18 minutes late on the third repetition.
$100.00 50 Jets 800 Employees 0% Overbooked
Note: This row of coded variables were all at their Low (or –1) settings
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
5656Basic Design of Experiments
Since we ran our DOE with 3 repetitions, and we want to analyze the variation in our DOE results, we need to prepare the worksheet by having Minitab calculate means and standard deviations.
First, we need to name some blank columns. Name a blank column StdDev Min Late, a second blank column Count Min Late, and a third blank column Mean Min Late.
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
5757Basic Design of Experiments
Now, we will have Minitab calculate the means and standard deviations.
Select Stat>DOE>Factorial>Pre-Process Responses for Analyze Variability
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
5858Basic Design of Experiments
Click in the box Store standard deviations in and select the column you named StdDev Min Late
Click in the box Store number of repeats in and select the column you named Count Min Late
Click in the box Store Means (optional) in and select the column you named Mean Min Late
Click OK
Click on Compute for repeat responses across rows, then click in the cell under Repeat responses across rows of: and select Min late 1, Min late 2, and Min late 3 from the columns pane.
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
5959Basic Design of Experiments
Minitab calculates means and standard deviations for each combination of factors. (Remember: there were 24, or 16, combinations.)
Minitab also determines the counts. (Remember: there were 3 data points at each combination, since we ran 3 repetitions at each setting of the DOE.)
Looking at this data Practically, there appears to be some significance to the factors, but nothing definitive…yet.
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
6060Basic Design of Experiments
Before we view the statistics, we always start with the graphs.
Select Stat>DOE>Factorial>Factorial Plots.
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
6161Basic Design of Experiments
Select Main Effects Plot.
Choose Setup and click on OK to go to next dialog box
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
6262Basic Design of Experiments
Select StdDev Min Late and Mean Min Late for Responses,and move all four factors from Available to the Selected box to have them included in the analysis. Click on OK.
>
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
6363Basic Design of Experiments
1-1
60
45
30
15
0
1-1
1-1
60
45
30
15
0
1-1
Training Dollars
Me
an
Jets
Employees %Overbooked
Main Effects Plot for Mean Min LateData Means
The Main Effects Plot shows that the number of Employees is the only driver for Mean Min Late
Looking at this data Graphically, it appears that Employees might be a significant factor influencing the Mean of Time Late
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
6464Basic Design of Experiments
1-1
8
6
4
2
1-1
1-1
8
6
4
2
1-1
Training Dollars
Me
an
Jets
Employees % Overbooked
Main Effects Plot for StdDev Min LateData Means
The Main Effects Plot shows that the number of Jets AND Employees are driving the StdDev Min Late
Looking at this data Graphically, it appears that Jets AND Employees might be a significant factor influencing the StdDev of Time Late
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
6565Basic Design of Experiments
Now we will run the analysis.
Select Stat>DOE>Factorial>Analyze Factorial Design
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
6666Basic Design of Experiments
Select Mean Min Late and StdDev Min Late as the Response.
Choose Terms to get to next dialog box
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
6767Basic Design of Experiments
Include Terms in the model up through second order (2). This will include the main effects and two-way interactions. Click on OK to go back to previous dialog box.
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
6868Basic Design of Experiments
Click on Graphs and select the Pareto Chart. Click OK in both dialog boxes.
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
6969Basic Design of Experiments
This chart confirms what we saw earlier in the Main Effects Plot – the number of Employees has a significant impact on Mean Min Late. We also see that the interaction term BD* is significant.
CD
AC
AB
A
B
BC
AD
D
BD
C
9080706050403020100
Term
Standardized Effect
2.57
A Training Dollars
B Jets
C Employ ees
D %O v erbooked
Factor Name
Pareto Chart of the Standardized Effects(response is Mean Min Late, Alpha = 0.05)
Pareto Chart forMean Min Late
* - BD is the interaction between the factors Jetsand %Overbooked.
This is the „Critical F-statistic‟ used to determine significance.
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
7070Basic Design of Experiments
A
D
CD
BD
AB
BC
AC
C
AD
B
43210
Te
rm
Standardized Effect
2.571
A Training Dollars
B Jets
C Employ ees
D % O v erbooked
Factor Name
Pareto Chart of the Standardized Effects(response is StdDev Min Late, Alpha = 0.05)
This chart confirms only part of what we saw earlier in the Main Effects Plot – the number of Jets has a significant impact on StdDev Min Latebut Employees does not.
Pareto Chart forStdDev Min Late
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
7171Basic Design of Experiments
Factorial Fit: Mean Min Late versus Training Dollars, Jets, ...
Estimated Effects and Coefficients for Mean Min Late (coded units)
Term Effect Coef SE Coef T P
Constant 30.42 0.3191 95.34 0.000
Training Dollars -0.71 -0.35 0.3191 -1.11 0.319
Jets -0.77 -0.39 0.3191 -1.21 0.279
Employees -53.89 -26.94 0.3191 -84.44 0.000
%Overbooked -1.28 -0.64 0.3191 -2.01 0.101
Training Dollars*Jets 0.68 0.34 0.3191 1.06 0.337
Training Dollars*Employees 0.12 0.06 0.3191 0.18 0.861
Training Dollars*%Overbooked 1.13 0.56 0.3191 1.77 0.138
Jets*Employees -0.81 -0.41 0.3191 -1.27 0.259
Jets*%Overbooked 2.14 1.07 0.3191 3.36 0.020
Employees*%Overbooked -0.10 -0.05 0.3191 -0.16 0.881
S = 1.27632 PRESS = 83.4049
R-Sq = 99.93% R-Sq(pred) = 99.28% R-Sq(adj) = 99.79%
This data shows Analytically that Employees and the Jets*% Overbooked interaction are statistically significant.
In the Session window, we see that Employees and the interaction Jets*%Overbooked are the only statistically significant factors for Mean Min Late. All other main effects and 2-way interactions have a p-value > 0.05.
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
7272Basic Design of Experiments
Factorial Fit: StdDev Min Late versus Training Dollars, Jets, ...
Estimated Effects and Coefficients for StdDev Min Late (coded units)
Term Effect Coef SE Coef T P
Constant 4.575 0.7241 6.32 0.001
Training Dollars -0.105 -0.052 0.7241 -0.07 0.945
Jets -5.930 -2.965 0.7241 -4.09 0.009
Employees -2.477 -1.239 0.7241 -1.71 0.148
% Overbooked 0.146 0.073 0.7241 0.10 0.923
Training Dollars*Jets -0.987 -0.493 0.7241 -0.68 0.526
Training Dollars*Employees 2.032 1.016 0.7241 1.40 0.219
Training Dollars*% Overbooked 2.786 1.393 0.7241 1.92 0.112
Jets*Employees 1.655 0.828 0.7241 1.14 0.305
Jets*% Overbooked -0.935 -0.468 0.7241 -0.65 0.547
Employees*% Overbooked 0.932 0.466 0.7241 0.64 0.548
S = 2.89641 PRESS = 429.527
R-Sq = 84.84% R-Sq(pred) = 0.00% R-Sq(adj) = 54.52%
In the Session window, we see that Jets is the only statistically significant factor for Stdev Min Late. The negative sign for Effectindicates that standard deviation decreases as Jets increases. All other main effects and 2-way interactions have a p-value > 0.05.
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
7373Basic Design of Experiments
Summarizing what we have found in the initial analysis: Employees and the interaction between Jets and %Overbooked
had a significant impact on Mean Min Late.
As seen from the Effects, increasing Jets or %Overbookeddecreases Mean Min Late.
In addition, when the product of Jets*%Overbooked is positive, Mean Min Late will increase. If the product is negative, Mean Min Late will decrease.
Jets had a significant impact on Stdev Min Late. As seen from its Effect, increasing Jets decreases Stdev.
Term Effect Coef SE Coef T P
Employees -53.89 -26.94 0.3191 -84.44 0.000
Jets -0.77 -0.39 0.3191 -1.21 0.279
%Overbooked -1.28 -0.64 0.3191 -2.01 0.101
Jets*%Overbooked 2.14 1.07 0.3191 3.36 0.020
Term Effect Coef SE Coef T P
Jets -5.930 -2.965 0.7241 -4.09 0.009
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
7474Basic Design of Experiments
The next step in the DOE analysis is to eliminate the insignificant terms. This is called “reducing the model.”
Every study is different; in this particular case, let‟s take the following approach:
Reduce the StdDev model to identify the needed setting for „Jets‟ since it:
is the only significant factor influencing StdDev
plays only a small role in driving the Mean (Coef = -0.39)
will determine where to set the factor % Overbooked in the interaction term.
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
7575Basic Design of Experiments
Select Stat>DOE>Factorial>Analyze Factorial Design
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
7676Basic Design of Experiments
1. Click on Terms
2. Remove all Selected Terms: except B:Jets
3. Select OK and OK
1
2
3
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
7777Basic Design of Experiments
The mathematical model is taken from the „Coef‟ column of the Session Window: StdDev Min Late* = 4.575 – (2.965 x Jets)
Factorial Fit: StdDev Min Late versus Jets Estimated Effects and Coefficients for StdDev Min Late (coded units)
Term Effect Coef SE Coef T PConstant 4.575 0.7792 5.87 0.000Jets -5.930 -2.965 0.7792 -3.81 0.002
S = 3.11695 PRESS = 177.653R-Sq = 50.84% R-Sq(pred) = 35.79% R-Sq(adj) = 47.33%
Conclusion: To reduce StdDev Min Late, we should set the factor „Jets‟ to the +1 level (55).
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
7878Basic Design of Experiments
Now, let‟s reduce the model for the response, Mean.
Select Stat>DOE>Factorial>Analyze Factorial Design
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
7979Basic Design of Experiments
1. Click on Terms
2. Remove all Selected Terms: except B:Jets, C: Employees, D: % Overbooked and the interaction BD.
3. Select OK and OK
1
2
3
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
8080Basic Design of Experiments
The mathematical model is taken from the „Coef‟ column of the Session Window:
Mean Min Late = 30.42 – (0.39 x Jets) – (26.94 x Employees)–(0.64 x %Overbooked) + (1.07 x Jets x %Overbooked)
Factorial Fit: Mean Min Late versus Jets, Employees, %Overbooked
Estimated Effects and Coefficients for Mean Min Late (coded units)
Term Effect Coef SE Coef T P
Constant 30.42 0.3354 90.71 0.000
Jets -0.77 -0.39 0.3354 -1.15 0.273
Employees -53.89 -26.94 0.3354 -80.34 0.000
%Overbooked -1.28 -0.64 0.3354 -1.91 0.082
Jets*%Overbooked 2.14 1.07 0.3354 3.19 0.009
S = 1.34148 PRESS = 41.8811
R-Sq = 99.83% R-Sq(pred) = 99.64% R-Sq(adj) = 99.77%
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
8181Basic Design of Experiments
Of the four factors that were investigated, two (Employees and Jets), plus the Jets*%Overbooked interaction, were significant.
Jets – to reduce variation, we need to increase Jets to 55.
Employees - to reduce the average late time from 30 minutes, we need to increase Employees from 850 to 900.
Training Budget - had no effect, and can be reduced to $100k as a budget savings.
% Overbooked - had marginal effect on time late and on variation – should be reduced to 0% to improve customer satisfaction.
What would you do if this were your organization?
Minitab: Airline DOE Example
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
8282Basic Design of Experiments
Is the Change Significant?
1. Conduct a hypothesis test.
2. Open the Capability Data worksheet within the Airline DOE Data.mpj file.
3. Since we have the baseline sample and the improved sample, select Stat>Basic Statistics>2-Sample t…
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
8383Basic Design of Experiments
Is the Change Significant? (continued)
4. Select „Samples in different columns‟
5. Select „Baseline Data‟ for First: and „New Data‟ for Second:
6. Select „Graphs…‟
4
5
6
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
8484Basic Design of Experiments
Is the Change Significant? (continued)
7. Select Boxplots of data
8. Click on OK
9. Interpret boxplot. Does there appear to be a graphical difference?
7
89
New DataBaseline Data
50
40
30
20
10
0
Da
ta
Boxplot of Baseline Data, New Data
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
8585Basic Design of Experiments
Is the Change Significant? (continued)
10.Review Minitabs Session Window output.
11.Can we state, with 95% confidence, that there is a statistical difference between our Baseline Data and the New data? (i.e. Does the Confidence Interval contain „0‟ or is the P-value less than 0.05?)
10
11
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
86
Helicopter SimulationPhase Two
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
87
Exercise: Helicopter Simulation
Customers at CHI (Cellulose Helicopters Inc.) have been complaining about the limited flight time of CHI helicopters
Management wants to increase flight time to improve customer satisfaction
You are put in charge of this improvement project
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
88
Exercise: Constraints
Project Mission: Find the combination of factors that maximize flight time
Project Constraints:
Budget for testing = $1.5 M
Cost to build one prototype = $100,000
Cost per flight test = $10,000
Prototype once tested can not be altered
See allowable flight test factors and parameters on the next page
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
89
Exercise: Test Factors and Parameters
Paper Type Regular Card stock
Paper Clip No Yes
Taped Body No 3 in of tape
Taped Wing Joint No Yes
Body Width 1.42 in 2.00 in
Body Length 3.00 in 4.75 in
Wing Length 3.00 in 4.75 in
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
90
Exercise: Roles & Responsibilities
Lead Engineer – Leads the team and makes final decision on which prototypes to build and test
Test Engineer – Leads the team in conducting the test and has final say on how test are conducted
Assembly Engineer – Leads the team in building prototypes and has final say on building issues
Finance Engineer – Leads the team in tracking expenses and keeping the team on budget
Recorder – Leads the team in recording data from the trials
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
91
Exercise: Phase Two Deliverables
Prepare a Phase Two Report showing:
Recommendation for optimal design
Predicted flight time at optimal setting
How much money was spent
Description of experimental strategy used
Description of analysis techniques used
Recommendations for future tests
Comparison of Phase One and Phase Two approaches
Which Team Has The Best Design?
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
9292Basic Design of Experiments
Takeaways
Types of experiments – Trial and Error, OFAT, DOE
Introductory DOE terminology
Benefits of full factorial vs. fractional designs
How to use Minitab to design, run, and analyze a DOE
Use DOE results to drive statistical improvements
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
93
What other comments or questions
do you have?
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
9494Basic Design of Experiments
References
Schmidt & Launsby, Understanding Industrial Designed Experiments