Statistics and experimental design

EPSRC, Cambridge March 2008

Statistics and Experimental Design

Shirley Coleman

Industrial Statistics Research Unit


Outline of Talk

Purpose of Stats and Experimental Design

History and Applications Skill set needed Examples

Importance of planning Subtleties

Summary


Purpose of Statistics and Experimental Design

Investigate Make objective decisions Experiment efficiently Ensure reproducibility Build a model Predict Monitor ……….


History of experimental design

Agriculture http://www.rothamsted.ac.uk/ Oldest agricultural research station Rothamsted, Park Grass, 1856 Cockle Park, Palace Leas, 1897

Palace Leas, Cockle Park, Northumberland

1897

Park Grass, Rothamsted

Herts, 1856

Long term grass experiments


Palace Leas, Cockle Park


Palace Leas

Experiment on effect of fertilisers on hay yield Can also look at root structure, species,…

Meteorological station in next field from 1898

14 plots of land 8 for an experiment with N,P and K 6 additional (some abandoned in WWII)

Results were apparent in1898

Fabulous set of data


Palace Leas plots

N P2O5 K2O1 20 17 30 342 203 2 20/0 17 30 344 2 20/05 4 40/0/0/0 17 30 3467 358 609 67

10 35 6011 35 6712 60 6713 35 60 6714 100 66 100

Plot Year of Cycle

FYM

t ha-1 Fertilizer (kg ha-1)


Palace Leas plots

N P2O5 K2O1 20 17 30 342 203 2 20/0 17 30 344 2 20/05 4 40/0/0/0 17 30 3467 358 609 67

10 35 6011 35 6712 60 6713 35 60 6714 100 66 100

Plot Year of Cycle

FYM

t ha-1 Fertilizer (kg ha-1)


Rothampsted, Park Grass

Term

Effect

AC

AB

ABC

C

BC

A

B

1400120010008006004002000

316Factor NameA NB PC K

Pareto Chart of the Effects(response is HAY, Alpha = .05)

Lenth's PSE = 84


Importance of publishing!

COLEMAN, S.Y., SHIEL, R.S & EVANS, D.A. (1987) The effect of weather and nutrition on the yield of hay from Palace Leas meadow hay plots, at Cockle Park

Experimental Farm, over the period from 1897 to 1980. Grass and Forage Science 42, 353-358.


corr with smd

hay k

g/ha

-0.30-0.35-0.40-0.45-0.50-0.55

7000

6000

5000

4000

3000

2000

NPK

PK

NK

NP

K

P

N

control

FYM

FYM

FYM

FYM

FYM

Scatterplot of hay kg/ ha vs corr with smd


History and Applications

Agriculture Given industrial slant by G.Taguchi

(b 1924, published 1951, in NE 2000) Used in manufacturing Gradually used in

Business, service, health, Finance, marketing,

Bespoke nomenclature


Skill set needed

Logical thinking Attention to detail Presentation skills Analytical tools Knowledge of where to go next

Optimisation, RSA, simulation……


Examples

Pressure, temperature, pointer setting, haul off speed, welding current, granule size, nozzle width,….

N, P, K

Beer experiment


Beer experiment

(What affects frothing when pouring beer?)



Management Methodology

Six Sigma

Define Measure Analyse Improve Control

Lean

PDCA


Lean Six Sigma

Lean focuses on removing complexity

Six Sigma focuses on process improvement

Lean Six Sigma attempts to combine the best of both

Lean involves less statistics and is very popular in

some applications, such as healthcare.


Define problem

QI tools, brain-storming, team roles Identifying factors and levels

Measurement issues

Decide what, how, when, who and where to measure Analyse

Use current knowledge, Set experimental design and pilot

Improve

Run experiment and analyse Control

Recommend method for best or least froth Look for other opportunities to use what has been learnt


Team roles

Secretary

Waiter

Pourer

Measurer

Observers


Factors

Materials (beer type, temperature of bottles)

Machines (glass shape)

Man (steadiness)

Milieu (pressure, humidity, temperature)

Method (angle, speed, height, time opened)

Measures (volume, height)


Experimental design

More information out requires more data in

However, statistically designed experiments

help reduce the number of trials with least

reduction in information

Eg up to 7 factors can be tested in 8 trials

Taguchi uses Plackett-Burman designs

Eg up to 11 factors can be tested in 12 trials


Saturated L8 design

Trial A B C D E F G

1 -1 -1 1 -1 1 1 -1

2 1 -1 -1 -1 -1 1 1

3 -1 1 -1 -1 1 -1 1

4 1 1 1 -1 -1 -1 -1

5 -1 -1 1 1 -1 -1 1

6 1 -1 -1 1 1 -1 -1

7 -1 1 -1 1 -1 1 -1

8 1 1 1 1 1 1 1


Taguchi saturated L8 design

Trial A B C D E F G

1 1 1 1 1 1 1 1

2 1 1 1 2 2 2 2

3 1 2 2 1 1 2 2

4 1 2 2 2 2 1 1

5 2 1 2 1 2 1 2

6 2 1 2 2 1 2 1

7 2 2 1 1 2 2 1

8 2 2 1 2 1 1 2


Identifying factors and levels 3 factors each at 2 levels

Factors - +

Beer type Belgian French

Glass type Flat Rounded

Glass angle Upright Tilted

Response Froth height (mm)


Orthogonal array

Trial Beer Glass Angle

1 -1 -1 -1

2 1 -1 -1

3 -1 1 -1

4 1 1 -1

5 -1 -1 1

6 1 -1 1

7 -1 1 1

8 1 1 1


Experimental trials

Eg Belgian beer poured into a flat

bottomed glass without tilting

Eg French beer poured into a round

bottomed glass without tilting


Randomisation

Reduces false replication and bias

Eg drug trial

first group of mice have treatment and second have placebo,

if mice are selected by grabbing from cage,

fittest are caught last and placebo can

appear to be better than the treatment


Team roles

Secretary : Read trial and register result

Waiter : Pick the right glass and bottle

Pourer : Pour the beer into the glass

Measurer : Use the ruler to read off the froth

Observers: Note that procedures are followed


Mean o

f Fr

oth

1-1

30

25

20

15

10

1-1 1-1

Beer Glass Angle

Main effects plot


Factors and factor levels

Factors - +

Beer type Belgian French

Glass type Flat Rounded

Glass angle Upright Tilted

Response Froth height (mm)


Interaction plot: glass*angle

Angle

Mean

1-1

40

30

20

10

0

Glass-11


ANOVA table for beer

Source DF SS MS F P

Beer 1 138 138 4.78 0.06

Glass 1 189 189 6.54 0.03

Angle 1 2233 2233 77.26 0.00

Glass*Angle 1 638 638 22.06 0.00

Beer*Angle 1 0 0 0 0.96

Beer*Glass 1 11 11 0.37 0.56

Error 9 260 29

Total 15 3468


Graphical analysis of significance

Standardized Effect

Perc

ent

5.02.50.0-2.5-5.0-7.5-10.0

99

95

90

80

70

605040

30

20

10

5

1

Factor NameA BeerB GlassC Angle

Effect TypeNot SignificantSignificant

BC

C

B

Normal Probability Plot of the Standardized Effects(response is Froth, Alpha = .05)


Residual

Perc

ent

1050-5-10

99

90

50

10

1

Fitted Value

Resi

dual

403020100

10

5

0

-5

-10

Residual

Fre

quency

7.55.02.50.0-2.5-5.0-7.5-10.0

8

6

4

2

0

Observation Order

Resi

dual

16151413121110987654321

10

5

0

-5

-10

Normal Probability Plot of the Residuals Residuals Versus the Fitted Values

Histogram of the Residuals Residuals Versus the Order of the Data

Residual Plots for Froth


Context and implications

Know which glasses to buy

Know how to pour

Adapt to alternative requirements, eg no froth

Apply methodology in other contexts


Other examples

Human Resources Training requirements Explore effects of commands given

Accounts Explore effect of timing and nature

of reminders Preferences

Explore trade-offs Conjoint analysis


Conjoint Analysis

Customers CONsider JOINTly and give their opinions,

trading off factors to reach a desired end. For example, to design a conference, consider:

In University or hotel 2 days or 3 days Evening speaker or not 20 minute or longer talks ….

Present the different options, eg in a questionnaire and analyse results

Helps determine what people value in different product features

or service attributes.


Format of questions (ENBIS)

‘How successful do you think the following

conference would be?’

An informal, applied conference held in a

conference suite that has an evening session,

it mostly features presentations involving

industrialists’

1= not very successful 5= very successful


Results - ENBIS

ANOVA with categorical responses gave

Applied vs theoretical and

Industrialists vs academics

as important factors


Results - ENBIS

Taguchi style analysis for the variability of

responses showed

significantly greater variation in views about

applied talks than for theoretical talks

significantly greater variation for

workshops than for presentations


Online Conjoint Analysis

Used in automobile feature testing to find the features

consumers are willing to give up in order to get something

they value more.

Outcomes help guide new product design, old product

redesign or repositioning decisions.

Used in travel industry to determine how much consumers

are willing to pay for a ticket in order to get more leg room.

Building “Voice of Customer” into Product DevelopmentSource: Siegel (2004)


Other examples

Kansei Engineering Incorporates emotion into design

Orthogonal array for products Semantic scales for emotion Sample of customers

Highly developed in Japan KENSYS 2003-6 in Europe

Do you think that people would be happy wearing these products? Please circle the number which is closest to your feelings for each picture. Picture Number

extremely unhappy

unhappy

neither unhappy or happy

happy

extremely happy

1 1

2

3

4

5

2 1

2

3

4

5


Analysis relates emotions to design

Model Response is happiness Design factors are colour, style, heel, etc

Aim is to advise designers Which shoes will give the desired emotional

response How to develop a balanced portfolio

Similar results for logistic regression or ANOVA


Design a waiting room

Design factors

Sofa or chair Lighting soft or bright Service desk Windows

Other factors

Waiting time max 30 minutes ……..


Please rate where you feel

the image fits on each of the

following semantic scales.

Comfortable Uncomfortable

At Ease Uneasy

Efficient Inefficient

Trustworthy Untrustworthy

Calm Stimulating

Boring Interesting

1 2 3 4 5


Design Rules

ComfortableComfortable

Seating (sofa)Windows (yes)Lighting (soft)Service desk (yes)

Seating (chair)Service desk (yes)Max Waiting Time (30min)

EfficientEfficient


Key Drivers of Satisfaction

[Ease of Use][Navigation, clarity, fresh/relevant content, etc.]

[Graphic Style][Colour, layout, print size, type, no. of photographs, graphics and animation.]

[Perceived channel advantage ][Price, Speed, etc.]

[Privacy and Security ] [Brand, reputation, appearance of the site (more

imp than security logos appearing on website)] [Fulfilment and Reliability]

[Timeliness of service, availability, breadth and depth of

products/services, responsiveness/access (availability of service

personnel, multiple communication channels) and personalisation.]

Source: Baur, Schmidt & Hammersmith, 2006


Website design process

The BriefThe Brief Trial PagesTrial Pages The PrototypeThe Prototype The PrototypeThe Prototype

Client requirements and goals

LaunchApprovalResponse and

refinement

CLIENT

WEB DESIGN FIRM

Strategic planning, engineering

Style book, training, quality tests

Final design, testing and coding

Design content and marketing

User SurveyConjoint Analysis

Kansei Design for

Emotional Appeal

Content & NavigationSPC

Building “Voice of Customer” into Design by courtesy of A.Parulekar

Web Designfor

Sticky Relationship


Statistics

Quantitative management

Software Graphical Tabular

Comparative tests, ANOVA

Statistical models

Regression Logistic regression

Multivariate analysis


Tutorials, eg MINITAB

These easy step-by-step tutorials introduce you to the

Minitab environment and provide a quick overview of

some of Minitab's most important features. The tutorials

are designed to explain the fundamentals of using Minitab:

how to use the menus and dialog boxes, how to manage

and manipulate data and files, how to produce graphs, …. Session One: Graphing Data Session Two: Entering and Exploring Data Session Three: Analyzing Data Session Four: Assessing Quality Session Five: Designing an Experiment


Data exploration, eg SAS JMP

Key processes and inputs associated with excessive variation in 60-minute dissolution

Recursive Partitioning Decision Tree


Importance of planning

Collecting all the data that is needed

Getting the right people involved

Getting the right materials in place

Ensuring time for input from others

Poke yoka


Subtleties include

Measurement issues

Sample sizes

Use continuous variables where possible Robustness

Bias and confounding, especially time trends

Piloting


Summary

All of the above

Thanks

Documents

Statistics and experimental design