View
228
Download
1
Tags:
Embed Size (px)
Citation preview
Research Methodology
EPH 7112
LECTURE 7: EXPERIMENTAL DESIGN
Contents
Synthesis Implement Solution Design Experiments Conduct Experiments Reduce Results
Scientific Method
Analysis
Synthesis
Validation
Hypothesis
Analysis
Describe Problem
Set Performance Criteria
Investigate Related Work
State Objective
Hypothesis
Specify detail and comprehensive solution
Assert expected results Define factors that will be varied Measure against performance metrics
Synthesis
Implement the solution And experiment To accomplish the goals And validate the
hypotheses
Synthesis Steps
Implement Solution Design Experiments Conduct Experiments Reduce Results
Implement Solution
Implement solution to test hypotheses Methods:
AcquireConstructCombination of both
Implement Solution
ConstructCustom made to meet requirements Time consumingExpensive
Implement Solution
AcquireQuick solutionCheaperMay not meet requirements
Implement Solution
Consider strongly to acquire the solution
Even if part of entire solution Consultants – acquired solution?
Implement Solution
Example: Optical Amplification in S-band
Construct using Thulium doped fiber Problem: Fusion Splicing is not
possible Solution: Use Angled Connectors Issue: Not specified during order
Implement Solution
Is your solution right or is it the right solution?
Careful implementation Step-by-step Troubleshooting Example: Constructing an optical
amplifier Problem: WDM is faulty
Design Experiments
To design a series of experiments Results used to estimate how good
solution to solve problem An experiment acquires data to
measure the performance of the solution under controlled conditions in a laboratory
Design Experiments
Always good to have a check list Objective Unit under Test Inducers Sensors Supervisor Channels Domain Knowledge Range Knowledge Solution
Experiments
Is it really necessary? How about theoretical or simulation
work? Experiment = verification Example: Find solution of two-
dimensional plane that satisfy certain conditions
Experiments
Simulation and modeling Verify against experimental results Example: Modeling of Optical
Amplifier Advantages of modeling
OptimizationAnalyzing the physical phenomena
Design Experiments
Planning:Specification Experiment LaboratoryDesign of experiment blocksDesign of protocolsAcquiring and managing data
Experiment Laboratory
Laboratory is where the experiment takes place
Large room with test & measurement equipments, units under test, chemical & mechanical apparatus, computers
Laboratory
Experiments can also take place:In an officeFieldManufacturing Plant
Laboratory
Depends on the experiment:ObjectiveSample Unit under TestInducersSensorsSolution
Laboratory: Safety
Watch out for moving or revolving parts (they don’t like necklaces and neck ties!)
Watch out for Electro-Static Sensitive Devices
Limit personnel into the laboratory Maintenance and cleaning personnel
can cause mishaps
Design Experiment Block
Process that takes place in the laboratory during the experiment
Important termsSampleFactors (independent variables)InducersSensors
Sample
Task unit consisting of objects, living plants, animals, humans that is the subject in the experiment
Factors
Condition or parameter of a task whose value is intentionally varied to measure its impact on the results of the task
Inducers
A device or mechanism that alters the task unit/ subject during the experiment
Sensors
Device that capture the results from the task/ unit or subject
Extent
Each factor is assigned a set of values
Extent of the factor space is total number of unique combination of values that may be assigned to each factor
FVa = number of values for factor a
Extent = FVa x FVb x FVc
Treatment
Each one of the unique combination of values that may be assigned to every factor is called a treatment
One instance in the entire factor space
Case Study
FactorsTypeface: 2 types (Serif and Sans
Serif)Noise Level: 12 levelsCharacter: 36
Extent = 36 x 12 x 2 = 864 Each combination = treatment
Block Design
If sample is a single object or device, then all the possible treatment must be assigned to it during the experiment
Example: Characterization of a Thulium doped Fiber Amplifier for different pump powers and wavelengths
Block Design
If sample is more than one, then the treatments may be distributed in some way among the sample
Important terms:Experiment trialExperiment block
Experiment Trial
Complete set of treatments applied to a sample during the experiment (sample is more than one)
Example: The combination of typeface: serif, character <A, B, C> and noise level <130, 140, 150>
Experiment Block
Set of experiment trials that provides a cover of the factor space that is appropriate and adequate for achieving the task objective
Block Design
What is the appropriate set of experiment trials
that provides an appropriate cover of the factor space
for the experiment?
Block Design
Three basic strategies:Enumerated block designSystematic block designRandomized block design
Enumerated Block Design
Assigns every possible treatment to every sample
Obvious strategy if sample = 1 If sample > 1, this is not practical Because total number of trials =
extent of factor space x number of sample
Too large !!
Systematic Block Design
Uses a deterministic algorithm to assign treatments to different sample in a systematic way
Eventually covers the entire factor space
Problem: Unintentional resonance between sample and treatment can be sparked
Systematic Block Design
Example: A marketing survey is carried out to
every 100th telephone numberThe chances a treatment assigned to,
say a number 03-2698 1100 belonging to a business entity
Is higher than say 03-2698 1024A bias towards response of business
entity may occur in the survey
Systematic Block Design
This block design should be avoided Unless this bias can be ascertained
Randomized Block Design
Similar to systematic block design Except that the treatment assigned to
the sample are sequenced randomly This can also reduce the risk of
systematic bias
Case Study
Enumerated block design is not practical
Total trials = 864 x 14! Each sample has to respond to 864
treatments! Fatigue ‘Peak Performance’
Case Study
Another disadvantage: Humans are smartEasily guess that factor space include
10 decimal digits and 26 Latin characters
Guess from elimination processBias the results
Case Study
Since all the license plate inspectors had same recognition skills
Not all treatment need to be assigned to every sample/ subject
Divide 864 treatments equally Reduce time for each subject Can a systematic block design do it?
Case Study
Systematic block design also has setbacks
Subjects can also detect the periodicity
Biased improved performance Randomized block design is solution Computer generated pseudo-random
assignment of treatments
Case Study
Decide how many and which sets of treatments would be randomly assigned to subjects
Combined to cover enough sets for each factor
To make up set of trials that cover entire factor space
Representation Factor
A factor that is not intended as a basis for measuring performance
However they are necessary for assigning values of a parameter
Example:Characters
Performance factor
A factor that is used as a basis for measuring performance
Example:TypefaceNoise Level
Case Study
Either assign each of two typefaces to half the subjects
Or assign both typefaces to all Former solution better to avoid
confusion among subject and more realistic
Case Study
Noise Level range 130 to 240 with increments of 10
How to distribute the treatment to subjects?
Condition:Interval must be sameSubsets differentSame average
Case Study
Subsets suitable:{130, 150, …, 210, 230}{140, 160, …, 220, 240}{130, 140, …, …, 230, 240}
Case Study
How many subjects? Access to 14 trained license plate
inspectors 2 for OP Pilot Number of subjects will determine the
combination of performance factor values
Case Study
Statisticians require at least 30 responses over entire experiment for each typeface and noise level combination
Characters can be assigned to obtain response
Must use whole set (36 characters) or multiple of whole set to avoid character bias
Case Study
Decided:6 subjects for OP Pilot8 subjects for experiment trialUsing block design in column B
Control Trial
Measures the performance of one set of task in the absence of another to isolate the effects of the included components on performance
Control Trial
To identify possible bias in the processes of the project task
Bias is a consistent tendency to behave in an inconsistent way under certain conditions
Example: A spring loses its memory when
elastic limit is exceeded
Control Trial
Case Study:The ‘Listening Rat’Disabled Power Brakes
Control Trial
To establish performance baselines for comparison
Without baseline, it is impossible to test the hypothesis of a solution that suggests a certain improvement or behavior
Example:To test if a new hand lotion is better
than not using any hand lotion
Case Study
Two control task was introduced:With characters but without noiseWithout characters, only noise
Case Study
The first control trial to ensure that each subject had sufficient experience with interface during practice sessions
So that the effects of the learning curve is negligible during test trials
If learning bias occurs, repeat practice session
Case Study
Second control trial to measure selection time in the absence of any characters
Pure guessing The statistics of this selection time
used as baseline for computing the confidence that subject did not purely guess in the test trials
Protocols
Step-by-step procedure to be followed during preparation and conduct of experiment
Main purpose:To ensure that experiment can be
accurately and precisely repeated
Protocols
Check list can help ensure uniformity in preparation of lab before experiment trial begins
Everyone involved must carry out protocol accordingly
Pilot trials can be used to plan and debug the protocol
Protocols
Anyone in contact with human subjects in an experiment trial should not expose the objective of the experiment
Protocols can be printed as flowcharts, pseudo-code or lists
Case Study
Characterization of EDFA Steps include:
Measure input signalMeasure output signalOSA will compute Gain, NF, PASE
Sequence of measurement is important
Data Management
Most critical and frustrating task Protect data Maintain logs of data (where it is kept,
which file is for what) Record ALL experiment data
Data Management
Do not preprocess the raw experiment data in any way before recording it
Establish clear organizational and documentation conventions for data files
Back-up !!
Conduct Experiments
Time to follow your plans Resist temptation to
improvise on the fly If doesn’t run well, stop and
revise Consider failed experiment
as pilot trial
Reduce Results
Performance values to validate the hypothesis cannot be drawn directly from raw results
The raw results must be reduced, combined or transformed to be meaningful
Case Study 1
Identify Faster Traffic on Highway Need to measure speed This is not raw results Reduced from compression signals,
time two pulses occurred
Case Study 2
Characterization of EDFA Need to measure Gain (dB) and Noise
Figure (dB) Not raw results Reduced from among others; Input
Signal Power (dBm), Output Signal Power (dBm), ASE Level (dBm)
Reduce Results
Data Reduction methods may change Performance metrics may be altered Important to record both raw and
reduced results
Q&A