Upload
arik-johnson
View
322
Download
5
Embed Size (px)
DESCRIPTION
Can Bayesian statistics really determine in advance if the bid you are offering will be the winner or just another loser? And, if the metrics forecast a loss, can the same algorithm tell you what to change in order to win instead? Competitive bidding is where big money sales opportunities are won or lost, and there are four (4) rules that can help you turn a losing situation into a winning sale. These four rules help you better understand what the customer wants, examine what competitors might do in response and how to beat them, while helping you to offer the best bid, optimized for yours and your prospective customer’s intended outcome. Statistical metrics evaluate your probability of success against the competition and help you more objectively determine how to win. But how can you get at the foundational issues that will determine who will win? Learning objectives: Learn the Four Rules that help you understand what will actually determine the customer’s decision. Visualize your bid head-to-head against the competition and employ objective metrics to determine if you will win. Identify weaknesses in your offer that must be improved for your bid to beat the competition. Bill Zangwill is a Professor, Emeritus, from the University of Chicago, Booth School of Business. He has authored four published books, one of which was selected by the Library Journal as “One of the Best Business Books of the Year,” and had over 50 papers in academic journals. In addition, he has had three articles published in the Wall Street Journal. His consulting engagements include top firms such as IBM, AT&T, Motorola, many smaller firms and the US government. He has also taught at the University of Illinois and the University of California, Berkeley. He is considered one of the most innovative thinkers in his field. Bill will present 30 minutes on how the four rules can help you turn a losing situation into a winning sale and will be joined by webinar moderator Arik Johnson, Founder & Chairman at Aurora WDC.
Citation preview
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
How Four Statistical Rules
Forecast Who Wins a
Competitive Bid
A Complimentary Webinar from Aurora WDC
12:00 Noon Eastern /// Wednesday 22 October 2014
~ featuring ~
Dr. Bill Zangwill Arik Johnson
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Bill Zangwill
Bill Zangwill is a Professor, Emeritus, from the University of Chicago, Booth School of Business. He has authored four published books, one of which was selected by the Library Journal as “One of the Best Business Books of the Year,” and had over 50 papers in academic journals. In addition, he has had three articles published in the Wall Street Journal. His consulting engagements include top firms such as IBM, AT&T, Motorola, many smaller firms and the US government. He has also taught at the University of Illinois and the University of California, Berkeley. He is considered one of the most innovative thinkers in his field.
Email: [email protected]
The Intelligence Collaborative is the online learning and networking community powered by Aurora WDC, our clients, partners and other friends and dedicated to exploring how to apply intelligence methods to solve real-world business problems.
Apply for a free 30-day trial membership at http://IntelCollab.com or learn more about Aurora at http://AuroraWDC.com – see you next time!
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
α Use the Questions pane on your GoToWebinar control panel and all questions will be answered in the second half of the hour.
α You are welcome to tweet any comments on Twitter where we are monitoring the hashtag #IntelCollab or eavesdrop via http://tweetchat.com/room/IntelCollab
α Slides will be available after the webinar for embedding and sharing viahttp://slideshare.net/IntelCollab
α To view the recording and download the PPT file, please register for a trial membership at http://IntelCollab.com.
Questions, Commentary & Content
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Agenda
► Introduction
► Risk, Bias, Errors: Caused by Usual Procedures, but Usually Missed
► Application to Situations in Business
► Summary, Q&A and Discussion
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
The Joy When We Win.
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
How do we avoid the grief and attain the joy?
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Novel Approaches. Two patents, third pending.
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Why are the patents and new approaches
needed? Because usual methods typically have hidden bias, hidden risks and hidden errors.
► MAJOR CONUNDRUM: We believe in the usual approaches and keep using them because the errors are hidden so are missed and not considered.► E.g., Behavioral research. If we believe the decision will be 70-80%
successful, likely it will be roughly 50%.
► First methodology that expressly helps warn of errors, bias, issues missed► Humans miss issues.
► 2 patents and third pending
► Follow the four rules
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Four Rules – Boost Win Rate
1. Stop using usual methodologies as they likely have hidden risk, bias, errors. They can cause losses despite our belief in them.
2. Do not assume. TEST.
3. Improve twice. Best ideas come after reviewing situation a couple times.
4. Follow procedure, otherwise bias and risks will slip in secretly, stealthily and surreptitiously.
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Risk, Bias, Errors: Caused by Usual
Procedures, but Usually Missed
Math. Bias. Surprises.
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Does Arithmetic Have a Hidden Secret?
Example: SUBTRACTION
100 $100,000,000
-90 -$90,000,000
----- ---------------
10 $10,000,000
WHAT CAN POSSIBLY BE HIDDEN HERE? WHAT MIGHT BE WRONG?
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
In Business Decisions, Most Data Are Not
Known That Perfectly
Example: SUBTRACTION ERROR
100
-90
-----
10
But error in value
100 +/_ 10 (90 – 110) ( 10% error)
-90
------
0 to 20 (100% error)
HUGE ERROR INCREASE FROM 10% to 100%
WORSE IF BOTH FACTORS HAVE ERRORS OR OVER TIME
Math causes error.
TYPICAL ALGORITHM: WE BELIEVE IT IS RIGHT YET INSTEAD PRODUCES ERROR
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
DCF and Other Financial Formula Can
Increase Error
Gordon growth model:
$10 million payout in one year, g = growth rate, r= rate of return
Value, DCF = $10/ (r – g) (in millions)
Example 1: r = 12% , g = 10%
Value = $10/ ( 0.12 - 0.10) = $10/(0.02) = $500 million
Example 2: r = 12% , g = 11% growth rate increased by 10%
Value = $10/ ( 0.12 - 0.11) = $10/(0.01) = $1 billion
10% error in input
100% error in answer.
Proprietary material. Do not disseminate. Patent pending and copyrights University of Chicago and Decision Command, Inc. SLAM(TM) is a registered service mark of Decision Command.
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
More Accurate Using Better Math
Two quantities, 40 and 10
What percentage is 40 of total?
40/( 40 + 10) = 40/50 80%
Suppose 25% error in 40. Could be 30 – 50.
What is percentage of the total now?
30/(30 +10) 75%
50/(50 + 10) 83.66%
Much reduced error relative to 80% base --– error less than 7%
ERROR REDUCED FROM 25% TO LESS THAN 7%.
THE MATH IN MOST ALGORITHMS IN COMMON USE CANNOT BE TRUSTED.
Must use better approaches that reduce error and attack the hidden secret.
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Improves Accuracy, Reduces Risk
Pij = conditional probability in a cell, Pj = probability of column
(Uses Bayes and Laplace approaches)
Suppose there is risk in a specific cell entry Plm
Consider risk (percent error) in the output Pm over risk in the input Plm.
The fractional risk in output is
The fractional risk in the input is
i ij
j
i ij
j
PP
P
m
lm
m
dP
dPP
lm
1P
Proprietary Information, FOUO
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Error Percentage Always Decreases,
Proprietary Information, FOUO
i ij i l im i im i l im
jm
2mlmi ij
jm lmlm lm
m i im
lm
i ij
j
i ij i l im i im i l im i ij i im i im i im
j j
lm
i im i im i ij i im
j j
i ij i im
j
i ij
( P ) P P PdP [
dPdP ( P )P dP
[ ](P ) ](P )P P1
PP
( P ) P P P ( P ) P P P
(P )
( P ) P ( P ) P
( P ) P
( P
j
i im
i ij
j
)
P1 1
( P )
Percentage error out less than percentage error in
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Serious Problems Often Cause Us to Lose: Most
Algorithms Increase Risk and Errors, and We Do Not
Know It
Bayesian Algorithm I Use Reduces Risk and Errors
1. “The model was previously shown to be surprisingly robust to obvious violations of
this independence assumption, yielding accurate classification models even when there are clear conditional dependencies.”
“Many researchers have noted the good performance of SBC, including Clark and Niblett, Knononenko, Langley and Sage, and Domingos and Pazzani.”
2. Beyond Independence: ..is commonly thought to assume that attributes are independent
given the class, but this is apparently contradicted by the surprisingly good performance it exhibits in many domains that contain clear attribute
dependences. No explanation for this has been proposed so far. In this paper we show that the
SBC does not in fact assume attribute independence, and can be optimal even when this assumption is violated by a wide margin
Proprietary. Confidential information. US Patent 7,676.446. Copyright University of Chicago and/or Decision Command.
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
BIAS – Frequent Cause of Not Winning
Bias and optical illusions are similar. What you
believe is not there.
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
Holding hands, or not?
INSIGHT: Statistically identify entries that are odd relative to facts of situation, outliers
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Bias and Hidden Risk
Biases, hidden risk, faulty assumptions tend to be at variance, out-of-kilter with facts of situation.
Anomaly, outlier or oddity is often the clue: risk, bias, faulty assumption
outlier
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Surprise – Must the Unexpected Always be
Unexpected?
INSIGHT: Identify contradictory and odd information. The greater the level of contradictory and odd information, the greater chance something odd is going on, something was missed.
MATH ADVANCEMENT– Add variable to represent surprise
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Bayes SystemFirst System to Warn of Risks/Surprises/Missed Issues
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Usual Methods Typically Have Hidden Error,
Risk and Bias, and Often We Will Not Know It.
► NO AUTOMATIC IDENTIFICATION OF RISK.
► NO AUTOMATIC MEANS TO PREVENT MATH ERRORS.
► NO AUTOMATIC CROSS-CHECKS ON ACCURACY.
► NO AUTOMATIC ANALYSIS OF RISK OR SURPRISE/MISSED
ISSUES.
► NO AUTOMATIC ESTIMATE OF THE PROBABILITY OF SUCCESS.
► NO PROCESS TO RESOLVE DISAGREEMENTS
NEED BETTER WARNING OF THE HIDDEN PROBLEMS
“YOU CAN’T DO A GOOD JOB WITH POOR TOOLS”
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Proof
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
TEST AT UNIVERSITY OF CHICAGO
Managers and executives made decisions
They compared quality of decision made this new way versus their
usual decision approach.
40% better and 20% faster
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Four Rules – Boost Win Rate
1. Stop using usual methodologies as they likely have
hidden risk, bias, errors.
They can cause losses despite our belief in them.
2. Do not assume. TEST.
3. Improve twice.
Best ideas come after reviewing situation a couple times.
4. Follow procedure, otherwise bias and risks will slither in
secretly, stealthily and surreptitiously.
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Application to Situations in Business
Use better ways to handle math, bias, risks
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Bidding to Invest in a Chilean Firm
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
INVEST IN EXCELLENT CHILEAN FIRM. PARTNER.
In private education, growing rapidly, wanted funds for international growth and acquisition
AAI could assist with funding growth. Gateway to future.
Several trips to Chile to speak with owner
AAI BIDDING AGAINST ARCH RIVAL ADVENT
Felt confident they would win—
got to know owner, understood situation
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Situation: AAI Confident of Winning
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
AAI WERE HIGHLY EXPERT AND DID TRADITIONAL
ANALYSIS WELL---SURE THEY WOULD WIN :
analyzed situation, obtained information, visited Chile,
brought in experts, financial analysis, discussions.
RESISTED USE OF SYSTEMATIC METHOD,
SUPERFLUOUS.
But agreed to use it because easy, fast, and I was a friend.
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Difficulties with Usual Methods
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
No goals from perspective of Santo Tomas.
• No metrics for those goals. Prob. ST will select AAI or Advent.
No independent warning of possible biases, risks.
• No probability of missed issues/surprise.
No iterations- building of better and better solns. (First analysis is always incomplete---Donnelly’s rule)
• No metrics to determine if decision is getting better.
No testing if decision excellent or not.
arguments, blind alleys.
Made decision as if they were making the decision.
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
5 Minute Role Play Real Decision Maker
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
MUST GET AT EMOTIONS. WHAT ARE THEY CONCERNED OR WORRIED ABOUT? Utilize goals, hopes, fears, problems of real decision maker. What could cause failure? What causes customer worries?
Santo Tomas goals.
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
We Fail (Missed before because traditional had
no means to detect.)
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
EXPOSED HUGE RISK. ASTOUNDED. BUT THEY HAD TO AGREE BECAUSE THEY DID THE EVALUATION.
WHAT SHOULD WE DO?
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
- Test: What are the risks that cause our loss?
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
Why we lose
TRADITIONAL NO WARNING OF RISKS
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Traditional Methods: Argue
►BEFORE: Argue who is right.
►SYSTEMATIC APPROACH: TEST.
Identify risks. Examine possible
improvements and check their metrics
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
New approach to win. BOOSTED METRICS:
success rate from 63% to 75% beating Advent’s 70%
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
REVISED AAI STRATEGY
METRICSAAI 63-75ADVENT 70
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Test—expose risks again. Still some issues, so
build higher – IMPROVE TWICE
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Build the win higher—test and improve twice
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Develop New Strategy - Won
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
► Bring in experts on international to our team and show Santo Tomas, that we are really best for global expertise. Create connection in Chile to build trust.
► Convince Santo Tomas our financial deal is really better. (less upfront, but more later)
► Ensure CEO comfortable (CEO is owner)
Conclusion: saved them from losing deal
Traditional means fail due to hidden errors, no warning.
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Four Rules – Boost Win Rate
1. Stop using usual methodologies as they likely have
hidden risk, bias, errors.
They can cause losses despite our belief in them.
2. Do not assume. TEST.
3. Improve twice.
Best ideas come after reviewing situation a couple times.
4. Follow procedure, otherwise bias and risks will slither in
secretly, stealthily and surreptitiously.
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
New Product Development at EG Insurance.
What is going on?
Large commercial insurance firm wants to sell to
mortgage originator insurance market
Traditional product based upon annual premium—
Exciting new product they expect to launch to clearly
differentiate themselves from competitors: Chubb,
CNA, Lloyds of London:
A transaction-based plan based upon monthly
volume. Client would only pay for what was actually
used.
WE’RE READY TO GO AHEAD.
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2013
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Role Play: What does customer think?
Get at worries.
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2013
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Test: Complete matrix, examine metrics (cannot be done in
traditional)
New product loses. Software saved them from failure! Forced analysis from viewpoint of actual decision maker.
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2013
METRICS: Prob. SuccessLow, 16%, 53%.Easy to miss this.
WAY UNDER GOAL OF 75%
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Warning. METRIC High surprise risk – “You can
do better than this, you missed something”
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2013
METRICS:Success, 16%, 53%Surprise 31%
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Examine Hidden Risks
-- Countering risks leads to new concept.
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2013
risks
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
NEW IDEA: Transaction based, but on closed
loans only. Should save money.
►TEST IDEA: DO NOT ASSUME: (cannot
be done with traditional because no
metrics)
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2013
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Test New idea: Transaction based, but on
closed loans only– Surprise still high
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2013
METRICS:Prob. of success is lower.Surprise drops to 20% from 31%
What to do now?Build
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Test: Check Risks
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2013
risks
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Check More Risks and Weaknesses
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2013
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
TEST another idea — Large increase in
estimated success probability metric
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2013
METRICS:Success 71%Surprise 16%
CLOSE ENOUGH TO 75%
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
TWO IMPROVEMENT CYCLES
Sizable improvement in probability of winning over
prior approach.
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2013
Value
Initial-New transaction idea($31)
Closed loans only($54)
ZERO
Cust. choice($74)
Traditional
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Saved Them From Losing Money.
Usual Means Have Hidden Error
$4.3 MILLION SAVED.
Over 30% improvement . Three cycles.
Exposed that initial evaluation was incorrect
“Having software like this around
to aid in making important
decisions is wonderful.”
“50% faster than regular method.”Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2013
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Four Rules – Boost Win Rate
1. Stop using usual methodologies as they likely have
hidden risk, bias, errors.
They can cause losses despite our belief in them.
2. Do not assume. TEST.
3. Improve twice.
Best ideas come after reviewing situation a couple times.
4. Follow procedure, otherwise bias and risks will slither in
secretly, stealthily and surreptitiously.
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
IT Competitive Bid
WE LOSE
Prob. of success
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Test: Despite ABC Winning, Maybe ABC Can
Be Beaten
Factors where XYZ beats ABC
where ABC beats XYZ
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Where are others strong or weak?
XYZ STRENGTHS SMS STRENGTHS
ATT STRENGTHS
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Improve XYZ to Beat ABC
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Test: XYZ Still Has Weaknesses
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
XYZ WEAKNESSES
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Improve XYZ Second Time to Beat ABC –
IMPROVE TWICE
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Thank you! Now how about a little Q&A?
Bill Zangwill
Email: [email protected]
The Intelligence Collaborativehttp://IntelCollab.com #IntelCollab
Powered by
Four Rules – Boost Win Rate
1. Stop using usual methodologies as they likely have
hidden risk, bias, errors.
They can cause losses despite our belief in them.
2. Do not assume. TEST.
3. Improve twice.
Best ideas come after reviewing situation a couple times.
4. Follow procedure, otherwise bias and risks will secretly,
stealthily and surreptitiously slither in.
Proprietary, do not disseminate, property of University of Chicago and Decision Command, copyright, 2014. Patented USPTO