Embed Size (px)
Citation preview
W2 Track 10/7/2009 11:30 AM
"What Would MacGyver Do?"
Presented by:
Harry Robinson Microsoft
Brought to you by:
330 Corporate Way, Suite 300, Orange Park, FL 32073 888‐268‐8770 ∙ 904‐278‐0524 ∙ [email protected] ∙ www.sqe.com
Harry Robinson
Harry Robinson is Principal Software Development Engineer in Test for Microsoft's Bing team. He has twenty years of software development and testing experience at AT&T Bell Labs, Hewlett-Packard, Microsoft, and Google as well as time spent in the startup trenches. While at Bell Labs, Harry created a model-based testing system that won the 1995 AT&T Award for Outstanding Achievement in the Area of Quality. At Microsoft, he pioneered the model-based test generation technology, which won the Microsoft Best Practice Award in 2001. Harry holds two patents for software test automation methods, maintains the site www.model-based-testing.org, and speaks and writes frequently on software testing and automation issues.
9/9/2009
1
What Would MacGyver Do?
Harry RobinsonMicrosoft
Principal SDET, Bing
What Would Ed Smylie Do?
Harry RobinsonMicrosoft
Principal SDET, Bing
9/9/2009
2
April 14, 1970
“The contraption wasn't very handsome, but it worked.”‐ James Lovell, Apollo 13
June 1, 2009
bing!
9/9/2009
3
Smart thumbs
yipes!
Popular now
Deals on underwater cameras Plan your next vacation today Obama education speech · <tacky phrase here> · Laura Bush · Tropica…
9/9/2009
4
Autosuggest
Deals on underwater cameras Plan your next vacation today Obama education speech · Congressional hearing · Laura Bush · Trop…
Spell correction
“porn viedo” “porn video”“porn viedo”
9/9/2009
5
Images/Shopping
We considered the usual options• Testers
• Automation scripts
• A huge test lab
• Relevance judges
• Bug bashes
• Fuzzers
C d i
… as well as some not‐so‐usual options
• Crowdsourcing
• Logged user queries
Desperation tends to make one sort of flexible – Angus MacGyver
9/9/2009
6
Welcome to the monkey trap“The trap consists of a hollowed‐out coconut chained to a stake. The coconut has some rice inside which can be grabbed through a small hole. The hole is big enough so that the monkey's hand can go in, but too small for his fist with rice in it to come out. The
k h i d i dd l t d
Well, he should … see if things he thought were important really were important and, well, stop yanking and just stare at the coconut for a while.
… what general advice would you give the poor monkey?
monkey reaches in and is suddenly trapped ...
Before long he should get a nibble from a little factwondering if he is interested in it.”
‐Robert PirsigZen and the Art of Motorcycle Maintenance
Wh hi ' b k h i hi i h iWhen something's broken, the easiest thing is to throw it away, forget about it. But if you just step back and take a look at what
you've got, you find a totally different way for it to work.‐ Angus MacGyver
9/9/2009
7
Components of creativity
Expertise
Creative‐thinking skills
Motivation
Creativity
“Creative thinking … refers to how people approach problems and solutions – their capacity to put existing ideas together in new combinations.”
‐ Teresa AmabileMotivating creativity in organizations
Motivation
9/9/2009
8
Expertise
• Lawyers
• Project managers
• Domain experts
• Feature testers
• Senior testers
• Automation specialists
Conceptual blocks
“… mental walls that block the problem‐solver from correctly perceiving a problem or conceiving its solution.” ‐ James L. Adams
Conceptual Blockbusting
Perceptual• Stereotyping
• Difficulty isolating the problem
• Delimiting the problem too closely
• Saturation
Emotional• Fear of risk• No appetite for chaos• Judging rather than generating ideas• Inability to incubate• Complacency
Cultural Environmental
p g
Cultural• Taboos
• Humor (lack of)
• Reason vs intuition
• Tradition vs change
Environmental• Supportive team
• Supportive management
• Incorporating feedback
• Hidden agendas
9/9/2009
9
Voices in our heads: automation
“Automated tests represent the automation of a manual process.” Harold F Tipton Micki Krause– Harold F. Tipton, Micki Krause
“The most important benefit of automated testing over conventional manual testing is the minimization of costs over repeated tests.”
– Markus Helfen, Michael Lauer
“We have found that it is essential for tests to execute very quickly; our target isWe have found that it is essential for tests to execute very quickly; our target is under 30 seconds for the typical test run.” – Shaun Smith, Gerard Mezaros
The less you know sometimes, the better. – Angus MacGyver
Voices in our heads: new bugs
“I have never been convinced that finding ‘new’ bugs is a realistic expectation for test automation.” – I.M. Testy
“After you have run your automated tests for the first time, you are done finding new bugs. Never again will you find a new issue. ” – Steve Rowe
“Running automated test scripts can not be used to find new bugs in the software …” – Cordell Vail
9/9/2009
10
Voices in our heads: test oracles
“Oracle: A mechanism to produce the predicted outcomes to compare with p p pthe actual outcomes of the software under test.” – BS 7925‐1
“A test oracle is a mechanism for predicting the expected results.”‐ Andreas Spillner; Tilo Linz; Hans Schaefer
“In real testing the outcome is predicted and documented before the test is run.” i i– Boris Beizer
Challenge assumptions
• Is it necessary to predict the outcome of a test?
• Must an automated test behave like a human?• Must an automated test behave like a human?
• Can random inputs be used well?
• Does a test program really need to run quickly?
• How do humans judge if a result is correct?
• How efficient must a test program be?
• Why can’t a test program be used for exploratory testing?Why can t a test program be used for exploratory testing?
Never laugh at what you don't know. – Angus MacGyver
9/9/2009
11
Blockbusting
• Challenge assumptions
• Investigate the problem
• Question all constraints
• Look at object attributes
• Brainstorm ideas
• Who else has done something like this?
• Imagine you have unlimited resources …
Imagination is the most important thing the human mind has. – Angus MacGyver
Part 1: Break the problem up
9/9/2009
12
Part 2: Smart thumbs/Popular now
• Smart thumbs:– Create explicit domain
• Popular now:– Validate upon data entry– Validate upon data entry
Part 3: Autosuggest
• Bug bash?G t d h k?• Generate and check?– Input from
• Log data• Exhaustive combinations
– Check with• Homemade utility• Homemade utility• PoliCheck
• Simple solution: Check source file
9/9/2009
13
Part 4: Spell correction
• Generate and check– Input from
• Log data
• Mutated combinations
– Check with• Homemade utility
• PoliCheck
Part 5: Images/Shopping
• Generate and check– Input from
• Log data
• Exhaustive combinations
– Check with• PoliCheck
If you don't have the right equipment for the job, you just have to make it yourself.– Angus MacGyver
• Homemade utilities
• Autonomation (people + machines)
9/9/2009
14
What should you do?
• Approach test as an engineering problem
• Study up on
– Test approaches
– Your product
– Available tools
– Creative problem‐solving
• Testing books: Beizer, Binder, Jorgensen
h l b• Papers: Test generation, heuristic oracles, combinatorics
Always carry around a knapsack to pick up things you find along the way. – Angus MacGyver
April 19, 2005
Ed Smylie accepts the first Great Moments in Engineering awardon behalf of NASA’s Crew Systems Division who improvised an air filterthat kept astronauts alive on the ill‐fated mission 35 years ago.
Any problem can be solved with a little ingenuity. – Angus MacGyver
9/9/2009
15
The most important tool for survival isn't duct tape or a Swiss Army Knife, it's your wits. Always keep them about you.
– Angus MacGyver
10/7/2009
1
What Would MacGyver Do?
Harry RobinsonMicrosoft
Principal SDET, Bing
What Would Ed Smylie Do?
Harry RobinsonMicrosoft
Principal SDET, Bing
10/7/2009
2
April 14, 1970
“The contraption wasn't very handsome, but it worked.”‐ James Lovell, Apollo 13
June 1, 2009
bing!
10/7/2009
3
Smart thumbs
yipes!
Popular now
Deals on underwater cameras Plan your next vacation today Obama education speech · tacky phrase goes here · Laura Bush · Trop…
10/7/2009
4
Autosuggest
Deals on underwater cameras Plan your next vacation today Obama education speech · Congressional hearing · Laura Bush · Trop…
Spell correction
“porn viedo” “porn video”“porn viedo”
10/7/2009
5
Images/Shopping
We considered the usual options• Testers
• Automation scripts
• A huge test lab
• Relevance judges
• Bug bashes
• Fuzzers
C d i
… as well as some not‐so‐usual options
• Crowdsourcing
• Logged user queries
Desperation tends to make one sort of flexible – Angus MacGyver
10/7/2009
6
Welcome to the monkey trap“The trap consists of a hollowed‐out coconut chained to a stake. The coconut has some rice inside which can be grabbed through a small hole. The hole is big enough so that the monkey's hand can go in, but too small for his fist with rice in it to come out. The
k h i d i dd l t dmonkey reaches in and is suddenly trapped ...
Welcome to the monkey trap“The trap consists of a hollowed‐out coconut chained to a stake. The coconut has some rice inside which can be grabbed through a small hole. The hole is big enough so that the monkey's hand can go in, but too small for his fist with rice in it to come out. The
k h i d i dd l t d
… what general advice would you give the poor monkey?
monkey reaches in and is suddenly trapped ...
10/7/2009
7
Welcome to the monkey trap“The trap consists of a hollowed‐out coconut chained to a stake. The coconut has some rice inside which can be grabbed through a small hole. The hole is big enough so that the monkey's hand can go in, but too small for his fist with rice in it to come out. The
k h i d i dd l t d
Well, he should … see if things he thought were important really were important and, well, stop yanking and just stare at the coconut for a while.
… what general advice would you give the poor monkey?
monkey reaches in and is suddenly trapped ...
Before long he should get a nibble from a little factwondering if he is interested in it.”
‐Robert PirsigZen and the Art of Motorcycle Maintenance
Wh hi ' b k h i hi i h iWhen something's broken, the easiest thing is to throw it away, forget about it. But if you just step back and take a look at what
you've got, you find a totally different way for it to work.‐ Angus MacGyver
10/7/2009
8
Components of creativity
Expertise
Creative‐thinking skills
Motivation
Creativity
“Creative thinking … refers to how people approach problems and solutions – their capacity to put existing ideas together in new combinations.”
‐ Teresa AmabileMotivating creativity in organizations
Motivation
10/7/2009
9
Expertise
• Lawyers
• Project managers
• Domain experts
• Feature testers
• Senior testers
• Automation specialists
Creative‐thinking skills
“Conceptual blocks are mental walls that block the problem‐solver from correctly perceiving a problem or conceiving its solution.”
‐ James L. Adams
Perceptual• Stereotyping
• Difficulty isolating the problem
• Delimiting the problem too closely
• Saturation
Emotional• Fear of risk• No appetite for chaos• Judging rather than generating ideas• Inability to incubate• Complacency
Conceptual Blockbusting
Cultural• Taboos
• Humor (lack of)
• Reason vs intuition
• Tradition vs change
Environmental• Supportive team
• Supportive management
• Incorporating feedback
• Hidden agendas
10/7/2009
10
Ping‐Pong Pipe Problem
A steel pipe is embedded in the concrete floor of a bare room… The inside diameter is 0.06” larger than the diameter of the ping‐pong ball (1.50”) that is resting at the bottom of the pipe. You are one of a group of six people in the room, along with the following objects:
• 100 feet of clothesline • A box of granola • A light bulb • A file
from Conceptual Blockbustingby James L. Adams
List as many ways as you can think of to get the ball out of the pipe without damaging the ball, the pipe or the floor.
• A carpenter’s hammer • A wire coat hanger • A chisel • A monkey wrench
Ping‐Pong Pipe Problem
A steel pipe is embedded in the concrete floor of a bare room… The inside diameter is 0.06” larger than the diameter of the ping‐pong ball (1.50”) that is resting at the bottom of the pipe. You are one of a group of six people in the room, along with the following objects:
• 100 feet of clothesline • A box of granola • A light bulb • A file
from Conceptual Blockbustingby James L. Adams
List as many ways as you can think of to get the ball out of the pipe without damaging the ball, the pipe or the floor.
• A carpenter’s hammer • A wire coat hanger • A chisel • A monkey wrench
Notice that having tools can distract you from a good solution!
10/7/2009
11
So, Are Tools Bad?
MacGyver Gets Lazy http://xkcd.com/444/
You can do more with a gun that just fire it. – Angus MacGyver
Voices in our heads: automation
“Automated tests represent the automation of a manual process.” Harold F Tipton Micki Krause– Harold F. Tipton, Micki Krause
“The most important benefit of automated testing over conventional manual testing is the minimization of costs over repeated tests.”
– Markus Helfen, Michael Lauer
“We have found that it is essential for tests to execute very quickly; our target isWe have found that it is essential for tests to execute very quickly; our target is under 30 seconds for the typical test run.” – Shaun Smith, Gerard Mezaros
The less you know sometimes, the better. – Angus MacGyver
10/7/2009
12
Voices in our heads: new bugs
“I have never been convinced that finding ‘new’ bugs is a realistic expectation for test automation.” – I.M. Testy
“After you have run your automated tests for the first time, you are done finding new bugs. Never again will you find a new issue. ” – Steve Rowe
“Running automated test scripts can not be used to find new bugs in the software …” – Cordell Vail
Voices in our heads: test oracles
“Oracle: A mechanism to produce the predicted outcomes to compare with p p pthe actual outcomes of the software under test.” – BS 7925‐1
“A test oracle is a mechanism for predicting the expected results.”‐ Andreas Spillner; Tilo Linz; Hans Schaefer
“In real testing the outcome is predicted and documented before the test is run.” i i– Boris Beizer
10/7/2009
13
Challenge assumptions
• Is it necessary to predict the outcome of a test?
• Must an automated test behave like a human?• Must an automated test behave like a human?
• Can random inputs be used well?
• Does a test program really need to run quickly?
• How do humans judge if a result is correct?
• How efficient must a test program be?
• Why can’t test automation be used for exploratory testing?Why can t test automation be used for exploratory testing?
Never laugh at what you don't know. – Angus MacGyver
Blockbusting
• Challenge assumptions
• Investigate the problem
• Question all constraints
• Look at object attributes
• Brainstorm ideas
• Who else has done something like this?
• Imagine you have unlimited resources …
I i j i d f M• Imagine you just arrived from Mars …
Imagination is the most important thing the human mind has. – Angus MacGyver
10/7/2009
14
Part 1: Break the problem up
Part 2: Smart thumbs/Popular now
• Smart thumbs:– Create explicit domain
• Popular now:– Validate upon data entry– Validate upon data entry
10/7/2009
15
Part 3: Autosuggest
• Bug bash?G t d h k?• Generate and check?– Input from
• Log data• Exhaustive combinations
– Check with• Homemade utility• Homemade utility• PoliCheck
• Simple solution: Check source file
Part 4: Spell correction
• Generate and check– Input from
• Log data
• Mutated combinations
– Check with• Homemade utility
• PoliCheck
10/7/2009
16
Part 5: Images/Shopping
• Generate and check– Input from
• Log data
• Exhaustive combinations
– Check with• PoliCheck
If you don't have the right equipment for the job, you just have to make it yourself.– Angus MacGyver
• Homemade utilities
• Autonomation (people + machines)
Part 6: Bing overall
• Use Best‐of‐Breed as a test oracle
Commercialfilter
Do theseresults differ?
Results
web
Results
10/7/2009
17
Narrator: “It had been an unavoidable accident; one of those rare times when fog forms right out of mid‐air and in five minutes the visibility goes from ten miles down to zero. There had been no airport nearby; he couldn’t land. Unavoidable.”zero. There had been no airport nearby; he couldn t land. Unavoidable.
Narrator: “It had been an unavoidable accident; one of those rare times when fog forms right out of mid‐air and in five minutes the visibility goes from ten miles down to zero. There had been no airport nearby; he couldn’t land. Unavoidable.”zero. There had been no airport nearby; he couldn t land. Unavoidable.
Outlaw: “[Your] rule number eight told him never to land away from an airport. In his last five minutes of visibility, he flew over eight hundred thirty‐seven landing places –smooth fields and level pastures – but they were not ‘designated airports’ … so he didn’t even think about landing …
10/7/2009
18
Narrator: “It had been an unavoidable accident; one of those rare times when fog forms right out of mid‐air and in five minutes the visibility goes from ten miles down to zero. There had been no airport nearby; he couldn’t land. Unavoidable.”zero. There had been no airport nearby; he couldn t land. Unavoidable.
“Outlaw”: “[Your] rule number eight told him never to land away from an airport. In his last five minutes of visibility, he flew over eight hundred thirty‐seven landing places –smooth fields and level pastures – but they were not ‘designated airports’ … so he didn’t even think about landing …
“It’s up to us to keep flight alive in a world of airplane drivers – the people who come out of your school, concerned only with moving straight and level from airport to airport. We’re trying to keep a few real pilots left in the air. “
Narrator: “It had been an unavoidable accident; one of those rare times when fog forms right out of mid‐air and in five minutes the visibility goes from ten miles down to zero. There had been no airport nearby; he couldn’t land. Unavoidable.”zero. There had been no airport nearby; he couldn t land. Unavoidable.
“Outlaw”: “[Your] rule number eight told him never to land away from an airport. In his last five minutes of visibility, he flew over eight hundred thirty‐seven landing places –smooth fields and level pastures – but they were not ‘designated airports’ … so he didn’t even think about landing …
“It’s up to us to keep flight alive in a world of airplane drivers – the people who come out of your school, concerned only with moving straight and level from airport to airport. We’re trying to keep a few real pilots left in the air. “
‐ Richard BachA Gift of Wings
10/7/2009
19
What can you do?
• Approach test as an engineering problem
• Study up on
– Test approaches
– Your product
– Available tools
– Creative problem‐solving
• Testing books: Beizer, Binder, Jorgensen
h l b• Papers: Test generation, heuristic oracles, combinatorics
Always carry around a knapsack to pick up things you find along the way. – Angus MacGyver
April 19, 2005
Ed Smylie accepts the first Great Moments in Engineering awardon behalf of NASA’s Crew Systems Division who improvised an air filterthat kept astronauts alive on the ill‐fated mission 35 years ago.
Any problem can be solved with a little ingenuity. – Angus MacGyver
10/7/2009
20
The most important tool for survival isn't duct tape or a Swiss Army Knife, it's your wits. Always keep them about you.
– Angus MacGyver
The most important tool for survival isn't duct tape or a Swiss Army Knife, it's your wits. Always keep them about you.
– Angus MacGyver
Thanks for attending!
