Embed Size (px)
Citation preview
WK2 Keynote 10/5/2011 10:00 AM
"I Didn't Know I Knew That: A Story of Self-Learning"
Presented by:
David Hayman Qual IT Software Testing Solutions
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073 888‐268‐8770 ∙ 904‐278‐0524 ∙ [email protected] ∙ www.sqe.com
David Hayman Qual IT Software Testing Solutions
David Hayman has been in IT for twenty-seven years with twenty-three of those focused on software testing. Originally from the UK, David has lived in New Zealand for eight years. He is currently the test practice manager for Qual IT Software Testing Solutions in Auckland, delivering testing solutions, and mentoring and training IT teams across a gamut of industries and applications. A contributing author to The Testing Practitioner, David is a thought-provoking and entertaining speaker at events in the UK, Europe, and Australasia. Actively involved in the ISEB/ISTQB certification program since 1998 as a trainer and exam writer, David is currently the Chair of the Australia and New Zealand Testing Board.
1
Welcome
I didn’t know I knew that; until I asked myself
1
David HaymanStarWest, Disneyland Hotel, Anaheim, California
October 2011
[email protected]+64 21 293 6604
Today’s ObjectivesIdentify what can be done by people using a process and a positive attitudep p
Differentiating the ‘what’ and the ‘how’
What were the objectives of the testing?
Look at the approaches taken
How test case design techniques were used
2
How test case design techniques were used
Consider what is really in a name
Share a few good and bad things I learnt
2
Can you test this please?A voice activated internal Directory system
We’ve got a couple of possible tools
It’ll need performance testing in conjunction with a DTMF system in production
Any business experience?
Any testers with experience?
3
Any testers with experience?
Any help at all?
Why are we doing this?
Why me?
Sure, now what was the question again?
What do I know?Wh d I k ?Who do I know?Where can I look for help?What have I done that is similar?How hard can it be?Why did I say yes?
“If k h t i th h I f il?” ☺
4
“If no one knows what success is then how can I fail?” ☺
On the other hand……
“If no one knows what success is then how can I succeed?”
3
It’s not the ‘what’ but the ‘how’
5
We know the ‘what’
HealthBankingInsuranceSurveysShoppingEd tiEducationCommunication[Anti] Social Media
6
4
The ‘how’ is different
We may not know exactly how but y ywe know whySometimes it’s just another name for the same thingArtificial boundaries and constraintsIf I need to be told to think I’m in the If I need to be told to think I m in the wrong job!
7
Constraints
I’ve learned more since I prepared p pthis presentationWe don’t all have the opportunities afforded to someDo we need to know everything about the ‘how’ before we can even consider starting – of course not
8
5
Sources of inspirationThe Internet[Anti] Social media[Anti] Social mediaISTQBBS7925-2; test case design techniquesUsersCustomersBA’s
9
BA sTesting GurusOther companies with voice systemsUsability groupsMe – I like a challenge
What didn’t we know?Risks to the project and productWas the testing we planned:Was the testing we planned:
ValidValuableComplete
As they say – “we didn’t know what we didn’t know” and it wasn’t just us!jEqually no one knew what they were signing off, but they did know what they wanted.
10
6
Key Considerations
Good IVR more ‘personal’ and easier pfor people with some disabilities e.g. the blind [or visually impaired], dyslexic or paralysed than DTMF.
Bad IVR can give you blood pressure g y pand make you throw things!
11
Business Requirements
5 seconds to answer the callGo round three times [became twice]Bad languageUnrecognised nameSilenceHang UpsHand off to operatorDepartment vs. individualAll lines busy – engaged toneRequired line busy – go to voicemail
12
7
State Transition Models
Turning requirements and call flows into d lmodels
Establishing 100% coverage Identify test conditions, input data and expected resultsUsed for impact analysis on change requestsUsed to identify key areas for regression
13
Used to identify key areas for regression
14
8
Wizard of Oz TestingFollow the yellow brick road
What is it?We tested usability in at the start rather than the usual at the end approachGap analysisTest driven designA review with a fancy name?
15
A review with a fancy name?
How does it work?Pseudo Scrum included Marketing
Did it work?Yup
Say SomethingWhakatane
k = wak a tane
In the Maori alphabet there are only 10 consonants and 5 vowels and no ‘F’It’s all about pronunciationWhat we say the way we say it and what What we say, the way we say it and what people hear
Whakatane= fuk a tar nay
16
9
Put the ‘Fun’ in FunctionalTypes of name
IndividualsIndividualsDepartments
Fun with test data:Mark EtingThe Reverend Ndabaningi Sithole John Smith/Joe Smith/Rob Smith/Bob Smith
17
Phonetics35 Accents [all within IT department] –wide involvement in the projectPhonetic vs. pronunciation
PlatformsHomeMobileMobileSkypeSpeakerHands FreePublic call boxes – 2 common issuesCar Hands free
18
Where is the caller, what is going on in the background
Background noise and how the IVR copes –again relate to ‘noise’ in a performance test but this can be done on an individual call
10
Actual Recordings
Wav filesFirst in NZ to use this toolUI and reporting issuesTool recognition thresholds – levels of correctness - NuanceCodec changes introduced ‘clicks’Jitter was an issue [Ji i l i Jitter was an issue [Jitter, an irregular time variation of period signal properties, such as small, unpredictable delays in scheduling]
19
Script #1Step/Actor Response
Step 1IVR Text to Speech "Hello and welcome to the StarWest Interactive Voice Response system. To
help us direct your call please choose one of the following options. Say "Speakers", “Tickets“ , "Free Gifts” or "Operator"
Step 2Caller Silence
Step 3IVR Text to Speech I'm sorry I didn't catch that. To help us direct your call please choose one of
the following options. Say "Speakers", “Tickets“ , "Free Gifts” or "Operator"
Step 4C ll "Ti k t "
20
Caller "Tickets"
Step 5IVR Text to Speech To confirm did you say "Tickets"?
Step 6 YesCaller
Step 7IVR Text to Speech Thank you. Putting you through to tickets
11
Call #1 – Well that’s a good start
21
Call #2 - Did you hear that?
22
12
Call #3 – Are you sure?
23
Sample problems
Roger RoffLearning could be fooledBad languageSilenceBackground noiseStepping over the thresholdEstablishing “confidence levels”
Example - Peter Gouws
24
13
Performance Testing
ToolsIBM/Ci R t t k it iIBM/Cisco Router network monitoringCyara ‘Cruncher’ Tool configured as a service not in-house – more realistic
Operational ProfileLimited to 120 lines and handled a mixture of DTMF and IVR – included a 4 hour ‘Soak Test’
25
Test’
Manage scripts to ensure silence added so that call was fully answered before the message started
Test Environment
Careful use of Production switches and network and phone linesTest environments – had to use external hardware, therefore control over quality was impossible.DTMF ‘tromboning’ issue caused problems with IVR throughput and line availabilityOther than that all good ☺
26
14
Other issues
Social engineeringg gHow to stop people phishingStutters, lisps and other speech impediments
27
The end result
Good functional and performance testing with the product supported by the business
BUTThe quality of the product was overridden by the feeling in the marketVoice systems at the time were getting a bad rap
28
bad rapTherefore a good system that could have improved the image of the company was lost to the general public because of the business risk assessment.
15
Good things that happened Part 1
Voice Talent – new careerPrevention rather than cure – always goodGap analysis – we missed how much?Fully Documented – OK I know!Enthused the business and others to get involved in testingVery formal test processVery formal test processConfigured the system rather than tested it
29
Good things that happened Part 2
Test driven design or test driven configuration?Replaced ‘Good Enough’ with
BEWECODOGITHELIOUS or BEWE TestingA Review by any other nameImproved the UI on the Cruncher toolMade some friendsMade some friendsLearnt a lot about the tool, myself and my teamNot all systems that pass acceptance testing and are essentially bug free will go into production!!! 30
16
AcknowledgementsAlok Kulkarni, Bonny Malik and Thomas Fejes @ Cyara Solutions www cyarasolutions comCyara Solutions www.cyarasolutions.com
Nick Brown and Piers Langridge are to testing what Batman and Robin are to crime fighting
31
Akash Jattan @ IBM New Zealand now in Oz
Anna Creery – IVR voice – available for work now ☺
Call #4 – The final straw
32
