Upload
icans-gmbh
View
4.839
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Citation preview
1
programs testing programs GUI-based test automation
1st Friday – 02.03.2012
Sascha Bartels
2
1. What‘s „test automation“?
2. Why do you do that?
3. How do I know, that I‘m doing it right?
4. Is our test automation cool?
3 3
01 | WHAT‘S „TEST AUTOMATION“?
4
A user browses with his browser on PokerStrategy.com
5
regression tests before each release of PokerStrategy.com
6 6
02 | WHY DO YOU DO THAT?
7
Possible objectives of test automation:
- find more bugs
- nightly regression tests
- reduce testing staff
- shorter test periods
- test more
Frequent problems of test automation: [1]
1. unrealistic expectations (of the management)
2. poor testing processes („Automating chaos just gives faster chaos“)
... etc.
Let us examine our goals more exactly!
1: Software Test Automation Effective use of test execution tools – Dorothy Graham & Mark Fewster
8
find more bugs
the idea:
manual testing finds bugs
automated tests are faster than manual
tests
so automated tests find more bugs
9
find more bugs
the practice:
- automated tests are repeated tests
- looks for unexpected side effects
- so automated tests can find
more regression bugs
- testers have more time for manual
tests
- poor tests are not getting better by
automating them
10
nightly regression tests
the idea:
There are unused ressources
(like test systems e.g.)
We could run automated tests „while
we sleep“
11
nightly regression tests
the practice:
the correct tests have to be selected
test results need to be analyzed
12
reduce testing staff
the idea:
The automation tool costs money, so we
should be able to save somewhere
else
We want to reduce costs and staff
is expensive
13
reduce testing staff
the practice:
tests have to be automated and maintained
test results must be analysed
test automation can make work easier
for testers
14
shorter test periods
the idea:
reduce deadline pressure
testing is a „bottleneck“, so we will save
time there
15
shorter test periods
the practice:
the goal can be achieved more easily:
run fewer tests, omit long tests,
cut regression testing
Automated tests take time:
creation and maintenance of test
cases
analysis of test results
software quality has the greatest influence on the time needed
for testing
16
test more
the idea:
achieve a better test coverage
testing is good, so more testing
has to be better
17
test more
the practice:
unimportant tests produce more
Maintenance instead of being gainful
some tests require higher
automation effort,
e.g. for technical reasons
18
Possible realistic objectives
- find more regression bugs
- automate the most importants tests
- a useful amount of tests run at night and / or weekend
- relieve testers and support manual testing
It is important to know why you automate, so you know what and how to
automate.
Objectives of testing should not get mixed up with the objectives of
automation.
Test automation is not a tool against bad test practices
That‘s No Reason To Automate! Why Good Objectvies Are Critical to Test Execution Automation – Dorothy Graham and Mark Fewster
19 19
04 | HOW DO I KNOW, THAT I‘M DOING
IT RIGHT?
20
measuring
metrics
Depending on the objective and the initial situation other metrics are
important.
Will this car fit into my garage?
Does this one consume as much petrol as this?
There might be a hole in the tank.
Yes?
21
1. DDP– Defect Detection Percentage
Is the relation between found and currently known bugs.
Example: A release contains 100 new bugs, the test run finds 70 bugs:
DDP: 70%
Weaknesses in this approach:
1. The absolute amount of bugs is unknown.
2. It is unknown at what time a bug has come into the product.
3. Not all bugs are equally critical.
4. Not all bugs are equally difficult to find.
5. No reflection of the absolute count.
It is easy to determine this measure and it can have a high significance.
22
1. ROI – Return on investment for process optimization
current process optimized process
Costs of test execution 10 000 € 20 000€
DDP 70% 90%
found bugs 700 900
bugfixing (100 € per bug) 70 000 € 90 000 €
bugs found after release 300 100
bugfixing after release (1000 € per bug) 300 000 € 100 000 €
total costs 380 000 € 210 000 €
savings: 170 000 € investment: 10 000€ ROI (savings/investment): 1700%
23
1. ROI – Return on investment in test automation
manual testing automated testing
Costs for test case design 6 000 € 6 000 €
costs for automation 0 16 000 €
total one-time expenses 6 000 € 22 000€
costs for test execution 5 000 € 1 000 €
amount of test execution cycles 3 3
costs per release 15 000 € 3 000 €
total costs of 4 releases a year 66 000 € 34 000 €
savings per year: 32 000 € investment: 16 000€ ROI (savings/investment): 200%
24
My test automation is much cooler than
yours!
25
How do you get that idea? - I have a lot of test cases - which run every night for several hours - and provide lots of test results
26
- Well, I have:
- saved more hours of manual testing (effectiveness) - spent less time on test case maintenance (maintainability)
- more bugtickets per fault report (reliability)
- less training time for new employees (usability)
27
Hm… - I need to edit many tests for each release, so that they can be executed - I need 4 hours a day to analyse my logs files - if I find a bug, it is most likely caused by my test environment - my new colleagues need 2 weeks of training before they can help me - and after every release we have to fix many bugs that were missed during testing
28
And then I have: - less effort to find the test case that tests a specific range (flexibility) - fewer test cases blocked by the same software bug (robustness) - less effort to run test cases on different test environments (portability)
29
30 30
05 | IS OUR TEST AUTOMATION COOL?
31
We have 18 test suites for:
- conversion path tests, submit poker accounts, support ticket system,
fraud check, upload images, buy points packages, video section…
- 865 performed tests
- 9 hours test run every night
However, we have two test systems (integration and system test) and almost
all tests run in Internet Explorer and Firefox.
- about 3400 test executions
- 36 hours of test runs in total
- 5 virtual machines for test execution
32
1. maintainability
central master suite with:
- object repository
- procedures for common test steps (e.g. login)
33
2. efficiency
- before the introduction of test automation:
12 hours of manual regression testing for each release and test
environment
- today:
4 hours of manual regression testing for each release and test
environment
3. reliability
solved problems with external dependencies (Facebook, Google ...)
- requests are routed via the host file on an internal web server
- answers with 1x1 pixel images or empty responses to scripts
34
4. flexibility
- a test suite per functional section
- each test suite can be started by pressing a button in Jenkins
- free choice of browser and the test environment
5. usability
- graphical abstraction of test scripts in QF-Test
- using Jython or Groovy scripts (database access, HTML)
6. robustness
- 3 test suites are currently on “failed", there are 2 bugtickets
- a test suite is unfortunately affected by a bug in Firefox
35
7. portability
- choice of the test environment and the browser via parameter
- at the moment QF-Test supports only Internet Explorer and Firefox
36
Any questions?
37
ICANS GmbH
Valentinskamp 18
20354 Hamburg
Germany
Phone: +49 40 22 63 82 9-0
Fax: +49 40 38 67 15 92
Web: www.icans-gmbh.com