Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
Testing – the Key to an Efficient and Predictable Software Engineering Process 28.05.2008
(c) AdNovum Informatik AG 1
Testing – the Key to an Efficient and Predictable Software Engineering Process
Dr. Tom Sprenger CIO AdNovum Informatik Dipl. Informatik-Ing. ETH
May, 28th 2008 @ ETH Zürich
2
AdNovum = Engineering Quality
Core Competence High-End Software & Security Engineering
Customers Financial services, Banks, Private banks, Swiss Post, Government, and Insurance
Key Data Est. 1988 170 employees, 70% graduate engineers Zurich (HQ), Bern, Budapest (Hungary) Strategic Partners
Sun, BEA, Swiss Post, HP, IBM, Oracle, SafeNet, Siemens Switzerland, Vasco
Testing – the Key to an Efficient and Predictable Software Engineering Process 28.05.2008
(c) AdNovum Informatik AG 2
3
Agenda
Starting Position
Motivation
Challenges in Practice
Quality Assurance System
Constructive Measures SW-Engineering Process Technology Management Risk Analysis Testing Toolbox
Analytical Measures Automatic Testing Continuous Testing (NightlyBuild)
Q&A
4
Starting Position
Testing – the Key to an Efficient and Predictable Software Engineering Process 28.05.2008
(c) AdNovum Informatik AG 3
5
Motivation
Improving quality of the results while reducing risk
Improving efficiency while keeping high quality
Manage distributed development
Near-/Off-shoring
Provide periodical information points for project management
Controlling
Provide measurements for assessing improvement efforts
Well established Testing/QAE is a differentiation in the market
6
Challenges in Practice
Time/Budget vs. Quality/Testing
Functionality vs. Quality/Testing
⇒ Testing/QA must be part of the offer and the development process!
Problem of testing today: “80% of the time is used to build up a suitable test environment; only 20% of the time remains to actually write and execute tests”
⇒ Establish an efficient testing infrastructure and make use of templates and blueprints
Tests must be 100% repeatable and reproducible
⇒ Prerequisite for efficient testing is a well defined software engineering process
Testing – the Key to an Efficient and Predictable Software Engineering Process 28.05.2008
(c) AdNovum Informatik AG 4
7
Quality Assurance System
8
Constructive Measures: SW-Eng Process
The core process 1. Analysis & design
2. Development / implementation
3. Test & integration
4. Manage & optimize
Each phase defines Roles & tasks
Deliverables & checkpoints
Testing – the Key to an Efficient and Predictable Software Engineering Process 28.05.2008
(c) AdNovum Informatik AG 5
9
Constructive Measures: SW-Eng Process
What about a Quality Assurance Process?
There is NO such thing as a QA process Avoiding the risk of working for the QA process (instead of SW-Eng Process)
Only process is the Software Engineering Process
We follow an “Aspect-oriented QA-Approach" define Q-Aspects along the Software Engineering Process keep the focus on producing software enable the developers for QA QA is an integral part of engineering software
Each Q-Aspect has one or more Q-Metrics attached Q-Metrics can be verified automatically
10
Q-Aspekt: Risk Assessment
Q-Aspekt: A&S Sign-off
Q-Aspekt: Code Metrics
Q-Aspekt: Test Coverage
Q-Aspekt: A&S Sign-off
Aspect-Oriented QA-Engineering
Testing – the Key to an Efficient and Predictable Software Engineering Process 28.05.2008
(c) AdNovum Informatik AG 6
11
Constructive Measures: Tech. Management
Software tool set 3 levels Strategic Tactical Operative
Central information system Explicitly show dependencies
Development from multiple locations
Expert know-how
Ability to anticipate
Cost control
Condition: automation
12
Constructive Measures: Risk Analysis Matrix
Testing – the Key to an Efficient and Predictable Software Engineering Process 28.05.2008
(c) AdNovum Informatik AG 7
13
Constructive Measures: Testing Toolbox I
Test Type Tool Description
Source Code Metrics PMD
JDepend
Checks Java code against a defined set of rules to detect common errors.
Generates design quality metrics for each Java package to measure the quality of a design in terms of its extensibility, reusability, and maintainability.
Functional Testing JUnit
Cactus
JTestCase
Tests units of Java code.
Simple test framework for unit testing server-side Java code (Servlets, EJBs, ...).
Enables a systematic separation of test code and test data.
14
Constructive Measures: Testing Toolbox II
Integration Testing HttpUnit
Selenium
Emulates basic browser behavior to test form submission, JavaScript, basic HTTP auth., etc.
Used for Ajax web sites or extensive JavaScript functionality.
Code Coverage Emma Discovers which parts of the code have not been executed during the tests.
Load Testing Sams / Grinder
Proxy Sniffer
Tests your system with heavy load, re-using your integration tests.
Load generator that supports customized load and stress testing of web applications.
Profiling JProbe Detects performance problems and memory leaks in Java code.
Testing – the Key to an Efficient and Predictable Software Engineering Process 28.05.2008
(c) AdNovum Informatik AG 8
15
Analytical Measures: Automatic Testing I
Automatic Testing is your friend!
Prerequisite
Homogenous environment over all projects in your company Environment supports script-based execution (no user interaction
required) Make consistent use of versioning (source and 3rd-party components) ⇒ These are the minimum requirements for reliable, reproducible builds
Defined project structure Defined set of mandatory build targets (e.g. init, build, test, etc.) with Well-known semantics
Well-known result/output
16
Analytical Measures: Automatic Testing II
(Automatic) testing starts with the specification Risk Analysis (minimize effort, maximize benefits) Use Cases Requirements (services, test data, etc.)
Make use of Testing Toolbox (JUnit, HttpUnit, Selenium, Grinder, ...)
Distinguish different aspects: functional, performance, load testing Use clever approach to enable reuse of functional tests for load testing
Distinguish between test scenario and test cases test scenario test code test case # test data
Separate test data from test code and enable reuse of test code
Define clear and reasonable quality and testing goals!
Testing – the Key to an Efficient and Predictable Software Engineering Process 28.05.2008
(c) AdNovum Informatik AG 9
17
Analytical Measures: NightlyBuild I
Continuous Integration Testing Framework
Builds all our active projects every 24 hours Repository checkout of the project head (latest version) Verification of source code quality (code-style metrics, pattern detection, etc.)
Complete build of the entire project Start-up of test environment
Project deployment Execution of tests (e.g. source quality, unit, integration, and load tests) Pull-down of test environment
Reporting
Statistics: Number of Builds: 292 Number of Jobs: > 5000 Number of Platforms: 15 Number of Hosts: 29
18
Analytical Measures: NightlyBuild II
Testing – the Key to an Efficient and Predictable Software Engineering Process 28.05.2008
(c) AdNovum Informatik AG 10
19
Analytical Measures: NightlyBuild III
: :
20
Analytical Measures: NightlyBuild IV : :
Testing – the Key to an Efficient and Predictable Software Engineering Process 28.05.2008
(c) AdNovum Informatik AG 11
21
Analytical Measures: Application Web
Display project dependencies as a graph
Investigate critical project dependencies
Get information about Software versions in use Unused versions …
Toolbox information
22
Quality Assurance Engineering Team
Project Support Testing opportunities consulting Setup of testing infrastructure
OpenSource Tool Productizing JDepend integration Emma extensions HttpUnit & Selenium extensions
Meta-Software-Engineering Autom. architecture validation Autom. risk-based code coverage AQUARIUS - Quality and Risk Unit
Sensing
Customer Projects
Testing – the Key to an Efficient and Predictable Software Engineering Process 28.05.2008
(c) AdNovum Informatik AG 12
23
Questions?