Embed Size (px)
Citation preview
corporate quality consulting | Phone: +49 (0) 2241 25021 0 | Mail: [email protected]
Kunde
Projekt
Referent
Datum
Karlsruher Entwicklertage
Stephan Salmann
06. Juni 2013
Agiles Testen in Großprojekten
Agenda
Problem
Big Picture of the Solution
Unternehmenspräsentation
Some Detail Views
2
Das Unternehmen
Mission Statement Wir bauen die Brücke zwischen Business und IT
Mitarbeiter
Rolle
Mediator: für ein gemeinsames Verständnis zwischen Business und IT sowie Kunde und Lieferant
Advisor: strategisch und operativ für Analytik und Konzeption
Man of Action: pragmatische und zielorientierte Lösungsumsetzung
0
20
40
60
80
100
120
2005(Gründung)
2007 2009 2011 2013
Hannover
Siegburg
Frankfurt
München
Hamburg
Wien
Zug
3
Fundierte IT-Management Beratung
4
Spezialisten
Freelancer
commodity Leistungen
operativ
strategisch
spezifische Leistungen
Outsourcer
Integratoren
Strategie- berater Wirtschaftsprüfer
corporate quality
Services & Trainings der CQ
IT-Linienmanagement
Enterprise Architecture
Excellence
IT-Portfolio- Management
IT-Produkt- management
IT-Controlling
Geschäftsprozessberatung in ausgewählten Branchen
Programm- & Projekt-
management IT-Architektur
Requirements Management
Test- management
Projektumsetzung im Managed Service Ansatz
Projekt
IT-S
trat
egi
e
Sou
rcin
g &
In
tern
atio
nal
isie
run
g
Go
vern
ance
, Ris
kman
agem
ent
& C
om
plia
nce
Qu
alit
ätsm
anag
emen
t
& P
roze
sse
Trusted Business Communications
Social Media Business Integration
Mobility Solutions Banking Open Enterprise
5
Agenda
Problem
Big Picture of the Solution
Unternehmenspräsentation
Some Detail Views
6
Conway´s Gesetz
Bedeutung für 3 Teams:
1. Drei Teams implizieren 3
Haupt-Software-Systeme
2. Es sind 3 Team-
Kommunikationen notwendig
7
Bedeutung für 17 Teams:
1. Zehn Teams implizieren 10
Haupt-Software-Systeme
2. Es sind 136 Team-
Kommunikationen notwendig
Die Kommunikation zwischen den
Hauptsystemen ist, nur so gut wie die
Kommunikation zwischen den Teams.
(𝑛 − 1)𝑛
2
Options for large scale agile Projects
8
Comuni-
cation
Steering Meeting
structure
Integration of
results
Scale effects
Option 1: one big
Agile Team
-- (many to many)
-- (missing roles
and hierarchy)
-- (to many
participants)
++ ++
Option 2: many
small agile Teams
&. scrum of
scrums
-+ -+ -+ (to undifferent)
-- (individual
procedures and
tools)
-- (in allocation,
architecture,
provisioning)
Option 3: many
small agile teams
with an adequate
program steering
++ ++ ++ (focussed)
++ ++ favorite
approach
How to organize big project with i.e. 30 - 400 team members
Build Problems
Growing complexity of build
Large variety of inconsistent
library versions
Different compiler versions
No staffing flexibility between
agile teams
Functionality
Incomplete
Redundant parts
Integration problems
Fingerpoint between teams
Missing technical/functional fit
No scale effects
technical/functional redundancies,
Tools cost
low staffing flexibility
Large Toolsets
High Maintenance Effort
Our Experience in large scale agile projects using scrum of scrums
9
Defined by individual agile Team
Overall incomplete or redundant
Defined by individual agile Team
High Maintenance costs by the variety of tools
No scale effects
Problems with interoperability (cross functions: PM, SCCM, QM, …)
Software of different teams can not be integrated with a proper
technical or functional fit
The integration is not a goal of an agile team
The integration leads to a fingerpointing between the agile teams
More and more desorientation over time
Specialists for tasks are bounded in Teams
Extensive costs for learning curves when staff is reallocated in
other agile teams
The split in teams serves for a missing view on the big picture so
that technical cross functions are neglected
The steering on the program level is indirect via different roles in
different agile Teams
Focus
of agile
Team
Processes
Tools
Integration
problems
Staffing
flexibility
Functional
Silos versus
Horizontal
functions
Big Problems
Fuzzy Evolution of Functionality in parallel Scrum Teams
cummulative
Functionalities
1. Agile Team
2. Agile Team
3. Agile Team
Overlapping Development
Characteristics:
Redundancies
Incompleteness
Every team chooses
Own Architecture
Own Design
Own Implementation
Own Test
Own Success
……….
Incompleteness !
10
Evolution of oriented Scrum Team Developement (adequate steering)
1. Iteration 2. Iteration 3. Iteration
Cummulative
Functionalities
1. Mandate
2. Mandate
3. Mandate
Characteristics:
No redundancies
Completeness
Every team has
One Major Architecture
Disjoint task set
Integrated Tests
One Success
measurement
1. Agile Team
2. Agile Team
3. Agile Team
11
Agenda
Problem
Big Picture of the Solution
Unternehmenspräsentation
Some Detail Views
12
Sequence of Tests
13
Program-
management
Agile
Team 1
Business
Archi-
tecture
Technical
Archi-
tecture
Programm Governance Advisory
Config.
& Release
Manage-
ment
Integration
Test
Quality
Manage-
ment
Integration Team
Iteration 1
Agile
Team 2 Iteration 1
Agile
Team 3 Iteration 1
Inte
gra
tio
n T
est
Iteration 2
Iteration 2
Iteration 2 In
teg
rati
on
Test
RF
T
RF
T
RF
T
Iteration 3
Iteration 3
Iteration 3
Inte
gra
tio
n T
est
RF
T
RF
T
RF
T
Man
date
M
an
date
M
an
date
copyright by CQ
Ma
cro
La
ye
r
Pro
gra
m/D
om
ian
Mic
ro L
aye
r
Pro
ject/A
pp
lica
tion
ST
Daily Build
RST
Daily Build
RST
Daily Build
RI RI
Accep
tan
ce T
est
ST ST DT
FT
DT
DT
DT
DT FT
FT
FT
FT
DT FT
DT FT
DT FT
DT FT
DT-Developer Test // FT- Functional Test // ST – Smoke Tests // IT – Integration Test // R<X> - Regressiontest
Operating Phase
Delegate
Ressources
Define
Technical
Domain
Architect.
Define
Business
Achtitect.
Setup the Organization
14
Define Major
Business
Capabilities Program Governance Advisory
Integration Team
AgileTeams
Agile Team 1
Agile Team 2
Agile Team 3
Ma
cro
La
ye
r
Pro
gra
m/D
om
ian
Mic
ro L
aye
r
Pro
ject/A
pp
lica
tion
M
M
M In
teg
rati
on
Test
Setup Phase
Man
date
s:
iterative
Path of the agile Solution Development for one Agile Team
Start
Mandates ~
Cones of Freedom
1. Iteration 2. Iteration 3. Iteration
Fu
nctio
nalit
y
Explanation:
Agile Team receives a
Mandate per Iteration
from the PGA
A Mandate consists of
the Application(s) to be
build and the
capabilities of be
realized
As long as the agile
stays in the cone of the
mandate it moves from
sprint to sprint
If it moves outside the
cone, it has to ask for
approval of the PGA
S1
S2
S3 S4
S1 S2
S3
S4
S1 S2
S3
S4
final
Application
S - Sprints Round Tables Mandates (Application/Capabilities) from the PGA
Time
15
Typical (functional) Testprocesses in large scale agile projects
16
Test Task Quality Criteria Test Object Test Basis Layer Team/Role Regression/
Capture
Regression/
Replay
Remarks
Developer Test
DT
functionality Class capability, user
story
Agile Team Developer always, while
testing
dailiy within daily
build & test
procedure
see J Unit;
special
frameworks
Fuctional Test
FT
functionality application in the
definied
integration
degree (per
component)
capability for the
definied
integration set
Agile Team Team
Member/Tester
when sufficient
stability is
reached, mostly
late in sprint
in the same sprint
and selectively in
the forthcoming
sprints (to proof
the stablity of
already
implemented
functionality) -
RFT
reuse of mock
ups and stubes;
utilization of
common
testing tools
Smoke Test
ST
executability Daily Build over
the components
in their released
status; integrity
of added
interfaces
compiler;
selected streams
through the
business process
model
all Agile Teams,
integrated
Testcase:
Developer;
Testexecution:
C&RM-Team
potentially daily,
but selective
potentially daily -
RST
end 2 end tests;
no proof of
correctiveness
Integration Test
IT
interface
integrity; Sematic
Integrity;
correctiveness of
state transitions
Technical Domain
(all applications
of theprogram) in
their defined
integration
degree
business process
model or
business domain
model (including
business objects
& functionality)
Program Layer Integration Test
Team
selective executed in
Integration + 1 -
RIT
end 2 end tests
incl. proof of
correctness
Acceptance Test
AT
functionality (of
the complete
System)
Technical Domain
(all applications
of the program) in
their final
integration
degree
business process
model or
business domain
model (including
business objects
& functionality)
Program Layer Business Units in
their common
organization
no no end 2 end,
incl.proof of
correctness
non functional tests based on the program risk assessment on all integration levels possible (L&P, usability, security, operations tests, …)
Agenda
Problem
Big Picture of the Solution
Unternehmenspräsentation
Some Detailed Views
17
General Operating Model
18
Enterprise Architecture
Integration Manager
… Project 1 Scrum Master
Project 2 Scrum Master
System Integration Team
Program Manager
M M
M
M
Project Config Mgr.
Business-,Information-, Application-, Integration- Architecture; Release Planning
Release Planning (Components & Capabilities)
Package delivery (Application)
Program CM System
all artefacts checked in & labeled
Baselining Building Packaging
Components of the environment
Run & maintain the environment
Archive (Package, Sripts, Data,…)
Single point of control
Project 3 Scrum Master
Product Owner
Customer
Team End-user
Test Environment Responsible
Integration Team
Orgchart Integration Team
19
Integration
Testmanager
Problem Resolution
Integration
Test Team
Integration Test
Envir. - Mgmt
Config.
& Release
Management
Config
Administration
Build Run the
Environment
Test Data
Deployment
• Test case determination
• Test data definition
• Test execution
• Test evaluation
• Regressions tests (Integration tests)
• Test case sellection
PL
Execution
ST, RST, RFT
• Scheduled Builds
• Build Automation
• CMDB
• Test Automation
• Scheduled Tests
• System
Administration
• System
Deployments
• Preprocess ing
Tickets
• Retrieving
Testdata
• Uploading
Testdata
Quality
Management
(not in scope)
correct
End 2 End
processing
accross all
results of
all
agile teams
all function receive the data they require
i.e. are the underlying securities sufficient for a loan offering?
….Security data are available and approved
all functions transform data only for defined states of the business objects
i.e. does a terminated customer still receive services?
… The customer status is „terminated“; all functions, except „renew membership“ are
not valid for this status
All data are interpreted the same way throughout the whole system/all applications
i.e. does credit rating „A“ mean highest soundness in all applications?
… the value „A“ provides the same meaning for all functions, which interpretes this
data
Which Quality Criteria do we Test with Integration Testing?
20
Interfaces
State Transition
Semantic Integrity
Remark: The correct implementation of functionality in applications is tested in functional testing. This
quality is given and has not to be tested again. Therefore Integration Testing focuses on….
Functional Testing versus Integration Testing
21
Data Object 1
Function 1 Function 2 Function 3
X
X
X
X
X
X
X
X
Integration Testing
• End 2 End workflow per data object
• Low coverage ove all functions
• High coverage for state transitions and
interfaces
• Often done with (migrated) productive data
• Labor intensive for automation
Functional Testing
• all features/user stories of this function
• in deep (high coverage)
• synthetic test data
• suitable for automation
Fct
Fct
Fct
Fct
Fct
Fct D
D
D
D
D D
D
Fct Fct
IT-Landscape
D
Fct D
Fct
D
Appl. 1 Appl. 2
Data Object 2
Data Object 3
Data Object 4
22
Environments, integration grade and activities on the envirnoments, push/pull mechanisms
Development System Test
Environment
Weekly Build & Test
Pre-Production Production
Component System System of Systems
(Technical Domain) Coupled
Systems
Coupled
Systems
Versioning
(CMDB)
Component Test
Comp. Integrat. Test System Test (ST)
RST
SMT, RSMT, RSIT
Acceptance Tests
(OAT, BAT)
Functional SIT
Environmanet
SIT (FIT, UT)
Non Functional
SIT Environment
SIT (SLT, SPST, FOT)
pull
push
Build
Engine
pull pull
pull
pull
pull pull
push
Maintenance
Environment
Maintenance Tests
(analog ST)
pull
► ST: System Test
► SMT: Smoke Test
► SIT: System Integration Test;
five Tasks
► AT: Acceptance Test (BAT,OAT)
► (R) Regression Tests
Consequences of the Daily Build&Test Process
► Build Overall Team Development
► Test Overall Team Development
Agile
Team 1 Iteration
Agile
Team 2
Agile
Team 3
Iteration
Iteration
► The Outcome is delivered to the
Agile Teams
► Interface Problems are discussed by
the Hotfix Team and Chief Architect
► Targets
compilable software
Running basic functionalities
Awareness of Team interaction
Architects awareness on software interaction
Early error indicates of design
Build automation
Deployment automation
Test automation
► Daily &Test as early as possible
ST RST RIT
Inte
gra
tio
nte
st
in Daily Build
23
Entry Exit Criteria
24
Iteration 2
For Application 1
Developer
Test/I Functional
Test/I
Functional Test Environment Developer
Environment
Integration Environment
Application 1
Application 2
Application 3
Smoke
Test/I
Regression
Smoke Test/I-1
Regression Inte-
gration Test/ I-1
Part of the Daily Build
Inte
gra
tio
n T
es
t
Regression
Functional
Test/I-1
Entry
Criteria
Exit
Criteria
Entry
Criteria
Test Complete
ness
Max.
Failures
DT approval
FT all user
stories
(0/A);
(15%/B)
RFT 20% of
all RFTs
(0/A);
(15%/B)
ST/RST 10% of
No. User
Stories
100%
execut-
able
RIT 30%
auto-
mated
(0/A);
(5%/B)
(Example)
Exit
Criteria
Integr.at.
Test
Completeness
Coverage All Business
Objects & States
Coverage All capabilities
Coverage All Interfaces
Max.
Failures
(0/A)
(15%/B)
(Example)
Remark:
SW is deployed in FTE and ITE in parallel
ST, RST, RIT are executed dialy after build
RST, RIT are reduced Test Sets
Wilhelmstraße 63
D-53721 Siegburg
Phone: +49 (0) 2241 25021 0 Fax: +49 (0) 2241 25021 29
Mail: [email protected]
Web: www.corporatequality.de
Rooseveltplatz 13/5
A-1090 Wien
Phone: +43 (0) 1 89 03 448 Fax: +43 (0) 1 89 03 448 15
Mail: [email protected]
Web: www.corporatequality.at
An der Lorze 11
CH-6300 Zug
Phone: +41 (0) 41 740 39 19 Fax: +41 (0) 41 740 39 20
Mail: [email protected]
Web: www.corporatequality.ch
© 2005-2013 corporate quality | Verwendung nur mit Zustimmung der corporate quality
Stephan Salmann, CEO
corporate quality consulting GmbH
Phone: +49 (0) 2241 25021 10
Mobile: +49 (0) 163 327 27 02
Fax: +49 (0) 2241 25021 29
Mail: [email protected]
Web: www.corporatequality.de
Thank you for your Attention….
Branch, Merge and Packaging Strategy
System of Systems,
Release and ST
approved Applications
System of Systems
In Weekly
Integration
Grade &
CT, CIT tested
Application
Under
Development
Released,
Application
(Release
Candidates)
Component
R1 R 1.1 R 2
Package R 1.1
Development, CT & CIT
Branch Merge &
Label
Package R1
System Integration Test
Package R1 Package R2
Package R2
Remark: Only Happy Path,
Foward Look, without
patching
System Test for Capabilities R1 of Application 1
Weekly Build &Test
