Software Testing in a Distributed EnvironmentNeil Langmead

Unrestricted

2

The Agile DevOps World

Many stakeholders Distributed across teams Huge potential for

misunderstanding Lost In Translation…• And this is just English!

3

The Distributed Agile World

Requirement to: Merge in parallel Stop bugs proliferating Testing Early and Often Fix Before the Merge! Time Zero Fixing Fast Desktop Feedback

/main branch

BL130Released to customer

devel: john

/main/task115New loading form

devel: pat

devel: pablo

/main/task114Typo in about form

/main/task116GDI resource leak

This check-in introduces a bug, but only “john” will get affected by it until he merges! He has a chance to fix it before affecting anyone

4

The Importance of Scalable SCM

DSCM systems require as a minimum: Scalability, to 80+ MLOC (e.g. PLM) Robustness, security Efficient branching and merging Automated Workflow, API Rapid Release Cadence

5

Process Transformation- Siemens CT

Strong correlation in net new coding violations ofgood practice (blue)along normalizedcode churn (orange).

Post Perforce, newviolations flatten to zero- why?

6

The Problem of Complexity

Complexity Manager is now an official role!

LOC estimated to increase by 20% in 2016

Multiple build targets, where only a few source files may differ, complicate the testing process

7

The Problem of Technical Debt

8

The Problem of Technical Debt

Like financial debt, difficult to control/ pay back

Accumulates even when we do nothing (e.g. lib versioning)

! The importance of the NULL

Release• How long does it take for your

organisation, if 1 LOC is changed, to up issue to a new version? • Can be weeks, or even

months…

9

How Do We Test?

Need Total Testing approach• Consider FOSS implications• Logic Flow/ Syntax / Semantic Analysis• Architecture• NFR (reliability ,security…)• Unit Testing / Coverage• BVA, Equivalence Partitioning• Fuzzing• Concurrency/ Threading issues• Proliferation through branching

10

How Do We Test?

“Connected Desktop” Developer IDE’s connected

to build server AND SCM Time Zero Approach• Don’t let the bug grow!

On the Fly Analysis• Eclipse, Visual Studio, etc

11

How Do We Test?

12

How Do We Test?

13

Change Based Testing (CBT)

How about we only test what’s changed?

BUT how do we identify changed components?

Tight integration between SCM, Jenkins, JIRA

High levels of Automation Continuous Testing

14

CBT & Invasiveness

• Define & Measure Invasiveness of a defect

Compute transitive closure of a change, to assess impact on the system

Can be done on feature, defect or enhancement level from the SCM repo

15

CBT & Invasiveness

16

CBT – Issue Invasiveness

• Set of Scripts to interrogate the Helix repository

• Selects on IssueID, Author• Cross Referencing Matrix

yields the required change metrics, which is then passed to the unit testing tools

17

CBT – Least Common Ancestor

18

CBT – Least Common Ancestor

19

Insights from Repository Analytics (RA)

Other insights become apparent from Helix RA:Comparison measures contractor productivity (violations of good practice, contributions, change impact) by project & location.

20

Insights from Repository Analytics

Other insights become apparent from Helix RA:Comparison measures contractor productivity (violations of good practice, contributions, change impact) by project & location.

21

Test Driven Development Analysis

Outlying project reveals one developer (ID #1040) who is contributing greatly (i.e. skilled and experienced) but adds few new tests. “1040” needs to on boarded regarding the TDD standard for for greater compliance and project quality.

22

Do we now need a Dynamic Architecture?

Architecture is the hardest part of a SW system to change

Often set early, not very adaptable

Expensive, technically challenging

“Elastic Architecture” methods required• Component based• Flexible API, “Plug and Play”

23

Conclusions

Complexity is here to stay Need advanced, efficient methods to reduce test effort Use Branch Per Task/ Test Per Task Change Based Testing Approach to reduce technical debt Time Zero Testing before the Merge Advanced Repository Analysis to gain additional insights Agility in the Architecture is the Next Frontier (Neil’s opinion…) “Continuous Testing” is here to stay

24

Demonstration

demonstration of the Total Testing Platform, in use at Siemens

Thank you for your attention!