Outline - so · PDF file–To verify scalability of cloud solutions –To iterate on complex configuration scenarios which are time consuming for manual testing

1

Outline

• Intro

• Introducing Continuous Integration on system level @Philips Lighting– Transition of an important R&D activity towards CI

• Technical achievements & backlog

2

Intro – Continuous Integration

• Lot of interest in continuous integration• Everybody has an opinion on what it is about

... but how to make it work

• Recommended literature– Continuous Delivery, J. Humble and F. Farley, Addison-Wesley

... certainly about test automation

... release process is key as well!

3

IntroUnderlying challenge

Difficult to organize efficient creation of

complex first-of-a-kind systems

To let teams work efficiently and motivated in the knowledge age

• Communicate Purpose• Give Autonomy … trust!• Mastery … opportunities to develop!

• https://www.youtube.com/watch?v=1SfmmuC9IWs

The challenge: “transition SW development at Philips Lighting” applying SAFe(Scaled Agile Framework)

4

SW Development for Philips Lighting Controls Some Background - Diversity in applications & markets

• Commissioning applications– Set up a network of lights, sensors & switches running

embedded FW– Configure light behavior through remotes, PC or cloud

applications

• Control applications– Switches, remotes– Apps on smart devices– Apps in the cloud

• Maintenance and emergency test applications– FW update

• Dashboards– Energy reports– Alert reports– Occupancy heat maps– Trajectory analysis– Asset tracking

Major R&D SW activities are on

• Connected street poles “citytouch”• Connected lighting for office

“interact”• Indoor controls “Dynalite”• Architectural lighting “TrustSite”

Larger projects exceed 60 engineers

Systems have to be developped quickly (preferably << 2 years), have to be cheap and easy to commission

5

Transition Example @Philips LightingStep #1 of CI introduction: SAFe trainings (http://www.scaledagileframework.com)

June 2015SAFe trainings

Learning “Without CI no SAFe”

Test approach June 2015

Manual test iterations every 4 weeks

Key components typically rejected after 2 days of testing• Releasing is difficult and happens ~once or twice a year• Hundreds of tickets• Frequent ticket “ping-pongs”• Inefficient CCBs• Project continuously in escalation mode

Many project manager changes (5 PMs in 2 years!)

I&V team gets negative attention• First I&V lead leaves after differences• Second I&V lead goes the same way for similar reasons• Lead integrator leaves

6

I&V / SystemTeam

"manual test team with architect, integrator, PM (10+)“ with gating responsibility (“industrial age” I&V)

self organizing FACILITATING team of specialists with part-time scrum master and product owner: framework engineer, test-suite engineer, simulation expert, security expert, testers focused on keep the release chain running

Process CCBs and meetings on issues and tickets taking hours every week

Short discussion (5 to 10 minutes) on “are all failing test-cases followed up” during daily team-of-teams stand-up; issues are fixed asap and tickets are filed if fixing takes long or detection happens late in release chain

Release Once or twice per year typically requiring lead engineers to fix issue on site

Every two weeks deploying to production cloud (almost fully automatic)

Up to 300 open tickets at once –never below 200on major components

Typically less than 40- All components

Tickets

Extremely motivated team in the beginning –

amazing spirit

After a while, feature requests started to

overload project while issues hampered stable

releases

Frustration rises between teams and at higher management

Corrective actions

Situation at start of transition

7

A Ticket Resolution Example

• Observed issues– Overview Below speedometers is shown : "kw"– Overview In graph when hovering, "Kw" is shown– Monitor In the energy graph, the Y-axis shows "KWh"– The correct abbreviations should be "kW" and "kWh"

• ResolutionComment by A [ 2015-01-29 ]

Tested with IE11.

Comment by B [ 2015-01-30 ]

[CCB] Can be done in a next release.

Comment by C [ 2015-02-05 ]

In the overview page it is changed to kWh.

Comment by C [ 2015-02-05 ]

This issue is fixed in Feb 5th build.

Comment by C [ 2015-02-20 ]

Kw is used for the current hour and after every hour it is calculated for kwh. and current, trend and targer is got changed to kwh as per the product manager comments.The fix shall be available with the FW12 build

Comment by D [ 2015-04-16 ]

Successfully verified on FW13 (Server Version : 1.1.000.20150324 )

Comment by PM [ 2015-05-07 ]

OK

5 people involved resolution: three

months!

8

Transition Example @Philips LightingStep #2 of CI introduction: Decision start with one new project


Summer 2015New project defined

First automateone project insteadof two half

- Test automation has high management attention- … no capable automation engineers available- … focus on one project is not accepted; management

wants all projects improved- … alternative suggestion from management “if automation

is so difficult, have more and shorter manual iterations”


9

Transition Example @Philips LightingStep #3 of CI introduction: Engineer joining team for test automation



2nd half 2015Engineer joining team for test automation

Work on CI environment starts

With one skilled engineer working on CI, situation remains brittle- No pull from project teams- Main claims on “we do CI”, but no automated

E2E tests on system level



10

Transition Example @Philips LightingStep #4 of CI introduction: Project restructured



May 2016Project restructured

New RTE with SW background,new team setup,SAFebecomes operational

Many improvements; still no break-through in E2E performance and adoption of continuous integration




SAFe terminology http://www.scaledagileframework.com

RTE = Release Train EngineerPO = Product OnwerART = Agile Release TrainART triangle = Product Manager, Architect, RTEPI = Program Increment


11

Transition Example @Philips LightingStep #5 of CI introduction: All teams to work on CI



July 2016All teams to Work on CI !!

RTE/ART triangle pulls emergency brake- All hands on deck to work

on CI for one PI!- Engineers struggle to set

up automated system tests- Many complaints on test

framework- Sudden need to scale

framework to all teams & set up release pipeline; ~10 environment

Getting ALL teams involved made CI happen







SAFe terminology http://www.scaledagileframework.com

RTE = Release Train EngineerPO = Product OwnerART = Agile Release TrainART triangle = Product Manager, Architect, RTEPI = Program Increment

12

Transition Example @Philips LightingStep #6: Feature teams




Jan 2017Feature teams

It helps to have a responsible engineer driving a feature E2E







13

March 2017: CI works!




Jan 2017Feature teams

March 2017CI works!







15

CI

Test Infra

BS

WG-pro

Sensic

Switch

BS

WG-pro

Sensic

Switch

Test Infra

Server

GW

Sensor

Switch

Test Infra

dev

Server

GW

Sensor

Switch

Test Infra

RE

Baselinedsystem SW& new test-casesper feature

bi-weeklyReleases

Develop cloud featuresagainst stable device FWDevelop test case perfeature “fx”Run regression suite perFeature “fx-1”

Scrum Masters responsible

Develop device featuresagainst cloud SW in CIDevelop test case perfeature “fx”Run regression suite perFeature “fx-1”

Scrum Masters responsible

development teams + system team

Regression testing incl. system experts / TVD

Manual testingNFR testingRelease reportRegression suite

Acceptance testing ofproduction keys & certificates

interact

Integrationper feature “fx”Run regression suite perFeature “fx”

RTE responsible

Cloud

baselined device FW

System PRs/CRs

PO’s

PM (SA, RTE)

PM (SA, RTE)

stage

Server

GW

Sensor

Switch

production keys& certificates

maintained by DevOps

development keys& certificates

HTC48 ceiling

HTC48 ceiling

Achievement: Release Pipeline Connected Building ART

Server

GW

Sensor

Switch

Test Infra Server

GW

Sensor

Switch

Test Infra

demo

Several times a day

17

Unit / product CI RE Stage

Main Objectives

Verify correctness of individual features

Verify correct integration of features on latest SW/FW against acceptance criteria

Verify NFRs on baselinedreleases against acceptance criteria

Verify (1) production certificates and FW images correctness, (2) database upgrade integrity

Framework & approach

Automated test setups using stubs at product boundaries

Automated E2E system tests running functional tests (starting with happy flows); ability to test in ceiling

(1) Automated E2E system tests running functional tests and completing NFRtesting; ability to test in ceiling(2) User acceptance testing

Upgrade from current production version n-x to release under test n; upgraded system can be (1) commissioned, (2) FW up-graded to n+1 release, (3) metrics reporting stays alive, (4) data-base integrity

Execution frequency

Upon check-in to main line

At least once a day Upon release request- Cloud per feature

(every day / week)- BCB and WG-pro cloud

when required- Sensors (~2 times a

year)

Once a baselined release passed the RE stage- Cloud: frequent in the future

desire to automate- Devices: low frequency in the future

less focus on automation for sensors

Unit tests User Acceptance Tests, Performance tests

Automated acceptance tests

interact

CB-ART Release Pipeline: High-Level Strategy

Match tests & stages!

Teams must always be able

to release!

19

CB-ART Dashboard: Release report follows dashboard structure

Same sequence as on dashboard

00x Smoke tests01x Commissioning

02x FW update03x Metrics04x Security

05x NFR

20

Test Framework

“Lego” approach fitting on developer

desk

Rasp-pi running webserver with

REST API to interact with sensor

Jenkinstestlink

Main test case language is Java plus

some phyton

Framework provides interfacing libraries

to developpers

Config per HW setup

Several hundred luminaires in the

ceiling

Jira

confluence

21

Don’t do it like this

... example from a labtable of a test framework proof of concept

22

The road ahead

• Long-term #1 amibition– Reduce amount of projects relying on manual testing only– Harmonize way-of-work between projects

• Improve easiness of framework deployment– Use docker to become machine independent to enable MAC installations for app developpers

• Increase usage of APIs just below UI – To speed up tests

• Increase usage of emulators / simulators for large scale scenarios– To verify scalability of cloud solutions– To iterate on complex configuration scenarios which are time consuming for manual testing

• Increase release frequency of HW teams– To shorten the release frequency towards CI

23

THANK YOU!

Documents

Outline - so · PDF file–To verify scalability of cloud solutions –To iterate on complex configuration scenarios which are time consuming for manual testing