Upload
others
View
7
Download
0
Embed Size (px)
Citation preview
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
How We Embraced Risk
To Catalyse Change
Case Study Presentation | 28/10/2020 | Zurich-Airport
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
Mass extinction events occur more frequently
than one might imagine. Throughout earth's
"recent" history, the so-called Big Five are the
most prominent.
These Great Resets always produce lots of
losers, but also the winners of tomorrow.
Winners embraced the risks and hardship that
arise from a global catastrophe and gradually
learned to thrive in a new, changed
environment. We should try to learn from that.
Millions of years before current time
Genera
extinction inte
nsity in % Our lucky
chance!
The dinosaurs'
opportunity
O-S
Ob.-D
P-Tr
Tr-JK-Pg
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
In January 2018, Swiss Air-Rescue Rega
started a project that aimed to introduce a
new helicopter mission control system.
The project was concluded in July 2019 on
time, on budget and with a very smooth
transition to operations.
We faced many risks along the way, which
we always confronted head-on. I want to
outline these risks, how we "embraced"
them and how we were able to catalyse
change by addressing them.
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
Shorthand for "Swiss Air-Rescue Rega"
Based in Switzerland
Privately run, non-profit organisation
Financed by around 3.5 million donors
Provides around-the-clock medical
assistance by air
12 helicopter bases throughout
Switzerland with 19 helicopters
3 repatriiation jets
16'782 missions in 2019
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
https://www.youtube.com/watch?v=n2JG-yrgvrM
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
Georg Hauzenberger, IT Project Manager
Helicopter Rescue
Originally born in Austria, now living in Zurich
Background in communication systems
technology
Worked in satellite communications,
telemetry systems design, requirements
engineering, systems architecture and cost
steering
Several years of experience in agile projects
and teams
Working for Rega since April 2018
Courtesy of www.bendik.ch
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
1. Moving from waterfall to agile
2. Making sure the users have their say
3. Which new communication system?!
4. Packing and leaving on-prem
5. Not everything is software
6. Testing the "real deal"
7. Going live and the aftermath
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
In Rega's context, the helicopter mission control system is used to
register incoming alarm calls, dispatch the "next best" rescue
helicopter, convey mission-critical information and store all mission
metadata for further processing.
The mission control system is operated by Rega's emergency
dispatchers, which act as a hub between the patient, the helicopter
crew and other emergency medical services, such as hospitals.
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
Rega's current mission control system was
already in operation since 2012. Zero downtime,
better usability and greater operational
efficiency together with diverse technical drivers
(e.g. Windows 7 phase-out) were the main
reasons for seeking a replacement.
In the summer of 2017, a tender was launched
in which several suppliers participated, including
the vendor of the current system. At the
beginning of November 2017, the decision was
made to switch to a new supplier. One key
agreement was that the project would be
conducted based on agile methodologies.
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
3+ Involved Suppliers (Dev, QM, Infrastructure, …)
Total project costs in the low seven digits (USD)
Core project team comprising 12 people
Project duration 18 months
"Scrumish" methodology
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
The mission control system enables Rega's
emergency dispatchers to register incoming
alarms, dispatch the "next best" emergency
helicopter and coordinate the mission involving
patients, Rega's helicopter crews and other
emergency service providers.
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
Map
Computer-aided
Dispatch (CAD)Rega GeoStore,
OSM, GWR, Swisstopo, …
PoI-Data
Mico
Radio system
Telephony and Radio
Helicopter positions, Helicopter status, "Panic button"
F24 SMS, FaxExchange
SAP Mission invoicing, helicopter and mission data
Mission Plattform
Crew alarms, Mission handover, helicopter status
BI Mission statistics
ETV Not
ETV Inside
Rega-App
Caller identification and position triangulation
Caller identification
Alarms via Rega-App
Client side: map web application
interacts with a fat client
dispatching application
Server side: georedundant,
virtualised application, geo object
and database servers
Whole range of systems needs to
be integrated
Interfaces to telephony, radio,
avionics systems, mobile apps,
etc.Rescuer & Mission
Database
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
ScopeCreep
Resources
User Acceptance
Integration
Test Coverage
New Data Centre
New Communication
System
New Client Infrastructure
New Methodology
Roll-out
You know those "Our" risks
12
34
56
6 7
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
1. Moving from waterfall to agile
New Methodology
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
Although it was agreed that the project should be conducted using agile
methodologies, it did not really start that way.
As I only joined Rega in April 2018, my boss, Lukas Müller, the head of IT,
had the onerous task of introducing change in the project. At first, the team's
agile transformation proved to be an uphill fight …
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
Keep it simple and stupid: our
development and testing method was
"born" on a paper napkin
Fast feedback: we iterated each week
in the beginning, then moved to two
weeks
Encourage self-organisation: we
quickly got rid of processes and
RASCI-matrices
Visualise: visualise your product,
current state of development, your
progress, your process, everything!
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
Sprint: 1 Week
Wednesday
Backlog Refinement &
Testing Planning
08:15 – 09:45
Break with Coffee and
Gipfeli 09:45 – 10:00
Project Mgmt Varia
10:00 – 10:30
On-prem Working
Session
from 10:30
Thursday
User Review
09:00 – 10:30
Break with Coffee and
Gipfeli 10:30 – 10:45
User Workshops &
Trainings
10:45 – 12:30
On-prem Working
Session
from 12:30
Tuesday
Daily
07:45 – 08:00
Friday
Daily
07:45 – 08:00
Monday
Deadline deployment user
stories and creation of test
cases for current Sprint
Testing Goal
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
0
100
200
300
400
500
600
700
800
10 12 14 16 18 20 22 24 25 26 27 28 29 30 31 32 33 34
Work
Rem
ain
ing
[SP]
Sprint
Measured & Projected PA 4
Measured Projected PA 5 (excl. Mico)
Measured & Projected PA 5 (incl.Mico)
Measured & Projected PA 7 (incl.Mico)
Sprint 30:105 SP (65 SP)Sprint 31: 50 SP (22 SP)Sprint 32: 23 SP (22 SP)Sprint 33: 100 SP (72 SP)
AddressObjects
ELS Heli(Pre-)
Production System
Mico
Radio
F24
SAP
Data Import
Mission Platf.
ARS
MicoTest System
Make call
Open mission
Get mission details
Create protocol entry
Replicate master data
Contact Data
ELS HeliInt
ELS HeliTest
IMPORT IMPORT
Done
In Progress
Open
Descoped
Legend
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
2. Making sure the users have
their say
New Methodology
User Acceptance
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
Originally, scope was documented with business
and IT requirements
Business requirements were clustered into Epics
Epics contained a "high-level" User Story, were
linked to the original business requirements and
had acceptance criteria that sketched the broad
scope
Epics were, in turn, clustered into User Stories
that detailed the broad scope (based on several
business requirements at times)
For each iteration, a focus and testing goal were
defined based on the Epics and User Stories
implemented
Epic Alarming rescue helicopters ("resources")
As an emergency dispatcher,
I'd like to alarm "resources" over
various channels and see their
status, so that I can initiate the
handling of a mission and check
upon its status.
Alarming Cantonal police
forces in case of a traffic
accident
User Story
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
User acceptance of an application is the ultimate goal. But
writing User Stories is not enough for that.
Already at the end of Sprint 1, a group of 4 emergency
dispatchers started to test "out-of-the-box" functionalities and
how they were adapted to their needs. User feedback was
afterward consolidated, inserted as User Stories into the Product
Backlog and, if prioritised high enough, considered in a Sprint.
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
For each Sprint, we assigned a
so-called "testing goal", e.g. "we
are testing mission handover to
SAP"
The Product Increment that was
deployed at the end of each
Sprint should allow the users to
accomplish the testing goal in
the following Sprint
Feedback (new User Stories or
Defects) were considered as
part of "blocker" User Stories
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
3. Which new communication
system?!
New Methodology
User Acceptance
New Communicatio
n System
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
During the analysis phase of the project, it became clear that a very fundamental assumption did not hold. The
communication system, which is THE central system in the operations centre, also needed to be replaced. A mere
three months later, in July 2018, it was decided that a new communication system will be developed from scratch.
Roughly halfway through the project, this posed a major risk. We decided to embrace it.
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
We grabbed the opportunity and decided that
this could be a major chance for our
emergency dispatchers to get the system that
they both need and want
We started off by synchronising the other
project stream to our 1-week Sprint cadence
Based on 5 guiding Epics, we started
integrating from the 1st common Sprint and
conducted common reviews
After a mere 5 weeks, we could already say
that the integration risk was significantly
reduced by using user feedback and agile
methodologies
Geo-
objects
ELS Heli(Pre-)
Productive System
Mico
Funk
F24
SAP
SAP Import
MissionPlatform
ARS
MicoTest System
Make call
Open mission
Get caller details
Generate protocol entry
Replicate dataContact
Data
ELS HeliInt
ELS HeliTest
IMPORT IMPORT
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
4. Packing and leaving on-prem
New Methodology
User Acceptance
New Communicatio
n SystemNew
Datacentre
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
Public safety organisations in Switzerland
traditionally hosted most of their IT
infrastructure in their own server rooms or data
centres. Nowadays complexity and security
demands mandate that "IaaS services" be
operated by professional organisations.
Accordingly, Rega's IT strategy foresaw that by
the end of 2019, all servers should be moved to
external datacenters.
So why let's not be the first ones to be there?
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
In the original waterfall project plan, a so-
called "Pre-SAT" was scheduled for the 20th
of September 2019
As the new environment would allow for a
truly georedundant architecture, we decided
to conduct the Pre-SAT in the new
environment
Back then, we still needed to have core
switches installed and even had no physical
WAN connection (and only 2 months time)
Having overcome those obstacles, we had a
running (earmarked to be) production system
already 9 months before go-live
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
Datacenter Zurich
Rega Center
Test-System
(ZH)
Int-System (ZH)
Prod-System
(ZH)
EZ Prod/Int/Test Clients
Datacenter Bern
Int-System (BE)
Prod-System
(BE)
"Classic" three-tier architecture moved to fully
virtualised environment
Automatic failover between georedundant
sites in Zurich and Bern
Fully separated production, integration and
test environments
Integration 1:1 image of production
environment
Frequent system patching combined with
regression testing ensures system stability
and security
Arbitrarily short release cadence
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
5. Not everything is software
New Methodology
User Acceptance
New Communicatio
n SystemNew
DatacentreNew Client Infrastructure
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
Building usable software means not only to
focus on the usability of the software, but also to
consider the devices that it will be used on and
the context of usage. Always keep ergonomics
in mind.
Once again: early prototyping and feedback is
vital. Once you got a first direction you need to
test, test and test together with your users and
iterate fast.
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
6. Testing "the real deal"
New Methodology
User Acceptance
New Communicatio
n SystemNew
DatacentreNew Client InfrastructureIntegrationTest
Coverage
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
Complex infrastructures can best be
integrated and tested using iterative
approaches
In general: the earlier you integrate, the
better
It pays off to build up test capabilities that
enable true end-to-end testing
The integrated systems should be
operated in an environment that mimics
production 1:1
Keep in mind: The closer we are to reality,
the less risk we face in production
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
Usability testing should equally occur with
the future setup in the environment that
the users will work in
Early in the mission control system and
the communication system projects, we
conducted workshops in which the future
interplay of applications was defined
We chose the new workstation together
with the users, showing them two variants
Testing ergonomics "in the field" should be
done for each user group, as even the
"best" system may prove to be unusable
there
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
7. Going live and the aftermath
New Methodology
User Acceptance
New Communicatio
n SystemNew
DatacentreNew Client InfrastructureIntegrationTest
CoverageRoll-out
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
Guiding principle number 1: prepare what you
can prepare upfront
Guiding principle number 2: test everything that
you can test upfront, in the process and in the
aftermath
Guiding principle number 3: aim for a lazy go-live
– the less you need to do, the better
After go-live, the system continued to be in use
without any further interruption / maintenance
A mere 3 non-critical incidents occured in July
DevOps and continuous improvement of ITIL
processes
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger
| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger