Godesic release management methodology

Tackling Transition Release Management Good Practices

Godesic Tackling Transition Paper 1

Table of Contents

Godesic Release Management Methodology Foreword Introduction

What do we mean by a ‘Release’? Release Management

Release Management Challenges Release Management Good Practices Approaches to Release Management Conclusion Appendix 1 Research


Godesic Release Management Methodology

This paper sets out Godesic’s experience

of transitioning projects into live operations

over the last fifteen years. It positions

Release Management as an increasingly

recognised discipline with an emerging set

of expertise that when done well can help

organisations reduce the risk of change

and increase customer and market impact.

Foreword

“Release management is an area that in

many organisations remains full of risk

and personal heroics…. insight insight

insight… recommendation to read on….”

Richard Bell

ex.CIO / COO… Barclays

[image headshot of Richard] would this

work Richard?


Introduction

Transitioning new capability into live operations remains a complex divide to cross with

increasing costs and customer intolerance for interruptions to operations. Current competitive

and regulatory pressures are driving the need for a regular and successful drumbeat of new

capability released into operations.

Against this backdrop those responsible for change and operational efficiency in organisations

are tasked with working out effective approaches to take the risk and cost out of maintaining

an up to date operation.

This paper sets out the challenges in this area to draw out some root causes behind the

headaches and then defines a number of good practices gained from interviewing leading

thinkers in this area together with drawing on the experience within Godesic.

What do we mean by a ‘Release’? This paper considers a release as the point where newly developed capability is transitioned

to an operational state. This definition of release is deliberately broad and covers all aspects

that enable the successful launch of new capability. As a transition point, a release has a

relatively short duration, yet has to squeeze in a lot of complexity, often with many tightly

timed critical tasks, a broad cast of participants, a large and broad set of stakeholders and a

heightened level of risk.


Figure 1 below depicts where release management fits in the wider context.

Figure 1 Release Management Bow Tie Risk & Ownership Profile

Release Management

Release management is typically talked about from a technical perspective in terms of

software applications, however to tackle a number of the areas of complexity this paper

deliberately has release management encompass a wider set of challenges. These are then:

1. Technology The infrastructure and application elements of a release

2. Process The operational elements of a release, encompassing process and people changes required to transition to new ways of working

3. People Communication and stakeholder management that ensures the release engages with execs, staff and customers in a way to maximise the chance of success

Using this broad definition positions the technology element in context as only part of the

story. Thinking more widely also exposes the similarities between vastly different problems

that share the same fundamental challenges. This definition of release management can

apply to a product launch as much as it can to an IT program in that it requires a high level of

critical coordination across multiple teams to achieve a common outcome.


With these definitions in place the content below will set out the strategylevel challenges that

organisations face where Release Management can make a difference. It will also look at

how Release Management can help to underpin the delivery of organisational strategy and

maintain the pace of its realisation.


Release Management Challenges

Considering the definition above the following typical challenges are faced by those

responsible for delivering it:

Significant Release Demand post the financial crisis organisations have more pressure than ever to transition improved capability into live operations to underpin the execution of

strategy; organisations have an average of c.1500 release events per month. This is

compounded in verticals such as Financial Services where often 90% of the queued releases

are related to mandatory regulatory change. At peak periods organisations like some major

universal banks have a 2 year release queue.

Scarce Release Capacity Organisations have a limited capacity to transition capability into live. Transitions often have to book very limited weekend or periodic slots, change freezes

exist that pause everything and maintaining strategic and regulatory go live dates is not

simple. Against this scarcity of opportunity any release slot that results in a rollback or failure

is a slot that is not recoverable; the organisation’s capacity for change is irrevocably

diminished for that period.

Downtime Intolerance the tolerance of customers (internal and external) for systems downtime is increasingly limited; downtime often leads to customers leaving services.

Downtime on critical services in Telco and Financial Services is also leading to significant

market fines. Releases of new capability that lead to incidents in live are extremely

expensive.

Team Geographic Diversity Cost pressures have driven organisations to place portions of their operations in diverse areas across the globe. Engaging these teams in the transition of

capability to live presents a communications problem. Also notifying, mobilising and repairing

any follow on incidents is not simple.

Ownership Gap. Transition of new capability to be released into live operations involved moving from the ‘build the company’ space to the ‘run the company’ space. This can lead to

ownership gaps due to lack of clear governance and accountability for taking capability across


this divide. Without effective definition and communications projects and operations teams

can assume the other party is responsible and disengage leading to problems evolving to

having significant impacts before being addressed.

Complexity. Releases often involve the interaction of new capabilities with old systems and procedures that are hard to bottom out outside of putting the capability into live operation.

This necessarily leads to lots of unknowns and interfaces to be tested. The complex nature of

these unknowns is a source of tremendous risk. This leads to some sobering statistics that

are often quoted to us in client discussions: often 10% of all releases/changes result in

incidents or roll back and 80% of all incidents in live operations result from change.

Poor Existing Solutions Service Transition takes place in real time with plans in hours and minutes. Existing enterprise project management toolsets such as MS Project break at this

level of granularity. These toolsets are also poor at enabling the rapid collection and

distribution of status information with the usual mechanisms being weekly timesheets.

Enterprise Release Management needs solutions that can take in data on issues with the

execution and status on performance in seconds. Currently those responsible in this area

integrate a lowest common denominator solution of Microsoft Excel, lots of relatively

expensive resources and lots of phone calls.


Release Management Good Practices

To help tackle the above challenges there are a number of good practices to help release a fit

for purpose capability to time and budget.

Right Organisation. The release is highly dependent on the team in place to lead and execute the introduction of the capability. Is the entire release responsibility on a single

release manager who has to act as project manager, facilitator between the project and

operations and communications manager to interact with all stakeholders from the business?

How can an effective team be put in place that is supported with the right tools to get the job

done. Good practice organisations are taking time to define release management as a

professional area with effective role descriptions and place within the organisational

landscape.

Communications. People are often tempted to tackle complexity in the management of a critical event like the transition of new capability into operations with more complexity. This

usually takes the form of extensively detailed plans, reports, artefacts and organisation. Best

practice organisations are seen to focus on improving communications in the same

circumstances to get better resulting performance. Those organisations that are able to

leverage a mix of communications technologies to enable realtime coordination, rapid status

distribution and notification of the need to act demonstrate the pace and agility needed to

launch and land complex and critical events.

Effective Governance and Process. Release are complex and appropriate governance is needed to ensure that any changes that the capability will introduce are assessed for impact,

and the right parties sign off and agree to permit progress. People need to know who decides

on what, when. This provides a machine that the organisation can rely on to take place and

this frees up capacity to tackle the tough problems that arise.

Plan. Plans can be seen as administrative but the process of discussion and iteration mitigates risk. Planning in detail with the people who will be executing the release in realtime

is invaluable to anticipate issues and put in place the right support model to ensure success.

A single coherent plan, a single source of the truth, as an outcome that is configuration


managed and understood across the organisation will prevent numerous potential obstacles

that could derail the risk.

Practice. Release failure and subsequent defects are costly. Investment of time with the team rehearsing the release runbook to ensure that activities are appropriately sequenced

and to put in place the branches of the plan that may be taken if things go off track.

Rehearsal and scenario planning lead to excellent preparation.

Configuration. An effective data set is required on the requirements, scope, architecture, interfaces relating to the capability to be released. Effective configuration of the management

information is also required such as the set of responsibilities and the maintenance of the

plan.

Support Model. Who are the set of people and organisations that will be on call for the live

surgery of deploying the capability. Are the on call arrangements in place to get a response

time that is acceptable to nip any risks in the bud. Is the support model documented and will

the release team have access to the right details to ensure calls can get through. Has the

support model been tested?

Audit Trail. Having the ability to look back after the release to understand the audit train on decisions and actions together with timings of the tasks presents excellent opportunities to

develop a world class release management capability in your company that can support a

robust capacity for change delivery.

Ownership. World class release management organisations breed a sense of collective

ownership of the endtoend release. Each person is measured to a degree on the successful

operation of the capability and this avoids interfacedelays where parties feel their job is

complete and the next party is not fully aware they have to take the next step. It also avoids

contention between the project and operations environments.


Approaches to Release Management

In our client experience and research we have seen a number of approaches to Release

Management. These approaches are not mutually exclusive but are most likely to suit

different types of organisations depending on aspects such as the tolerance any live

experimentation.

Make the change/transition very safe set up the change to avoid it being all that different. We have seen organisations force changes down a process that rejects anything out of the

ordinary and packages it so its 95% similar to things that have worked to date. This seems to

work well for organisations that are amending via limited updates that can be simulated in

environments that are understood to be almost mimics of live operations. It has downsides in

not being able to handle the widespread changes that are sometimes required. It is also

difficult to achieve in circumstances where the reality of live operations is hard to mimic in

tests, simulations and scenarios.

Make changes fast but put them into operation selectively. Here the organisation is not limited attempting to simulate the impact on live operations to test before deployment; rather

they are using slivers of their operational base to act as the confirmation that the change will

not detrimentally affect the live estate. We have seen organisations such as Facebook

maintain pace in this manner through exposing a limited number of users in live operations to

the new way of doing things. Changes are implemented quickly with groups such as internal

employees being the guinea pigs before then going out to wider customers. In each case

metrics and methods are required to understand whether to turn up deployment or

immediately roll back if a large snag is hit.

Planned deployment but extensive control. Here organisations are usually performing the required testing and quality review to safeguard the release of changes but those changes are

not necessarily modified to ensure for similarity with the existing components of the system.

Emphasis here is placed on effective critical event control for the transition to ensure that

deployment takes place in an orderly fashion and the inevitable inflight engineering during

the transition period is effectively controlled to coordinate diverse global teams to land the

changes successfully.


There is no single silver bullet approach but rather there are interesting aspects of each

approach that organisations can learn from to improve their ability to usher in new ways of

operating without causing significant issues with the existing operational estate.


Conclusion

Our research found that the need for releasing new capability into the operations of the organisation to deliver on strategy and comply with regulations remains a significant organisational priority. Against that demand the capacity for releasing capability is scarce and often results in either a failed transition or costly incidents in live operations. In that context the practice of release management remains tough in itself due to the additional facets of increasingly geographically dispersed teams and increasingly complex systems that are hard to simulate and model changes on. Good practice organisations are focusing on a number of good practices to tackle this backdrop. These range from defining the specific role of the Release Manager, engaging the wider business through effective governance, deploying a range of communications technologies, planning and rehearsing to anticipate issues and emphasising ownership to align the project and operations areas of the business. In each of the above well thought out mechanisms people are moving from considering the transition period to be a point on the project lifecycle to being a specific high risk area that requires additional attention to safeguard the successful delivery of new capability. Organisations not adopting these good practices are often reliant on personal heriocs of individuals whilst suffering significant shrinkage in their flow of successfully deployed capability. In addition to the good practices a number of overarching approaches have evolved to suit the nature of particular businesses. These range from evaluating and shaping changes for similarity to avoid transitioning capability that may lead to incidents in live to almost continuous deployment to a select slice of the operational base. In all of the above it is hoped that the collective set of research presents food for thought in considering additional steps to be taken to maintain and improve the organisation’s ability to get new capability in place to underpin compliance and competitive advantage.


Appendix 1 - Research

This report is based on interviewing over 80 organisations on their approaches to transitioning

new capability into live operations. The sectors included are as follows: Financial Services,

Government, Telco & HiTech, Major Sporting Events, Manufacturing and Energy.

The personnel interviewed range from Release Managers to CIO’s, COOs and Chief

Executives.


Documents

Godesic release management methodology