Unleash the agile power bridging the gap between development and operations - xebialabs

Unleash the Agile Power: Bridging the Gap between Development and Operations

Dr. Wilco Koorn Senior Consultant and Scrum Master

April 2010

Unleash the Agile Power: Bridging the Gap between Development and Operations 2

Acknowledgements

We kindly thank KLM/Air France for allowing us to use their operational data in this white paper.


Table of Contents

Summary 4

The Gap in Enterprises 5

Agile in Enterprises widens the Gap 6

Self-service bridges the Gap 8

Case study: Self-service deployments at KLM/Air France 11

XebiaLabs’s Vision on Deployment Automation 14

About XebiaLabs 15

Relevant literature 16


Summary

Development and Operations departments have different, and at times opposite, objectives. The objective of Development is to produce a continuous stream of value by providing change, whereas the objective of Operations is to produce a continuous stream of value by providing stability. Trends like increased automation of business functions, introduction of a service-oriented architecture paradigm, ever more heterogeneous middleware environments and the emerging use of virtualization in datacenters have increased pressure on both Development and Operations. Moreover, business simply demands more agility from the IT organization. Development’s response to this demand is to adopt Agile methodologies, allowing them to provide more changes in a shorter timeframe. As a result, the Operations department has to cope with many more software updates to the development, test and production environments. This makes Operations’ objective to provide a stable infrastructure to end users an increasingly difficult task. Studies indicate that especially the hand-over of software packages from Development to Operations causes a lot of idle waiting time and therefore lowers overall productivity. Hand-overs not only involve developers explaining to Operations what has changed in their code and how this might impact other components, but also rewriting deployment scripts and then retesting the application on a shadow server. These activities require a substantial amount of time. Eliminating hand-overs can be achieved by adopting a “self service” deployment model in the Development department. A necessary prerequisite for introducing self-service deployments is to first automate the entire deployment process. A case study made by airline company KLM/AirFrance proves the value of automating the deployment process and introducing self-service deployments. Without hiring additional Operations staff, KLM/AirFrance was able to cope with a 500% increase in the number of deployments per month. Thanks to deployment automation and self-service deployments, they now save $1 million annually in their entire software development and delivery chain.


The Gap in Enterprises

Enterprise organizations traditionally make a distinction between the “Development” speciality, focusing on the creation and testing of software, and the “Operations” function, focusing on the management of the IT infrastructure needed to run the software. Although their goal of supporting the business functions is the same, the methods of achieving this differ significantly between Development and Operations. Over the years, both departments have developed different ways of working, optimized to support their own objectives. Let’s take a closer look at the Development department and the Operations department to get a better understanding of their specific aims: The objective of Development is to produce a continuous stream of value by providing change. This boils down to an enhanced software product with fewer bugs and more features, and sometimes even to a completely new software product. The objective of Operations is to produce a continuous stream of value by providing stability. Organizations need a stable and reliable IT infrastructure for their day-to-day operations. To reduce the risk of interrupting the ongoing business, Operations departments usually maintain a strictly controlled change process. This difference in objectives is the root cause of the gap between Development and Operations: Impact of new trends The last couple of years, this gap between Operations and Development has widened due to emerging trends in the domain of IT. To start off, we see that the sheer number of software applications is increasing, as more and more business functions become (partially) automated. The footprint and importance of IT within companies has expanded tremendously. Secondly, a trend that started several years ago is the adoption of “service oriented architectures” or SOAs. The idea behind SOAs is to split up large, monolithic applications into a set of loosely-coupled, reusable web services to create a more flexible IT infrastructure. On top of that, ever more heterogeneous middleware environments have increased complexity, while the emerging use of virtualization in datacenters means that some parts of infrastructure management will be pulled into the realm of deployment. Also, business has become more demanding. As a result, the Development department has to build more applications for end users in a shorter period of time. By introducing Agile methodologies and working practices, Development tries to adapt to the business requirement of providing more changes in a shorter timeframe. As a result, the Operations department has to cope with many more software updates on their production environments. More business applications, usually divided into several large sets of web services, spread out over a variety of middleware environments have made the Operations department’s objective - maintaining stability - a nightmare.

Development’s focus is on change: ‘we need new things, and we need them fast’ Operations’ focus is on stability: ‘we need to keep things running, just like yesterday’


Agile in Enterprises widens the Gap

Introducing Agile methodologies into an IT organization brings a competitive advantage by cutting cost and shortening time to market. Agile methods generally promote a disciplined project management process that encourages continuous inspection and adaptation. This is underpinned by a leadership philosophy that encourages teamwork, self-organization and accountability, a set of engineering best practices intended to allow for rapid delivery of high-quality software, and a business approach that aligns development with customer needs and company goals. Development cycles or iterations in the Agile philosophy can be as short as one or two weeks, leading to a constant stream of new software packages. This means there will be many small software deliveries rather than one big one as was the case in the traditional waterfall approach. Various case studies have pointed out that development projects can gain enormous time savings when adopting Agile. These savings can result in an up to fivefold reduction in development time compared to the time taken by similar projects using the waterfall approach. The iterative strategy leads to a nearly continuous stream of software packages that Operations has to deploy to the production environment as working software. So as soon as Development starts to embrace change by using Agile methods in their software engineering process, the Operations department is impacted as well.

Operations usually has implemented a controlled change process, meant to maintain a stable production environment. Because Development now chooses to adopt Agile methods as a response to the increased business need to provide more changes at a faster pace, Operations feels the pressure building up on their controlled change process. Software changes (in the form of installable packages) have to be deployed to Test and Acceptance environments at a much higher frequency, without compromising Operations’ objectives of maintaining a stable infrastructure for business users.


More changes, smaller changes Working Agile results in many smaller updates as opposed to a few larger ones. A major release usually affects many other components, applications, servers and configuration settings. The chance that one of these steps fails, causing the whole system to malfunction, is therefore significant for a major release. A small update is less likely to bring a production environment to a halt than a major release. This means that the Agile approach brings down the overall risk for carrying out updates, while at the same time increasing the frequency and number of updates. This implies that the Operations department will have to do more routine work, of a lower complexity, than before. However, Operations cannot drop their control procedures for small updates. This would undoubtedly lead to an increase in interrupts of the running business systems. The Operations team simply has to be as strict as ever before in applying their pre-production verification tests. The entire change process has to remain intact, also for smaller updates. What does this mean for the gap between Development and Operations? This gap will only become wider in an Agile environment. With Agile delivering a productivity gain of 400% to 500% to Development, Operations needs to become more productive as well. If Development delivers 4 to 5 times as many packages per unit of time, Operations consequently has to deploy 4 to 5 times as many packages. This could be realized by hiring 5 times as many people, or by applying a less strict change process. The first option would dramatically increase the cost of Operations, the latter dramatically reduce the stability of the IT landscape, which would again lead to very high costs to remedy resulting problems. So neither option is viable. Operations is more or less forced to also embrace change, but its hands are still tied by heavy-weight control procedures. In the end, introducing Agile methods in the IT organization feels like a “catch-22”. The enterprise will lose most of the competitive advantage gained by introducing Agile in Development because Operations cannot cope with the productivity increase. Can enterprises bridge this gap, and if so, how?


Self-service bridges the Gap

Based on our experience and analyses carried out across our customer base, we found that the hand-over of software packages from Development to Operations causes a lot of idle waiting time and thus lowers overall productivity. These hand-overs are a consequence of the strict controlled change process, where Operations takes full responsibility for keeping the upstream test (and sometimes also development) systems as much as possible synchronized with the production environments. These pre-production environments serve as a buffer for the production systems. Only tests executed on systems identical to the production systems are truly meaningful when trying to minimize risks on production environments. Why do hand-overs of software packages cause a lot of idle waiting time? Let’s give an example to explain this: suppose a developer has to deliver a software update before 4pm so the deployer can install it on a test environment. The developer faces some nasty bugs and works late to fix these and sends his work to the deployer only at 9pm. The deployer, having waited in vain till 6pm, has already gone home. As a result, the deployment to the test environment will have to wait till the next day. The next morning, however, the deployer faces some problems in installing the package correctly, because it is not well-documented and some configuration details are wrong. The developer, having worked late last night, only arrives at 11am. He immediately starts to hand over his knowledge of the missing configuration details to the deployer, but in the end, it is noon by the time the application has been deployed to the test environment. Sounds all too familiar? In fact, the organization did not just lose 2 hours of idle waiting time, i.e. productivity lost by the deployer, but also increased its time to market by at least 18 hours (from 6pm on day 1 till noon the next day). And of course this happens not just once, but many more times, across many projects… Hand-overs not only involve developers explaining to Operations what has changed in their code and how this might impact other components, but also rewriting deployment scripts and then retesting the application on a shadow server. Working Agile leads to more updates which in turn leads to more hand-over moments, thus more work for both departments. Development and Operations become mutually dependent on each other – on the developer to explain what has changed in the code, and on the

KLM/AirFrance Case Study: Hand-over error rates of 90% KLM/AirFrance performs load tests on all web applications before go-live, in order to preserve the required operational stability. The availability of properly set-up test systems is a bottleneck. In most cases, the applications were not deployed to the test systems in time, resulting in idle waiting times and inefficient test cycles. KLM/AirFrance investigated this issue further and found that the late deployment of applications was mainly due to the error sensitivity during the hand-over between developer and deployer (the Operations person responsible for deploying the application). Error rates of 70% - 90% during hand-over were common. This led to a very high bounce rate of yet-to-be-deployed packages, lots of rework for Development, idle waiting time for the Operations department and sub-optimal use of the test infrastructure and hardware.


deployer to do the actual deployment. The mutual dependency requires Operations and Development to synchronize their duties and activities. Since this synchronization in reality is not always optimal, idle waiting times for both departments occur. IT organizations that are able to eliminate these inefficient hand-over processes will save a lot of time and money in both Operations and Development. Self-service to the rescue Eliminating hand-overs can be achieved by adopting a “self service” deployment model in the Development department. This allows a software engineer to integrate intermediary test deployments in his daily development work. In order to preserve the required stability on the live production environments, this self-service model should be managed by the Operations department. As argued above, the buffer environments, i.e. the development and test systems, should be kept synchronized as much as possible with the live production systems. Operations wants to make sure the correct change control procedures are still enforced on these buffer systems. In other words, the Operations department needs to define exactly “who may do what where”, in the strictest sense of the word. For example, a software developer delivering a package containing a new version of an application is allowed to import from a repository and then deploy it himself, but only to a particular subset of the infrastructure. The same developer however, is not allowed to modify any configurations of the middleware or security systems, nor add or change any resources needed by the application. In fact, software developers effectively get their own sandbox in which they can safely carry out intermediate (test) deployments of their own packages without interfering with other systems or configurations. Self-service deployments require automation A necessary prerequisite for introducing self-service deployments is to first automate the deployment process. Automation of application deployment has the advantage that the deployment process can be completely mapped out in advance, strictly defining authorization levels and access rights, as well as fall-back scenarios in case something goes wrong. Without automating your deployment process first, introduction of self-service deployments would be equal to giving everybody access to all systems. This would lead to a rather chaotic scene in which every developer (and deployer) would have to use his own ‘best guesses’ as to how to solve deployment problems. Of course, deployment manuals and scripts can help here, but they are not water-tight and rather brittle. Manuals might not be read properly or may not be used at all. Scripts must be kept in-synch at all times, but since this often is a manual task, it’s easy to overlook a change or two. Automating your deployments would prevent this. The benefits for the Operations department are obvious. Deployment automation combined with a self-service deployment model eliminates non-critical, less-interesting, routine deployment tasks. As a side effect, Operations has more time left for its other, non-deployment related work. When we introduce an additional tool in the domain of Operations, the Operations department suddenly becomes the enabler of change to the Development department while retaining full control. If we implement such a tool without compromising the controlled nature of the change process in Operations we bridge the gap between the two departments and fully unleash the potential of Agile! For the Development department, this approach of automating a part of their development work is not fundamentally different to what they are already used to. Applying a deployment


automation tool for their self-service deployments is just a natural extension to the tool chain they already use, such as IDEs, source code management systems, continuous integration tools, automated testing systems, automated quality control systems, and automated build tools. The benefit of self-service deployments to software developers is obvious: there is no dependency on, and thus no waiting for, the Operations department anymore. The self-service model further streamlines and optimizes the overall software development process. The next chapter describes the real-life case study of KLM/AirFrance, a company that successfully implemented this approach to deliver automated self-service deployments.


Case study: Self-service deployments at KLM/Air France

Introduction KLM/AirFrance is one of the largest airline companies worldwide. In the very competitive airline market, KLM/AirFrance wanted to introduce a new service to their customers. This service allowed air passengers to opt for an ‘economy comfort’ seat with more leg space for an additional fee. The new service required changes to several software applications. To realize this, a number of development projects were defined, jointly referred to as the “Y35” program. The Y35 program was outsourced to an Indian company that employed several teams to work on the projects. Because of the intense time pressure, teams were working on a 24x7 schedule. And since Agile development methodologies were applied, an almost continuous stream of software packages had to be deployed on Test and Acceptance environments. All deployments had to be done on IT systems in the Netherlands due to the necessary interconnectivity with other systems. The Y35 program had a very challenging deadline. Still, it was completed and delivered on time, without working overtime or hiring additional temporary staff in the Operations department! How did KLM/AirFrance achieve this remarkable result? Solution Several months before the Y35 program started, KLM/AirFrance had introduced Deployit as a new deployment automation tool in their Operations department. This tool enabled their developers to carry out self-service deployments. Although the deployment automation tool was introduced to increase efficiency, rather than being driven by the introduction of Agile in Development, the case study highlights the ability of the Operations department to keep pace with the increased output of Agile Development. Let’s have a look at the data on the number of monthly deployments at KLM/AirFrance in 2009.

From the table above we can infer that about five times as many deployments were carried out in November 2009 than in previous months. In November, just prior to the project delivery date of December 1st, Development released a massive number of small changes, all of which had to be deployed to the respective test environments. We also see the workload is back to normal in December, indicating “Y35” is finished.


If we take a step back and observe what happened, we see a sudden fivefold increase in the number of deployments. This was caused by the increased output of the Development department, working in an Agile way on a 24x7 schedule. However, no additional staff had to be hired in the Operations department in November, nor did the Operations staff have to work overtime. The majority of the intermediate deployments were handled by the developers themselves. This was made possible because KLM/AirFrance had already implemented the self-service deployment model early in 2009, using Deployit as its deployment automation tool. This led to changes in their deployment cycles, as illustrated by the following figure:

Before self-service deployments

After self-service deployments

We can conclude from this that KLM/AirFrance managed to bridge the gap between Development and Operations. The Operations department became a true enabler of change since they already had the right infrastructure and the right control procedures in place to flexibly scale up to more than 5 times the average number of deployments without hiring any additional resources or even working overtime. This not only meant that Operations facilitated a smooth and in-time delivery of the final Y35 project; they also saved significantly on their temporary personnel costs.


Automated self-service deployments were a major factor in turning the internal IT organization of KLM/AirFrance into a reliable partner for the business departments. Moreover, by having eliminated the countless hand-over moments between Development and Operations, KLM/AirFrance now saves about $ 1 million annually thanks to their very efficient overall software development and delivery process!


XebiaLabs’s Vision on Deployment Automation

Application deployment is the process of getting an application running correctly in a middleware environment. Deploying all the artifacts that make up the application to the correct target middleware is a complicated process that is very hard to get right when done in an ad hoc fashion. In the meantime, the pressure to get deployment done right is increasing. Agile development techniques are resulting in more frequent deployments while service oriented architectures and ever more heterogeneous middleware environments are increasing the complexity of the deployment process. And the emerging use of virtualization in datacenters means that some part of infrastructure management will be pulled into the realm of deployment. These trends quickly stretch manual deployment procedures and in-house scripts to breaking point. Clearly a more mature and scalable solution is needed.

Organizations might look to one of the existing tools in their extensive tool suites to solve this issue. Some might try and extend a build automation system with deployment capabilities while others might try and use server automation tools to "provision applications" just like they provision new infrastructure. But deployment is not only about development and it is not only about production. Deployment is about taking the artifacts delivered by a development team and deploying those to the infrastructure managed by operations. Therefore a new solution is needed that directly addresses the need for deployment automation.

Companies need a deployment automation product that can be trigged by a continuous integration platform, that can get approval from a change management system, and that can gather information from a configuration management database. It can import artifacts from a build automation system and can interface with server automation tools to do prerequisite provisioning. It connects to middleware systems to automatically carry out the deployments, according to the best practices as laid out by middleware vendors and in deployment manuals. It has out of the box functionality for the major middleware systems and its standard functionality can be extended and tweaked to conform to company standards. It has a security system that allows you to configure limit who can deploy what application to which environment.

Deployit is this product. It is not a generic script engine and it does not just concern itself with moving configuration files around. It is not bolted on top of a build automation system and it is not vendor specific. Deployit is a deployment automation platform that is built around a Unified Deployment Model that includes concepts such as deployment packages, middleware environments, and promotion of a deployment package through those environments. Deployit includes built-in support for IBM WebSphere, Oracle WebLogic, JBoss, and Tomcat middleware stacks and Deployit has an extensive plugin system that allows you to extend its functionality to support even more middleware. Deployit can scale from a single development deployment to many large concurrent production deployments, all using the same methodology that ensures your deployments happen in a predictable, timely manner.


About XebiaLabs

XebiaLabs is the leading provider of Deployment Automation solutions. With its flagship product Deployit, XebiaLabs automates Java EE deployments in an enterprise environment. Deployit allows organizations to manage their middleware environments by predictably executing and automating their Java application deployments.

XebiaLabs was founded in 2008, and currently serves various customers around the world. Headquartered in the Netherlands, XebiaLabs is an international company with development and sales offices in the USA, France, Germany and India. In other countries, XebiaLabs works through a network of local representatives. For more information, visit the XebiaLabs website at http://www.xebialabs.com.

http://www.xebialabs.com/


Relevant literature

Beck, K. (2000). eXtreme Programming explained - Embrace Change. Addison Wesley, ISBN 201-61641-6.

Partington, V. (2010). Java Deployments in an Enterprise Environment. Xebialabs. http://www.xebialabs.com/java-deployments-in-an-enterprise-environment-whitepaper

KLM/AirFance (2009). KLM/AIR FRANCE speeds up Java application deployment with XebiaLabs’ Deployit. XebiaLabs. http://www.xebialabs.com/sites/default/files/1/KLM%20customer%20story.pdf

http://www.xebialabs.com/java-deployments-in-an-enterprise-environment-whitepaper

http://www.xebialabs.com/sites/default/files/1/KLM%20customer%20story.pdf

Education

Unleash the agile power bridging the gap between development and operations - xebialabs