DevOps Workshop - Addressing Quality Challenges of Highly Complex and Integrated Applications

© 2015 IBM Corporation

DevOps Technical Workshop -Addressing Quality Challenges of Highly Complex and Integrated Applications

6204 -DevOps Workshop

Bill Hudacek (IBM)Vinit Kutty (IBM)Andrew Williams (IBM)

Workshop Objectives

• Interact with other clients and explore case studies on how to use new approaches to testing to reduce waste and increase speed and agility

• Share and capture best practices in new testing processes, tools and capabilities

• Walk away with an action plan developed with insights from other clients and subject matter experts

1

Agenda

• DevOps and Quality• Concepts & Terminology

• Scenario Discussion• Practices & Challenges• Solutions

• Best Practices Discussion

2

DevOps and Quality

3

The reality our customers face every day…

Testing Labor Test Lab Complexity

Heterogeneous Environments

Public CloudPrivate Cloud

Data Warehouse MainframeEnterprise

Service Bus

Directory Identity

File systems

Collaboration

Web/Internet

Routing Service

Third-partyServices Portals

ContentProviders EJB

SharedServicesArchives

Business Partners

Messaging Services

TraditionalTesting

Other costs

Testing costs

of customers experience

production delays

>45%

to deliver validated test environments*

~2-8 weeks

of large organizations spend approximately 25%–30% of project budget on testing **

>60%

Concepts & Terminology

4

Development Velocity

Clients are experiencing increasing complexity …while demanding cycle reductions


5

ContinuousCustomer

Feedback & Optimization

Collaborative Development

Continuous Release and Deployment

ContinuousMonitoring

ContinuousBusiness Planning

ContinuousTesting

Operate Develop/ Test

Deploy

Steer

DevOpsContinuous Feedback

Introducing IBM DevOps

6


Acceleratesoftware delivery –

for faster time to value

Balance speed, cost, quality and risk –

for increased capacity to innovate

Reduce time to customer feedback –

for improved customer experience

Eliminate testing bottlenecks through virtualized dependent services

Reduces the cost of testing by increasing levels of automation

Accelerates time to delivery through shorter test cycle times

r

Remove defects earlier through advanced defect management

Interconnected Technology

Operate

DataDefects

Reports

BUILD

EXECUTE / VIRTUALIZE

PREPARE

ANALYZE & REPORT

Automated Process

DevOpsContinuous

Testing

Environment

Application

TestExecution

EnterpriseDashboard

DevOps delivers an accelerated test approachInterconnected and seamless

7


Continuous Testing features

Continuous Testing

Test Automation

• Run unattended• Validate business

flows across channels• Durable, maintainable,

resilient to change

Orchestration from build to production deployment• Reliable • Repeatable • Traceable

Test Environments

• Virtual services• Realistic & complete - (production like)

• Available on demand• Inexpensive to setup

Manage Quality across the lifecycle

• Collaborate • Govern • Report

8

DevOps Foundation

… and interlocked with other DevOps solutions

At least 1/3 of The Continuous Delivery Pipeline

is testing!

Everything between development &production is a test stage!


9

Scenario Discussion

The world before adopting DevOps and Continuous Testing…

Introduction to some Practices and Solutions

10

Thursday: Phil finds a test environment but it’s not an exact match; he identifies back-ends from other testenvironments that can be used but now tests will need to be scheduled carefully to avoid conflicts.

Monday: Mike discovers that the requested test environment and a few Core Banking Solution back-end services are not available yet.

Mike calls Phil (Chuck’s boss)because Chuck (Env guy) is on leave and it’s been a week. An urgent message is left.

!Test Environment NOT delivered

MIKEDev Lead

Cost

Productivity

An every day scenario Testing a Core Banking Solution

11

Practices and Solutions

!PHILEnv Lead

Tuesday: Phil gets the message and estimates it will take 2 weeks to set up the test environment and make all required Core Banking Solution back-ends available. Phil drops everything to find an alternative…

BTW Test team is ready

SUETest Lead

Friday: Test Execution is rescheduled!

Friday: The Test Team starts test environment validation but identifies significant issues –some infrastructure-related, and some from conflicts due to another team’s testing hitting the same back-end services

The following Monday, There’s no test automation so testing is manual. After three daysof testing, 450 defects are reported –most turn out to be invalid.

1 week later: Sue gives the green light

one weeklate!

Wednesday: Test Team is still ready to test but stops work on preparation until an alternative environment is identified

THE Test Team

Let’s analyse the scenario

12

!

IMPACT

CAUSE

SOLUTION

• Test delays and disruptions• Higher risk of component failure• Manual, inefficient and inaccurate• More unplanned IT work

• High level of re-work• T&M costs increased• Delays to implementation of new functionality

• Manual provision of test environments• Manual gaps between automated processes• Manual install/configure • Manual test setup (back end providers, data setup & resource dependency)

?


Automated Integration Testing

Service Virtualization

Deployment Automation

Continuous Testing

User Interface

Integration

Component

Unit Tests

13

Testing Pyramid – Automation is Key, but where ?

Integration Testing and Virtualization


Using virtual componentsContinuously test what you want, when you want, how you want


Progressively replace Virtual Services with ‘live’ ones…Progressively replace Virtual Services with ‘live’ ones…

Actual Service/App

Virtual Service/App

• Service Virtualization is an enabler for continuous testing

• Services, applications, and systems are introduced into the continuous integration cycle in a prioritized, controlled fashion.

• Controlled integration testing helps isolate defects for faster resolution.

14

Automated Integration Testing & VirtualizationAn Example

Authorization system

IFX over TCPIP

ISO8583 over TCPIP

SOAP over HTTPS

Rational Test Workbench / Rational Test Virtualization Server

ATM

Virtual authorization system

ATMATMATM

15

VISA


SystemEnterprise Payment

System

WebSphere MQ

ATM device driver (Java)

ISO8583 over TCPIP

SOAP over HTTPS

Enterprise Payment System

WebSphere MQ

ATM device driver (Java)

+

Legend:

• Automated, consistent deployments• With rollbacks of application

deployments• Orchestration of changes across

servers, tiers and components• Configuration and security differences

between environments

Deployment by using UrbanCode Deploy

• Multi-Tier Application Models• Easy Process Designer• Track what is where:

Inventory• Scalable Distributed

Automation• Quality Gates and Approvals• Integrations Replace Custom

Scripting

Key Features

16






Continuous Testing

;;

Service Virtualization with IBM UrbanCode Deploy Facilitates rapid creation of environment / applications from patterns

Mobile App

Middleware

Backend Systems of Record

Automate your multi-platform deployments


QA

SIT

UAT

PROD

IBM UrbanCode Deploy

17

Best Practices

18

Do not wait for the UI to automate your Tests Test early and often

Leverage single functional tests in (regression) test

suites and in performance tests

Create Virtual Services to enable full functional

testing even without all back-end Service Providers

running ‘live’

Automated Integration Testing Best Practices

19

Best Practices




Continuous Testing

Service Virtualization Best Practices

Consumer-centric rather than Producer-centric

Governance: choose your Organizational Model• Access to all• Tools Maker/specialist• CoE• Federated CoE

Virtual Services at tier N• Service providers at deeper

tiers tend to be smaller, simpler, more stable

• Virtual Services with more “behaviors”, change more often

Make Virtual Services small/self-contained and data-driven when possible

Less Logic is more• The more logic in a Virtual

Service, the more it starts to resemble an application itself

• Remember Rule #1 of software development• “All Software Has Bugs”

You define reality: more powerful negative testing• The mainframe looks like it’s

down!• Services that respond in 15

minutes…an hour… a day• Return invalid output – or test

using invalid inputs

20

Best Practices




Continuous Testing

Automate everything Tie deployments to the rest of the lifecycle

Provide visibility into the deployment process

Consider enabling technologies like “cloud”

Deployment Automation Best Practices

21

Best Practices




Continuous Testing

Continuous Testing

Test Automation

Security

Mobile

IBM’s continuous testing solutionContinuous Testing requires a strong platform

22

IBM RationalTest VirtualizationServer

IBM RationalTest VirtualizationServer

IBM InfoSphereOptim Test Data Management

IBM RationalQuality Manager

IBM RationalQuality Manager

IBM RationalPerformanceTest Server

IBM RationalPerformanceTest Server

IBM MobileQuality Assurance

Perfecto Mobile

IBM MobileQuality Assurance

Perfecto Mobile

IBM Security AppScan

IBM Security AppScan

IBM RationalTest Workbench

IBM BPM

IBM RationalTest Workbench

IBM BPM

IBM UrbanCode DeployIBM UrbanCode Deploy

Test Data Management

Quality Management

Environment Provisioning


Performance

Best Practices

A CT workflow engine and enterprise dashboardSignificant reduction opportunities in cycle time and effort

23

Environment Verification

Test Execution & Service Virtualization

Environment Provisioning

Defect Management

Intelligent testing

Notify

Data setup

Create defect

enterprise dashboard

workflow engine

Best Practices

Intelligent Testing within CTOn a scheduled basis, CT identifies modified code within a test environment by using a client’s version control tool combined with structured content searches of design documentation– with the objective of down selecting regression test cases for execution. This allows daily verification of modified code against a clients test automation suite.

VALUE

• Reduce time to analyze

changed components

• Reduce cycle time

• Automatic application

validation

Source Code tool

ACTIVITIES CONNECTS TO

.

Design documents

Environment

Data

Execution

Report

Defect

Application

PREPARE

ANALYZE• Analyze design documents corresponding to changes• Identify associated System test case• Down select key Regression Test Cases using tags text match

EXECUTEEnvironment

Data

Execution

Report

Defect

Application

• Initiate test of the selected Regression test cases• Create defect on failed tests

• Schedule Intelligent Test Execution• Identify changed code from source control tools

24

Test Management tool

Create defect

Best Practices

Defect Management with CTDefect triage will be guided by CT and change defect priority for blocking defects, send timed reports based on status change and kick off a retest of any fixed defect with an associated automated test case

REPORT

EXECUTE

• Send a timed report for aged defects• Send a timed report for Deferred, Rejected or

duplicate defect

• Initiate Retest of any defect in “Retest Ready”

ACTIVITIES

.

Environment

Data

Execution

Report

Defect

Application

CONNECTS TO

Test Automation tool

Test Management Tool

Test Management tool

ANALYZE

Environment

Data

Execution

Report

Defect

Application

1. Regularly identify defect status change 2. Change priority value for blocking defects

Best Practices

25

Business objectives• Accelerate test delivery• Increase efficiency• Reduce delivery risk

Automation

Virtualization

Analytics

• Online service catalogue for automated test suites

• Dashboard and analytics with automated production of test reports

• End-to-end regression test suite fully automated and interlocked with environment, reporting and defect management workflows

• Environment provisioning and virtualization services to produce test labs on time and improve productivity

IBM solution

Delivered outcomes

0

10

20

30

40

50

60

70

80

90

Baseline Effort inhrs

HATc Effort in hrs

40% effort reductionin test execution times

50% effort reductionin report preparation times

50% reduction* in test environment provisioning

Provisioning is one simple step vs. 40+ hours

Continuous Testing Case StudyIn 2014, a Global Insurance firm was provided with Continuous Testing service from IBM

26

Best Practices

LOB LOBLOB

Dev & Test EnterpriseLocation BLocation B

LOB LOBLOB

�Test Delivery Manager �Test Agile Manager�Test Architect

Location ALocation A

Test Enterprise

Core Test Delivery

Flex

Input Output

� Test Delivery Manager � Test Planner� Test Infra Environment� Test Architects / Performance � Test Analysts

Test Management

� Test Architects� Test SMEs� Test Analysts

�Test Delivery Manager �Test Architect

Location BLocation BLocation ALocation A

Transform

Organization Transformation: A DevOps approach requires a realignment of skills and resources

Specialist Teams

* 2015 European Bank

Fun

ctio

nal

Str

ateg

icO

pera

tiona

l

Test Management

Flex� Test Architects� Test SMEs� Test Analysts

Core Test Delivery� Test Manager � Test Planner� Test Analysts � API Developer

Input Output

Virtualization / Automation� Virtualization Lead� Virtualization Specialist� Workflow specialist� Performance / Automation

Test Environment� Cloud Architect� Cloud Specialist� Test Infra Environment� TDM Specialist

Mobile Test� Mobile Test Architect / Leader� Mobile Test Specialist� Mobile Test Automator

Fun

ctio

nal

Str

ateg

icO

pera

tiona

l

Best Practices

27

Scenario Discussion

After adopting DevOps and Continuous Testing

Workshop free discussion period

28

DevOpsContinuous

Testing

Monday: Mike requests CT to build Test Environment and deploy the application. 1 hour later it is ready and automatically validated by CT. Virtual Services are deployed (for unavailable back-ends).

Wednesday: CT schedules re-tests related to the fixed code from the previous day’s defects.

Test Environment

deliveredTuesday - Thursday : CT automatically issues reports to core stakeholders and the Defect Manager who works proactively via CT with Mike’s Dev Team.

Tuesday: Susan monitors CT which is running automated Tests against the Core Banking Solution. CT automatically raises 20 defects against the Release

Costs

Productivity

A solution with CT with real benefits

Friday: Test Execution is

completes ahead of time

Friday: Stakeholder review based on CT automatically issued reports

Monday: Core Banking Release is ready for deployment with zero High-severity defects

29

The Benefits….� Avoidance of

business impact� More reliable

applications� Proactive problem

prevention� More effective and

quicker testing� Less re-work� More efficient and

happier workers� Lower operational

costs and improved top-line

Open Discussion

What Quality and Complexity issues are you facing?Are they worse? Not as severe?

Describe the issues you are facing not touched upon today?We would love to hear about new aspects where DevOps/CT might help.

So…. what is working today for you?

30

Scenario discussion

Actions today

31

Identify area of significant benefit and engage IBM

Let’s meet one on one for a more focused discussion

Talk to Andrew about our Continuous Testing Service

Arrange IBM to deliver a Proof of Technology

Contact Info:

Bill Hudacek [email protected] Kutty [email protected] Williams [email protected]

Notices and DisclaimersCopyright © 2015 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from IBM.

U.S. Government Users Restricted Rights - Use, dupli cation or disclosure restricted by GSA ADP Schedule Contract with IBM.

Information in these presentations (including information relating to products that have not yet been announced by IBM) has beenreviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY. IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided.

Any statements regarding IBM's future direction, in tent or product plans are subject to change or with drawal without notice.

Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.

References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business.

Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation.

It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law.

32

Notices and Disclaimers (con’t)

Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED,INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right.

• IBM, the IBM logo, ibm.com, Bluemix, Blueworks Live, CICS, Clearcase, DOORS®, Enterprise Document Management System™, Global Business Services ®, Global Technology Services ®, Information on Demand, ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, SoDA, SPSS, StoredIQ, Tivoli®, Trusteer®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.

33

Thank YouYour Feedback is

Important!

Access the InterConnect 2015 Conference CONNECT Attendee Portal to complete your session surveys from your smartphone,

laptop or conference kiosk.

34