DevOps Best Practice for Oracle SOA and BPM

Rubicon Red

The Facts

Founded in 2009, Rubicon Red’s mission has been to lead customers to success on their Oracle middleware journey.

This is the origin of our name; helping customers cross the “Rubicon” in their successful adoption of the Oracle Middleware “Red” stack.

Largest dedicated Fusion Middleware consulting and managed support services business in Australia.

Offices in Brisbane, Sydney, Melbourne, Adelaide and Redwood Shores California

Offshore development and support centre in Hyderabad India.

One of the founding members of the Red Expert Alliancehttps://www.redexpertalliance.com

Recognised global leader in Oracle Fusion MiddlewareThought Leadership. Innovation. Unrivalled Expertise.

Oracle Customer Advisory Boards• Oracle SOA• Oracle BPM• Oracle WebLogic

Rubicon RedRecognised global leader in Oracle Fusion MiddlewareThought Leadership. Innovation. Unrivalled Expertise.

Oracle A/NZ Specialised Partner of the Year 2014

Middleware

2nd year

I N N O V A T I O N A W A R D 2 0 1 4

I N N O V A T I O N A W A R D 2 0 1 1

I N N O V AT I O N A W A R D 2 0 1 0

I N N O V AT I O N A W A R D F I N A L I S T

2 0 1 5

x3

x3

The value organizations deliver through products & services is increasingly defined by the software that underpins them

Software is a Competitive Advantage

Organizations are in a Digital Race

Solution not delivered by a single systemRather a patch work of applications, each one performing a particular business function

5

Oracle Middleware provides the platform to combine these business apps, into an integrated solution

Majority of IT projects fail to deliver on-time and on-budget.

Software project waste 40%+ of resources

Gene Kim - Why Everyone Needs DevOps Now: 15 Year Study Of High Performing Technology Orgs

The Human Cost…

The Impact of Technology FailuresHigh Profile Technology Failures Increasingly Common

NYSE Says Wednesday Outage Caused by Software Update

July 10th 2015

United Airlines blames grounding of hundreds of flights on computer glitch

July 8th 2015

U.S. visa system will be offline until at least next week

June 17th 2015

• On average - a single major technology failure can cost a business as much as $US 10.8 million.• 45% of organisations experience a loss of brand equity or market share following a major

technology failureTechradar – Dec 2014

DevOpsA better way?

200x

*over 3 years

Faster lead times than their peers

60x Fewer failures

50% Higher market cap growth*

30x Deploy code more frequently

• Featured on 800 CEO Reads Top 25: What Corporate America Is Reading June, 2013

• Novel on how DevOps can be used to• Address many of the process issues that

plague IT Delivery / Operations• Align Development & Operations• Align IT with the Business• Help the Business Win

“3 Ways” of DevOps• The First Way: Systems Thinking emphasizes the performance of the entire

system as opposed to the performance of a specific silo of work or department• The Second Way is about creating and amplifying the right to left feedback loops• The Third Way is about creating a culture that fosters both continual

experimentation, taking risks and learning from failure; and understanding the repetition and practice is the prerequisite to mastery.

13

Work in Small BatchesGetting Feedback Fast – FAIL FAST• Faster Feedback• Validate assumptions

• Solve Problems Quickly• Easier to find & fix defects

• Reduce Overhead• Organizations get better at things

they do frequently

• Reduce Risk

Release

Get FeedbackImprove

Release small chunks of functionality frequently

Get regular validation on the production readiness

of your application

Incorporate feedback rapidly. Fail fast and learn

fast

Breaking the Bottlenecks In The Flow1. Environment Creation2. Code Deployment3. Test setup and run4. Overly tight architecture5. Development6. Product Management

Leading A DevOps Transformation: Lessons Learned – Gene Kim

Takes weeks/months to provision and configure Oracle Middleware Environments

Manual build & deployment of codePainful, Resource Intensive and Stressful stage of Oracle Middleware Implementations• Manual deployments

• SLOW & ERROR PRONE• Inconsistent across environments• Neither repeatable nor reliable• Require extensive documentation (often outdated)

• Which leads to • Long delivery times• Ongoing Project delays• Significant risk of Major Production Issues with

associated financial and brand damage

“What do you mean, ‘it’s not working in production?’ I TESTED IT BEFORE WE

RELEASED!”

17

Delaying integration means…Invalid assumptions discovered late in the process!

• Integration designs are full of “un-identified” assumptions• Need to integrate with actual systems

to validate• Correcting issues with core design

patterns can result in significant time delays• Frequent and early releases into SIT

critical for integration projects. Systems implemented in isolation make assumptions about other

systems with which they will integrate.

Minimise differences between environmentsVersion Control all Middleware Configuration

18

Issues caused by Configuration Drift are often the most difficult to diagnose and result in many wasted weeks/months of man effort to resolve.

Configuration Drift !

19

Test Teams standby Idle ….Waiting for deployment errors to be fixed

Test Team Idle whilst Deployment is being performed Performing smoke tests and

resolving deployment issues Turning around blocking defects etc.

• Lack of Testing

“How long would it take your organization to deploy a change that involved just one

single line of code?”Mary and Tom Poppendieck, Implementing Lean Software Development

“The test team has approx. 18% down-time, waiting for deployment issues to be resolved, every 8 days of delay is costing me 1 FTE” Programme Manager - Major Bank

Defects discovered late in deliveryCost of defect removal increases exponentially as the development lifecycle progresses.

The later defects are found and fixed, the greater the risk to the business they pose.

21

Outage In Production….Waiting for deployment errors to be fixed

60% of Production failures caused by

human error or lack of automation

80% of all IT services outages caused by unauthorized configuration changes

99% of DR failures caused by Config Drift

• Lack of Testing

“Configuration drift and unauthorized configuration changes account for nearly

80% of all IT service outages”source: Gartner Research

Technical DebtEvery time we take a “short cut” we create technical debt:• Manually configure an environment• Create code without automated tests• Manually do deployment

Technical debt is what you feel the next time you want to make a changeGene Kim – Author The Phoenix Project

23

DevOps; The First Way…Accelerate Flow to Production

Work in Small Batches Create Safety / Flow Through Automation

Automated Environments, means identical DEV, TST, PRD

Continuous Integration / Testing Automated Regression Testing Continuous Delivery / Deployment Blue / Green Deployment

Define a Reference Architecture Should “Target” Flow, i.e. reduce Technical

Debt

• Lack of Testing

“The process for releasing/deploying software MUST be repeatable and reliable”

Continuous Delivery is a Key Practice of DevOps

24

Code Build Integrate Test Release Deploy Operate

Agile Development

Continuous Integration

Continuous Delivery

Continuous Deployment

DevOps

Reduce the cost, time and risk of delivering incremental changes into Production

Automating Environment Creation

Lessons Learned on our Journey of Discovery

A declarative based solution for the install,

configuration and continuous deployment of

Oracle Fusion Middleware and

Applications.

• Still need to manually build first environment to create “Gold” Image.

• Significant amount of manual work to update clone with environment specific details.

• Any changes to “gold” environment requires new gold image to be created.

Cloning / Gold Images

100% Automated Rollout of Oracle FusionNo way to propagate changes to previously created environments.

• Significant Investment to implement and maintain

• Scripts get “big” very quickly, complex to maintain.

• Brittle, tied to a particular topology/configuration.

• Need re-write /re-factor each time Oracle make a new release.

WLST

Scripting Software Defined Environments

Automating Oracle Middleware Platform LifecycleSoftware Defined Environments

Fully automated provisioning of Oracle

Middleware Environments in

minutes, at the push of a button

Platform Provisioning

Fully automated ongoing patching and

configuration management of your Oracle Middleware

environments.

Lifecycle Management

Establish a standard process for delivering

Oracle Middleware environments

on-premise and on-cloud.

Enables Hybrid Cloud

Delivering Platform as a ServiceDrives Strong Governance & Consistency

• Activate push button automation of Environment provisioning / tear

down Configuration updates Continuous Integration

• Implement governance and version control for Platform Configuration

• Define Integration Reference Architecture

• Standardise on single integration technology stack for on-premise that can in future move to the cloud

• Define & implement consistent FMW blueprint for all environments.

Software Defined EnvironmentsDrive Strong Governance & Consistency

29

Use a consistent Platform Blueprint across ALL environments.

PRODCI PRE-PRODSIT

Environments become more production-like

Version Control and Promote Configuration Changes

DEV

Issues caused by Configuration Drift are often the most difficult to diagnose and result in many wasted weeks/months of man effort to resolve.

Platform Blueprints• Define standard topology• Ensure consistency across all

environments• EDG Templates provided out of the

box

Platform Models• Map Blueprint to Specific

Environment• Defines environment specific details

Blueprints & Models version controlled• Control promotion to new

environments• Easy to rollback to previous

environment

Fully automated provision• Provision new environments in

minutes

Demo: Install & Configure

Automating the Path to Production

SOA & BPM EnvironmentsPath to Production

Develop & Test Code

Automated Build

CIDeploy & Test

SITDeploy & Test

UATDeploy & Test

CAPACITYDeploy & Test

PRE-PRODDeploy & Test

PRODDeploy & Test

• Dev Environment• Developers build, fix & unit test code

prior to committing

• CI Environment• Automated Build & Deployment of Code• Automated Tests

• Non-Production Environments• Automated Deployment• Automated Tests• Long running / manual tests come last

• PROD

Automating the Software Delivery Pipeline

33

“How long would it take your organization to deploy a change that involved just one single line of code?”Mary and Tom Poppendieck, Implementing Lean Software Development

The set of processes, procedures and tools used to promote code from development into production.

34

Orchestrating the Delivery PipelineLeverage CI Tools such as Jenkins, Hudson, BambooOrchestrates Delivery Pipeline• Pipeline consists of various jobs

• Build, Deploy, Test, etc

• Split out by phase / environment• Dev, CI, SIT, UAT, etc

For Example• DEV Pipeline

• Triggered whenever code change committed

• CI Pipeline• Scheduled Nightly

• Non-Prod Pipelines• Triggered Manually – when business decides.

Build AutomationBuild Binaries Once… Deploy to Multiple Environments

Keep environment specific configurations OUT of the buildThis includes environment specific SCA Configuration Plans, OSB Customization Files

36

Environment Delivery PipelineFor Non-Production Environments

Re-Provision Middleware

Platform

Configure Release

Configure Middleware

PlatformDeploy Release

Smoke Test Environment Specific Tests

• Re-Provision Middleware Platform• Prevents Configuration Drift

• Configure Release• Update environment specific config, e.g.

Web Service Endpoint

• Configure Middleware Platform• Resources, e.g. Data Sources, JMS• Create & Configure Adapters

• Perform Environment Specific Tests• Longer running tests• Manual Test Performed as late in the

processes as reasonable

Application PromotionMinimise differences between environments

37

Use a consistent Platform Blueprint across ALL environments.

Re-Provision Non-Prod Platforms to eliminate

configuration drift.

PRODCI PRE-PROD

CAPACITY

UAT

SIT

Environments become more production-like

Increasing Confidence in build’s production readiness

DEV

Issues caused by Configuration Drift are often the most difficult to diagnose and result in many wasted weeks/months of man effort to resolve.

Automated DeploymentDrives Strong Governance & Consistency

Automated Builds• Compilation and packaging of code

centrally in an automated fashion

Application Blueprints• Defines which artefacts, resources

make up a release package• Build once, deploy to multiple

environments.

Application Models• Captures environment specific

details.

Fully Automated Build and Deployment

Quickly Deploy and Manage Releases across Environments

39

Build Quality In“Cease dependence on mass inspection to achieve quality. Improve the process and build quality into the product in the first place”

W. Edwards Deming

40

Different Kinds of Testing• Treat test code as production code

• Includes Test Data

• Not all tests call external systems• Make use of Mock Services

• Run smoke tests first• Perform short running tests before

long running tests• Perform Manual Testing after

Automated Tests• Continually maintain your tests

Functional Acceptance Tests

Journey Tests

ShowcasesUsability Testing

Unit testsIntegration tests

System tests

Non-Functional Acceptance Tests

Performance, scaling, …

Automated

Automated Manual / Automated

Technology Facing

Business Facing

Supp

ort p

rogr

amm

ing

Critique Project

Diagram invented by Brian Marick

Manual

41

Design Code to Simplify Testing …Use a layered reference architecture

• BPM Projects• Mock Service Layers• Use Automated Testing

• SOA Project• Same Approach• Mock IVS Layer

Service layering, use of Mock Services & Automated Testing Increase productivity and improves code quality.

Blue / Green Deployments• Allows time to validate new

release is behaving as expected before go-live• Zero down-time for go-live.• Makes rollback very simple

Blue Environment

Release N

Green Environment

Release N+1

Router

43

Continuous Deliver Tool Chain For…Oracle SOA Suite and Oracle BPM Suite

Pipeline Orchestration

Source Code Management

OSB, SCA Build Automation Software Repository

Oracle Middleware Platform

Provisioning

Configure Oracle Middleware Platform

Configure & Deploy OSB, SOA, BPM

ArtefactsAutomated Testing

Nexus

44

Benefits of DevOpsReduce Risk, Decrease Costs, Speed Up Time to Market

Reduce Risk.

Reduce risk of projects delays, provide better

visibility.Significantly reduce

risk of defects in production.

Decrease Cost.

Reduce waste and increase developer and operational

efficiency.

Provide the business with a strategic advantage in its ability to be more responsive in delivering new solutions faster, cheaper and more often.

Speed Up Time to Market

Increase agility of development and test

teams. Shorten the

development lifecycle.