Experiences of SOACS

Copyright © 2014, eProseed and/or its affiliates. All rights reserved. | Confidential

Implementing the UK’s 1st Production

SOA Cloud Service

Kiran Tailor

Global Lead DBA / BI Solutions Architect, CIMA

Simon Haslam

Technical Director, eProseed

1

© CIMA 2

Me and CIMA

• Kiran Tailor , Global Lead DBA/BI Solutions Architect

• Blog : http://blog.puredba.co.uk/

•Twitter : @KiranTailorUK

CIMA and the AICPA are joining forces to create a new

association to represent the entire breadth of the

accounting profession.*

600,000 members and students worldwide. *Offices and Staff across the world. *

http://blog.puredba.co.uk/

Simon Haslam

Technical Director, eProseed UK

• Platform / Infrastructure Architect

• Using Oracle products since ~1995 (Oracle7)

• Formerly UKOUG App Server & Middleware SIG Chair

Copyright © 2016 eProseed and its affiliates. All rights reserved. © CIMA 4

• Our Previous and New Architectures

• Provisioning DBCS and SOACS

• Tailoring Cloud Services

• Experiences and Oracle Cloud Tips

• Monitoring in the Cloud and Reporting

Our Previous System – All Running On-Premises

Oracle Databases

CRM

Biztalk

GP

SQL Server

Hybrid Cloud

Public Cloud

Private Cloud

New System

SiebelCRM

Oracle Databases

Exalytics

Oracle ERPOracleDBCS

SOACS

Oracle Database Cloud Service

Oracle SOA Cloud Services

Financials

Our Integration

SOA Composites

Fusion Cloud Services

Connectivity S

erv

ices

SOA Infra

Integration Tables

Customer Processes Invoice Processes

Activity Services

On-Premises

Payment Processes

Connectivity S

erv

ices

SQL Server

© CIMA

Batch

UpdateNotifications Fault

Handling







Copyright © 2016 eProseed and its affiliates. All rights reserved.

WHAT THIS PRESENTATION IS IS NOT

• This is not a step-by-step guide about provisioning SOACS or DBCS manually through console

• We focus on some of the decisions you have to make, lessons learnt and tips

• You will notice some of these are same as for on-prem… many of your old skills are still useful


TOPOLOGIES

• Before eProseed CIMA were in development as single instance SOACS:–WebLogic Managed Server directly open to internet

–No clustering / failover options (other than VM restart)

–Can’t easily add OTD later without re-provisioning

–Not really an ‘Oracle intended’ design for production

10

SOAWebLogic

DBCS


TOPOLOGIES

• So… we introduced OTD–OTD is only internet-facing component

• Hardened, good track record/few security patches

• Option for WAF features etc

–Option to add second OTD node• Note: actually independent – different to on-prem config.

–Allows future scale-out and rolling patching withoutchanging any end points• Scale out could even be done online

11

SOAWebLogic

DBCS

OracleTraffic

Director


TOPOLOGIES – ACROSS ENVIRONMENTS

• Hard to justify OTD cost on Dev but we want all envs to behave the same

• Wire Dev SOACS MS into Test OTD - unsupported but quite practical

12

SOAWebLogic

DBCS

OracleTraffic

Director

PROD

SOAWebLogic

DBCS

OracleTraffic

Director

TEST

SOAWebLogic

DBCS

DEV


STORAGE TIPS

• Both DBCS and SOACS need Oracle Cloud Storage for backups

– Note: SOACS runs on “full PaaS” JCS – no virtual image option

• Provisioning containers – how to subdivide & manage storage?

– We provision per instance – exactly same name for simplicity

– Drop storage with instance (if you don’t want it make sure you don’t pay for old backups)

• Initial Sizing

– Size bigger rather than smaller

– Can extend in DBCS later though but you do need a restart of the database


OPC USER TIPS

1. ONLY use lower case email addresses. If mixed case:

– Some displays, e.g. user listing, show it as lower case

– Some interfaces are happy with lower case, (probably authentication) insist on it mixed

2. Use a non-email name for the primary provisioning account

– All objects are created with this in their identifier, e.g.• Better to have to reference /Compute_exampleco/exampleco/…

• Than /Compute_exampleco/[email protected]/…

• (especially with multiple admins, plus admins leave)

3. Limit the use of the provisioning/owner account to the provisioning tools

– All admins should have their own logins

14


TLS (SSL)

• SOA CS (JCS) – WebLogic and OTD has SSL configured but only with demo certificates

=> not suitable for production usage, especially OTD since that’s public

• Inputs to configuring SSL:

– Signed certificate including any intermediates: either by public CA (simplest) or internal CA

– Private key: keep it safe, but keep it outside SOACS for simplicity (to save re-signing… at least for now)

– Any additional certificates to be trusted: typically the internal CA

• We currently switch from OPSS Key Store Service to Identity and Trust JKS on disk

– In domain Custom Identiy & Cutom Trust | JKS | <DOMAIN_HOME>/security/trust-v5.jks

– setUserEnv.sh -Djavax.net.ssl.trustStore=${DOMAIN_HOME}/security/trust-v5.jks

– (we may consider OPSS later since java can reference KSS too)

15

eProseed Accelerator for Oracle Cloudconfigures your SSL for OTD HTTPS listener,

Admin Server & Node Manager listening port


SSL TRUST TIP

16

DO NOT INSTALL CERTS FROM 3rd PARTY SERVICE PROVIDERS… YOU ARE PROBABLY MISSING SOMETHING!

Otherwise you will build a brittle integration that could break outside

of your control

Beware blogs and even MOS notes or SRs advising otherwise!!!

Cloud providers typically use wildcard certificates.Wildcards are not enabled out of the box

See next slide…


WEBLOGIC CONFIGURATION FOR WILDCARD SSL

• When checking SSL certificate, WebLogic looks at the CN

– CN = login.eproseed.com <= this works fine by default

– CN = *.eproseed.com <= this is a wildcard certificate

• By default, even in 12.2.x WebLogic will not trust wildcards or certificate SANs

– Use this weblogic.security.utils.SSLWLSWildcardHostnameVerifier in domain Custom Hostname Verifier for Admin Server and all Managed Servers.

– Don’t ask me why this is not default by now!

• This hostname verifier has been available since 10.3.6 (probably backports for earlier, otherwise roll your own)

• Same is true for Subject Alterative Names (SANs) which is what we’re supposed to use instead of wildcards


AUTOMATED PROVISIONING – REST API

• All operations (& more) that can be done through console can also be done through REST APIs

• Can call REST API via most modern tools, e.g. curl, Postman in Firefox etc

• Authentication

– Most APIs use header tokens

– Compute Service APIs need you to call authentication first & get a token

18


REST API

19


EXAMPLE PAYLOAD

{

"serviceName": "soacs-unit-test-1",

"level": "PAAS",

"topology": "soa",

"trial": false,

"subscriptionType": "MONTHLY",

"description": "soacs-unit-test-1",

"provisionOTD": true,

"cloudStorageContainer":"Storage-orclnnsoa/soabackup",

"cloudStorageUser":"soacs.Storageadmin",

"cloudStoragePassword":"welcome1",

"parameters": [

<see next slide>

]

}


EXAMPLE PAYLOAD (CONTD.){

"version": "12.1.3",

"edition": "SUITE",

"managedServerCount": "1",

"templates": "full",

"adminUserName": "weblogic",

"adminPassword": "welcome1",

"connectString": "example.com:1535:orcl12c",

"dbaName": "sys",

"dbaPassword": "fmwpwd1",

"shape": "oc3m",

"VMsPublicKey": "ssh-rsa ...",

"type": "weblogic"

},{

"listenerPortsEnabled": true,

"loadBalancingPolicy": "LEAST_CONNECTION_COUNT",

"otdAdminUserName": "otdadmin",

"otdAdminPassword": "welcome1",

"shape": "oc3",

"type": "OTD"

}


IaaS VM

EPROSEED ACCELERATOR FOR ORACLE CLOUD

22

SOA

CS

Oracle Cloud Service Managers

JCS

DB

CS

Sto

rage

Net

wo

rk

Co

mp

ute

Identity

eProseedAccelerator

for Oracle Cloud

Creation

Vir

tual

M

ach

ine

Vir

tual

M

ach

ine

Vir

tual

M

ach

ine

Config. Report


AUTOMATION & PUTTING IT ALL TOGETHER

23

Env masterJSON

BlueprintJSON

Oracle Cloud configJSON

Cookbook(Recipes)

PayloadJSON

Env type

Env no

Blueprint type

DBCS override

Instances

Runlist

Sizing

etc

OPC DC

API URLs

Customer DetailsJSON

Domain, Auth, DCRESTcalls

© CIMA 24







TAILORING CLOUD SERVICES

• Additional SSH keys

• User accounts (Linux & WebLogic)

• Firewall / VPN

• (Backup schedule)

25

Connecting from Windows

User and Key Management

# useradd ukoug

# mkdir /home/ukoug/.ssh

The next step we copy and paste in the public key we have generated for

the new user:

# echo "<key here ssh-rsa…" > /home/ukoug/.ssh/authorized_keys

To check# cat /home/ukoug/.ssh/authorized_keys

Compute Cloud Service



Tunneling

SQL Connection


SSH TUNNELLING IS 1ST STEP BUT SEE…

Tomorrow, 11:35, Hall 6B








FAILING BACKUPS

1. Domain locks

2. Admin user password change

34


1. FAILING BACKUPS IF ADMIN HAS LOCK ON WEBLOGIC CONFIGURATION

• Backup tool takes a domain lock at start, releases at end

– Primarily (presumably) to stop you changing domain during backup

– Also if an administrator has started a session there could be changes in configuration that have not yet been activated

• If an Admin has the lock then the backup tool can’t get it

A better approach might be just to force discard of changes & release of lock? Debatable.

35

Backups – Domain Configuration Lock




2. FAILING BACKUPS AFTER PASSWORD CHANGES

• OPC Passwords (i.e. those in an identity domain) expire monthly

• You can change the password when logging into the cloud console

• But…

39

DON’T IGNORETHESE EMAILS!

Backups – Password Changed


• Same change in DBCS


• Update the Wallet

- sudo –s

- /var/opt/oracle/bkup_api update_wallet --password=new-password

• Validate in oss.cfg




• Tailoring Compute Cloud Service




EM CC MONITORING

44 DB CS instance

SOA CS instance

44

OTD

SOA

Oracle Financials

Cloud (ERP)

DB EE

CIMA Data Centre

SQL Server

Paypaletc

Siebel

EM 13cCloud Control

Hybrid CloudGateway

EMAgent

EMAgent

EMAgent

• Prod• Test• Dev


MONITORING & NOTIFICATIONS

• Target types

– SOA

– WLS

– OTD

– Database

– Hosts

– Exalytics

– TimesTen

45


ALERTING

46


REPORTING

• Daily Report about integrations

• Scheduled emails to IT Team/Managers

• Automatically raises incidents (check?)

47


EXPERIENCE & LEARNING POINTS

• Create a Hybrid Cloud Agent EM user and private key (ssh-keygen -b 2048 -t rsa)

• Add the EM Server (IP Hostname) to host

• Bug 23013302 in EM OPC Agent - 'fproxy forwarder' process is utilizing 95 to 100% CPU

• According to Oracle certifications SOACS 12.1.3 is not supported with EM 13.1

• Error Hospital Fix – Copy Jar files manually

• Weblogic Domain Refresh

• OTD SNMP Port for availability status?/ Firewall

• Downtime (Oracle Patching and Changes)

48

Now certifiedFixed in 13.2

https://support.oracle.com/epmos/faces/BugDisplay?id=23013302&parent=SrDetailText&sourceId=3-13105306851




• Tailoring Compute Cloud Service



© CIMA 50

Benefits

• Provisioning – 50% Faster

• Enhanced monitoring – 20% Reduction in Man-hours

• 50% Reduction in TX completion time

• Security

• Backups

• Scale Up – Scale Down

• Reduce Head Count ??

© CIMA 51

Oracle Excellence Award 2016 – Cloud Innovation

謝謝！Спасибо!

Dziękujemy!

!شكرا لك

Thank you!

Kiran Tailor

[email protected]


HOW TO CONTACT US

53

@simon_haslam@kirantailoruk

Technology

Experiences of SOACS