Oracle GoldenGate 12 - DOAG Deutsche ORACLE … · Oracle GoldenGate Administration Client (adminclient) Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | [oracle@ogg123

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Oracle GoldenGate 12.3 Was ist neu an der Microservices Architecture

Thomas NiewelSystemberaterOracle Deutschland B.V. & Co K.G.November 2017


Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

3


Agenda

Oracle GoldenGate

Oracle GoldenGate Classic Architecture vs. Microservices Architecture

WebPage Interfaces

Directory Structure

Rest Api

1

2

3

4

5


Cap

ture

Trai

l

Ro

ute

De

live

r

Pu

mp

Oracle GoldenGate Use cases

New DB/HW/OS/APP

Zero Downtime Upgrades & Data Migration

Fully Active Distributed

DB

High Availability& Disaster Recovery

ApplicationOffloading

Query & Report Offloading

Big Data, DW & Marts

Real-time BI, Hadoop Data Staging, Data Ingestion

Event Driven Architecture, SOA/JMS, Coherence

Message Bus& Data Grid

Data SynchronizationAcross the Enterprise

Global Data Centers

Real-time Analytics& Massive Parallelization

DataStreaming

GoldenGate

Real-timeData Delivery


Oracle GoldenGate Flexible Deployment Models

UnidirectionalQuery OffloadingZero-Downtime Migration

Bi-DirectionalHot Standby orActive-Active for HA

Peer-to-PeerLoad BalancingMulti-Master

BroadcastData Distribution

Integration/ConsolidationData Warehouse JMS, Flat File

Oracle GoldenGateApplication Adapters

HDFS,HIVE,FLUME,Kafka,

HBASE. JDBC

Oracle GoldenGate for Big

Data


GoldenGate: Heterogeneous Replication

7


Agenda

Oracle GoldenGate


WebPage Interfaces

Directory Structure

Rest Api

1

2

3

4

5

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 9

OGG 12.3 Overview• 2 Installation Packages

– Classic Architecture (CA)

–Microservices Architecture (MA)

• Upgrading to 12.3

– Classic Architecture – as usual

–Microservices Architecture – must be fresh install


Oracle GoldenGate Classic Architecture

DYNAMICPORTLIST


Administration, Distribution, Receiver, Metrics Services with RESTful Service Interfaces

GoldenGate 12.3Microservices Architecture

Source Target

Trail Files

Extract

Admin Server

Trail Files

Replicat

Distribution Server

Receiver Server

SourceDatabase

TargetDatabase

HTTPS RESTFulService Interfaces

Admin Server

Metrics Server

Service Manager

Service Manager

Metrics Server


• Admin Server

– Replacement of the mgr process

• Distribution Server

– Replacement of the pump process

• Receiver Server

– Replacement of the collector process

• Metrics Server

– Real-time performance monitoring

Classic Architecture vs. Microservices Architecture


• Enables remote configuration, monitoring, and administration

• Provides formal RESTful service interfaces

– Industry-standard communications protocol: HTTPS, JSON

– Enables Applications to embed GoldenGate

• Secure – Authorize identity of clients with username / password or SSL client certificates

• No web server needed

Common Key Attributes and Benefits of Each Service


Setup with oggca.sh



Setup with oggca.sh



Solutions For EveryoneAdministrators Traditionalists DevOps

• Simple to use Web interfaces

• Get up and running quickly

• Lower learning curve

• Command Line access via adminclient – run anywhere

• Similar GoldenGatecommands as in GGSCI

• Interaction the same as previous releases, just thin client

• Uses RESTful APIs

• Run anywhere

• Roll your own monitoring


Standard TLS/SSL based authentication and Client Authorization

Security Model

Client Authorization

• Identity via SSL user certificates or via username/password

• Roles

– Security User

– Administrator

–Operator

– User

GoldenGateServices

SSL Server Certificate

Authorized Clients

Clients

SSL User Certificate

HTTPS

User/ Passwd


Remote and secure administration and data communication in cloud environments

Cloud Ready

GoldenGateServices

DeploymentReverse

Proxy wss

HTTPS

• Remote Administration and Configuration

• No OS access required after installation

• Secure administration and data transport

• Works with all standard HTTP proxy and reverse proxy servers

GoldenGateServices

Deployment

Internal Network DMZ


Agenda

Oracle GoldenGate


WebPage Interfaces

Directory Structure

Rest Api

1

2

3

4

5


Access via URL http(s)://<hostname>:<port>

WebPage Interfaces


Access via URL (http(s)://<hostname>:<port>i.e. http://localhost:8209

Use Administration User for access:i.e. oggadmin/********

• Maintains inventory and configuration information about all deployments

• Watchdog process for Other OGG Services

• Runs as a daemon• Auto-Restart

Oracle GoldenGate Service Manager


• Diagnostic Page• Provides access to responses

that the ServiceManagerreceives

• $OGG_DEPLOYMENT/var/log/ServiceManager.log

Oracle GoldenGate ServiceManager


Example: Security Roles

Oracle GoldenGate Administration Server - Housekeeping


Oracle GoldenGate Administration Server - Housekeeping


Access via URL (http(s)://<hostname>:<port>)i.e. http://localhost:8509

• Manage Extracts/Replicats• Create, delete, start/stop, reports,

statistics

• View critical events/messages

• View notifications per page

Oracle GoldenGate Administration Server - Overview



• Manage Extracts/Replicats• Create, delete, start/stop, reports,

statistics

• View critical events/messages

• View notifications per page

Oracle GoldenGate Administration Server - Overview


Example: Create Extract process

Oracle GoldenGate Administration Server - Configuration





Example: Create Extract process (cont’ed)






Oracle GoldenGate Administration Server - Details







• Replacement for the Extract Pump process

• Quickly identify source, trail file, and destination

• Distributes one or more trails to one or more destination

• Does not support transformations

• Supports multiple transport protocols• Tunable per network path

Oracle GoldenGate Distribution Server


Industry standard HTTP(S) initiated full-duplex streaming protocol

Communication Protocol

Trail Files

Extract Trail Files

Replicat

Distribution Service

Receiver Service

• SSL based security

• Can seamlessly traverse through HTTP forward/reverse proxy servers

wssws

udt ogg


• Supports 4 transfer protocols – OGG, WS, WSS, UDT

Note: For WS/WSS, target credential needs to be setup that connects to the receiver service

• UDT • UDP transport for wide-area networks• Faster than TCP over wide-area with

high latency

• OGG• inter-operability with traditional

deployments

Oracle GoldenGate Distribution Server - Setup


• Quickly see path information• Status• Source/Target trails• DB Name• Trail name

• Configure Application Options

• Configure TCP Options

Oracle GoldenGate Distribution Server - Details


• Quickly see path information• Status• Source/Target trails• DB Name• Trail name

• Configure Application Options

• Configure TCP Options

Oracle GoldenGate Distribution Server - Details



• Replacement for Collector

• Single source that handles multiple incoming trail files

• View statistics of path

• Tune network parameters

Oracle GoldenGate Receiver Server


• See Network Statistics and File IO Statistics

• Statistics include:• Trail file name• Port• Bytes received• Bytes written out• Wait times

Oracle GoldenGate Receiver Server - Details


Metrics Service• Separate the metrics collection and repository

from the administrative layer

• All GoldenGate processes send metrics via UDP to the metrics server

• Metrics server provide RESTful API to query metrics. Responses in either

– XML format (12.2)

– JSON format

• Supports integration with third party tools

• MUST have Oracle Management Pack for Oracle GoldenGate

Performance Metrics Server


• Metrics captured in real-time• No Repository

Oracle GoldenGate Metric Server


• New thin client command line tool

• Offers similar functionality as GGSCI, now includes rlwrap

• Runs from $OGG_HOME/bin

– $OGG_HOME can be on any machine

• Supported on Windows and Linux

Oracle GoldenGate Administration Client (adminclient)


[oracle@ogg123 sf_VM]$ ./adminclientOracle GoldenGate Administration Client for OracleVersion 12.3.0.1.0 OGGCORE_MAIN_PLATFORMS_170318.0742

Copyright (C) 1995, 2017, Oracle and/or its affiliates. All rights reserved.

Linux, x64, 64bit (optimized) on Mar 18 2017 13:30:38Operating system character set identified as UTF-8.

GG (not connected) 1> connect http://localhost:16000 deployment Atlanta as oggadmin password ********

OGG (http://localhost:16000 deployment_1) 2>

Address and Port for Service Manager

Deployment to connect to

Administrator name and password

AdminClient (connecting)


• Interact with multiple deployments from a single command line tool

• Runs from any machine using RESTful calls

Administration Client (adminclient)


AdminClient (deployments)


AdminClient (help)


Agenda

Oracle GoldenGate


WebPage Interfaces

Directory Structure

Rest Api

1

2

3

4

5


Old File System layout – Classic Architecture

Directory Overview

• $OGG_HOME is read-write

• Can support multiple process groups but only one installation

• Everything in one directory – CREATE SUBDIRS

• Upgrade path - overwrite binaries, manual moving file


New File System layout – Microservices Architecture

Directory Overview

• $OGG_HOME should be read-only

• One to Multiple deployments can share same $OGG_HOME

• Deployment Homes can be customized

• One step upgrade

• New terms/variables to be used

• DEPLOYMENT_HOME

• $OGG_ETC_HOME / $OGG_VAR_HOME


New Directory Structure

• Number of Deployments: (ServiceManager + N)


Oracle GoldenGate Deployment Home – etc example

Deployment Homes

$OGG_ETC_HOME$OGG_VAR_HOME


Oracle GoldenGate Deployment Home – var example

Deployment Homes

$OGG_ETC_HOME$OGG_VAR_HOME


Agenda

Oracle GoldenGate


WebPage Interfaces

Directory Structure

Rest Api

1

2

3

4

5


Oracle GoldenGate RESTful APIs• Representational State Transfer (RESTful) Application Program Interface

(API)

• Uses HTTP requests• GET, PUT, POST, and DELETE

• Can be used with many different programming languages• PL/SQL• Perl• Shell• JSON• Etc…

• Simple interaction from command line using CURL


Oracle GoldenGate RESTful APIs – JSON Documents • JavaScript Object Notation (JSON)

• An open-standard file format used to transmit data objects consisting of attribute-value pairs and array data types

• Media Type: application/json

• Example – create credential alias:

{"$schema":"ogg:credentials","userid":"ggate","password":"ggate"

}


Oracle GoldenGate RESTful APIs – GET • GET option allows you to retrieve items back from a service

• Example – GET associated services

curl -u gg123:gg123 -H "Content-

Type:application/json" -H

"Accept:application/json" -X GET

http://localhost:8209/services/v2/deplo

yments/ggsrc/services | python -

mjson.tool

.......

"$schema": "ogg:collection",

"items": [

{

"$schema": "ogg:collectionItem",

"links": [

{

"href":

"http://localhost:8209/services/v2/deployments/ggsrc/services",

"mediaType": "application/json",

"rel": "parent"

},

{

"href":

"http://localhost:8209/services/v2/deployments/ggsrc/services/adminsr

vr",

"mediaType": "application/json",

"rel": "canonical"

}

],

"name": "adminsrvr"

},

{

......


Oracle GoldenGate RESTful APIs – POST • GET option allows you to retrieve items back from a service

• Example – POST stats command



"Accept:application/json" -X POST

http://localhost:8509/services/v2/extra

cts/GGSRC/command -d

'{"command":“STATS"}' | python -

mjson.tool

"$schema": "ogg:stats",

"stats": [

{

"operations": [

{

"count": 0,

"operation": "insert"

},

{

"count": 3,

"operation": "update"

},

{

"count": 0,

"operation": "delete"

},

{

"count": 0,

"operation": "discard"

},

{

"count": 3,

"operation": "operations"

}

],

"since": "2017-11-13T15:29:09.298Z",

"sourceTable": "GG123.GG_HEARTBEAT_SEED",

"targetTable": "GG123.GG_HEARTBEAT_SEED"

}

.....


Oracle GoldenGate RESTful APIs – POST • GET option allows you to retrieve items back from a service

• Example – POST stop extract



"Accept:application/json" -X POST

http://localhost:8509/services/v2/extra

cts/GGSRC/command -d

'{"command":"STOP"}' | python -

mjson.tool

"messages": [],

"response": {

"$schema": "er:commandResult",

"reply": "PENDING",

"replyData": {}

}

}


Additional Information

• Documentationhttps://docs.oracle.com/goldengate/c1230/gg-winux/index.html

• OTNhttp://www.oracle.com/technetwork/middleware/goldengate/downloads/index.html


Oracle Cloud Workshops auf der DOAG Konferenz 2017

• Dienstag 21.11.2017

– 10.00 – 13.00 Uhr – Oracle Datenbank und GoldenGate in der Oracle Cloud

– 13.00 – 16.00 Uhr – IT Systems Management mit der Oracle Management Cloud

• Mittwoch 22.11.2017– 10.00 – 13.00 Uhr – Agile Entwicklung von Microservices in der Oracle Cloud

– 13.00 – 16.00 Uhr – Oracle Datenbank und GoldenGate in der Oracle Cloud

• Donnerstag 23.11.2017

– 10.00 – 13.00 Uhr – IT Systems Management mit der Oracle Management Cloud

– 13.00 – 16.00 Uhr – Agile Entwicklung von Microservices in der Oracle Cloud

65

Im Foyer Shanghai-Seoul – Einstieg in die jeweilige Session jederzeit möglich!
