Upload
alina-anchidin
View
227
Download
0
Embed Size (px)
Citation preview
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 1/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Jon Mead, Rittman MeadSeptember 2012
Event Driven Real Time Analytics
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 2/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Introductions
• Jon Mead
‣CEO/co-founder of...
•Rittman Mead Consulting
‣Oracle BI & DW Consultancy
‣Gold Partner
‣
Long(est) running Oracle BI blog‣ Annual BI Forum
‣OBIEE Oracle Press book
•Customer-facing FTSE listed
‣UK based and leading
‣ Internet based‣Retail based
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 3/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
The point of thispresentation isto give you an
idea of how toapproach a realtime eventdriven BIsystem using
Oracle's currenttoolset.
Agenda
•Understanding the Project
‣ Legacy architecture
‣Proposed architecture
‣Reporting requirements
•Technical Infrastructure
‣Hardware and Software
•
Data Warehouse Architecture‣ Adopting the Oracle reference architecture for real
time
•Design Challenges
‣De-queuing
•Operational
‣ODI Logging‣Multi-threading and scalability
•Further thoughts
‣Middleware or memory based applications
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 4/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Understanding the Project
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 5/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Business Goal
• Part of a major re-architecture program
• Covering ERP, CRM and BI
Driver: single view of customer
Delivered by: channel consolidationinto single enterprise data warehouse
• Data migration• Enterprise Architecture
• Enterprise Service Bus
• Real-time reporting• Legacy reporting
• BAU reporting
• Revenue and Profit• Liability and risk
• Up/cross-sell
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 6/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Legacy Architecture
• Legacyarchitectureconsisted of twocompletelyseparate
systems
• Retail storedshop basedtransactions
• Online storedtransactionsgenerated online
Retail trading systems
Online trading systems
Online trading systems
Online trading systems
24 hour batch (DTS)
Retail Data Warehouse
SQL Server 2005
6TB
24 hour batch (DTS)
Online Data Warehouse
SQL Server 2008
3TB
Retail trading systems
Retail trading systems
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 7/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Proposed Architecture
Online trading systems
Online trading systems
Enterprise Architecture
Online trading systems
Real-time feed
TIBCO QueueReal Time Data
Warehouse
Exadata
Real-time feed
Real-time feed -
transactions
Real-time feed -
reference data
DR
Exadata
ODI
ODI
Retail Data Warehouse
SQL Server 2005
6TB
Online Data Warehouse
SQL Server 2008
3TB
D a t a m i g r a t i o
n
- o n
c e o f f O
D I
D a t a m i g r a t i o n
- o n c e o f f
O D I
Retail trading systems
Retail trading systems
Retail trading systems
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 8/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Proposed Architecture
Online trading systems
Online trading systems
Enterprise Architecture
Online trading systems
Real-time feed
TIBCO QueueReal Time Data
Warehouse
Exadata
Real-time feed
Real-time feed -
transactions
Real-time feed -
reference data
DR
Exadata
ODI
ODI
Retail Data Warehouse
SQL Server 2005
6TB
Online Data Warehouse
SQL Server 2008
3TB
D a t a m i g r a t i o
n
- o n
c e o f f O
D I
D a t a m i g r a t i o n
- o n c e o f f
O D I
Retail trading systems
Retail trading systems
Retail trading systems
Current state to future
state includes a data
migration
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 9/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Need tounderstand thedifferent driversfor each of these needs and
the valueprovided by realtime reportingduring the
running of hightransactionevents
Reporting Requirements
•Real time monitoring
‣Risk and liability‣Profit and loss
• Analytic reporting
‣Consolidated analytics
‣ Legacy reporting
•Operational reporting
‣Detail level
‣Support analytical reports
‣Drill through
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 10/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Technical Infrastructure
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 11/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Volumetrics
• Initially processing data from 2500 shops, scaling to capacity
‣ 8 TB of migrated data‣Processing 1.8 million transactions a day
‣Processing 4,000 reference data items a day
‣ Approximately 9 million transaction rows being processed a day
‣ All transactions read from a TIBCO queue
‣ Approximately 200,000 reference data changes a day
‣ 30,834 transaction processing cycles a day (one every ~2.8s)
‣ 2,701 reference data cycles a day
‣ 680,000 recalculations a day
•Online transactions will follow
‣ 2 million transactions a day‣Comparable downstream figures
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 12/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Exadata and ODI
•Standard X2-2 Quarter Rack
‣ 2 compute nodes‣ All databases split across the nodes
•The storage is configure in dual redundancy mode to offer up about 9TB of usablespace, however, we use a couple of TB of that for backups and archive redo logs
•The flash storage has been set up as 250GB on each node as a local cache and 110GBfrom each being used to provide a 160GB flash disc.
•The database version is 11.2.0.3 and the client have the tuning and diagnostics packand Heterogeneous Services on top of the usual Exadata software set.
‣Both the 11.2.0.2 and 11.2.0.3 Oracle Homes still exists.
•ODI Agents for UAT and PROD running off Node 2
•Running up to 30 ODI Agents for PROD to get the speed to read off the TIBCO queues.Each agent running with 512MB with the calling agent running 1GB.
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 13/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Exadata and ODI
NODE 1
DEV01
DEV02
NFT
PROD
NODE 2
Compute Nodes
OracleDatabases
ODI AGENTS
DEV/ NFT
ODI AGENTS
PROD
UAT
ODIInstalls
PROD ODI
WORK
SCHEMA
UAT
REPOSITORY
PROD
REPOSITORY
UAT ODI
WORK
SCHEMA
UAT
SSD
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 14/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Exadata and ODI
NODE 1
DEV01
DEV02
NFT
PROD
NODE 2
Compute Nodes
OracleDatabases
ODI AGENTS
DEV/ NFT
ODI AGENTS
PROD
UAT
ODIInstalls
PROD ODI
WORK
SCHEMA
UAT
REPOSITORY
PROD
REPOSITORY
UAT ODI
WORK
SCHEMA
UAT
SSD
Currently only one Exadata
server available, so shared
platform
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 15/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Data Warehouse Architectur e
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 16/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Key Drivers
•Part of integrated Enterprise Architecture
•The enterprise data model was designedand developed in Enterprise Architect by themiddleware architects
•The architects wanted to base the approach
on the Oracle Reference Data Warehousearchitecture
•There were different reporting needs for real time and business as usual reporting
•Write performance was likely to be as big afactor as read performanc
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 17/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Oracle Reference Architecture
•Simplified view of Oracle’s
Data Warehouse Reference Architecture
•Enterprise Architecture wasXML based
Active Data Warehouse
Staging
Foundation
O
B I E E
PerfromanceAudit and
Reconciliation
Analysis
Operationaland real-
time
Enterprise Architecture
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 18/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Oracle Reference Architecture
•Simplified view of Oracle’s
Data Warehouse Reference Architecture
•Enterprise Architecture wasXML based
Active Data Warehouse
Staging
Foundation
O
B I E E
PerfromanceAudit and
Reconciliation
Analysis
Operationaland real-
time
Enterprise Architecture
One of design drivers was that the
foundation layer reflected the
enterprise data model
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 19/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Limitations
•The ODS would reflect the
enterprise architecture‣Non-database centric view
•Considerable processing to getdata into ODS
‣Data processing from Stagingto Foundation was too
complex to support SLAs•Performance layer also to
reflect existing more datawarehouse structures
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 20/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Limitations
•The ODS would reflect the
enterprise architecture‣Non-database centric view
•Considerable processing to getdata into ODS
‣Data processing from Stagingto Foundation was too
complex to support SLAs•Performance layer also to
reflect existing more datawarehouse structures
Result was non-performant and
unusable structures for real-time
reporting
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 21/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Alternative Architecture
• Split the processing between real-time and BAU
‣Process 1: Staging to Performance (real-time)
‣Process 2: Staging to ODS to Foundation
ODI
STG_xxx
STG_xxx
STG_xxx
STG_CTL
Staging
T I B C O
ODI
Microbatch ETL
BETBETSLIPDecompositionand Aggregate
tables
PerformanceFoundation
ODI Real time ETL
Real timeETL
O B I E E
SQL
SQL
Real time
query
Near realtime
BETBETSLIPDimension and
fact tables
ODI
Microbatch ETL
SQL
Near realtime
BETBETSLIP
3NF tablesBETBETSLIP
3NF tables
• Independentcontrol of either process
• Mechanism tohandle peaks indata
• Needed to ensureconsistencybetweenprocesses
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 22/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Design Challenges
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 23/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
De-queuing
•Concern that ODI would not be able to de-queue
‣ A lot of fluctuations, depending on events
•XML messages were verbose
‣ Large amount of processing time for each batch of messages
•Scalability provided by creating more agents
‣What would the limitations be in terms of RAM
‣What would the limitations be in terms of connections
‣What would the limitations be in terms of management
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 24/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
2 Queues
•Queue are unstructured so
the data can arrive in anyorder
‣Difficulty of processingbusiness logic
‣ Timestamps not alwaysaccurate
‣Keys not always present
•One of most challengingareas of the project
‣Often need to do manuallookup of keys
Transactions
STG_CTL
T I B C O
Real timeETL
Referencedata
Recycle
STG_xxx
STG_xxx
STG_xxx
Solution: recycle
mechanism
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 25/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
High number of writes
•The real-time write process generated a very high number of writes
•Exadata optimised for bulk reads•Contention for REDO logs (see also the ODI Logging)
•Exadata configured for more of an OLTP system than Data Warehousing system
‣However both share the same server
•Resolution: lots of work by the DBAs to optimise database configuration
Isn't this a little bit like an
OLTP system?
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 26/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Operational Challenges
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 27/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
ODI Parser
•XML Parser
‣ The XML data definition files were dynamically generated.‣ The current version of the XML Parser does not do a double pass of the definition file
‣ Any referenced complex definitions needed to be defined in the order they wereaccessed
‣ The software generating the XML definition files did not do this
•Resolution
‣Build a Java program to re-parse the XML data definition file and output a correctlyordered version
‣ This is a once per release process
‣ This behaviour if fixed in 11.1.1.7 of ODI (I think)
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 28/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
De-queuing Performance
•ODI struggled to keep up with, or fell behind the queue at peak times
‣Volumes of messages were not regular
•We also found agents failing
‣Hence we needed a resumption mechanism
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 29/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Scaling agents
•Because of the failing agents, we couldn’t just
increase their number
•Set up parent agents
‣One for each queue
‣One for monitoring and maintenance scripts
•Each parent agent ran a number of child agents
‣Each child agent was actually two agents
‣Second agent acted as redundancy
‣ Agents killed after 50 executions
P1 P2 M&M
A1 A2 A3 A4 C6
A3 A4
C3 C4
A1 A2
A1 A2
C1 C2
Q1 Q2
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 30/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Connections:
Every time anODI agent readfrom thequeue a newconnectionwas createdand destroyed.
There didn’tseem to beany pooling.
Impacts of Multiple Agents
•Memory
‣Parent agent 1024MB‣Child agent 512M
•Total number of agents used
‣ 3 parents
‣ 18 child (primary) and 18 child (secondary)
‣ Total: 39 = 21504MB (approx 21GB)
•However we didn’t get anywhere near linear scaling
‣Max TPS = 176
‣Max queue TPS = 480
•Second option is to increase the number of queues
‣Split by functional area
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 31/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
ODI Logging
•ODI Logging
‣ The ODI processes create 900GBof log files a day
‣ODI logging needs high IOPS
‣Exadata, by default not allocatingenough IOPS resource to the ODIlogging
‣ODI logging then becomes alimiting factor on the databaseperformance
‣ Target is SNP_SESS_TASK_LOG
‣ Log writer process cannot keepup
‣Number of active processes meanthe database will be performingas hard as it can and more activitywill slow everything down.
!
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 32/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
ODI Logging
•ODI Logging
‣ The ODI processes create 900GBof log files a day
‣ODI logging needs high IOPS
‣Exadata, by default not allocatingenough IOPS resource to the ODIlogging
‣ODI logging then becomes alimiting factor on the databaseperformance
‣ Target is SNP_SESS_TASK_LOG
‣ Log writer process cannot keepup
‣Number of active processes meanthe database will be performingas hard as it can and more activitywill slow everything down.
!
ODI does the same logging but the volumepreserved is reduced with lower levels of logging.
So in fact, lower levels of logging could be more
IO demanding as more data is deleted.
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 33/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
SNP_SESS_TASK_LOG
•The most demanding
SQL statement on thesystem is and always hasbeen the update toSNP_SESS_TASK_LOG
•This table holds 3 CLOB
columns. The update is"lazy", all columns areupdated each time. Thuseach update canpotentially update:
‣ the table
‣
three clob indexes‣ three clob tables
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 34/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
ODI Logging Impact
• Initially the system was
throttled on theSNP_SESS_TASK_LOG.
•ODI IOPS demand wasmaxing out the physical discIOPS capability of the box
•MoveSNP_SESS_TASK_LOG andSNP_SESS_TASK to a new ASM diskgroup created fromthe SSD storage in Exadata
!
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 35/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
ODI Logging Impact
• Initially the system was
throttled on theSNP_SESS_TASK_LOG.
•ODI IOPS demand wasmaxing out the physical discIOPS capability of the box
•MoveSNP_SESS_TASK_LOG andSNP_SESS_TASK to a new ASM diskgroup created fromthe SSD storage in Exadata
!
Simple solution for the ODI
Logging problem is to movethe database to another
server
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 36/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
ODI temporary tables
•The ODI real-time processing
creating a large number of I$ andother internal tables
•Once the processing around theseis complete, they are put theRecycle Bin
•The Recycle Bin become either
full or unmanageable
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 37/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
ODI temporary tables
•The ODI real-time processing
creating a large number of I$ andother internal tables
•Once the processing around theseis complete, they are put theRecycle Bin
•The Recycle Bin become either
full or unmanageable
There is a wider issue here
that affects scalability of the
whole solution, discussed innext slides
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 38/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Future Challenges
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 39/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Accessing datain memoryreduces the I/Oreading activity
when queryingthe data whichprovides faster and more
predictableperformancethan disk
Scalability
•The main bottleneck we are experiencing is I/O
‣High number of writes to the database‣ODI $ internal tables
‣ODI logging
•We should address this problem by making better use of memory
•We also have a constraint on the amount of memory Exadata can provide the agents
‣ Any allocated memory has the opportunity costof not be used by the database
•We should also explore other ‘logical’ approachesto solving this problem
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 40/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Process Flexibility
•The source system splits the data into real
time and batch‣ESB provides 2 separate queues
•The XSD is the same for both queues
•The processing for both queues isconstantly running in a loop
‣ The batch queue is much larger than the
real time queue.‣ The foundation layer requires data from
both
•The data from each queue lands in thesame stage tables partitioned by thequeue name
•Entire process controlled by maintainingBATCH_IDs
SRC SystemsFeed
Non Critical Data Real Time Data
Stage Schema
DQProcess
DQProcess
Event StageTables
STG_ Tables
Performance Schema
Foundation Schema
Reporting Tables
ODS LoadProcessing
Real Timeprocessing
Foundation LayerTables
CTL_EVENTbatch_idbatch_typeODS_processedRTF_processedSTG_processed
CDC ProcessCDC Process
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 41/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
In-Memory Processing
•Will require re-writing of the Knowledge Modules
‣Should also persist connections
•Option 1: Remove the writes to the $ tables and attempt to do moreoperations on-the-fly
‣Potential loss of audit trail and reconciliation points
‣Currently all outer joins are materialised
‣Will need to perform
•Option 2: Use In-Memory database?
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 42/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
In-Memory Processing
•Will require re-writing of the Knowledge Modules
‣Should also persist connections
•Option 1: Remove the writes to the $ tables and attempt to do moreoperations on-the-fly
‣Potential loss of audit trail and reconciliation points
‣Currently all outer joins are materialised
‣Will need to perform
•Option 2: Use In-Memory database?
Exalytics?
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 43/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Conclusion
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 44/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
The objectivesof the projectwhereachieved. Theresulting datais being usedon a dailybasis andproving
significantvalue to theorganisation
Conclusion
•The Oracle Reference Data Warehouse architect can
support real time event driven ETL, however it mayneed modifications
• IDO has some rough edges and kinks that need to beironed out for it to act at this kind of enterprise level
•Don’t underestimate the effort of doing a data migration
• Its important to understand the implications anddifferences of middleware centric data models andprocessing compared with databases centric ones
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 45/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Questions?
Sunday, 30 September 12
7/30/2019 Event Driven Real Time Analytics
http://slidepdf.com/reader/full/event-driven-real-time-analytics 46/46
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com
Jon Mead, Rittman MeadSeptember 2012
Event Driven Real Time Analytics