Upload
amity-meadows
View
38
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Metadata Driven Data Services for SOA. Paul Anderson Technical Sales Director 30 October 2014. Agenda. SOA Review Data Integration Data Services Use Cases Wrap-Up. Overview. - PowerPoint PPT Presentation
Citation preview
Metadata Driven Data Services for SOA
Paul Anderson
Technical Sales DirectorApril 20, 2023
2
Agenda
• SOA Review
• Data Integration
• Data Services
• Use Cases
• Wrap-Up
3
Overview
Data architecture and its management are evolving toward an data services management system that delivers information as a service.
4
SOA Timeline
• Personal involvement since 1998– “Candle's Roma product of 1998 is the ESB's most
direct ancestor" (Roy Schulte, Gartner)
• SOA concepts have existed for a long time– Business needs did not drive SOA adoption– Fragmented technology slowed adoption
• Gained widespread acceptance in last 5 years• Web Service Technology has enabled broad
adoption of the concept
5
What is SOA?
• Gartner– "Web services are about technology specifications, whereas SOA is a
software design principle.” (Yefim V. Natis)
• SOA Concepts– Decoupling of service production from consumption– Service Interchange– Service Discovery
• SOA is an architectural pattern– The architectural pattern has/does appeared in many guises– ESB tools represents the most widely accepted toolset for SOA
implementation– Just because you use an ESB does not mean you have an SOA
architecture• SOA benefits from the platform neutrality of Web Services
– Enables interchange of services– Requires decoupling– Provides discovery
• The changing business environment is driving SOA– The scope of business functionality is changing
6
SOA Business Drivers
• Business Process enhancements– Migration of customer management to relationship management– Risk Management across/between organizations– Order/Shipment tracking across/between organizations– Tighter Partner Management
• Cross-Sell / Up-sell– Based on history/demographics/relationship/value
• Business Agility– Services are re-usable building blocks– Cataloging of available services– Enhanced time to market– Focus on information rather than data
• Integrated Businesses– Transactions span multiple organizations
7
Characteristics of SOA
• Requires Horizontal integration– Scope crosses lines of business boundaries– Scope crosses corporate boundaries
• Requires Horizontal understanding– Requires understanding of multiple lines of business– Requires development of common vocabulary
• Require concept generalization– Exploitation of commonality– Drilldown into specifics
• Requires easy access to information– Need to remove the complexity of differing semantics,
location, and access methods
8
SOA often slowed by
• Vertical Fragmentation– Differing technologies across verticals– Differing semantics across verticals
• Infrastructure Fragmentation– Application fragmentation – Hardware fragmentation– Data fragmentation
• LOB focus• Multiple technologies• Replicated data
• Organizational fragmentation– Ownership Issues– Organizational issues– Governance issues
9
Common Misconceptions #1
SOA is an technology issue:
• Web Services are a suite technology specifications
• You can implement a web service based stove pipe
• SOA does not have to be web service based
• Looks like an arcitectural diagram but is just a list of standards
10
Common Misconceptions #2
SOA is an application issue:
• Focuses on services but ignores the underlying data
• Characteristics of an application layer are very different from the characteristics of a data layer
• Focuses on reuse of functionality and ignores reuse of data
11
Common Misconceptions #3
An ESB tool is all you need:
12
Common Misconceptions #3 (Continued)
• Fragile implementation– Data access is embodied in the business process– Adding an additional data source requires business process
change– The data services are source specific
• Poor performance– Data reduction is likely happening in the Business Process layer– XML well suited for business process layer but not as well suited
to the data services layer
• Unmanageable– Difficult to audit data access– Difficult to secure data access
13
SOA reality
• SOA is fueled by data– SOA combines People, Machines – SOA embodies Business Process– SOA is based on data
• SOA is fueled by metadata– You can only reuse what you know exists– You can only reuse what you understand
• SOA based applications need to co-exist with existing applications
14
DataServices
BusinessServices
Applications Processes Users
Get customer Close account
Archive history Get single view of product
Calc lifetime value
Request for data operations:(Type, Format, Latency, Quality)
Response: (Data or
Metadata)
DataDataData
Metadata Sync
Access Profile Move
TransformAggregate
Data Integration Implementation Evolves Toward Data Services – Gartners View
15
Deriving Value From Enterprise Information
Leading organizations incorporate information assets into their portfolio management process. – Treat information as an organizational asset
• Optimize availability and quality
• Balance accessibility vs. security
• Improve information supply chain performance and integration
– Squeeze more value out of information• Optimize and eliminate overlapping processes
• Improve enterprise performance
• Trade information for goods and services
• Strengthen enterprise relationships (partners, employees, customers/constituents, suppliers)
Information must be managed as a currency — save it, invest it, spend it, manage it and account for it.
16
Agenda
• SOA Review
• Data Integration
• Data Services
• Use Cases
• Wrap-Up
17
??
Enterprise Information Problem
Enterprise InformationEnterprise InformationSourcesSources
Enterprise InformationEnterprise InformationConsumersConsumers
customerscustomers
salessales
supplier datasupplier data
billingbilling financialfinancial
accountsaccountsshippingshipping
customerscustomers
Data ServicesData Services
The current state of corporate data
Disparate Information
Silos
18
Data Integration Challenges
• Organizations cannot address basic questions– What information do I have?– Where is it?– How do I access the most current information?– How do I manage it?– What is the impact of change during and after
implementation of my system?
• Information is stored in myriad data stores across the Enterprise. – Different formats/data types– Different structures/semantics– Different access methods (API’s)
19
Data Integration Landscape
EAI
CRM ERP
EAI - Target is the application Even Driven data movement between apps Data movement pre-wired Message based. Requires application coordination / workflow. Requires complex translation / programming.
CRM ERP ETL
ETL - Target is the database Scheduled extraction of data Bulk loading of data warehouses. Batch driven, Large Volumes. Inflexible, unable to meet business agility needs Historical data analysis. One-way data movement, read-only access.
EAIDW /ODS
Reporting /Dashboard
Data Services - Target is the end-user
On-demand delivery of information Real-time integration of disparate data. Universal data access layer. Integrates multiple data sources. Push / Pull any data across the enterprise. Single View of Customer. Supports dynamic / evolving reporting. Supports / Enables SOA / Web Services.EAI
CRM ERP ETL
DW /ODS
DS
Web Services
BI BAM Apps
20
Integration Technologies
ETL
EAI
DataService
s
Data ProcessIntegration Style
Data
Inte
gra
tion T
imelin
ess
Real Time
Batch
21
What if……….
CustomApplication
s
Web Services,Business Processes
Packaged Applications
Reporting, Analytics
Each client has their own desired “view”
Clients issue queries against their “view”
EAI, ESB, BPM
Client sees single database containing enterprise data
Clients are freed from the specifics of underlying sources
Enterprise Data Enterprise Data ServicesServices
<sale/> <value/></ sale >
Virtual Data
22
Enterprise Data Services
XML DWDBMS FILESAPPS
Applications /Dashboards
ESB/Portals
Reporting / Analytics
• High Performance Infrastructure
• Universal data access layer.
• Integration of structured/unstructured data.
• Data access security enforcement.
• Insulates data consumers from sources.
• Real-time integration of disparate data.
• Federation of operational & historical data.
• Provides a virtual representation of data.
• Extract & Combine information on-demand.
• Supports dynamic reporting / dashboards.
• Read & Write to any data source.
Web Service RelationalWeb Service/
Relational
23
model, describe, relate
share, version, manage, discover
package, deploy, version
model, describe, relate
MetadataCatalog
ModelerModeler
access & updatereal-time
Custom Apps
Web Services,Business Processes
Packaged Apps
Reporting, Analytics
EAI, Data warehouses
xml
databases
warehouses
spreadsheets
services
<sale/> <value/></ sale >
geo-spatial
Enterprise InformationEnterprise Information
rich media
…
Model-driven Data servicesUnderstand, relate, harmonize, rationalize, and use
Data ServerData Server
<sale/> <value/></ sale >
Virtual Data Server
real-timediscover,access,update
24
Agenda
• SOA Review
• Data Integration
• Data Services
• Use Cases
• Wrap-Up
25
SOA Challenges
• Majority of data is held in relational sources– Relational structure optimized for LOB usage– Probably do not want the existing data structure to drive data
service structures– Performance concerns may rule out XML federation
• Granularity Issues– Granularity of data needed to support SOA may be different
than current use cases– Typically there will be data reduction moving from existing data
sources to data service presentation• Web-Service data is XML based
– Data is organized hierarchically– Need to map from relational to Hierarchical
• Scope of application typically wider that traditional applications– Spanning lines of business and/or corporate boundaries
26
SOA Challenges (continued)
• Performance– XML based data is exceptionally verbose– Exposing existing data in XML clearly not viable
• Security– Broader access to data drives security concerns– Applications that span political boundaries may be
subject to local confidentiality regulations– May need to integrate data security with application
security infrastructure
• Auditability– Regulations demand access audit trails
28
What is a Data Service?
MasterData
SAPDivisionalOperationalApplication
Data Service
SQL SQLAPICall
SOAP/XML
• Decouple data sources from application– Data implementation shielded
from application
• Semantic/Format Mediation– Standard vocabulary
• Single access point– Web Service/XML– SQL
• Federation• Scalability
– Security, performance
Bridge theGap
29
Data Service Layer in SOAClient Process & Applications
Data Sources
Data Services Layer
Message Services (ESB)
Business Services
Business Process Services
App App App App App App
Data Service Data Service Data Service Data Service Data ServiceData Service
30
Obtain XML From Non-XML Sources
?
«XML»
<customerPositions> <accounts> <account ID=…>
…
</account> </accounts></customerPositions>
«Application»
«Relational»
<<XML Doc>>
NEED:Mapping from Data to XML
GIVEN: Data Sources containingInformation to integrate
GIVEN: Fixed XML SchemaWANT:Data complying to schema
31
XML Modeling to achieve business agility
The Problem:The Problem:
TT
<X>
</X>
<X>
</X>
…
XML Document
EIS Data
XML XML SchemaSchema
TT
<X>
</X>
<X>
</X>
…
XML Document
EIS Data
XML XML SchemaSchema
TT
<X>
</X>
<X>
</X>
…
XML Document
EIS DataThe Solution:The Solution:
32
Data Service DesignThree ‘typical’ use scenarios:• Bottom-up: expose relational-style sources in ‘table-like’ form• Business view: starting from XSD/XML-based business views• Top-down: starting from WSDL definition of Web service operations
Web SvcOperatio
n
XSDIN
XSDOUT
<<X>>
Import sources
Source models
Relational views
XML views
XSDs - in/out
Web service operation
VD
B C
onta
iner
VD
B C
onta
inerWSDL
WSDL descriptor
BottomUp
Start
33
Data Service DesignThree ‘typical’ use scenarios:• Bottom-up: expose relational-style sources in ‘table-like’ form• Business view: starting from XSD/XML-based business views• Top-down: starting from WSDL definition of Web service operations
Web SvcOperatio
n
XSDIN
XSDOUT
<<X>>
Import sources
Source models
Relational views
XML views
XSDs - in/out
Web service operation
VD
B C
onta
iner
VD
B C
onta
inerWSDL
WSDL descriptor
BusinessView
Start
34
Data Service DesignThree ‘typical’ use scenarios:• Bottom-up: expose relational-style sources in ‘table-like’ form• Business view: starting from XSD/XML-based business views• Top-down: starting from WSDL definition of Web service operations
Web SvcOperatio
n
XSDIN
XSDOUT
<<X>>
Import sources
Source models
Relational views
XML views
XSDs - in/out
Web service operation
VD
B C
onta
iner
VD
B C
onta
inerWSDL
WSDL descriptor
TopDown
Start
35
Data Service DesignThree ‘typical’ use scenarios:• Bottom-up: expose relational-style sources in ‘table-like’ form• Business view: starting from XSD/XML-based business views• Top-down: starting from WSDL definition of Web service operations
Web SvcOperatio
n
XSDIN
XSDOUT
<<X>>
Import sources
Source models
Relational views
XML views
XSDs - in/out
Web service operation
VD
B C
onta
iner
VD
B C
onta
inerWSDL
WSDL descriptor
BottomUp
BusinessView
TopDown
Start
Start
Start
36
Data Services – Design Time
XML DWDBMSAPPS
• Model and Manage metadata.
• Combines technical & business metadata
• Enrich models with custom metadata.• Infer relationships.• Infer ownership.
• Maintain consistent data dictionary
• Understand overlaps in business areas
• Reduce time to perform impact analysis.
• Metadata open standards
• MOF• XMI
• Import metadata from :
• Rational Rose• ERWIN• System ArchitectUML
MetadataModeler
MetadataRepository
RuntimeMetadata
Virtual Databases
Security
MetadataReports
Metadata Relational View
Design timeMetadata
37
Data Services – Run Time
XML DWDBMS FILESAPPS
MetaDataModeler
QueryBuilder
MetadataRepository
Connector Framework
Query Processing / Optimise Engine
Data Cache
Virtual Databases
Security/Audit
Console(Admin)
• Robust security and access control• Row & column level• Supports existing authentication solutions
• Connectivity to• Databases• ERP• MOM• Data Warehouses• Apps / Legacy• Files
• Distributed query optimization & processing
• Data Caching vs. Real Time
• Model driven Integration• GUI based modeler• Metadata Repository
• Data Connectivity via• SQL-92• JDBC• ODBC• SOAP/HTTP• SOAP/JMS
38
• On-demand information– Real time data integration– Information sharing between business units
• Federation of disparate Information– Structured, unstructured– Relational + XML + Enterprise Apps + Legacy
• Faster time to market– Integrated information in days, weeks– Tight coupling of design & implementation phases– Leveraging the skill-set of the data architects for integration
• Costs across application lifecycle reduced– Model-driven abstraction layer between information sources and
applications eases development and maintenance
Data Services benefitsData Services benefits
39
Agenda
• SOA Review
• Data Integration
• Data Services
• Use Cases
• Wrap-Up
40
Typical Use Cases
• Financial Services – Market Reference Data
– Risk Management: BIS - Basel II
– Transaction Monitoring: Anti-money Laundering, Patriot Act
• Supply Chain– Visibility
– Reporting
• Customer Service– Single View of Customer
– Account Aggregation & Cross-marketing
• Financial Reporting– Corporate Governance & Compliance
– Sarbanes-Oxley, Executive Dashboard
• Homeland Security– Watch Lists
41
Securities Reference Data
MetaMatrix Server
Challenge• Securities data is inconsistent• Consistent pricing & other data needed for
better trades & accounting• “Hard-coding” was too expensive and inflexible
Solution• Models define common “view” of securities data• Transform data to XML formats• Publish to pub/sub system• Support over 50 front-end applications
ROI• Consistent data = better trades, better
accounting, better risk management• New data sources online faster (weeks)• $3 million in IT savings
SecurityPrice
SecurityPrice
SecurityDescription
IBM MQ Series
Front-End Applications
42
Implementing Web Services & Service Oriented Architecture
• Business Drivers – Information sources available and discoverable as location-
independent Services on the CSFB Network
– CSFB Lines of business have access to information when and where it is needed
– Web Services Loosely Coupled IT Architecture
• Technology Use Case – Universal Data Services– Web Service consumers integrated to discoverable Services
– Encapsulate existing/legacy functionality
• Solution Benefits – ROI – 70% of IT Budget spent on integrating CSFB systems
– Found new uses for old data
– Squeezed more value out of legacy systems
– Embrace heterogeneity
– Increase business agility
43
Single View of Customer
Challenge• Call center application• Full view of customer interaction• Data distributed across multiple systems• Data inconsistent• Need a enterprise-wide data layer for more
applications
MetaMatrix Solution• Real-time access• Virtual views of data• Federated queries across systems• Common re-usable data definitions• Enterprise-wide data layer for re-use
ROI• Better service of customers• Lower data integration costs
MetaMatrix
Call Center Application
Data Sources
44
Integration
XML XML / Binary
Application
XML XML / BinaryXML /Binary
Da
ta Q
ua
lity
Rules
Enterprise Information Integration
Information Bus
Extract, Transform and Load
Information Transport & Protocols
ODS(structured)
Near Real Time
Content(unstructured)
LOBApp(s)
LOBData
DataStaging
ExternalData
DataWarehouse
Met
adat
a M
anag
emen
t
MetaData
Info
rma
tion
Ass
ets
Info
rma
tion
Ser
vice
sIn
form
atio
n C
onsu
mpt
ion
DataMarts
Conceptual Architecture
45
Agenda
• SOA Review
• Data Integration
• Data Services
• Use Cases
• Wrap-Up
46
Recommendations
• Start small, incrementally grow toward information as a service
• Build the infrastructure on a project by project basis
• Data security must be a top priority• Ensure performance as you go• Keep it in SOA context — information fabric is a
means to an end, promoting information as a service
• But ability to access and use information is a longer-term vision
48