Exploring Data Integration Capabilities of the WSO2 Platform

June 2015

Jagath AriyarathneTechnical Lead

Ravi UndupitiyaSenior Software Engineer

About the Presenters๏ Jagath Ariyarathne - jagatha@wso2.com

Jagath is a Technical Lead at WSO2. Prior to joining WSO2, he worked for DirectFN (Pvt) Ltd, where he was involved in the design and development of DirectFN Financial Data dissemination back-end system. His main focus was on real-time data distribution with low latency, high performance and bandwidth friendly solutions. He holds a BSc First Class degree in Computer Science and Engineering from University of Moratuwa, Sri Lanka.

๏ Ravi Undupitiya - ravi@wso2.com

Ravi is a senior software engineer in the integration technology team where he mainly focuses on the WSO2 Enterprise Service Bus. Prior to joining WSO2 he interned at the Intelligent Agents and Multimedia Research Group at the University of Southampton where he worked on a smart grid project which was aimed at helping people make smarter decisions when consuming energy. Additionally he also dived into the world of Natural Language Processing and carried out an evaluation of several text classification algorithms.

*

Agenda๏ Background to Data Integration

๏ SOA based approach to Data Integration

๏ WSO2 Data Services Server

๏ WSO2 Enterprise Service Bus

๏Demo

๏ Security ๏ Transactions

*

Why we need Data Integration?๏ Data is an important asset.

๏ Found in various sources and often siloed within applications.

๏We must leverage this data.

๏ Data consumers on various devices and platforms. ๏Security and privacy.

๏ Needs to be flexible for change.

๏ Legacy system support.

*

Data Integration Solutions ๏ Ad-hoc approach to data integration not good!

๏ Unmaintainable and messy.

๏ Not flexible enough for changes.

๏ Need data source specific implementations

๏SOA approach better!

๏ Data exposed as a services.

๏ Applications consume the data through services.

.NET J2EE C++

Data Sources

.NET J2EE C++

Data Sources

*

Data Services on WSO2 Platform๏ WSO2 Data Services Server (DSS)

๏ Allows rapid development of data services.

๏ Data services are decoupled from the infrastructure where it is stored.

๏ Wizard based UI to generate data service description language.

๏Data service description language

๏ Maps service requests to queries operating on data store.

๏ Maps query results to XML schema definitions.

๏ Supports validation, transformation, eventing etc…

DSS Key Features - Any Data Store

DSS Key Features - Data Federation

DSS Key Features - Server Side Batch Processing

DSS Key Features - Client Side Batch Processing

DSS Key Features - Array/UDT/Complex Types

DSS Key Features - Distributed Transactions

DSS Key Features - Notifications๏Event sources, subscribers, topics๏Allows a subscriber to be notified of a specific activity in a data service

๏Event Triggers๏ Input Event Trigger๏Output Event Trigger

DSS Key Features - REST Support

DSS Key Features - Caching Support

DSS Key Features - Security Support

DSS Key Features - Transformation

*

Demo 1๏Exposing MySQL data table as a service.

๏ SOAP

๏ RESTful

*

Data Integration Solutions ๏ Ad-hoc approach to data integration not good!

๏ Unmaintainable and messy.

๏ Not flexible enough for changes.

๏ Need data source specific implementations

๏SOA approach better!

๏ Data exposed as a services.

๏ Applications consume the data through services.

.NET J2EE C++

Data Sources

.NET J2EE C++

Data Sources

*

Layered Approach๏SOA alone is not enough. Point-to-Point service integration. ๏Enterprise Service Bus for a cleaner layered approach.

Point-to-Point Service Integration (Spaghetti Integration) ESB in Action

What is an ESB? ๏An ESB is a middleware solution that enables interoperability, among heterogeneous environments using a service oriented model. ๏Stateless and Seamless Integration๏Standard protocols/formats like SOAP, REST, JSON ๏Transports like HTTP/S, JMS, TCP, VFS (and many more)

The WSO2 ESB๏Lightweight and high performance๏Feature rich and standards compliant๏ SOAP and WS-* standards๏REST support๏Domain specific transport support (FIX, HL7)๏User friendly and highly extensible ๏Configuration driven! ๏100% free and open source

WSO2 ESB๏Apache Synapse ๏A lightweight and open source ESB implementation from ASF๏Makes up the mediation engine of WSO2 ESB๏Multithreaded and asynchronous message processing๏Based on a number of well known open source projects๏Apache Axis2 ๏Apache HTTP Components

WSO2 ESB Fundamentals๏ Routing๏ Filtering๏ Transformation๏ Protocol Switching๏ Load Balancing ๏ Quality of Service

More Features๏100% EIP support๏Connectors๏Message Stores/Processors๏JSON Support

๏ JMX based monitoring and control๏ Statistics collection๏ Priority based message mediation๏ XSLT, XPath, XQuery, Smooks, JSONPath๏ Caching, Throttling๏ Scripting languages (JavaScript, Groovy)๏ JDBC connectivity๏ Drools๏ Clustering

Functional ComponentsMediator Sequence Endpoints Local Entry Proxy Service REST API Message Stores/Processors Templates Connectors Tasks Priority Executors Registry

Functional Components๏Each functional component serves a specific purpose

๏Functional components can be mixed and matched to implement various integration scenarios and patterns

๏Configuring WSO2 ESB for a given scenario requires: ๏ Identifying the right set of components๏Putting them together in the most optimal manner

ESB and DSS

๏ Why not access data stores directly from ESB?

๏ ESB has rudimentary DB support๏DBLookup Mediator๏DBReport Mediator

๏ ESB’s function is to ensure low latency with high throughput for stateless transactions. Data intensive processing done on ESB could hinder ESB performance.

๏ Separate JVM/servers to handle data intensive tasks.

๏ Can deploy DSS features on top of ESB (thanks to WSO2 Carbon platform).

Layered Approach

๏ Why access DSS through ESB?

๏ Perfectly ok to access DSS data services directly.

๏ Handling DSS invocations through ESB provides a layer of abstraction over data layer for integration.๏ Interception point๏ Service Chaining๏ Parallel processing๏ Load balancing ๏ Simple transformations๏Handling Multiple protocols

๏ A merger causes an organization to consolidate their account databases.

๏ Leverage WSO2 DSS to expose data from multiple data sources as services.

๏Leverage WSO2 ESB to provide a secured uniform interface and aggregation.

Demo 2 Use Case

MySQL Database

Excel Data Sheet

*

Demo 2Aggregate data from multiple WSO2 DSS Data Services using WSO2 ESB

Excel Data Service

RDBMS Data Service

Excel File

MySQL DB

Data Services ProxyClient Service

● Aggregate Mediator● Clone Mediator● Header Mediator

Request

Response

Multiple Sources

Demo 3 Use Case

๏ A fund transfer between two accounts need to occur in a transacted manner.

๏ Leverage WSO2 DSS for distributed transaction using boxcarring feature.

๏ Leverage WSO2 ESB for service chaining.

*

Demo 3Boxcarring sample with WSO2 DSS and ESB

begin_boxcar

execution 1

execution 2

………

execution 10

end_boxcar

begin_boxcar

execution 1

execution 2

abort_boxcar

execution 10

end_boxcar

*

Demo 3 cont

Bank Data Service

Bank DB (Book Store’s)

Payment Services ProxyBook Store

Application

● Customer’s Account Number● Book Store’s Account Number● Payment Amount

Bank DB (Customer’s)

● Customer’s Account Number● Payment Amount

● Book Store’s Account Number● Payment Amount

begin_boxcar

end_boxcar

credit_bookstore

debit_customer

Service Chaining

Where to begin..

๏ WSO2 ESB Documentation ๏ https://docs.wso2.org/display/ESB481/WSO2+Enterprise+Service+Bus+

Documentation

๏ WSO2 DSS Documentation๏ https://docs.wso2.com/display/DSS322/WSO2+Data+Services+Server+D

ocumentation

Further Reading๏Master Data Management using WSO2 Platform ๏ http://wso2.com/library/webinars/2013/10/mdm-with-wso2-enterprise

-middleware-platform/

Contact us !