Data Virtualization & Information As A Service ( IaaS )

www.synerzip.comwww.synerzip.com

Data Virtualization&

Information As A Service (IaaS)

By Anil AllewarSenior Solutions Architect - Synerzip

1


About Me!!

Confidential 2

Anil Allewar

Senior Solutions Architect @ Synerzip

Technology Evangelist & speaker

Core interests: JEE, EAI, EII


• Use cases

Agenda

• What does it mean?

• Implementation Frameworks• Demo• Questions?

3

• Architecture explained


Why it makes sense?

Confidential 4


Use Cases

Data Warehouse

ETL

Financial Data

OLTP Data

ETL

3rd Party Data

Data Mart

ETL

Web Service 1

Web Service 2

Legacy Data

Custom Program

Excel files

Confidential 5


Traditional Data Integration

Enterprise Information System

ETL

Source System

Source System

ETL

Confidential 6

Business Applications


Problems with ETL

Confidential 7

More than 1 copy of data for staging

Intermediate data => Errors

Lead time to add new source

Domain knowledge for mapping

Batch Process => No real time data


Problems with DBMS consolidation

Confidential 8

Alternate approach => Single EIS (say RDBMS)

Extensive changes to existing apps

Might not satisfy everyone’s requiremets


• Use cases

Agenda



9



Data Virtualization & Federation

Confidential 10

Single API to access data

Only metadata stored at virtualization layer

Real time access without

copying/moving dataFederate data across hetero/homogenous

sources


Data Virtualization

Confidential 11


• Use cases

Agenda



12



Architecture

Confidential 13

UserApplication

Com

mon

Acc

ess

API

Connector 1

Connector 2

RUNTIME & QUERY ENGINE

VirtualDatabase

Translator 1

Translator 2


• Use cases

Agenda



14



Vendors• Commercial Products

– Composite Software• http://www.compositesw.com/data-virtualization/

– Denodo• http://www.denodo.com/en/product/overview.php?n=h

– IBM• http://www-03.ibm.com/software/products/en/ibminfofedeserv

– Informatica• http://www.informatica.com/us/data-virtualization/

– Red Hat• http://www.redhat.com/products/jbossenterprisemiddleware/data-virtualizati

on/

• Open Source– Jboss Teiid

• http://teiid.jboss.org/

Confidential 15

http://www.compositesw.com/data-virtualization/



http://www.denodo.com/en/product/overview.php?n=h



http://www-03.ibm.com/software/products/en/ibminfofedeserv



http://www.informatica.com/us/data-virtualization/

http://www.informatica.com/us/data-virtualization/

http://www.redhat.com/products/jbossenterprisemiddleware/data-virtualization/



http://teiid.jboss.org/

http://teiid.jboss.org/


Selected Platform – JBoss Teiid

Confidential 16

Open Source

Number of relational/NoSQL/ERP/CRM data stores

JEE standards

Add custom EIS support using JEE

components

Active & responsive community Synerzip contribution: Defect

discovery, root cause analysis, feature verification


Teiid Components• Virtual Database

– container for components used to integrate data from multiple data sources

• Source Models– structure and characteristics of physical data sources

• View Models– structure and characteristics of abstract structures you want to expose

to your applications

• Teiid Designer– Eclipse based UI to dynamically discover data

source objects and apply data federation– Generate virtual database from 1 or more

sources

Confidential 17


Teiid Components• Translator

– Provides abstraction later between Teiid Query Engine and source system

– Convert Teiid SQL commands to source specific execution commands

– Convert result data from source system to Teiid specific format

• Resource Adapter– Provides connectivity to the physical data

source– Integration provided through Java Connector

Architecture (JCA) APIConfidential 18


Teiid – Supported EIS• Amazon SimpleDB• Apache Accumulo• Apache SOLR• Cassandra• File• Google Spreadsheet• JPA• LDAP• Excel – as file• SalesForce

• JDBC– MS access, DB2, derby, excel-

odbc, greenplum, h2 , hive(for accessing Hadoop), oracle, teradata and most RDBMS

• MongoDB• Object• OData• OLAP• Web Services• SAP Netweaver Gateway

Confidential 19


Performance Characteristics• Access same data using Oracle and Teiid drivers

– Retrieval times comparable when accessing tables having no Blobs

Confidential 20

0 0 0 1 13693

8,652

505,005

505,007

720,850

859,457

1,003,798

1,103,9180

5,000

10,000

15,000

20,000

25,000No. of rows Vs Time: No Blobs

Oracle-JDBCTeiid-JDBC

No. of rows

ms


Performance Characteristics

– Teiid slower when accessing Blob data• Can be tuned

Confidential 21

0 0 2 42 21,804 32,531 185,4540

5,000

10,000

15,000

20,000

25,000

30,000

No. of rows Vs Time: Blobs

Oracle-JDBCTeiid-JDBCm

s

No. of rows


• Use cases

Agenda



22



Demo

Confidential 23

JDBC Client JD

BC API

RDBMS Resource Adapter

MongoDB Resource Adapter

TEIID RUNTIME & QUERY ENGINE

Federated VDB

mySQL Translator

MongoDB Translator mySQL


Demo-Steps• Pre-requisites

– mySQL server 5.5+ installed– MongoDB 2.4.x+ installed

• Steps– Load the mySql and MongoDB database with sample

data– Setup environment – JBoss, Eclipse– Create Teiid project in Eclipse using Teiid designer

• Import source model using JDBC• Create the virtual model and federate data from the

source model• Create a virtual database (VDB) and deploy to JBoss

– Access data using JDBC client or through browser using OData

Confidential 24


Demo – Scenario

Confidential 25

FederatedData


Demo – Connection Profile

Confidential 26


Demo – Source Model

Confidential 27


Demo - Source Model Generation

Confidential 28


Demo – Map Source To View

Confidential 29


Demo - Association

Confidential 30


Demo – Data Federation

Confidential 31


Demo – Source Code

• Source code–https://github.com/Synerzip/JBoss-Teiid

–Contains• Configuration files• Instructions• “How-to” videos• VDBs, source models and view models

Confidential 32

https://github.com/Synerzip/JBoss-Teiid

https://github.com/Synerzip/JBoss-Teiid


Conclusion

• Data Virtualization and Federation is a rapidly emerging technology that solves traditional BI/ETL problems.

• It provides lower time to market, distributes data across the enterprise as a service and provides real time access to enterprise data.

Confidential 33


• Use cases

Agenda



34



Contact Me

• [email protected]

Confidential 35

mailto:[email protected]

www.synerzip.comwww.synerzip.com• 3

684

Questions?

www.synerzip.comHemant Elhence

[email protected]


Synerzip in a Nutshell1. Software product development partner for small/mid-sized technology companies

• Exclusive focus on small/mid-sized technology companies, typically venture-backed companies in growth phase• By definition, all Synerzip work is the IP of its respective clients• Deep experience in full SDLC – design, dev, QA/testing, deployment

2. Dedicated team of high caliber software professionals for each client• Seamlessly extends client’s local team, offering full transparency• Stable teams with very low turn-over• NOT just “staff augmentation”, but provide full mgmt support

3. Actually reduces risk of development/delivery• Experienced team - uses appropriate level of engineering discipline• Practices Agile development – responsive, yet disciplined

4. Reduces cost – dual-shore team, 50% cost advantage5. Offers long term flexibility – allows (facilitates) taking offshore team captive – aka “BOT” option


Our Clients

http://www.itko.com/

http://www.yagnaiq.com/index.htm

http://www.clearorbit.com/index.php?sess_id=b0a39bdaf17f0f7c5cd34f8cc3d49d10

http://www.zafesoft.com/index.asp

http://www.hallsettlementsystems.com/Default.aspx

http://www.hallsettlementsystems.com/Default.aspx

http://www.fd2s.com/home.php

http://www.repliqa.com/

http://www.broadlane.com/index.html

http://www.neofirma.com/Default.aspx

http://www.sevayatra.com/index.html

http://www.fuelquest.com/

http://www.hartic.com/index.php

http://www.coretrace.com/

http://www.healthcarepayment.com/home.cfm

http://www.sourcecorp.com/Home.aspx


Call Us for a Free Consultation!

Hemant Elhence [email protected]

469.322.0349

Thanks!

Documents

Data Virtualization & Information As A Service ( IaaS )