Master Guide for SAP HANA Smart Data Integration and SAP ... · PDF file1 Getting Started SAP HANA smart data integration and SAP HANA smart data quality provide a set of tools and

PUBLIC

SAP HANA Platform SPS 12Document Version: 1.0 – 2016-05-11

Master Guide for SAP HANA Smart Data Integration and SAP HANA Smart Data Quality

Content

1 Getting Started. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.1 Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 About This Document. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2 Use Cases. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

3 Deployment Options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63.1 Deployment in High Availability Scenarios. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

4 Architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

5 Components. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

6 Summary of Workflow and Tasks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14

2P U B L I C© 2016 SAP SE or an SAP affiliate company. All rights reserved.

Master Guide for SAP HANA Smart Data Integration and SAP HANA Smart Data QualityContent

1 Getting Started

SAP HANA smart data integration and SAP HANA smart data quality provide a set of tools and processess that let you connect to any source, provision and cleanse data, and load to SAP HANA on premise or in the cloud.

1.1 Overview

The SAP HANA smart data integration and SAP HANA smart data quality options provide tools to access source data, and provision, replicate, and transform that data in SAP HANA on-premise or in the cloud.

The smart data integration and smart data quality options let you enhance, cleanse, and transform data to make it more accurate and useful. These options let you efficiently connect to any source to provision and cleanse data for loading into SAP HANA on-premise or in the cloud, and for supported systems, write back to the original source.

Capabilities include:

● A simplified landscape, that is, one environment in which to provision and consume data.● Access to more data formats including an open framework for new data sources.● In-memory performance, which means increased speed and decreased latency.

Feature area Description

SAP HANA smart data integration

Real-time, high-speed data provisioning, bulk data movement, and federation. Provides built-in adapters plus an SDK so you can build your own.

Includes the following features and tools:

● Replication Editor in the SAP HANA Web-based Development Workbench, which lets you set up batch or real-time data replication scenarios in an easy-to-use web application

● Transformations presented as nodes in the application function modeler delivered with SAP HANA studio and SAP HANA Web-based Development Workbench, which lets you set up batch or real-time data transformation scenarios

● Data Provisioning Agent, a lightweight component that hosts data provisioning adapters, enabling data federation, replication, and transformation scenarios for on-premise or in-cloud deployments

● Data Provisioning adapters for connectivity to remote sources● Adapter SDK to create custom adapters● SAP HANA Cockpit integration for monitoring Data Provisioning Agents, remote subscrip

tions, and data loads

SAP HANA smart data quality

Real-time, high-performance data cleansing, address cleansing, and geospatial data enrichment. Provides an intuitive interface to define data transformation flowgraphs in SAP HANA Web-based Development Workbench and SAP HANA studio.

Master Guide for SAP HANA Smart Data Integration and SAP HANA Smart Data QualityGetting Started

P U B L I C© 2016 SAP SE or an SAP affiliate company. All rights reserved. 3

1.2 About This Document

This Master Guide is the central starting point for the technical implementation of SAP HANA smart data integration and SAP HANA smart data quality.

This Master Guide includes the following information:

● Overview● Architecture● Software components● Deployment scenarios


Master Guide for SAP HANA Smart Data Integration and SAP HANA Smart Data QualityGetting Started

2 Use Cases

You can use SAP HANA smart data integration and SAP HANA smart data quality to replicate or transform datea from remote sources.

Use case Description

Replication You can replicate data (batch and real time) into an SAP HANA system (on premise or in the cloud).

Transformation You can transform data (batch and real time) on the way to the SAP HANA system (on premise or in the cloud). An example of transforming data is cleansing data using smart data quality.

Master Guide for SAP HANA Smart Data Integration and SAP HANA Smart Data QualityUse Cases


3 Deployment Options

Common deployment options for SAP HANA systems, Data Provisioning Agents, and source systems are described.

There are two common deployment landscapes that we recommend:

Landscape Description

Distributed landscape ● System 1: SAP HANA Server● System 2: Data Provisioning Agent● System 3: Source system

Combined landscape ● System 1: SAP HANA Server● System 2: Data Provisioning Agent and the source system

SAP HANA on premise vs. SAP HANA in the cloud

Using SAP HANA on premise or in the cloud is a choice of deployment. Here are some things to keep in mind when deciding which deployment to use. If your deployment includes SAP HANA in the cloud and a firewall between SAP HANA and the Data Provisioning Agent:

● The Data Provisioning Proxy must be deployed. This is done by downloading and deploying the HANA_IM_DP delivery unit.

● The Data Provisioning Agent must be configured to communicate with SAP HANA using HTTP. This is done using Data Provisioning Agent Configuration tool.

Other deployment considerations

When planning your deployment, keep the following in mind:

● You may not have one Data Provisioning Agent registered in multiple SAP HANA instances.● You may have multiple instances of the Data Provisioning Agent installed on multiple machines. For example,

a developer may want to have a Data Provisioning Agent installed on their computer to work on a custom adapter.


Master Guide for SAP HANA Smart Data Integration and SAP HANA Smart Data QualityDeployment Options

3.1 Deployment in High Availability Scenarios

In addition to installing SAP HANA in a multiple-host configuration, you can use agent grouping to provide automatic failover and load balancing for SAP HANA smart data integrationand SAP HANA smart data quality functionality in your landscape.

Auto-failover for the Data Provisioning Server

In a multiple-host SAP HANA system, the Data Provisioning Server runs only in the active worker host. If the active worker host fails, the Data Provisioning Server is automatically started in the standby host when it takes over, and any active replication tasks are resumed.

NoteLoad-balancing is not supported by the Data Provisioning Server.

For more information about installing SAP HANA in a multiple-host configuration, see the SAP HANA Server Installation and Update Guide.

Auto-failover for the Data Provisioning Agent

Agent grouping provides automatic failover for connectivity to data sources accessed through Data Provisioning Adapters.

When an agent that is part of a group is inaccessible for a time longer than the configured heart beat time limit, the Data Provisioning Server chooses a new active agent within the group and resumes replication for any remote subscriptions active on the original agent.

Any remote subscriptions between the Queue and Distribute states at the time the agent became unavailable are logged as exceptions, and must be manually reset with new Queue and Distribute commands.

Load balancing for the Data Provisioning Agent

Agent grouping also provides load balancing between individual agent hosts by using a round robin policy.

For example, with multiple agents in the group and a remote source configured on the group, each smart data access request to the remote source is load balanced by rotating through the agents in the group.

For changed-data capture and real-time operations, an agent in the group is selected when a QUEUE command is executed on a remote subscription. Any following CDC requests are directed to the agent selected during the QUEUE command.

For complete information about configuring agent groups, see the Administration Guide for SAP HANA Smart Data Integration and SAP HANA Smart Data Quality.



Related Information

SAP HANA Server Installation and Update Guide (HTML)SAP HANA Server Installation and Update Guide (PDF)Administration Guide for SAP HANA Smart Data Integration and SAP HANA Smart Data QualityAdministration Guide for SAP HANA Smart Data Integration and SAP HANA Smart Data Quality



http://help.sap.com/saphelp_hanaplatform/helpdata/en/aa/c4bed2390045f59dd8fdeab3ffafb3/frameset.htm

http://help.sap.com/hana/SAP_HANA_Server_Installation_Guide_en.pdf

http://help.sap.com/saphelp_hana_options_eim/helpdata/en/06/756fc260e548ed9f64c439b8649ced/frameset.htm

http://help.sap.com/download/multimedia/hana_options_eim/SAP_HANA_EIM_Administration_Guide_en.pdf

4 Architecture

These diagrams represent common deployment architectures for using smart data integration and smart data quality with SAP HANA.

In all deployments, the basic components are the same. However, the connections between the components may differ depending on whether SAP HANA is deployed on premise, in the cloud, or behind a firewall.

Figure 1: SAP HANA deployed on premise

Master Guide for SAP HANA Smart Data Integration and SAP HANA Smart Data QualityArchitecture


Figure 2: SAP HANA deployed in the cloud or behind a firewall

The following tables explain the diagram and the network connections in more detail.

Outbound Connections

Client Protocol and additional information Default Port

Data Provisioning Agent When SAP HANA is deployed on premise, the Data Provisioning Server within SAP HANA connects to the agent using the TCP/IP protocol.

To manage the listening port used by the agent, edit the adapter framework preferences with the Data Provisioning Agent Configuration tool.

5050

Sources

Examples: Data Provisioning Adapters

The connections to external data sources depend on the type of adapter used to access the source.

C++ adapters running in the Data Provisioning Server connect to the source using a source-defined protocol.

Java adapters deployed on the Data Provisioning Agent connect to the source using a source-defined protocol.

Varies by source



Inbound Connections

Client Protocol and additional information Default Port

Data Provisioning Agent When SAP HANA is deployed in the cloud or behind a firewall, the Data Provisioning Agent connects to the SAP HANA XS engine using the HTTP/S protocol.

NoteWhen the agent connects to SAP HANA in the cloud over HTTP/S, data is automatically gzip compressed to minimize the required network bandwidth.

For information about configuring the port used by the SAP HANA XS engine, see the SAP HANA Administration Guide.

80xx

43xx

Related Information

SAP HANA Administration Guide (HTML)SAP HANA Administration Guide (PDF)



http://help.sap.com/saphelp_hanaplatform/helpdata/en/00/0ca1e3486640ef8b884cdf1a050fbb/frameset.htm

http://help.sap.com/hana/SAP_HANA_Administration_Guide_en.pdf

5 Components

SAP HANA smart data integration and SAP HANA smart data quality include a number of components that you need to install, deploy, and configure.

Component Description

Data Provisioning Server The Data Provisioning Server is a native SAP HANA process. It is built as an index server variant, runs in the SAP HANA cluster, and is managed and monitored just like other SAP HANA services. It provides out-of-the-box native connectivity for many sources and connectivity to the Data Provisioning Agent.

The Data Provisioning Server is installed with, but must be enabled in, the SAP HANA Server.

Data Provisioning Agent The Data Provisioning Agent is a container running outside the SAP HANA environment, but it is managed by the Data Provisioning Server. It provides connectivity for all those sources where the driver cannot run inside the Data Provisioning Server. Through the Data Provisioning Agent, the preinstalled Data Provisioning Adapters communicate with Data Provisioning Server for connectivity, metadata browsing, and data access. The Data Provisioning Agent also hosts custom adapters created using the Adapter SDK.

The Data Provisioning Agent is installed separately from SAP HANA server or client.

HANA_IM_DP delivery unit The HANA_IM_DP delivery unit bundles monitoring and administration capabilities and the Data Provisioning Proxy for when connecting to SAP HANA in the cloud.

The delivery unit includes the Data Provisioning adminisration application, the Data Provisioning Proxy, and the Data Provisioning monitor.

Data Provisioning admin application

The Data Provisioning administration application is an XS application that manages the administration functions of the Data Provisioning Agent with SAP HANA in the cloud.

This component is delivered via the HANA_IM_DP delivery unit.

Data Provisioning Proxy The Data Provisioning Proxy is an XS application that acts as a proxy to provide communication between the Data Provisioning Agent and Data Provisioning Server when SAP HANA runs in the cloud. When SAP HANA is in the cloud, the agent uses HTTP(S) to connect to Data Provisioning Proxy in the XS Engine, which eliminates the need to open additional ports in corporate IT firewalls.

This component is delivered via the HANA_IM_DP delivery unit.

Data Provisioning monitor The Data Provisioning monitor is a browser-based interface that lets you monitor agents, tasks, and remote subscriptions created in the SAP HANA system. You can view the monitors by directly entering the URL of each monitor into a web browser or by accessing the Data Provisioning tiles in the SAP HANA cockpit, a web-based launchpad that is installed with SAP HANA Server.

Enable Data Provisioning monitoring functionality (for agents, data loads, and remote subscriptions) by creating the statistics tables and deploying the HANA_IM_DP delivery unit.

SAP HANA Web-based Development Workbench Replication Editor

The SAP HANA Web-based Development Workbench, which includes the Replication Editor to set up replication tasks, is installed with SAP HANA Server.


Master Guide for SAP HANA Smart Data Integration and SAP HANA Smart Data QualityComponents

Component Description

SAP HANA Web-based Development Workbench Flowgraph Editor

The SAP HANA Web-based Development Workbench Flowgraph Editor provides an interface to create data provisioning and data quality transformation flowgraphs.

Application function modeler The application function modeler provides an interface to create data provisioning and data quality transformation flowgraphs.

Application function modeler is installed with SAP HANA studio.

Master Guide for SAP HANA Smart Data Integration and SAP HANA Smart Data QualityComponents


6 Summary of Workflow and Tasks

Implementation and use of SAP HANA smart data integration and SAP HANA smart data quality require the installation, enablement, and deployment of various components across the SAP HANA landscape. Some of these components are required and some are optional, depending on how you want to use these options.

The following table provides you with an overview of the various tasks that administrators and developers need to perform to enable smart data integration and smart data quality.

Task Sub-task Performed by Tool to use Where to find more information

Notes

Enable the Data Provisioning Server

Administrator SAP HANA studio Administration Guide for SAP HANA Smart Data Integration and SAP HANA Smart Data Quality

The DP Server is disabled by default.

Enable the Script Server


The Script Server is disabled by default. Enable it only if you plan on using smart data quality.

Install smart data quality Cleanse and Geocode directories

Administrator Administration Guide for SAP HANA Smart Data Integration and SAP HANA Smart Data Quality

Add user roles and privileges to perform various activities


Deploy the HANA_IM_DP delivery unit

SAP HANA studio Deploy SAP HANA Content

Install the Data Provisioning Agent

Administrator DP Agent Installer Administration Guide for SAP HANA Smart Data Integration and SAP HANA Smart Data Quality

Install on Windows or Linux


Master Guide for SAP HANA Smart Data Integration and SAP HANA Smart Data QualitySummary of Workflow and Tasks

http://help.sap.com/saphelp_hanaplatform/helpdata/en/69/0fc250ac574b7d98a4fe35ff6ded50/frameset.htm

http://help.sap.com/saphelp_hanaplatform/helpdata/en/69/0fc250ac574b7d98a4fe35ff6ded50/frameset.htm


Notes

Register Data Provisioning Agent with SAP HANA

Administrator DP Agent Configuration tool

Administration Guide for SAP HANA Smart Data Integration and SAP HANA Smart Data Quality

Register Data Provisioning adapters with SAP HANA

Administrator DP Agent Configuration tool


Create custom adapters

Install Data Provisioning Framework

Adapter developer SAP HANA studio Adapter SDK Guide for SAP HANA Smart Data Integration and SAP HANA Smart Data Quality

Import Launch Configuration


Install PDE in order to export developed plugin


Export plug-in project as JAR file


Deploy custom adapters (.jar) to Data Provisioning Agent

Administrator Data Provisioning Agent Configuration tool


Register custom adapter with SAP HANA

Administrator Data Provisioning Agent Configuration tool





Notes

Create remote sources


Configure data movement

Replication, using SAP HANA Web-based Development Workbench

ETL Developer / Application Consultant

SAP HANA Web-based Development Workbench

Configuration Guide for SAP HANA Smart Data Integration and SAP HANA Smart Data Quality

Transformation, using SAP HANA studio

ETL Developer / Application Consultant

SAP HANA studio (application function modeler)


Begin replication / transformation by invoking a stored procedure generated during activation

Administrator / ETL developer

SAP HANA studio or SAP HANA Development Workbench Editor



Monitor Data Provisioning agents, tasks and remote subscriptions

Administrator SAP HANA Cockpit (Web and SAP HANA studio)


Process remote subscription exceptions

Administrator SAP HANA studio / SAP HANA Development Workbench Catalog


Related Information

Components [page 12]



Important Disclaimers and Legal Information

Coding SamplesAny software coding and/or code lines / strings ("Code") included in this documentation are only examples and are not intended to be used in a productive system environment. The Code is only intended to better explain and visualize the syntax and phrasing rules of certain coding. SAP does not warrant the correctness and completeness of the Code given herein, and SAP shall not be liable for errors or damages caused by the usage of the Code, unless damages were caused by SAP intentionally or by SAP's gross negligence.

AccessibilityThe information contained in the SAP documentation represents SAP's current view of accessibility criteria as of the date of publication; it is in no way intended to be a binding guideline on how to ensure accessibility of software products. SAP in particular disclaims any liability in relation to this document. This disclaimer, however, does not apply in cases of wilful misconduct or gross negligence of SAP. Furthermore, this document does not result in any direct or indirect contractual obligations of SAP.

Gender-Neutral LanguageAs far as possible, SAP documentation is gender neutral. Depending on the context, the reader is addressed directly with "you", or a gender-neutral noun (such as "sales person" or "working days") is used. If when referring to members of both sexes, however, the third-person singular cannot be avoided or a gender-neutral noun does not exist, SAP reserves the right to use the masculine form of the noun and pronoun. This is to ensure that the documentation remains comprehensible.

Internet HyperlinksThe SAP documentation may contain hyperlinks to the Internet. These hyperlinks are intended to serve as a hint about where to find related information. SAP does not warrant the availability and correctness of this related information or the ability of this information to serve a particular purpose. SAP shall not be liable for any damages caused by the use of related information unless damages have been caused by SAP's gross negligence or willful misconduct. All links are categorized for transparency (see: http://help.sap.com/disclaimer).

Master Guide for SAP HANA Smart Data Integration and SAP HANA Smart Data QualityImportant Disclaimers and Legal Information


http://help.sap.com/disclaimer/

go.sap.com/registration/contact.html

© 2016 SAP SE or an SAP affiliate company. All rights reserved.No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP SE or an SAP affiliate company. The information contained herein may be changed without prior notice.Some software products marketed by SAP SE and its distributors contain proprietary software components of other software vendors. National product specifications may vary.These materials are provided by SAP SE or an SAP affiliate company for informational purposes only, without representation or warranty of any kind, and SAP or its affiliated companies shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP or SAP affiliate company products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty.SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE (or an SAP affiliate company) in Germany and other countries. All other product and service names mentioned are the trademarks of their respective companies.Please see http://www.sap.com/corporate-en/legal/copyright/index.epx for additional trademark information and notices.

https://go.sap.com/registration/contact.html

https://go.sap.com/registration/contact.html

http://www.sap.com/corporate-en/legal/copyright/index.epx

http://www.sap.com/corporate-en/legal/copyright/index.epx

Documents

Master Guide for SAP HANA Smart Data Integration and SAP ... · PDF file1 Getting Started SAP HANA smart data integration and SAP HANA smart data quality provide a set of tools and