Upload
rohit-sharma
View
58
Download
7
Tags:
Embed Size (px)
DESCRIPTION
SAP HANA
Citation preview
SAP HANA & XS Engine
Overview
Internal
© 2013 SAP AG. All rights reserved. 2 Ramp-Up Knowledge Transfer Customer
Outline
HANA
• Introduction
• Architecture
HANA Studio
• Introduction
Modeled Views
• Attribute, Analytic, Calculation
XS Engine
• Introduction
• Repository
• Architecture
XS Engine Applications
• Application Containers
• Regi and Workspaces
• Deployment
Resources
© 2013 SAP AG. All rights reserved. 3 Ramp-Up Knowledge Transfer Customer
Introduction to HANA
“SAP HANA allows for the processing of massive quantities of real-time data in main memory to
provide immediate results”
How?
• Multi-core architectures and algorithms allowing for high process parallelization
• In-memory design
• Efficient data storage
(column store, compression, insert only deltas, no more materialized aggregates, …)
© 2013 SAP AG. All rights reserved. 4 Ramp-Up Knowledge Transfer Customer
Multi-Core and In-Memory
Traditional Databases
• Bottlenecked by disk I/O due to limited RAM
HANA
• With multi-core CPUs and RAM no longer being
limited, bottleneck shifts to data transfer
between CPU cache and main memory
© 2013 SAP AG. All rights reserved. 5 Ramp-Up Knowledge Transfer Customer
In-Memory Design
Traditional Databases
• Separate data management and applications into a DB and application layer
• Force data to travel from DB to application layer before it can be analyzed or modified
HANA
• Moves data intensive application logic to the DB layer
• Extends data processing capabilities of the DB
• Avoids layer boundary bottleneck
• Embedding is done via extensions to SQL (ex. SQLScript)
© 2013 SAP AG. All rights reserved. 6 Ramp-Up Knowledge Transfer Customer
Efficient Data Storage
Traditional Databases
• Even with data in-memory, CPU spends ½ of execution time in stalls (waiting for data to be
loaded from RAM to CPU cache) or cache misses
HANA
• Cache-aware memory organization – minimizes CPU stalls and cache misses
• Maximize spatial locality of data through the use of column stores
© 2013 SAP AG. All rights reserved. 7 Ramp-Up Knowledge Transfer Customer
Column Store Advantages
• Higher data compression rates when redundancy is high
Ex. run-length encoding, cluster coding, dictionary coding
• High performance for column operations
Ex. searching/aggregations implemented as loops over arrays stored contiguously
• Elimination of additional indexes
Storing in columns is like getting a built-in index for each column
• Parallelization
Column store is already vertically partitioned, so operations on diff columns can be easily
processed in parallel
• Elimination of materialized aggregates
In-memory column stores allow calculating aggregates on-the-fly, which simplifies data
model, app logic, and concurrency
© 2013 SAP AG. All rights reserved. 8 Ramp-Up Knowledge Transfer Customer
Column Vs Row Store
Use column store when:
Use row store when:
• Calculations are typically executed on single or a
few columns only.
• The table is searched based on values of a few
columns.
• The table has a large number of columns.
• The table has a large number of rows and
columnar operations are required (aggregate, scan,
etc.).
• High compression rates can be achieved because
the majority of the columns contain only few distinct
values (compared to number of rows).
• The application needs to only process a single
record at one time (many selects and/or updates of
single records).
• The application typically needs to access a
complete record (or row).
• The columns contain mainly distinct values so that
the compression rate would be low.
• Neither aggregations nor fast searching are
required.
• The table has a small number of rows (e. g.
configuration tables).
Deciding whether to use column or row stores depends on the situation…
© 2013 SAP AG. All rights reserved. 9 Ramp-Up Knowledge Transfer Customer
HANA Architecture
Developed mainly in C++ and designed to run on Linux servers
© 2013 SAP AG. All rights reserved. 10 Ramp-Up Knowledge Transfer Customer
HANA Architecture
Index server
• Contains data stores and engines for processing data
Preprocessor server
• Used by index server to analyze text data
Name server
• Contains info about topology of HANA system
• In distributed setup, knows where components are running and what data is on which
server
Statistics server
• Collects info about status, performance, and resource usage from all components
XS engine
• Maps persistence model in DB to a consumption model exposed to clients via HTTP
© 2013 SAP AG. All rights reserved. 11 Ramp-Up Knowledge Transfer Customer
Distributed HANA
HANA setups are usually distributed over multiple hosts for scalability and availability
HANA system is identified by a system id (SID)
HANA instances are identified by instance ids (IID)
XS engine and statistics server only exist once per system
© 2013 SAP AG. All rights reserved. 12 Ramp-Up Knowledge Transfer Customer
Index Server Architecture
© 2013 SAP AG. All rights reserved. 13 Ramp-Up Knowledge Transfer Customer
Index Server Architecture
Authentication
• Invoked when new connection established
• Users can be authenticated by HANA or an external provider
Authorization Manager
• Invoked by other DB components to check whether user has required privileges to execute
requested operations
Transaction Manager
• New sessions are assigned to a new transaction
• In HANA, each SQL statement is processed in the context of a transaction
• The transaction manager coordinates and tracks transactions
Request Processing and Execution Control
• Analyzes, executes, and forwards client requests
Planning engine
• Allows financial planning apps to execute basic planning ops in the DB layer
• Ex. create new version of dataset as a copy of existing one while applying filters and
transformations
© 2013 SAP AG. All rights reserved. 14 Ramp-Up Knowledge Transfer Customer
Index Server Architecture
Calc Engine
• Implements features like SQLScript, MDX and planning operations
Metadata Manager
• Manages metadata
• Ex. definitions of tables, columns, views, indexes, SQLScript functions, object stores
Relational Stores
• Subsystems of HANA DB which include in-memory storage and components that manage that storage
Persistence Layer
• Responsible for durability and atomicity of transactions (ex. DB restore on restart, log storage)
Repository
• Provides management and persistence of app specific objects
Ex. modeled views, stored procedures, web content exposed via XS engine
• Provides namespaces, versioning, and content import/export support
• Objects are stored with associated metadata (ex. type and version)
© 2013 SAP AG. All rights reserved. 15 Ramp-Up Knowledge Transfer Customer
HANA Studio
Both the development environment and admin tool for HANA
• Dev: create modeled views or stored procedures
• Admin: start/stop services, manage system
• Developed in Java and based on Eclipse platform
Directory structure when connected to HANA
Catalog
• HANA‘s data dictionary
• Contains all data structures, tables, and data
Contents
• Design-time repository
• Hold all models created with modeler
Note:
• “Contents“ are physically stored in DB tables under “Catalog“
• Contents node just provides a different view on the same
physical data
© 2013 SAP AG. All rights reserved. 16 Ramp-Up Knowledge Transfer Customer
Modeled Views
• Created using modeling tools in HANA studio
• Design-time artifacts stored in the repository
• For runtime execution, column store views are generated from modeled views
Advantages:
• Additional metadata can be provided (ex. attribute descriptions)
• Easier to use modeling tools compared to writing complex DDL
• Can be versioned in repository
• Easily consumable by customers
Modeled Object (design-time) Database View (run-time)
Attribute View Column view of type JOIN
Analytic View Column view of type OLAP or column view of type
CALCULATION on top of OLAP view
Calculation View Column view of type CALCULATION
© 2013 SAP AG. All rights reserved. 17 Ramp-Up Knowledge Transfer Customer
Attribute Views
Define joins between tables and/or projections on a table
• Ex. joining multiple tables together to create a single dimension table when using star schemas; this can
be joined to a fact table via analytic view to give meaning to the data
Join employees
to org units, then
join result to
sales transaction
via analytic view
© 2013 SAP AG. All rights reserved. 18 Ramp-Up Knowledge Transfer Customer
Analytic Views
• An OLAP cube (n-dimensional array of data)
• Set of physical tables interconnected in a star
schema (dimension tables around a fact table)
• Fact table provides info the user is actually
interested in (aka metrics, measures, or key figures)
• Dimension tables provide categories (classes, or
attributes) by which facts can be grouped and are
modeled as attribute views to enable reuse in
multiple analytic views
• Fact table provides content of OLAP cube
ie. “what” to group
• Dimension tables specify cube’s dimensions
ie. “by what” to group
• Cardinality of fact to dimension is N:1
© 2013 SAP AG. All rights reserved. 19 Ramp-Up Knowledge Transfer Customer
Calculation Views
• Used to provide composites of other views (join or union of multiple data flows) or invoke built-in or
generic SQL functions
• Used the same way as analytic views
• Able to perform complex calculations
Graphical
• Modeled using graphical modeling features of HANA modeler
Scripted
• Created as sequence of SQL statements
© 2013 SAP AG. All rights reserved. 20 Ramp-Up Knowledge Transfer Customer
Application Programming Options
Modeled Views (Calculation, Analytic, Attribute) & SQL Views
• Views are for used for defining application specific data models, for defining data flows and to structure
and reuse queries.
• Modeled analytic views and attribute views can be created with HANA studio without programming.
• Complex data flows can be modeled as calculation views using SQLScript or a visual modeler.
SQLScript
• The rich stored procedure language of the HANA.
• Procedures may contain SQL statements and call other procedures.
• Is used to write procedural orchestration logic and to define complex data flows.
R
• HANA also supports stored procedures written in “R” for statistical computing.
© 2013 SAP AG. All rights reserved. 21 Ramp-Up Knowledge Transfer Customer
Application Programming Options
L
• Dedicated imperative programming language for implementing stored procedures and functions (known
as scalar functions) that can be used in an SQL query.
• Stored procedures in L are typically used to implement operators that are called from SQLScript
procedures.
• L code is compiled to native code, and thus very efficient.
C++
• Performance critical code can be developed in C++ as part of an application function library.
• Using C++ is restricted to specialists who cooperate closely with HANA developers.
FOX
• The planning engine supports planning operations written in the FOX formula language.
© 2013 SAP AG. All rights reserved. 22 Ramp-Up Knowledge Transfer Customer
XS Engine Introduction
• Provides access to HANA DB by
transforming persistence model stored
in the DB into consumption model for
clients exposed via HTTP
• Uses SAP ICM for HTTP server
• Hosts system services that are part of
the HANA DB (ex. Search service, built-
in web server that provides access to
static content in the repository)
• Optional component of HANA
• Does not store data
© 2013 SAP AG. All rights reserved. 23 Ramp-Up Knowledge Transfer Customer
Application Specific Code: Where?
Both the index server and XS engine can store application specific code. So what’s the difference?
Index server
• All data-intensive calculations that need to be done close to data in index server (using
SQLScript, modeled views, L procedures, C++).
XS engine
• Just provides consumption model for client apps (non-performance-critical tasks, invokes views
and stored procedures that are executed in the index server).
© 2013 SAP AG. All rights reserved. 24 Ramp-Up Knowledge Transfer Customer
Index Server Vs XS Engine
XS engine is similar to
an index server without
data stores and with
an XS layer on top
When XS layer needs
to access data:
1. Opens a local DB
connection and
sends an SQL
statement to the
local SQL proc.
2. SQL proc
determines
location of data
and delegates
execution to index
server that has the
data.
© 2013 SAP AG. All rights reserved. 25 Ramp-Up Knowledge Transfer Customer
Repository
• Repository tables and content reside on the index server like all other tables
• Used for storing and versioning source code of apps for the XS engine
• Provides lifecycle management
• Repo objects uniquely identified by a combination of its package name, object name, and object type
Advantages:
• Save / edit intermediate development results without affecting runtime objects
• Developers work with abstract repo objects and don’t need to worry about corresponding representation
in the HANA DB (ex. Writing DDL statements)
• Unified local dependency and version management (no need for external system, and helps in delivering
content to customers and among different HANA instances)
• Allows multiple users to work on multiple inactive versions of the same object in parallel
© 2013 SAP AG. All rights reserved. 26 Ramp-Up Knowledge Transfer Customer
Parallel Development in Repository
• Repository uses lock-free approach
• Others must merge with the current active version before activation
• Repository checks inactive “based on” version equals current active version before allowing activation
© 2013 SAP AG. All rights reserved. 27 Ramp-Up Knowledge Transfer Customer
XS Engine Architecture
© 2013 SAP AG. All rights reserved. 28 Ramp-Up Knowledge Transfer Customer
XS Engine Architecture
1. Request handler uses URL mapper to determine the component that processes the request.
If there is a declarative consumption model in the repo for the requested resource, the OData
service is invoked. Otherwise, URL mapper determines which app in which container is
responsible for executing the request.
2. Request handler invokes the session manager (which also authenticates session)
3. Request handler checks with authorization manager to see if user is authorized to execute the
request
4. Request handler forwards request to OData service or responsible app container (app
container loads the app code from the repo)
5. Container triggers execution of the app code (app gets request object as input, processes
request and populates response object with output)
© 2013 SAP AG. All rights reserved. 29 Ramp-Up Knowledge Transfer Customer
XS Engine Application Development
• Guideline for applications is to expose a consumption model via REST services based on OData
• The XS engine currently contains an OData service that supports a declarative mapping between
the persistence and consumption models
• When additional logic is required, application specific code needs to be written for processing client
requests
• The XS layer contains multiple application containers (executing environments)
© 2013 SAP AG. All rights reserved. 30 Ramp-Up Knowledge Transfer Customer
XS Engine Application Containers
Currently, there are two major application containers:
C++
• Not used for general app development (special purposes only)
• Apps have same lifecycle as HANA DB and must be part of DB build
• Currently used for system apps
• Ex. web server for access to web resources stored in the repo
JS
• Based on Mozilla JS engine
• Currently used for pilot projects
• Due to lack of abstraction and encapsulation mechanisms, not recommended for complex
applications
??? (currently codenamed SLang)
• Additional app container is planned for the future that will have the benefits of JS, but also handle
existing JS shortcomings
© 2013 SAP AG. All rights reserved. 31 Ramp-Up Knowledge Transfer Customer
XS Engine Application Development
Need a way to access repository in XS engine
• Regi is the currently recommended repo access tool
• Used to load and activate repo objects which includes both DB objects (ex. Tables) and app resources
(ex. xs*, html)
• Helps to track object state within the repo
• Provides workspaces on local dev machines for XS application development
© 2013 SAP AG. All rights reserved. 32 Ramp-Up Knowledge Transfer Customer
Regi Workspace Structure
Below is an example of what a regi workspace looks like on your computer
packageTopLevel/
…
packageBottomLevel/
db/ Persistence
*.schema
*_TBL.sequence
*_TBL.table
logic/ Backend logic
*.xsjs
*.xsjslib
odata/ OData
*.xsodata
*.view
ui/ UI related
*.png
*.css
index.html
*.xsa
*.xscfgm
*.xscfgd
*_ext.xscfgd
© 2013 SAP AG. All rights reserved. 33 Ramp-Up Knowledge Transfer Customer
XS Engine Application Files
Based on the workspace example:
Root
*.xsa
• Application configuration (JSON)
*.xscfgm
• Configuration model
• Describes config variable types
• Like a template
*.xscfgd
• Configuration data
• Specifies config values
*_ext.xscfgd
• Configuration data
• Overrides / extends configuration from base *.xscfgd file
index.html
• Main file for application
Persistence
*.schema
• Defines database schema
*_TBL.sequence
• Defines a sequence (JSON)
*_TBL.table
• Defines a table (XML)
© 2013 SAP AG. All rights reserved. 34 Ramp-Up Knowledge Transfer Customer
XS Engine Application Files
Logic
*.xsjs
• XS JS
*.xsjslib
• XS JS library
• Contains functions that can be called from XS JS files
• Use ‘repo’ object of the XS JS API to include the library:
repo.import(<package name>, <object name>);
ex. repo.import(“sap.app.logic”, “config”);
• Call function from library with:
$.<package name>.<library>.<function>(<parameters>);
ex. val = $.sap.app.logic.config.getConfig(configParam);
OData
*.view
• Defines a view (JSON)
• Necessary if you wish to get data from multiple tables
*.xsodata
• Defines OData table (XML)
© 2013 SAP AG. All rights reserved. 35 Ramp-Up Knowledge Transfer Customer
Inactive Vs Active Objects
Inactive Objects
• When committed to the repo, only visible to user who created it
• Can have multiple inactive versions of the same object in the same repo (differentiated by workspace)
• Using an attribute view as an example, contains description of the attribute view (ex. which tables to join)
Active Objects (may or may not include runtime objects)
• When an inactive object is activated, it becomes visible to all repo users
• If applicable, corresponding runtime objects are created using SQL DDL
• Using an attribute view as an example, this would be a column view of type “join view” (CREATE
COLUMN VIEW…)
• Other repository objects (ex. web content) do not have runtime objects created during activation
© 2013 SAP AG. All rights reserved. 36 Ramp-Up Knowledge Transfer Customer
Deploying an XS Engine Application
1. regi track <package name>
Add package to set of tracked packages of the workspace
2. regi commit
Commit local changes to the repository (specify a list of files, or commit all tracked packages in your
workspace)
3. regi activate
Activates inactive versions of objects in the repository (specify a list of objects, or activate all inactive
objects in your workspace)
© 2013 SAP AG. All rights reserved. 37 Ramp-Up Knowledge Transfer Customer
Resources
HANA Blue Book
https://portal.wdf.sap.corp/irj/go/km/docs/corporate_portal/WS%20PTG/Product%20Architecture/Knowl
edge%20Transfer/Bluebook/The%20SAP%20HANA%20Database.pdf
Views
https://tdwiki.pgdev.sap.corp/download/attachments/100634778/Analytic_Views_What_are_they_and_h
ow_do_we_use_them.pptx?version=1&modificationDate=1333127736180
XS Wiki
http://trexweb.wdf.sap.corp:1080/wiki/index.php/XS
HANA Studio Installation Files
file://vantipsoftware.pgdev.sap.corp/software/SAP/HANA/
HANA (includes XSE) Installation Files and Instructions
file://trextest/trex_test/xsengine
http://trexweb.wdf.sap.corp:1080/wiki/index.php/XS_Installation_Windows
HANA XSE Sample Application Tutorial
http://trexweb.wdf.sap.corp:1080/wiki/index.php/Sample_Application_Development_Drop8
© 2013 SAP AG. All rights reserved. 38 Ramp-Up Knowledge Transfer Customer
© 2013 SAP AG. All rights reserved.
No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG.
The information contained herein may be changed without prior notice.
Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.
National product specifications may vary.
These materials are provided by SAP AG and its affiliated companies ("SAP Group") for informational purposes only, without representation or
warranty of any kind, and SAP Group shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP Group
products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing
herein should be construed as constituting an additional warranty.
SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in
Germany and other countries.
Please see http://www.sap.com/corporate-en/legal/copyright/index.epx#trademark for additional trademark information and notices.