0101 foundation - detailed view of hana architecture

SAP HANA Detailed View of HANA Architecture

© 2011 SAP AG. All rights reserved. / Page 2

Disclaimer

This presentation outlines our general product direction and should not be relied on in making a purchase decision. This presentation is not subject to your license agreement or any other agreement with SAP.

SAP has no obligation to pursue any course of business outlined in this presentation or to develop or release any functionality mentioned in this presentation. This presentation and SAP's strategy and possible future developments are subject to change and may be changed by SAP at any time for any reason without notice.

This document is provided without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. SAP assumes no responsibility for errors or omissions in this document, except if such damages were caused by SAP intentionally or grossly negligent.


Agenda

1. Architecture Overview2. Row Store3. Column Store4. Persistency Layer


ERP

Architecture OverviewIn-Memory Computing Engine and Surroundings

LogERP DB

In-Memory Computing Engine

Clients (planned, e.g.) BI4 Explorer

Dashboard Design

SAP BI4 universes (WebI,...)

Request Processing / Execution Control

MS Excel

BI4 Analysis

SQL Parser MDXSQL Script Calc Engine

Transaction Manager

Session Management

Relational EnginesRow Store Column Store

Persistence LayerPage Management Logger

Disk StorageLog VolumesData Volumes

Authorization Manager

Metadata Manager

In-Memory Computing Studio

Administration Modeling

Load Controller

Replication Agent

Replication Server

SAP Business Objects BI4

Data Services Designer

SBO BI4 servers

( program for client)

SBO BI4 Information Design Tool

Other Source Systems

SAP NetWeaver

BW3rd Party

Data Services


ERP

Architecture OverviewThe Engine

LogERP DB

Clients (planned, e.g.) SBOP Explorer 4.0

Xcelsius SAP BI universes (WebI,...)

MS Excel

SBOP Analysis

IMC Studio


Load Controller

Replication Agent

Business Objects Enterprise


SBO server programs for clients

SBO Information Design Tool


SAP NetWeaver

BW3rd Party

Data Services


Request Processing / Execution ControlSQL Parser MDXSQL Script Calc Engine

Transaction Manager

Session Management





Metadata Manager

Replication Server


ERP

Architecture OverviewLoading Data into SAP HANA

LogERP DB



Transaction Manager

Session Management





Metadata Manager



Load Controller

Replication Agent

Replication Server



SBO BI4 servers




SAP NetWeaver

BW3rd Party

Data Services


Dashboard Design


MS Excel

BI4 Analysis


ERP

Architecture OverviewData Modeling

LogERP DB



Transaction Manager

Session Management





Metadata Manager



Load Controller

Replication Agent

Replication Server



SBO BI4 servers




SAP NetWeaver

BW3rd Party

Data Services


Dashboard Design


MS Excel

BI4 Analysis


Clients (planned, e.g.)

ERP

Architecture OverviewReporting

LogERP DB



Transaction Manager

Session Management





Metadata Manager



Load Controller

Replication Agent

Replication Server



SBO BI4 servers




SAP NetWeaver

BW3rd Party

Data Services

BI4 Explorer

Dashboard Design


MS Excel

BI4 Analysis


ERP

Architecture OverviewAdministration

LogERP DB



Transaction Manager

Session Management





Metadata Manager



Load Controller

Replication Agent

Replication Server



SBO BI4 servers




SAP NetWeaver

BW3rd Party

Data Services


Dashboard Design


MS Excel

BI4 Analysis


DB Server

SAP High-Performance Analytic Appliance 1.0

SAP HANA

JDBC ODBC ODBO SQL DBC

SAP In-Memory Computing Engine

ReplicationServer

SAP In-Memory Computing Studio

SAP Business Application

ReplicationAgent

SAP BusinessObjects Data Services 4.0

Any source

SAP BusinessObjects

BI 4.0

Repository

SAP BusinessObjects BI clients

SQ

L

MD

X

BIC

S

Aut

hent

icat

ion

Con

tent

mgm

t

sync

Adm

in &

mod

el

load (optional)

(optional)

(optional)

(existing)


Request Processing and Execution ControlConceptual View

Standard SQL Processed directly by DB engine

SQL Script, MDX and planning engine interface

Domain-specific programming languages or modelsConverted into calculation models

Calc EngineCreate logical execution plan for calculation modelsExecute user defined functions

Relational EngineDB optimizer produces physical executing planAccess to row and column store


Calc Engine for Dummies

The easiest way to think of Calculation Models is to see them as dataflow graphs, where the modeler can define data sources as inputs and different operations (join, aggregation, projection,…) on top of them for data manipulations.

The Calculation Engine will break up a model, for example some SQL Script, into operations that can be processed in parallel (rule based model optimizer). Then these operations will be passed to the database optimizer which will determine the best plan for accessing row or column stores (algebraic transformations and cost based optimizations based on database statistics).


Calc Engine for DummiesExample


Agenda



In-Memory Computing EngineHigh Level Architecture

Row StoreOne of the relational enginesInterfaced from calculation / execution layerPure in-memory store

Persistence managed in persistence layer

SAP in-memorycomputing engine

HANA


Row Store ArchitectureRow Store Block Diagram

Row Store Block DiagramTransactional Version Memory

Contains temporary versionsNeeded for Multi-Version Concurrency Control (MVCC)

SegmentsContain the actual data (content of row-store tables) in pages

Page ManagerMemory allocationKeeping track of free/used pages

Version Memory ConsolidationThink ‘garbage collector for MVCC’

Persistence LayerInvoked in write operations (log)And in performing savepoints checkpoint writer


Row Store ArchitectureHighlights

Write OperationsMainly go into “Transactional Version Memory”“INSERT” also writes to Persisted Segment

Read Operations

Write Operations

Transactional Version Memory

Main Memory

Persisted Segment

Data that may be

seen by all active

transactions

Recent versions of changed records

Version Memory Consolidation

Version ConsolidationMoves “visible version” from Transaction Version Memory into Persisted Segment (based on Commit ID)Clears “outdated” record versions from Transactional Version Memory

Memory HandlingRow store tables are linked list of memory pagesPages are grouped in segmentsPage size: 16 KB

Persisted SegmentContains data that may be seen by any ongoing transaction Data that has been committed before any active transaction was started)


Indexes for Row Store TablesPrimary Index / Row ID / Index Persistence

Each row-store table has a primary indexPrimary index maps ROW ID primary key of table

ROW ID: a number specifying for each record its memory segment and page

How to find the memory page for a table record?A structure called “ROW ID” contains the segment and the page for the recordThe page can then be searched for the records based on primary keyROW ID is part of the primary index of the table

Secondary indexes can be created if needed

Persistence of indexes in row storeIndexes in row store only exist in memory

No persistence of index dataIndex definition stored with table metadataIndexes filled on-the-fly when system loads tables into memory on system start-up


Agenda



In-Memory Computing Engine High Level Architecture

Column StoreOne of the relational enginesInterfaced from calculation / execution layerPure in-memory store

Persistence managed in persistence layer

Optimized for high performance of read operationGood performance of write operationsEfficient data compression

SAP in-memorycomputing engine

HANA


Column Store ArchitectureColumn Store Block Diagram

Column Store Block DiagramOptimizer and Executor

Handles queries and execution plan

Main and Delta StorageCompressed data for fast read Delta data for fast writeAsynchronous delta merge

Consistent View Manager

Transaction Manager

Persistence Layer


Column StoreHighlights

Storage Separation (Main & Delta)Enables high compression and high write performance at the same time

Delta Merge OperationSee next slide

Read Operations

Write Operations

Main

Main Memory

Delta

Write optimized

Compressed and

Read optimized

Read OperationsAlways have to read from both main & delta storages and merge the results.Engine uses multi version concurrency control (MVCC) to ensure consistent read operations.

Data Compression in Main Storage

Compression by creating dictionary and applying further compression methodsSpeed up

Data load into CPU cacheEquality check Search

The compression is computed during delta merge operation.

Write OperationsOnly in delta storage because write optimized.The update is performed by inserting a new entry into the delta storage.


Column Store Delta Management

Delta Merge OperationPurpose

To move changes in delta storage into the compressed and read optimized main storageCharacteristics

Happens asynchronouslyEven during merge operation the columnar table will be still available for read and write operationsTo fulfil this requirement, a second delta and main storage are used internally

Read Operations

Write Operations

Main

Before Merge

Delta

Read Operations

Write Operations

MainNew

After Merge

DeltaNew

Read Operations

Write Operations

Main

During Merge

Main New

DeltaNewDelta

Merge Operations


Agenda

1. Architecture Overview2. Row Store3. Column Store4. Persistence Layer


Persistence Layer Purpose and Scope

Why Does An In-memory Database Need A Persistence Layer?Main Memory is volatile. What happens upon…

Database restart?Power outage?...

Data needs to be stored in a non-volatile wayBackup and restore

SAP in-memory computing engine offers one persistence layer which is used by row store and column store

Regular “savepoints” full persisted image of DB at time of savepoint

Logs capturing all DB transactions since last savepoint (redo logs and undo logs written)restore DB from latest savepoint onwards

Ability to create "snapshots"used for backups


Persistence Layer System Restart and Population of In-memory Stores

Actions During System RestartLast savepoint must be restored plus…

Undo logs must be read for uncommitted transactions saved with last savepointRedo logs for committed transactions since last savepoint

Complete content of row store is loaded into memory

Column store tables may be marked for preload or notOnly tables marked for preload are loaded into memory during startupIf table is marked for loading on demand, the restore procedure is invoked on first access


Thank you!


No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice.Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.Microsoft, Windows, Excel, Outlook, and PowerPoint are registered trademarks of Microsoft Corporation. IBM, DB2, DB2 Universal Database, System i, System i5, System p, System p5, System x, System z, System z10, System z9, z10, z9, iSeries, pSeries, xSeries, zSeries, eServer, z/VM, z/OS, i5/OS, S/390, OS/390, OS/400, AS/400, S/390 Parallel Enterprise Server, PowerVM, Power Architecture, POWER6+, POWER6, POWER5+, POWER5, POWER, OpenPower, PowerPC, BatchPipes, BladeCenter, System Storage, GPFS, HACMP, RETAIN, DB2 Connect, RACF, Redbooks, OS/2, Parallel Sysplex, MVS/ESA, AIX, Intelligent Miner, WebSphere, Netfinity, Tivoli and Informix are trademarks or registered trademarks of IBM Corporation.Linux is the registered trademark of Linus Torvalds in the U.S. and other countries.Adobe, the Adobe logo, Acrobat, PostScript, and Reader are either trademarks or registered trademarks of Adobe Systems Incorporated in the United States and/or other countries.Oracle is a registered trademark of Oracle Corporation.UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group.Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of Citrix Systems, Inc.HTML, XML, XHTML and W3C are trademarks or registered trademarks of W3C®, World Wide Web Consortium, Massachusetts Institute of Technology. Java is a registered trademark of Sun Microsystems, Inc.JavaScript is a registered trademark of Sun Microsystems, Inc., used under license for technology invented and implemented by Netscape. SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, Clear Enterprise, SAP BusinessObjects Explorer and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and other countries.Business Objects and the Business Objects logo, BusinessObjects, Crystal Reports, Crystal Decisions, Web Intelligence, Xcelsius, and other Business Objects products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP France in the United States and in other countries. All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary.The information in this document is proprietary to SAP. No part of this document may be reproduced, copied, or transmitted in any form or for any purpose without the express prior written permission of SAP AG.This document is a preliminary version and not subject to your license agreement or any other agreement with SAP. This document contains only intended strategies, developments, and functionalities of the SAP® product and is not intended to be binding upon SAP to any particular course of business, product strategy, and/or development. Please note that this document is subject to change and may be changed by SAP at any time without notice.SAP assumes no responsibility for errors or omissions in this document. SAP does not warrant the accuracy or completeness of the information, text, graphics, links, or other items contained within this material. This document is provided without a warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability, fitness for a particular purpose, or non-infringement.SAP shall have no liability for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials. This limitation shall not apply in cases of intent or gross negligence.The statutory liability for personal injury and defective products is not affected. SAP has no control over the information that you may access through the use of hot links contained in these materials and does not endorse your use of third-party Web pages nor provide any warranty whatsoever relating to third-party Web pages.

© 2011 SAP AG. All Rights Reserved