Hi Perf Analytics in SAP NetWeaver 2004s

8/7/2019 Hi Perf Analytics in SAP NetWeaver 2004s

1/64

Session ID: IM205High Per form anc eAnaly t ic s in SAP

Net Weaver 2004s


2/64

Klaus Nagel, SAP AG

Alexander Peter, SAP AG

Ron Silberstein, SAP Labs, LLC

Marc Bernard, SAP Labs, LLC


3/64

SAP AG 2005, SAP TechEd 05 / IM205 / 3

Learn ing Objec t ives

As a resul t o f t h is w ork shop, you w i l l be ab le t o : Explain the significant performance improvements delivered

with the High Performance Analytics capabilities deliveredwith SAP NetWeaver 2004s

Understand that HPA also reduces the maintenancesubstantially

Specify appropriate scenarios for HPA and their easy

application on top of existing data models and processes Understand SAPs strong commitment to investing in

performance innovations, simultaneously keeping TCO at aminimum


4/64

Overview

HPA Configuration

HPA Administration

HPA in Detail

Summary

Appendix: Some Performance Features 2004s


5/64 SAP AG 2005, SAP TechEd 05 / IM205 / 5

Ent erpr ise Business Int e l l igenc e: Chal lenges

Enterpr ise

Business

In te l l igence

RapidlyGrowing

Data Volumes

Requirements:

Speed andFlexibility

IncreasingNumber of

Users

Administration challenges Potential performance issues

Increasing demand for more,better, and actionableinformation

Natural data growth,RFID, POS, EDW,retention policies,

Right-time information,fast query execution,ad hoc queries,

More application areas,more casual users

Challe

nges

forres

ourc

e

allo

catio

n:ext

ensive

proa

ctiv

e

maint

enanc

eeffort

require

d

Challenges for resourceallocation: updating capacity

planning may result inadditional hardware investment



Som e Ways t o Im prove Query Per form anc e

Per formanc e Feat ures

HighPerformanceAnalytics

Aggregates OLAP

Cache

ReportingAgent /

InformationBroadcasting

Summarized (or subset)

materialized views ofInfoCube data

Accelerates response for

similar queries by cachingquery result sets, andreading from cacheinstead of DB

Run popular queries in background

(off hours) and push summaryviews of updated data to users Information Broadcasting



Trad i t iona l Aggregat es Back ground / Contex t

Genera l ly , aggregat es ... Are BI objects designed to improve query response times

Structurally look the same as an InfoCube

Are subsets of fact table data

(Typically) store summary data in new, smaller, child InfoCubes

Improve query response times by reducing the volume of data read

Are transparent to users

If an aggregate is available for a request, the system uses it automatically



How Many Aggregat es Do You Need?

You have to dec ide w h ic haggregat es you need

To ensure good performance

Pre-calculate aggregates for main queriesand reports

Restrict scope and frequency of reporting

This is a trade-off between flexibility andperformance

You cannot create aggregatesfor all possible queries

Thus, the typical strategy is to buildsomewhat general aggregates that morequeries can utilize, but the performancebenefit per query is less significant

# characteristics

#aggreg

ates

Aggregate Combinations



A Searc h Engine for Analy t ic s?

A new approac h The latest computer architectures make more intensive use of

increasingly inexpensive memory resources

Search engines deliver fast responses by giving read accesspriority over index updates

But can an engine first conceived for text search be used toimplement fast search for structured data in tables?

Yes SAP adapted its TREX search engine to implement

A c apab i l i t y fo r h igh per fo rmanc e ana ly t ic s !



HPA Capabi l i t y for SAP Net Weaver BI

High Per form anc e Analy t ic s (HPA)A new transparent approach based on TREXto boost BI query performance

Performance speedup factor between 10 and 100Without changing the BI user experience (transparent to users)

HPA Engine

Storage Query ProcessingDBMS TREXSAP NetWeaver Business Intelligence

Database


11/64


Termino logy

TREX and HPA TREX : Search and classification engine, mainly for unstructured data

HPA is built using TREX technology (indexing, retrieving, compression...) to handle structured data

Note: HPA and TREX are two different installations

The HPA engine cannot be used for standard KMC functionality

A TREX installation cannot be used for high performance analytics

Inside info: technically there is a lot of commonality but TREX-HPA role

switching is not supported

Index

HPA index: Structure containing replicated data and indexes (one index

per table) that represents the BI star schema for the HPA engine Indexing: Transfer of InfoCube data to the HPA server and processing

and compressing it into the HPA index


12/64


HPA Arc h i tec t ure

SAP NetWeaver BIHigh Performance

Analytics Capability

Data Acquisition

InfoCubes

BI AnalyticEngine

BusinessExplorer

Any Tool

AnySource

Query &Response

Indexing

HPA engine responds to queries:joins and aggregates in run time

HPA indexes loadedinto memory

HPA creates and storesindexes for InfoCubes

Q E t i


13/64


Query Ex ec ut ion

Query ExecutionWithout HPA

Pre-calculation /Information Broadcasting

Query ExecutionWith HPA

Pre-calculation /Information Broadcasting

OLAP Cache

HPA

1

OLAP Cache

Aggregate

InfoProvider

2

3

4

The BI system checks the listed repositories (in theorder shown) to obtain the query result set

HPA R t i l


14/64


HPA Rat ionale

High Performance Analytics

Very fast queryresponse time

Stable queryresponse time

High scalability Low maintenance

Performance

improvementsby factor 10 100

Independent of

DB optimizer,aggregates, ...

No aggregatemaintenance, minimized

roll-up/change run

Planned support of newhardware paradigms like

grid/adaptive computing

Increasedend-user

satisfaction/ExtendedBI reach

SignificantTCO

reduction

HPA S i


15/64


HPA Sc enar ios

Ready for h igh data vo lumes Queries that routinely involve access to many millions of records and

may involve up to billions of records

Examples: retail, utilities, telephone companies

Chal lenging response t im e SLAs

Example: service level agreements for call center operators demandshort response times for good closure rates

Unpred ic t ab le t ypes o f quer ies

Far more different data sets and aggregations than traditionaloptimization and caching strategies can handle

Excellent response times for any drill-down, slice & dice,

Examples: on-demand reporting for different user groups,ad hoc analyses

HPA Impac t on Main tenance


16/64


HPA Impac t on Main tenance

How HPA reduc es ma in t enanc e e f fo r t Aggregate maintenance

Automated caching strategies

Building database indexes

Improves implementation and administration Reduces TCO

The relational aggregate/caching approach is notobsoletewith HPA, and is suitable for some scenarios:

Smaller data volumes

Predictable query patterns

Relaxed response time constraints

Caching reduces potential high load on the HPA engine

HPA Test Resul t s


17/64


HPA Test Resul t s

Pre l im inary Lab Test Resu l t sLab tests with real SAP NetWeaver BI customer data

MultiProvider with 9 InfoCubes

About 850 million records together Customer aggregates used

The 7 most important, critical queries tested

Improvement factor 25

Hardware example:

At SAPPHIRE 2005 in Boston, SAP Executive Board member Shai Agassi demonstrated the

new capability for high performance analytics by launching complex queries against abillion data records and getting results back in seconds

For the demonstration, the HPA engine ran under 64-bit Linux on 8 blades, each with dual3.6 GHz Intel Xeon processors and 8 GB of RAM, mounted above a HewlettPackardProLiant 640 server with 4 Intel Itanium processors running SAP NetWeaver 2004s BI under

HP-UX, and an HP StorageWorks filer, all in a standalone cabinet

BI Perform anc e Tec hniques : Speed vs Re usabi l i t y


18/64


BI Perform anc e Tec hniques : Speed vs. Re-usabi l i t y

HPA of fers h igh re-usabi l i ty t oget her w i t h fast per formanc e in

a manner not possib le w i th o t her tec hn iques!

Canned Report s

Query Cac he

Summ ary Tables

Database

HPA

Less

fast

Fastest

Performance

Lowest Highest

Re-usability

High re-usability Very few specific objects (or sets) can be used toaccelerate many different queries


19/64


20/64

Overview

HPA Configuration

HPA Administration

HPA in Detail

Summary


HPA Inst a l la t ion


21/64


HPA Inst a l la t ion

Ins t a l la t ion for HPA in a box

During ramp-up of SAP NetWeaver 2004s, the underlying TREXsoftware for HPA will only be available preinstalled in a box

The box may be standalone or fit into an existing customer rack

The box will contain blade servers with 64-bit Intel processors

The OS for the blades is Linux SLES 9

The blades will be delivered with the complete HPA engine

preinstalled During the ramp-up phase, the box will contain Intel Xeon CPUs in

a choice of Hewlett-Packard or IBM hardware

Sizing for HPA (Ram p-Up Delivery)


22/64


Sizing for HPA (Ram p Up Delivery)

Sm al l T-shir t Size

20 parallel user sessions

250 million rows total

500 byte / row

Mediu m T-shir t Size

50 parallel user sessions

500 million rows total

500 byte / row

Large T-shir t Size

100 parallel user sessions 1,000 million rows total

500 byte / row

Intel-based blade technology under 64-bit Linux SLES9

HPA Index Main t enanc e 1


23/64



Ac c ess ing HPA Index Ma int enanc e

Similar to aggregatemaintenance viacontext menu for aparticular InfoCube

Or direct access viatransaction RSDDV



24/64


de a e a c e

Conf igur ing RFC Dest inat ion for in t he BI syst em

Configure HPA setting to the SM59 setting(this needs only be done once)

HPA Rol l-up and Change Run


25/64


p g

Roll-up

Update of newly loaded data to HPA index

Similar to roll-up process for aggregates

But there is only one HPA index to adjust

So the process is much easier and faster

Same process for HPA and aggregates: process chains do not need to bechanged after implementing HPA

Change Run

Update HPA index after master data/hierarchy changes

HPA index has same granularity as the InfoCubeNo c hanges needed fo r t ransac t ional da ta

HPA index includes some master data tables which have to be updated within

minimal runtime Adjustment of HPA index for key date changes of time-dependent attributes is

not necessary

Same process for HPA and aggregates: process chains do not need to bechanged after implementing HPA


26/64

Overview

HPA Configuration

HPA Administration

HPA Engine in Detail

Summary


Demo


27/64


Demo

Zero Admin is t ra t ion


28/64


HPA Goal : Zero Adm in is t ra t ion

HPA will feature automatic

Monitoring

Configuration

OptimizationSelf-repair

TREX Alert Server

Preconfigured checks

New checks can be added

Feeds any alerts to BI

TREX Admin Tool

Offers all admin options

Access to relevant optionsvia BI planned

Any alerts shownin portal view

SAP NetWeaver BI

TREX

HPAEngine

HPA Arc h i t ec t ure : H igh Avai lab il i t y and Sc alab i l i ty


29/64


RFCServerRFCServer

IndexServerBackup

HPAEngine

Index

ServerMaster

IndexServer

Primary

HPAEngine

Index

ServerCohort

IndexServerNode

HPAEngine

Index

ServerCohort

IndexServerNode

HPAEngine

Name

ServerMaster

NameServer

Primary

NameServer

Backup

Filer

Index

ServerCohort

HPA Fast Index Update


30/64


Queries Index updates

TREX

Main Index

Delta Index

Benefits: Smaller files

updated

Faster replication

HPA Engine

Queries runagainst both

indexes Merge delta indexes

as background job Triggered by size Per schedule Or manually

TREX Adm in is t ra t ion Tool


31/64


The TREX admin tool can be used for many tasks

The task shown here is landscape configuration

Aler t s in TREX Adm in Tool


32/64


Click here to view alerts


33/64

Overview

HPA Configuration

HPA Administration in TREX

HPA Engine In Detail

Summary


HPA Index Bui ld T ime


34/64


SAP NetWeaver BI High PerformanceAnalytics Capability

Data Acquisition

InfoCubes

BI AnalyticEngine

BusinessExplorer

Any Tool

Parallel indexing of InfoCube datavia standard BI processes

Vertical decomposition& compression to minimize

memory footprint of HPA indexes

AnySource


35/64

HPA Query Run T im e


36/64


SAP NetWeaver BI High PerformanceAnalytics Capability

Data Acquisition

InfoCubes

BI AnalyticEngine

BusinessExplorer

Any Tool

AnySource

Aggregation on the fly

Merging and resultspreparation for BI queries

HPA Arc h i tec t ure: TREX In dex Server and HPA Engine


37/64


HPA Engine Indexes BI InfoCubes

Materializes joins

Aggregation on the fly

Index Server

HPAEngine

Indexes

TREX

RFC

TREX Index Server

Indexes and searches

Contains HPA engine

Why HPA is so fastVertical decomposition

Horizontal partitioning

Smart compression

HPA Arch i t ec t ure : HPA Ver t ic a l Dec om posi t ion


38/64


HPA

stores tables by column

To find all instances of an attributevalue:

Go to the attribute column

Read its row values

Classic a l DB

stores tables by row

To find all instances of an attributevalue:

Go to the first row

Check the attribute value Go to the next row

Check the attribute value

Repeat for each row in the table

Att1Att1

Tuple2

Tuple1Att2

Att2Att2

...

If you dont havea predefined indexready you can wait

a long time

HPA Arc h i tec t ure: HPA Hor izonta l Par t i t ioning


39/64


Fact TableFact Table

PhysicalIndex

Server

IndexPart 1

PhysicalIndex

Server

IndexPart 2

PhysicalIndex

Server

IndexPart N

LogicalIndex

Server

IndexMetadata

Storage

TREX Services

Index

Part 1

Part 2

...

Part N

Metadata

Fact Table

Part 1

Part 2

...

Part N

LargeTables

HPA Arc h i t ec t ure : Sm ar t Com press ion


40/64


Compress ion

Dictionary-based compression

Size of table keys significantly reduced

Lab Resul t s Lab results on real data have shown compression factors up to 20

The HPA engine can handle huge data volumes in main memory without need for costly disk accesses

HPA Arc h i tec t ure: HPA Eng ine


41/64


HPA Queri es

Automatically join required indexesas specified in the HPA index

Aggregate large amounts of data inparallel

Restrict aggregated rows withboolean query expressions

Process more than one booleanexpression at once

Very fas t aggregat ion a lgor i thm

Exploits integer coding formax speed and min I/O

Designed to run on latest blade

servers and grid landscapes

Optimizes usage of memory andcache resources

Optimized for BI

Compressed m em ory index

Many million documents can fit in1 GB memory

Stores by column (not by row)

Can split indexes horizontally for

parallel processing

Hybr id dynamic index

Enables indexing of huge volumesof structured data

Keeps only required columns inmemory

HPAEngine

TREX

Engine

Engine


42/64

Overview

HPA ConfigurationHPA Administration in TREX

HPA In Detail

SummaryAppendix: Some Performance Features 2004s

HPA Sum m ary


43/64


Query proc ess ing in mem ory

Sophisticated compression

Horizontal and vertical data partitioning

Enabled for grid and adaptive computing

Based on 64 bit architecture

HPA is t ransparent fo r t he app l ic a t ions and 3rd par ty t oo ls

Available for all applications and tools that

Use SAP NetWeaver BI for reporting and analytics

Are based on SAP NetWeaver 2004s

Easy migration from existing aggregate setup to HPA

Queries automatically use HPA

Data maintenance via BI, minimum admin for HPA


44/64


45/64

Overview

HPA ConfigurationHPA Administration

HPA In Detail

SummaryAppendix: Performance Features 2004s

OLAP Cac he Enhancem ent s 1


46/64


Delt a Cac hing

A query reads data from the OLAPCache as far as possible

Additional requests will be readfrom the InfoCubes (F) fact table

These additional requests willbe updated into the OLAP Cache

Restrictions

If the InfoCube is compressed, delta cannot be determined Cache entrywill be built up completely

OLAP Cache

Request 1

InfoCube (F)

Request 2

Request 3

Request 1

Request 2

Request 3

Request 4

Request 5



47/64


Delt a Cac hing

Setting per query

Transaction RSRT

Activation of deltacaching



48/64


Cac hing for Mul t iProvider Quer ies

Invalidation of the basis InfoProviders depends on the groupingtype setting

No InfoProvidergrouping: complete query

result is kept and will bediscarded if one if theInfoProvider changes

Grouping: one group storesall InfoCube results, other

objects in one or moregroups (uses deltacapabilities of infoCubes)

Grouping with

separate basisInfoCubes: every InfoCuberesult is stored separately

Separation of allInfoProviders

Query Ex ec ut ion


49/64


Query Ex ec ut ion

All query splits can beparallelized

Splits of MultiProviderinto InfoProviders

Splits of InfoProviderinto Aggregates

Splits ofAggregates/InfoProviderinto E and F tables

Splits in analytical engine (constant selection, cell calculations) areparallelized with the exception of non-cumulative calculations

Number of parallel processes = min (6, max (0, #free processes 3))

not more than 6 parallel processes

F E

MultiProvider

InfoProviders

Aggregates

F/E tables

Rol l -up and Change Run 1


50/64


Roll -up and Change Run

Parallelized across aggregates

Sequence (with regards to aggregate hierarchy) is obtained

Several change runs can run in parallel if there is no common

InfoCube Settings can be made in the

Batch Manager Framework(transaction RSBATCH)

Rol l -up and Change Run 2


51/64


Ex am ple for para l le l c hange run/ro l l -up

First step: calculate all basis aggregates (if there are more processesavailable than basis aggregates, not all processes are used)

Once one job is finished, the system checks for the next level ofaggregates

The following graphic is simplified as not all jobs end at the sametime some jobs might be entangled

3 available parallel

processes

Step 1InfoCube

Basis

Aggregates

Aggregates

BA1 BA2

A1 A2 A3 A4

BA1 BA2

Step 2A1 A2 A3

Step 3 A4

Compression


52/64


Compression

General Optimizations

Inserts will be used automatically in case InfoCube is empty or therespective time value is new

Automatic DB parallelism via MERGE/UPSERT (also implemented in

DB6)

Generation of compression programs has been reducedperformance improvement especially for small requests



53/64


Selec t ive Dele t ion

Deleting in partitions

System automaticallydetects a databasepartition which matches

a subset of the recordsto be deleted

Scenario: you archive abunch of records and

expect performanceimprovement bydropping completepartitions

Recommendation: usethis flag for large datasets; for small sets theselective deletion might

work faster than the initial check for partitions

Par t i t ion ing


54/64


Enhanced part i t ion ing

Additional database platforms support partitioning

DB2 / UDB: Multi-dimensional Clustering

Clusters can be built out of several keys (e.g. month and org unit)

Benefits: improved query performance, small indexes, fast insertion/deletion, less

table reorganization Supported tables: PSA, Fact tables and DataStore object tables

Further SAP optimizations: automatic statistics (no job needs to be scheduled),automatic tablespace enhancements

DB2 / AS400: Range Partitioning

Maximum 1000 partitionsMS SQL Server: Range Partitioning

Fact tables (E/F), PSA

ORACLE

Partitioning is not used if partitioning option is not installed this is necessary forSAP NetWeaver installations without usage type BI

Repart i t i on ing 1


55/64


Repar t i t ion ing

Scenarios: after the initial partitioning you want someadditional partitions; after archiving data you want to mergesome earlier partitions

Three use cases of repartitioning tool

Append new partitions

Merge partitions

Complete repartitioning

Repart i t i on ing 2


56/64


Repar t i t ion ing

Append new partitions


57/64


Repar t i t ion ing

Merge partitions


58/64


Repar t i t ion ing

Complete repartitioning

MonthFiscal Year Variant


59/64


Dat a Load Per formanc e Improvem ent s

New Data Transfer Process

High scalability by enhanced parallelism

DataStore Object

Write-optimized DataStore object

Improved activation process

Improved rollback process

Centralized Monitoring

via Administration Cockpit Pro-active support of SAP

NetWeaver BI administrator instatus tracking andperformance optimization(data load, reportingand planning)

Enhanced data load & queryperformance statistics

SAP Net Weaver BI Near-L ine St orage St rat egy

AccessFrequency


60/64


q y

Age of Data

Online

Near-Line Storage

Near-Line St oragein SAP NetWeaver 2004s BI

SAP NetWeaver BI analyses havedirect access to archived data in

various storage media

Availability of historic data whilereducing costs

Physical decoupling of frequentlyused and less frequently or rarely

used data

Reloading of data into theInfoCube or DataStore Objectonly necessary in exceptional

cases

Furthe r Informat i on

P bl i W b


61/64


Publ ic Web:www.sap.com

SAP Developer Network: www.sdn.sap.com Business InformationWarehouse SAP BW Performance Tuning Knowledge Center

SAP Customer Services Network: www.sap.com/services/

NetWeaver Developers Guide: www.sdn.sap.com/sdn/developersguide.sdn

Re la ted Work shops /Lec t u res a t SAP Tec hEd 2005IM251, Enterprise Data Warehousing Enhanced Administration &Monitoring in SAP NetWeaver 2004s, 2h Hands-on

IM252, Enterprise Data Warehousing Enhanced Modeling Capabilities inSAP NetWeaver 2004s, 2h Hands-on

Relat ed SAP Educ a t ion Tra ining Oppor tun it ieshttp://www.sap.com/education/

Quest ions?


62/64


Q& A


63/64


Please complete your session evaluation.

Be courteous deposit your trash,and do not take the handouts for the following session.

Feedback

Thank You !


64/64

Documents

Hi Perf Analytics in SAP NetWeaver 2004s