© 2015 IBM Corporation

IBM Platform Data Manager for LSF Intelligent, Managed Data Staging

Gábor Samu, Portfolio Marketing Manager Software Defined Infrastructure gsamu@ca.ibm.com

© 2015 IBM Corporation 2

The IBM Platform LSF Family

IBM Platform

RTM

IBM Platform Analytics

IBM Platform Process Manager

IBM Platform

Application Center

IBM Platform Dynamic Cluster

IBM Platform License

Scheduler IBM

Platform Session

Scheduler

Platform LSF

Scheduling Efficiency

MapReduce Accelerator Platform HPC

Docker Connector

Hadoop Connector

IBM Platform

Data Manager

Platform MPI

Platform PCM

© 2015 IBM Corporation 3

Challenges with data management in HPC environments • Data is not available on the compute resources when needed •  Transfers of data done “in-band” – idle CPUs waiting for transfers • Wasted bandwidth, storage on duplicate transfers • What is the state of data transfers? •  Single user copying the same data repeatedly • Multiple users transferring the same data repeatedly

Compute power requires data to operate on

© 2015 IBM Corporation 4

Intelligent, managed data staging

§ Workload independent, managed data transfers

§ Eliminate duplicate transfers § Lower storage costs § Visibility of data transfer traffic

IBM Platform Data Manager for LSF

Whether in the cloud or working locally, ensure that your data is in the right location at the right time with

intelligent caching and out-of-band transfers, helping to reduce costs and overall time to solution.

© 2015 IBM Corporation 5

Unique data staging capabilities

•  Fully integrated with IBM Platform LSF •  Managed movement of data within and between Platform LSF clusters, with control over policies and

priority.

•  Control over out-of-band movement of data •  Preventing wasted compute cycles.

•  Eliminate redundant transfers with intelligent caching of data. •  Data affinity

•  For environments consisting of multiple IBM Platform LSF clusters, factor in data availability in scheduling

•  Configurable file transfer mechanism •  Administrators may configure IBM Platform Data Manager to use the underlying file transfer

mechanism (e.g. scp, gridftp, IBM Aspera)

© 2015 IBM Corporation 6

Data affinity – a closer look

Cache (file TUV789)

Platform LSF Cluster C Platform Data

Manager

Cache (file XYZ456)

Platform LSF Cluster B Platform Data

Manager

Cache (file XABCD123)

Platform LSF Cluster D Platform Data

Manager

Cache

Platform LSF Cluster A Platform Data

Manager

Platform Data Manager makes data availability a factor when forwarding to remote clusters!

My job requires file XABCD123

Job forwarded to Cluster D, where requested data file is

cached

© 2015 IBM Corporation 7

IBM Spectrum Scale (Persistent Data Source)

Flash (Data Manager Cache)

COMPUTE HOSTS

Transfer Job

Platform LSF job asks for input data file on

slow shared file system

Platform Data Manager pre-stages the file into fast flash cache before

job execution

Job accesses the data file from the cache and

writes its output to flash

Platform Data manager drains the output from

the cache into persistent storage after

job execution.

HPC perspective – Burst Buffers

Platform LSF &

Platform Data Manager

Transfer Job

© 2015 IBM Corporation 8

Thank you For more information: http://www.ibm.com/systems/platformcomputing/products/lsf