Container Support in IBM Spectrum LSF

Container Support in LSFRunning Jobs in Containers

Spectrum LSF Development

IBM Systems

Linux Containers vs. Virtual Machines

2

A “container” delivers an application with all the libraries, environment, and dependencies needed to run.

IBM Systems

What problem are we solving?

We are enabling users to run containerised HPC applications within their HPC environment.

We are NOT providing generic container management or running a container cloud.

| 3

IBM Systems

What difference does this make?

Without Docker:

Before the user can run an application…

$bsub “my_app < input > output”

… someone else needs to install the application, on the right OS, and all the associated libraries, resolve conflicts with all the other packages installed.

With Docker (or other container support):

The user just runs their containerised app

$bsub –a “docker(image)” “my_app < input > output”

The application and it’s library & OS dependencies are installed on demand.

Note: the syntax above is from our original support for Docker from 2014

| 4

IBM Systems

LSF’s original Docker integration (from 2014)

| 5

Definitions from lsb.queues, lsb.limits, lsb.applications apply to the job starter, not the job within the container

IBM Systems

Challenges with Docker/Containers in a Production Environment

Multiple different container technologies – Docker, Shifter, Singularity, rkt, etc.• Containers may be portable, but the runtimes have different usage

Many Businesses have concerns over allowing users to “install” and run an arbitrary application• Auditability, accountability, security of data.

Security of the run time environment:• Docker users have to be placed in the “docker user” group• Docker group members have “sudo” privileges – run jobs as root• Not really an issue when docker is being used to run services started by root - but a potentially serious

issue when end users are allowed to run arbitrary jobs.

| 6

LSF 10.1.0.1

IBM Systems

Enhanced Container Support in LSF 10.1.0.1

• LSF starts containers as admin (or root) • processes in container run as job owner• end users do not need to be in the “docker” group

• Administrators control (via lsb.applications) which images can be used in the cluster• LSF tracks all resource usage and enforces limits and policies• Consistent end user interface for Docker, Singularity and Shifter

| 7

$bsub –app xyzzy a.out –in in.dat –out out.dat

Begin ApplicationNAME = xyzzyCONTAINER = docker[image(corphost:5000/xyzzy) \

options(--rm --net=host --ipc=host \-v /gpfs/u/fred:/REF ) starter(lsfadm)]

DESCRIPTION = Approved XYZZY containerEnd Application

IBM Systems

Running Containerized Workloads

| 8

bsub -app ctn job

lsb.applications

Begin ApplicationNAME = ctnCONTAINER = docker [image(userx/ubuntu)]DESCRIPTION = docker jobEnd Application

Begin ApplicationNAME = ctnCONTAINER = shifter[image(userx/ubuntu)]DESCRIPTION = shifter jobEnd Application

Begin ApplicationNAME = ctnCONTAINER = singularity[image(/image/ubuntu)]DESCRIPTION = singularity jobEnd Application

Run LSF job in container with application profile

• Transparent to end users. End users submit jobs to application profile in common interface.

• Administrators can also customize the container runtime environment on demand.

• LSF Docker integration also addresses well-known privilege escalation issues

IBM Systems

User Specified Image Names (LSF 10.1.0.2) (1)

• Optionally, administrator can allow the user to pass JUST the desired image name• restrict users to use images in a trusted repository

| 9

$bsub –env “LSB_CONTAINER_NAME=bambam” –app docker a.out –in in.dat –out out.dat

Begin ApplicationNAME = dockerCONTAINER = docker[image(corphost:5000/$LSB_CONTAINER_NAME) \

options[--rm --net=host --ipc=host \-v /gpfs/u/fred:/REF ) starter(lsfadm) \starter(lsfadm)]

DESCRIPTION = Docker generic containerEnd Application

IBM Systems

User Specified Image Names (LSF 10.1.0.2) (2)

• Instead of having users set an environment variable, can write a simple esub to do it• below, esub myopt sets LSB_CONTAINER_NAME=bambam and passes it to the launcher

| 10

$bsub –a “myopt(bambam)” –app docker a.out –in in.dat –out out.dat


options[--rm --net=host --ipc=host \-v /gpfs/u/fred:/REF ) starter(lsfadm)


© 2017 International Business Machines Corporation 11

IBM Research

Now Available at https://github.com/IBM/ubiquityAlpha release to gain experience with users and their use cases

Support on a best effort basis

Ubiquity Container Volume Service for IBM Spectrum ScaleSecure, Portable, High Performance

• Provide provisioning and management of Spectrum Scale (POSIX) and CES NFS• Applications access file system just like native bare-metal• Benefits of IBM Spectrum Scale and Containers

– Launch any application from laptop to supercomputer– Break free of Supercomputer Linux distro, Linux kernel, or library restrictions– Multi-Tenant Data Access - Applications mount only their data and nothing else– High-Performance Data Access

• Just like bare-metal, avoid VM overheads

• Key Use Cases– HPC and Scientific Computing– AI and Deep Learning

• Users can identify datasets by name (no need for file path)– Expose biomed input read-only dataset to users as ‘biomed1’– Create storage space for a user named ‘DevOp1’ for use in all their applications

• Single naming of datasets across all orchestration frameworks Docker Datacenter

https://github.com/ibm/ubiquity

IBM Systems

Ubiquity Docker Volume Driver with LSF (1)

• We can specify the volume driver and volume mapping statically within the app definition

| 12

$bsub –a “myopt(fred)” –app docker a.out –in in.dat –out out.dat


options[--rm --net=host --ipc=host \--volume-driver ubiquity --volume my_volume:/data \starter(lsfadm)]


LSF 10.1.0.2

IBM Systems


• Coming in LSF 10.1.0.3: support custom script to generate container launch options• runs as lsfadmin and gets a copy of the job environment• output is appended to the container launch options string

• Use the script to dynamically perform volume mounting

| 13


options[--rm --net=host --ipc=host starter(lsfadmin) \@/usr/local/lsf/etc/volume_mapper ]

DESCRIPTION = Generic container with mapped volumesEnd Application

IBM Systems


| 14


options[--rm --net=host --ipc=host starter(lsfadmin) \@/usr/local/lsf/etc/volume_mapper ]

DESCRIPTION = Generic container with mapped volumesEnd Application

#!/bin/bash#/usr/local/lsf/etc/volume_mapper script runs as lsfadmin, gets passed all job env vars#does some magic, returns the mappingecho “--volume-driver ubiquity --volume my_volume:/data”

IBM Systems

Benefits of IBM Spectrum LSF container support

Enable Docker jobs to run on IBM Spectrum LSF managed clusters• Straightforward Docker enablement. Easy to install and easy to manage• Allows HPC users to run Docker jobs in batch mode in the same way as running other jobs.• All features in existing LSF V10.1 release are available to Docker jobs• Addresses security concerns with Docker

Enables system administrators to withhold root privileges from HPC user. • On compute cluster: HPC users are no longer in Docker user group• On development node(s) : HPC users remain in Docker user group to develop apps.• Effectively prevented potential security and privacy violations from Docker users

Seamless integration with the rest of LSF family of software

15

Thank you.

IBM Systems

ibm.com/systems

|

IBM Systems

Copyright © 2016 by International Business Machines Corporation. All rights reserved.

No part of this document may be reproduced or transmitted in any form without written permission from IBM Corporation.

Product data has been reviewed for accuracy as of the date of initial publication. Product data is subject to change without notice. This document could include technical inaccuracies or typographical errors. IBM may make improvements and/or changes in the product(s) and/or program(s) described herein at any time without notice. Any statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business. Any reference to an IBM Program Product in this document is not intended to state or imply that only that program product may be used. Any functionally equivalent program, that does not infringe IBM's intellectually property rights, may be used instead.

THE INFORMATION PROVIDED IN THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER OR IMPLIED. IBM LY DISCLAIMS ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NONINFRINGEMENT. IBM shall have no responsibility to update this information. IBM products are warranted, if at all, according to the terms and conditions of the agreements (e.g., IBM Customer Agreement, Statement of Limited Warranty, International Program License Agreement, etc.) under which they are provided. Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. IBM makes no representations or warranties, ed or implied, regarding non-IBM products and services.

The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents or copyrights. Inquiries regarding patent or copyright licenses should be made, in writing, to:

IBM Director of LicensingIBM CorporationNorth Castle DriveArmonk, NY 1 0504- 785U.S.A.

Legal notices

|

IBM Systems

Information and trademarksIBM, the IBM logo, ibm.com, IBM System Storage, IBM Spectrum Storage, IBM Spectrum Control, IBM Spectrum Protect, IBM Spectrum Archive, IBM Spectrum Virtualize, IBM Spectrum Scale, IBM Spectrum Accelerate, Softlayer, and XIV are trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at http://www.ibm.com/legal/copytrade.shtml

The following are trademarks or registered trademarks of other companies.

Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.

IT Infrastructure Library is a Registered Trade Mark of AXELOS Limited.

Linear Tape-Open, LTO, the LTO Logo, Ultrium, and the Ultrium logo are trademarks of HP, IBM Corp. and Quantum in the U.S. and other countries.

Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.

Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates.

Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license therefrom.

ITIL is a Registered Trade Mark of AXELOS Limited.

UNIX is a registered trademark of The Open Group in the United States and other countries.

* All other products may be trademarks or registered trademarks of their respective companies.

Notes:

Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here.

All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions.

This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult your local IBM business contact for information on the product or services available in your area.

All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.

Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. |

http://www.ibm.com/legal/copytrade.shtml

Software

Container Support in IBM Spectrum LSF