24
1 www.prace-ri.eu Docker SWARM: Light-weight orchestrator HPCW @ ISC2019 Abdulrahman Azab Dept. of Research Computing University of Oslo, Norway

Docker SWARM: Light-weight orchestrator - QNIBqnib.org/data/hpcw19/4_SCHED_1_SWARM.pdfDocker SWARM: Light-weight orchestrator HPCW @ ISC2019 Abdulrahman Azab Dept. of Research Computing

  • Upload
    others

  • View
    18

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Docker SWARM: Light-weight orchestrator - QNIBqnib.org/data/hpcw19/4_SCHED_1_SWARM.pdfDocker SWARM: Light-weight orchestrator HPCW @ ISC2019 Abdulrahman Azab Dept. of Research Computing

1 www.prace-ri.eu

Docker SWARM: Light-weight orchestrator

HPCW @ ISC2019

Abdulrahman Azab

Dept. of Research Computing

University of Oslo, Norway

Page 2: Docker SWARM: Light-weight orchestrator - QNIBqnib.org/data/hpcw19/4_SCHED_1_SWARM.pdfDocker SWARM: Light-weight orchestrator HPCW @ ISC2019 Abdulrahman Azab Dept. of Research Computing

2 www.prace-ri.eu

PRACE 6IP WP6.2.3: The deployment of containers and full virtualised tools into HPC infrastructures

Page 3: Docker SWARM: Light-weight orchestrator - QNIBqnib.org/data/hpcw19/4_SCHED_1_SWARM.pdfDocker SWARM: Light-weight orchestrator HPCW @ ISC2019 Abdulrahman Azab Dept. of Research Computing

3 www.prace-ri.eu

Docker UCP

Page 4: Docker SWARM: Light-weight orchestrator - QNIBqnib.org/data/hpcw19/4_SCHED_1_SWARM.pdfDocker SWARM: Light-weight orchestrator HPCW @ ISC2019 Abdulrahman Azab Dept. of Research Computing

4 www.prace-ri.eu

Page 5: Docker SWARM: Light-weight orchestrator - QNIBqnib.org/data/hpcw19/4_SCHED_1_SWARM.pdfDocker SWARM: Light-weight orchestrator HPCW @ ISC2019 Abdulrahman Azab Dept. of Research Computing

5 www.prace-ri.eu

Page 6: Docker SWARM: Light-weight orchestrator - QNIBqnib.org/data/hpcw19/4_SCHED_1_SWARM.pdfDocker SWARM: Light-weight orchestrator HPCW @ ISC2019 Abdulrahman Azab Dept. of Research Computing

6 www.prace-ri.eu

[centos@ucp-manager-0 ~]$ UCP_IP=<ucp-manager-0 IP>[centos@ucp-manager-0 ~]$ UCP_FQDN=<ucp-manager-0 FQDN>[centos@ucp-manager-0 ~]$ docker container run --rm -it --nameucp \-v /var/run/docker.sock:/var/run/docker.sock \docker/ucp:3.1.0 install \--admin-username admin \--admin-password adminadmin \--san ${UCP_IP} \--san ${UCP_FQDN}

Page 7: Docker SWARM: Light-weight orchestrator - QNIBqnib.org/data/hpcw19/4_SCHED_1_SWARM.pdfDocker SWARM: Light-weight orchestrator HPCW @ ISC2019 Abdulrahman Azab Dept. of Research Computing

7 www.prace-ri.eu

Page 8: Docker SWARM: Light-weight orchestrator - QNIBqnib.org/data/hpcw19/4_SCHED_1_SWARM.pdfDocker SWARM: Light-weight orchestrator HPCW @ ISC2019 Abdulrahman Azab Dept. of Research Computing

8 www.prace-ri.eu

Orchestration

Page 9: Docker SWARM: Light-weight orchestrator - QNIBqnib.org/data/hpcw19/4_SCHED_1_SWARM.pdfDocker SWARM: Light-weight orchestrator HPCW @ ISC2019 Abdulrahman Azab Dept. of Research Computing

9 www.prace-ri.eu

Page 10: Docker SWARM: Light-weight orchestrator - QNIBqnib.org/data/hpcw19/4_SCHED_1_SWARM.pdfDocker SWARM: Light-weight orchestrator HPCW @ ISC2019 Abdulrahman Azab Dept. of Research Computing

10 www.prace-ri.eu

Page 11: Docker SWARM: Light-weight orchestrator - QNIBqnib.org/data/hpcw19/4_SCHED_1_SWARM.pdfDocker SWARM: Light-weight orchestrator HPCW @ ISC2019 Abdulrahman Azab Dept. of Research Computing

11 www.prace-ri.eu

Container NetworkOperations

Page 12: Docker SWARM: Light-weight orchestrator - QNIBqnib.org/data/hpcw19/4_SCHED_1_SWARM.pdfDocker SWARM: Light-weight orchestrator HPCW @ ISC2019 Abdulrahman Azab Dept. of Research Computing

12 www.prace-ri.eu

Page 13: Docker SWARM: Light-weight orchestrator - QNIBqnib.org/data/hpcw19/4_SCHED_1_SWARM.pdfDocker SWARM: Light-weight orchestrator HPCW @ ISC2019 Abdulrahman Azab Dept. of Research Computing

13 www.prace-ri.eu

Page 14: Docker SWARM: Light-weight orchestrator - QNIBqnib.org/data/hpcw19/4_SCHED_1_SWARM.pdfDocker SWARM: Light-weight orchestrator HPCW @ ISC2019 Abdulrahman Azab Dept. of Research Computing

14 www.prace-ri.eu

Page 15: Docker SWARM: Light-weight orchestrator - QNIBqnib.org/data/hpcw19/4_SCHED_1_SWARM.pdfDocker SWARM: Light-weight orchestrator HPCW @ ISC2019 Abdulrahman Azab Dept. of Research Computing

15 www.prace-ri.eu

Use case:Containerised HPC

With HTConcor and swarm

Page 16: Docker SWARM: Light-weight orchestrator - QNIBqnib.org/data/hpcw19/4_SCHED_1_SWARM.pdfDocker SWARM: Light-weight orchestrator HPCW @ ISC2019 Abdulrahman Azab Dept. of Research Computing

16 www.prace-ri.eu

HTCondor

Page 17: Docker SWARM: Light-weight orchestrator - QNIBqnib.org/data/hpcw19/4_SCHED_1_SWARM.pdfDocker SWARM: Light-weight orchestrator HPCW @ ISC2019 Abdulrahman Azab Dept. of Research Computing

17 www.prace-ri.eu

HTCondor VM universe

universe = vmexecutable = vmware_sample_joblog = simple.vm.log.txtvm_type = vmwarevm_memory = 64 vmware_dir = C:\condor-testvm_checkpoint = true queue

Page 18: Docker SWARM: Light-weight orchestrator - QNIBqnib.org/data/hpcw19/4_SCHED_1_SWARM.pdfDocker SWARM: Light-weight orchestrator HPCW @ ISC2019 Abdulrahman Azab Dept. of Research Computing

18 www.prace-ri.eu

HTCondor Docker universe

universe = docker docker_image = debianexecutable = /bin/cat arguments = /etc/hosts output = out.$(Process) error = err.$(Process) request_memory = 100M queue 10

Page 19: Docker SWARM: Light-weight orchestrator - QNIBqnib.org/data/hpcw19/4_SCHED_1_SWARM.pdfDocker SWARM: Light-weight orchestrator HPCW @ ISC2019 Abdulrahman Azab Dept. of Research Computing

19 www.prace-ri.eu

Containerised HTCondor on Swarm

Page 20: Docker SWARM: Light-weight orchestrator - QNIBqnib.org/data/hpcw19/4_SCHED_1_SWARM.pdfDocker SWARM: Light-weight orchestrator HPCW @ ISC2019 Abdulrahman Azab Dept. of Research Computing

20 www.prace-ri.eu

Page 21: Docker SWARM: Light-weight orchestrator - QNIBqnib.org/data/hpcw19/4_SCHED_1_SWARM.pdfDocker SWARM: Light-weight orchestrator HPCW @ ISC2019 Abdulrahman Azab Dept. of Research Computing

21 www.prace-ri.eu

Page 22: Docker SWARM: Light-weight orchestrator - QNIBqnib.org/data/hpcw19/4_SCHED_1_SWARM.pdfDocker SWARM: Light-weight orchestrator HPCW @ ISC2019 Abdulrahman Azab Dept. of Research Computing

22 www.prace-ri.eu

Page 23: Docker SWARM: Light-weight orchestrator - QNIBqnib.org/data/hpcw19/4_SCHED_1_SWARM.pdfDocker SWARM: Light-weight orchestrator HPCW @ ISC2019 Abdulrahman Azab Dept. of Research Computing

23 www.prace-ri.eu

Page 24: Docker SWARM: Light-weight orchestrator - QNIBqnib.org/data/hpcw19/4_SCHED_1_SWARM.pdfDocker SWARM: Light-weight orchestrator HPCW @ ISC2019 Abdulrahman Azab Dept. of Research Computing

24 www.prace-ri.eu