Embed Size (px)
Citation preview
Container Migration All Around The World
Adrian Reber <[email protected]>Mike Rapoport <[email protected]>
Open Source Summit Europe 2017October 23, Prague
This project has received fundingfrom the European Union’s Horizon2020 research and innovationprogramme under grant agreementNo 688386
. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Container Migration▶ criu (https://criu.org/) based migration
Checkpoint/Restore In Userspace▶ runC (https://runc.io/) based containers▶ Xonotic (http://xonotic.org/) in the container
The Free and Fast Arena Shooter
Container Migration All Around The World 2/25
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Local Migration Demo# migrate xonotic rhel02runc checkpoint --image-path image xonotic finished after 0.58second(s) with 0Giving floating IP to rhel02DUMP size: 366M /runc/containers/xonotic/imageTransferring DUMP to rhel02DUMP transfer time 0.17 secondsrunc restored xonotic successfully
Container Migration All Around The World 3/25
. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Local Migration Setup▶ yum install --releasever 7.4 --installroot
/runc/containers/xonotic/rootfs xonotic-server▶ oci-runtime-tool generate --args "/usr/bin/darkplaces-dedicated"
--args "-userdir" --args "/tmp" --tmpfs /tmp --rootfs-readonly--linux-namespace-remove network | jq 'del(.linux.seccomp)' >config.json
▶ runc run xonotic -d -b /runc/containers/xonotic/ &> /dev/null </dev/null
▶ RHEL 7.4 (ish)runc-1.0.0-12.1.gitf8ce01d.el7.x86_64criu-2.12-2.el7.x86_64
Container Migration All Around The World 5/25
StuttgartStrasbourg
. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Remote Migration# migrate xonotic rhelfr truerunc checkpoint --pre-dump --image-path parent xonotic finished after0 second(s) with 0PRE-DUMP size: 351M /runc/containers/xonotic/parentTransferring PRE-DUMP to rhelfrPRE-DUMP transfer time 22.95 secondsrunc checkpoint --image-path image --parent-path ../parent xonoticfinished after 0.15 second(s) with 0Giving floating IP to rhelfrDUMP size: 20M /runc/containers/xonotic/imageTransferring DUMP to rhelfrDUMP transfer time 1.09 secondsrunc restored xonotic successfully
Container Migration All Around The World 7/25
. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Remote Migration Setup▶ Layer 2 OpenVPN tunnel▶ Floating IP address using keepalived▶ Pre-Copy Migration▶ Still RHEL 7.4 kernel and CRIU▶ Using runC git checkout▶ Two python scripts to manage the migration steps
https://people.redhat.com/areber/criu/migratehttps://people.redhat.com/areber/criu/migrate-server.py
Container Migration All Around The World 9/25
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Optimizations - Pre-Copy
memoryprocesstableentrySource
System
DestinationSystem
InitiateMigration
QuiesceProcess
transfer
ResumeProcess
Time
memoryprocesstableentry
Migration DurationProcessDowntime
FinishMigration
Figure: Pre-Copy Migration
Container Migration All Around The World 10/25
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
CRIU - Implementation - Checkpointing▶ Collect information about the process(es) from /proc
File descriptorsMemory MapsPID from process group leader and all children…
▶ Seize all processes using ptrace▶ Insert Parasite Code to dump memory from within
Container Migration All Around The World 11/25
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
CRIU - Implementation - Restoring▶ Restore memory pages from image files▶ Use /proc/sys/kernel/ns_last_pid to influence the next PID▶ Fork to re-create all required processes▶ Re-Open file descriptors and seek▶ Switch to restorer blob▶ Remap pages to right location▶ Jump into restored code
Container Migration All Around The World 12/25
MontrealStrasbourg
. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Remote Migration to Montreal# migrate xonotic rhelca truerunc checkpoint --pre-dump --image-path parent xonotic finished after0 second(s) with 0PRE-DUMP size: 351M /runc/containers/xonotic/parentTransferring PRE-DUMP to rhelcaPRE-DUMP transfer time 7.39 secondsrunc checkpoint --image-path image --parent-path ../parent xonoticfinished after 0.23 second(s) with 0Giving floating IP to rhelcaDUMP size: 21M /runc/containers/xonotic/imageTransferring DUMP to rhelcaDUMP transfer time 1.36 secondsrunc restored xonotic successfully
Container Migration All Around The World 14/25
. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .
Singapore
Montreal
. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Remote Migration to Singapore# migrate xonotic rhelsg truerunc checkpoint --pre-dump --image-path parent xonotic finished after0 second(s) with 0PRE-DUMP size: 351M /runc/containers/xonotic/parentTransferring PRE-DUMP to rhelsgPRE-DUMP transfer time 11.75 secondsrunc checkpoint --image-path image --parent-path ../parent xonoticfinished after 0.19 second(s) with 0Giving floating IP to rhelsgDUMP size: 21M /runc/containers/xonotic/imageTransferring DUMP to rhelsgDUMP transfer time 3.15 secondsrunc restored xonotic successfully
Container Migration All Around The World 17/25
. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .
Singapore
Strasbourg
. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Remote Migration to Strasbourg# migrate xonotic rhelfr truerunc checkpoint --pre-dump --image-path parent xonotic finished after3 second(s) with 0PRE-DUMP size: 351M /runc/containers/xonotic/parentTransferring PRE-DUMP to rhelfrPRE-DUMP transfer time 13.76 seconds runc checkpoint --image-pathimage --parent-path ../parent xonotic finished after 0.59 second(s)with 0Giving floating IP to rhelfrDUMP size: 21M /runc/containers/xonotic/imageTransferring DUMP to rhelfrDUMP transfer time 3.88 secondsrunc restored xonotic successfully
Container Migration All Around The World 20/25
. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Further Optimization▶ Lazy Migration
Based on userfaultfdSince CRIU 3.5 (released September 2017)Requires runC git master branch
Container Migration All Around The World 22/25
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Optimizations - Post-Copy
memory
memoryprocesstableentrySource
System
DestinationSystem
InitiateMigration
QuiesceProcess
transfer transfers on page fault
ResumeProcess
Time
processtableentry
Migration DurationProcessDowntime
FinishMigration
Figure: Post-Copy Migration
Container Migration All Around The World 23/25
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
Post-Copy Migration# migrate xonotic rhel02 true truerunc checkpoint --pre-dump --image-path parent xonotic finished after 0 second(s)with 0PRE-DUMP size: 351M /runc/containers/xonotic/parentTransferring PRE-DUMP to rhel02PRE-DUMP transfer time 0.1 secondsrunc checkpoint --image-path image --parent-path ../parent --lazy-pages--page-server localhost:27 --status-fd /tmp/postcopy-pipe xonoticReady for lazy page transferrunc checkpoint --image-path image --parent-path ../parent --lazy-pages--page-server localhost:27 --status-fd /tmp/postcopy-pipe xonotic finished after0.08 second(s) with 0Giving floating IP to rhel02DUMP size: 204K /runc/containers/xonotic/imageTransferring DUMP to rhel02DUMP transfer time 0.15 secondsrunc restored xonotic successfully
Container Migration All Around The World 24/25
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
...
.
The end.Thanks for listening.
▶ https://rhelblog.redhat.com/2017/10/12/container-migration-around-the-world/
▶ https://people.redhat.com/areber/criu/2017-xonotic-migration-all-around-the-world.mp4
Container Migration All Around The World 25/25
