1

Copyright © 2017 Samsung. All Rights Reserved.

Dockerized Tizen Platform

2

Abstract

update

Cloud

Tizen  Pla.orm  ECO  System

Cloud

Management  (Update)  

Build    CI

Monitoring

Store  Data  (DB)

Docker

Container  ECO  System

Cloud  Infrastructure  

Dockeriza:on

3

Agenda

•  Platform Management •  Docker Introduction •  Embedded Container •  Dockerization •  Demo •  Challenges

4

PLATFORM MANAGEMENT Why We Research ?

5

•  Development, Deployment & Operations – Tizen Platform have life-cycles ? – Platform builder – F/W upgrade – Remote control

Platform Mgt. | Situation

6

•  In IoT devices, The platform should be –  Simple as a single application –  Faster to create application –  Easy to distribute –  Support remote control (update, monitoring, …) –  Safe for system failure

•  Docker can be a solution ?

Platform Mgt. | Challenges

7

DOCKER INTRODUCTION What is Docker.

8

Docker | Introduced in 2013

h1ps://blog.docker.com

9

Container -  Similar to VM

-  but, based on Linux system call (no Virtual OS)

-  OCI (Open Container Initiative)

-  Isolated name space with executable packages

Docker (Container platform) -  Build container image, Run container

-  ECO system for container image

-  Services (deploy, management)

Docker | Basic Concept

h1ps://www.docker.com/what-­‐container  

10

Docker | Basic Workflow

h1ps://docs.docker.com/engine/docker-­‐overview  

11

Orchestration Management -  Connection to cloud server

-  Device Clustering

Docker | Extended Workflow

container container container  A

container  B

replica9on

container  C

container  D

POD-­‐A

Docker

NODE-­‐A

container  E

Docker

NODE-­‐B

container  E

Docker

NODE-­‐C

container  E

Docker

NODE-­‐D

HW-­‐A HW-­‐B HW-­‐C

SERVICE Node  Cluster

SERVICE Cloud  Server

12

Docker | Services

Monitoring -  Host : CPU load, Memory, Disk Space, Running containers / Host UP time

-  Containers : CPU load, Memory, Disk I/O, Network I/O

Container Deploying -  Rolling update, Rollback

Logging -  System log, Containers log

Container Mgt. -  Scaling, load balancing

13

EMBEDDED CONTAINER Why We Use Docker.

14

Embedded Container | Concept

•  Docker in embedded device •  Container has a initializer (/sbin/init instead of /bin/bash) •  Running container with privileged permission

–  Full HW resources

Lightweight  Host  OS

Linux  Kernel

Embedded  Pla.orm    Container  

Docker

15

Embedded Container | Usage

container  A

container  A

container container container  A

container  B

Docker

Server  Infra  structure

container  A

container  A

container container container  A

container  B

Docker Lightweight  Host  OS

PlaHorm  +  App  Container  A  

Docker

Lightweight  Host  OS

PlaHorm  +  App  Container  A  

Docker

Cloud  Service

Cloud  Service U:lize

•  Service  oriented  (regardless  of  physical  device)  •  Homogeneous  app  containers  in  server  infra  

•  Device  oriented  •  Homogeneous  app  containers  in  different  device  •  Proper  to  IoT  system  

Exis:ng  Usage  for  Server New  Usage  for  Embedded  Device

16

•  Platform Managements with Docker

•  Tizen Platform as a Embedded Container

Embedded Container | Tizen Platform

Docker  service  features Pla.orm  management  tools

Build CreaMon/modificaMon

Deployment DistribuMon

Update Upgrade

Docker-­‐registry PlaHorm  store

Lightweight  Host  OS

Linux  Kernel

Tizen  PlaHorm  

Dockeriza:on

Linux  Kernel

Container  

Docker

17

DOCKERIZATION What We Are Trying

18

Cloud  Server  

Overall Architecture

Docker  Registry  

[Host  OS]    Mzen-­‐minimal  

/  bare-­‐os

[Container]  Mzen-­‐headless

dockzen-­‐  launcher

Linux  kernel

docker-­‐client

docker-­‐daemon

containerd    

OCI::runc

container-­‐shim

container-­‐ctr

docker-­‐engine swarm

ca-­‐cerMficate

dockzen  -­‐agent

update

security

monitor

Network  (Wi-­‐Fi)

kernel  +  Host

Mzen-­‐headless

kernel  +  Host

Mzen-­‐headless

Mul:media  fw

kernel  +  Host

Mzen-­‐headless

Voice  App

Create  images  (+  fw)

Create  images    (+App)

[Tizen  Pla.orm  Containers]

19

Dockerization | Kernel Patches

Enable  cgroup   “FATA[0001]  Error  starMng  daemon:  Devices  cgroup  isn't  mounted”  Fix  :  {  CONFIG_CGROUP_DEVICE=y,  CONFIG_CPUSETS=y,  CONFIG_BLK_CGROUP=y}

iptables  error “FATA[0002]  Error  starMng  daemon:  Error  iniMalizing  network  controller:  Error  creaMng  default  "bridge"  network:  Failed  to  program  NAT  chain:  Failed  to  inject  docker  in  PREROUTING  chain:  iptables  failed:  iptables  -­‐-­‐wait  -­‐t  nat  -­‐A  PREROUTING  -­‐m  addrtype  -­‐-­‐dst-­‐type  LOCAL  -­‐j  DOCKER:  iptables:  No  chain/target/match  by  that  name.”  Fix  :    {CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=y  }

rooHs  mount  error “error=oci  runMme  error:  container_linux.go:247:  starMng  container  process  caused  "process_linux.go:359:  container  init  caused  \"rooHs_linux.go:54:  mounMng  \\\"  to  rooHs  \\\"  at  \\\"  caused  \\\  such  device\\\"\"“”  Fix  :  {CONFIG_POSIX_MQUEUE=y}

cgroup  memory  path  error “ERRO[0187]  containerd:  noMfy  OOM  events                                  error=cgroup  path  for  memory  not  found  panic:  standard_init_linux.go:175:  exec  user  process  caused  "exec  format  error“”  Fix  :  {CONFIG_MEMCG=y,  CONFIG_MEMCG_SWAP=y,  CONFIG_MEMCG_KMEM=y}

Enable  Overlayfs Fix  :  {CONFIG_OVERLAY_FS=y}

docker-­‐runc  keyring  failure “error=oci  runMme  error:  container_linux.go:247:  starMng  container  process  caused  "process_linux.go:359:  container  init  caused  \"could  not  create  session  key:  funcMon  not  implement“  Fix  :  enable  keyctl  syscall  compaMbility  for  32bit  with  64bit  kernel  {CONFIG_KEYS_COMPAT}  

Host  OS

Kernel Docker  &  FW

Container

•  Kernel Has Docker Dependencies

20

•  Required Packages in Host-OS –  rootfs – cgroup – Network – Certification – Docker & Frameworks

Dockerization | Host OS

[Host  OS]    Mzen-­‐minimal  

/  bare-­‐os dockzen-­‐launcher

Linux  kernel

docker-­‐client

docker-­‐daemon

containerd    

OCI::runc

container-­‐shim

container-­‐ctr

docker-­‐engine

swarm

ca-­‐cerMficate

dockzen-­‐agent update security monitor

Network  (Wi-­‐Fi)

Host  OS

Kernel Docker  &  FW

Container

21

•  Two Candidates –  Using Tizen subset (Tizen minimal) –  Create New for docker (BareOS)

Dockerization | Host OS

cerMficate  :  1MB

docker  :  52MB

wifi  netconfig  base    

:  about  60MB

　 Tizen  minimal BareOS Arch  type arm arm Size  (ROM) 123MB 66M Size  (RAM)   250MB  (run  dockerd  :  311MB) 53M  (run  dockerd  :  113M) Kernel  version 4.4.19 4.4.19 Docker  version  v1.13.1 v1.13.1 Init  system  systemd sysVinit Package  manager  tpk None Filesystem  ext4 ext4

cerMficate  :  1MB

docker  :  52MB

wifi  /  base    :  about  13MB

Tizen  minimal  RAM  Size  (113  MB)

BareOS  RAM  Size  (66  MB)

Host  OS

Kernel Docker  &  FW

Container

22

•  Manage docker life-cycle •  Manage Container life-cycle •  Monitoring APIs

Dockerization | Dockzen-launcher

command

API

docker  engine

dockzen-­‐launcher

device dockerd  connect config  file

systemd

content

state

Service  MainLoop

test

json  parser API  parser

dockzen-­‐agent

Host  OS

Kernel Docker  &  FW

Container

23

•  Binding as a Container •  Connection to Cloud

–  Manage Device uuid –  Authentication

•  Configure Update Policy

Dockerization | Dockzen-agent

web  connecMon  <<back-­‐end>>

API

agent

converter connect

dockzen-­‐agent

Server

dockzen-­‐launcher

Host  OS

Kernel Docker  &  FW

Container

24

Containerization | Initial Creation Host  OS

Kernel Docker  &  FW

Container

•  Platform Binaries to Tizen Container Image In Host PC 1.  Download platform binaries (https://download.tizen.org/) 2.  Loopback mount using mnt-img.sh

•  $ ./mnt-img.sh mount tizen-common_xxx_common-wayland-3parts-armv7l-artik.tar.gz

3.  Compress tarball •  $ sudo tar --xattrs -cvf ../[tar-name] .

In Target 4.  Docker-import

•  $ cat [tar-name] | docker import – [local-container-name]

5.  Push into Docker-Hub •  $ docker tag [local-container-name] [dockerhub-id]/[image-name] •  $ docker push [dockerhub-id]/[image-name]

25

•  Docker-Build with Dockerfile 1.  Install yum pkg-mgr

•  Add yum into base container image

2.  Case Study •  Add curl application à New Image

Containerization | Re-Creation

###      base_packages.repo  [base_packages]  name=base_packages  type=rpm-­‐md  baseurl=h1ps://download.Mzen.org/snapshots/Mzen/base/latest/repos/arm/packages  enabled=1  gpgcheck=0  sslVerify=false  

###  common_packages.repo  [common_packages]  name=common_packages  type=rpm-­‐md  baseurl=h1ps://download.Mzen.org/snapshots/Mzen/common/latest/repos/arm-­‐wayland/packages  enabled=1  gpgcheck=0  sslVerify=false  

v  yum  package  files ###  dockerfile  for  added  yum_pkg  and  exampleApp  ###  FROM  base-­‐image  #  install  yum  #  ADD  yum/yum_pkg  /usr/tmp/yum_pkg/  RUN  rpm  -­‐Uvh  -­‐-­‐nodeps  -­‐-­‐force  /usr/tmp/yum_pkg/*.rpm  ADD  yum/*.repo  /etc/yum.repos.d/  

###  install  rpm  pkg  and  exampleApp  ###  FROM  base-­‐image-­‐yum  #  install  packages  #  RUN  yum  install  curl  

Host  OS

Kernel Docker  &  FW

Container

26

•  Tizen uses Smack Security –  Extended attributes : security.SMACK64, security.capability –  Need to check xattr operations in docker

•  patch#1 : Capability error –  Failure in Tizen Container running –  Occurred permission error checking “CAP_MAC_ADMIN” –  In OverlayFS, upper layer can’t sync into lower layer as permission –  http://www.spinics.net/lists/linux-unionfs/msg00593.html

•  patch#2 : xattr copy error –  Failure in docker commands (commit, push, …) –  Extended attribute lists doesn’t be copied (in case of overlay, not overlay2)

Issues | Smack Security

27

•  /sbin/init (systemd) vs. /bin/bash –  Much discussions about “systemd in docker”

•  systemd requires privileged permission –  Initialize overall services regarding HW devices –  Necessary in Tizen container

•  Patches adding “-- privileged” –  Docker-build –  Docker-service

Issues | Privileged Container

28

•  Union file system –  Handled by layer architecture –  Avoid duplication and isolation

•  History –  Early 2013 : AUFS –  Late 2013 : Device Mapper –  Early 2017 : Overlay

•  Apply for Tizen –  OverlayFS –  Stability / mainline support –  Performance

Issues | Union File System

29

•  Security –  Need to minimize privileged permission

•  Fail safe –  Robust Host-os –  Container can be recovered(reboot)

•  Resource management –  Violation occurred in network resource –  CPU and memory is separated –  Disk is controlled by same journaling thread

Quality Inspection

30

DEMO What We Have Done.

31

Scenario Structure

Developers

Docker  Registry    (official  /  public)  

Docker  Registry  (public  /  private)  

Release

Service Server

Docker Registry

Build Tizen Container Image

Register Devices

Push New Image

Update Images

Dash-board •  Update •  Monitoring

<3rd Party Develop> Product Container Image

Embedded Device

(ARTIK710)

Web UI •  Image Repository

32

Demo Video Bring up

Update Dashboard

33

Demo Structures

Container  /  Mzen-­‐headless

dockzen-­‐launcher

Linux  kernel  4.4

docker-­‐engine

dockzen-­‐agent

Container  /  others

ARTIK7

[dockzen-­‐OS]    base  on  

Mzen-­‐minimal

docker  api

IPC

PoC  Server

H1p  Server

server

dockzen-­‐backend agent backend websocket

registry

registry-­‐web

H1p  Server

Docker  Registry  

container  mgt.

rest

Docker-­‐registry  Web  Dash-­‐board  Web

websocket

Target  Device

34

•  Packages – Artik7 boot&kernel – Host os

•  docker-engine •  docker framework

– Tizen container image

•  Instructions – Download boot&kernel – Download host os – Execute Tizen container image (only first time)

Development

35

CHALLENGES What We Try.

36

•  Improvement – Extend target device (raspi-3) – Create Tizen 4.0 reference container images – Optimize host-os embedded on Docker

•  Serviceability – Service to support Tizen docker is in development – 3rd Party can deploy Tizen docker in the future

Next

37

•  Github organization : https://github.com/dockzen – Docker source-code (patched for tizen)

•  https://github.com/dockzen/docker •  https://github.com/dockzen/containerd •  https://github.com/dockzen/runc

– Docker framework •  https://github.com/dockzen/dockzen-launcher •  https://github.com/dockzen/dockzen-agent

– Host-os : https://github.com/dockzen/dockzen-os – Artik7 kernel : https://github.com/dockzen/linux-artik7-docker

•  Docker-hub containers •  https://hub.docker.com/u/dockzen/

Contributing…