Introduction to automated environment management with Docker Containers - for platform specialists...

Lucas Jellema

AMIS Platform SIG - 1 October 2015

Introduction to automated environment management with Docker Containers - for platform specialists and developers

2

Overview

3

Overview

• Docker – what is behind the hype?• A grasp on terminology such as Container, Build, Ship and Run

– Images, registry, micro services, • Enough to participate in the discussion• How to get started and going with Docker

– On Windows (or Mac OS)• What can Docker mean to me

– In my personal R&D environment– In my daily work situation

• What’s Oracle’s position regarding Docker?• What is Docker’s current status and near by future• Handson resulting in

– A Docker enabled environment– Several containers running (Ubuntu, Wordpress, Oracle 11g XE DB, JDeveloper)– A stepping stone for creating additionals containers and interdependencies

4

What can Docker give us (developers/platform specialists)

• An efficient way – in terms of work and physical machine resources – to rapidly create, easily manage isolated environments– That can be created and deleted easily– That start and stop rapidly– That can be exchanged simply– That consume minimal system resources

• Use cases:– Quickly try out some software (release) – without upsetting your laptop– Easily create and distribute a complete training environment– Prepare an environment for deployment to a cloud environment– Locally construct and test a potentially complex environment that is then transfered to

a shared infrastructure– Implement a micro services infrastructure– Emulate/Test complex clusters on a single host

5

Docker - Run

6

Demo – Run our first container

dockerhost

port 80

IP 172.17.0.6

7

Demo – Run a second container based on same image

8

Demo – two running NGINX containers; port forwarding for one

port 80

port 80

port 90

IP 172.17.0.7

IP 172.17.0.6

9

Demo: run container in interactive mode using shell

dockerhost

/tmp/mynewfile.txt

10

Demo: restart container and attach to locate my file

dockerhost

/tmp/mynewfile.txt

11

Final status demo: Running Multiple Containers in a Single Linux Host

port 90 port 80

port 80

/tmp/mynewfile.txt

/var/lib/docker/aufs/diff/… 6af69dd33af4…/tmp/mynewfile.txt

12

Run

• Docker Container runs Linux – as does the host• Container is isolated - feels as

stand alone run time environment– Directory structure, IP address, users and groups

• Shared resources with underlying host (and therefore other containers)– memory, CPU, host

• Light weight:– Quick starting up and stopping– Leverages underlying Linux, only adds what is different/additional

• Clusters of containers– Dynamic adding/removing containers from clusters can be done very quickly (Google)– Especially when containers are stateless – (no shared session state in containers; possibly in joint cache, shared file system or

NoSQL database)• Management tools – to monitor and manage individual containers and

clusters of containers (dynamically scale up/scale down)

Docker HostDocker Container

Docker Container• ip address• directories & files• users & groups• process table

13

When it walks and talks like a Virtual Machine…

• Virtual Machine vs Container

14

Containers vs Virtual Machines

• Containers do not run through a hypervisor– They are logically isolated from the host and each other– They run the same kernel as the host (i.e. Linux!)

• Containers share resources (the kernel) with host– the rest of the OS can be unique per container (namespaces on host in mounts,

process table, network resources)• Containers are based on images defined in a layered file system; all

containers based on the same image share the image definition– Adding a copy-on-write file system layer– 50 containers based on the same image require same file system resources as 1

container – until they start changing or creating files• Containers start, stop, restart in seconds• Containers do not have physical resources (memory, CPU) pre-allocated

– Resources are allocated at run time

15

Docker is a high level API on core Linux Kernel features

• Solaris Zones and Windows Server offer similar isolation capabilities

16

Micro Services

• Architect the application into a set of collaborating services. • Each service implements a set of narrowly, related functions. • The services are elastic, resilient, composable, minimal, and complete.• Services communicate using standard protocols such as HTTP/REST• Services are developed

and deployed independently of one another.

• Each service manages its own state

17

Micro Services

• With Docker, each Micro Service is implemented with a single container– The micro service is not just encapsulated functionality that needs to be deployed

onto some platform (such as an ESB or BPEL engine) – instead it ships complete with the fully configured engine that runs in the

standardized container platform• All you need to run is:

– Start container. Period.

Linux Host + Docker Engine

18

A more complex network of microservices…

19

Micro Services and containers

• Containers can run on the same host– Or on different hosts

• Containers can be run as a single node– Or scale up to run as many nodes

• Containers can easily be replaced by other containers– That can be connected to in the same way

dockerhost

my-little-container

my-tiny-container

my-data-container

http

Simple Docker UI

GUIs for monitoring & managing a Docker runtime environment

21

GUIs for monitoring & managing a Docker runtime environment

• Dockland - Yet another docker web UI– https://github.com/dynport/dockland

• Shipyard – a web UI for Docker– https://github.com/ehazlett/shipyard

• DockerUI – Web interface to interact with the Docker Remote API– https://github.com/crosbymichael/dockerui

• Simple Docker UI– https://github.com/felixgborrego/docker-ui-chrome-app

• Panamax– https://github.com/CenturyLinkLabs/panamax-ui

22

Docker – Ship and Run

23

Demo ShipRun MySQL and Wordpress

24

Running Wordpress instance by pulling two public images

port 8080 port 80

dockerhostvmIP: 10.10.10.29

Docker Hub

25

Commit container as image and push to registry

dockerhostvm

Docker Hub

/tmp/mynewfile.txt

Dockersig-trial:1.0

26

Image published on public Docker Hub registry

27

Run container based on my published image

dockerhostvm

Docker Hub

/tmp/mynewfile.txt

Dockersig-trial:1.0

Dockersig-trial:1.0

28

Containers in the world of physical logistics

29Docker Containers Ship microservice – app, libs, configuration, platformready to run on any Linux Host+Docker

30

Docker Registry

• Images can be published to Public and Private Registries – Docker Hub is the default registry– Docker Hub contains official repositories from many projects and vendors– Private Registries can be created in the cloud and on premises

• Containers can be started from such images

31

Public Docker Registry

Docker Hub

Registry

push

Private Docker Registry

Docker Hub

pull &run

push

pull &run

32

Docker &Cloud

Public Docker Registry

Docker Hub

pull &run

33

How to Ship a Stand-Alone product?

• Create Installers + Configuration Instructions?• Make your product success dependent on platform configuration and OS

settings?

• Ship as a container – everything set up and ready to run!

• For example:

– RubiconRed – Preferred way to deliver their tool MyST: as Docker Container (image)

34

Docker – Build, Ship and Run

35

Demo Build

36

Demo run container after build

37

Demo Build – slightly more advanced Dockerfile

38

Demo – run image created in build process

39

Building a Docker Container

• The Build Steps are prescribed in Dockerfile– Could be called the container recipe

• Docker commands specify: – Start from Base Image– Perform commands such as ADD, COPY, RUN

that modify the starting situation– The Build Context contains the contents of the folder on the Docker host that

contains the Dockerfile (including sub folders)• This can become quite large• During build – files can be downloaded over HTTP using standard wget (but not copied from the host outside

the build context)• All files added to the container become part of its history (and increase the size)

– Specify the default CMD to run when the container is started

– Optionally: tag the container as image, push it to registry and start one or more containers from the new image

40

Building a Docker Container

• Docker files specifies all build steps– With fairly low level commands

• Start from base image - each step in the Docker Script adds a layer

• A layer is– A logical ‘savepoint’ in the container history

• That marks an intermediate ‘image’

– A physical directory somewhere on the Docker Host

Base ImageUbuntu 14.04

FROM Ubuntu:14.04COPYRUNWORKDIRRUNEXPOSECMD

COPY

RUN

RUN

Final Image

41

Run a Docker Container

• When a Docker Container is run based on an image then all these “physical” layers are combined into a “logical” read-only file system– Union file system– Copy-on-write– Like a stack: higher layers may override lower layers

• A writable layer is added for the running container

Base ImageUbuntu 14.04

COPY

RUN

RUN

Base ImageUbuntu 14.04

COPY

RUN

RUN

Writable Layer

run

Base ImageUbuntu 14.04

COPY

RUN

RUN

Writable Layer

run

42

StandardImage,

locally built

Build

• In addition to 10Ks of reusable images to start containers from• There are a zilion Dockerfiles to leverage for building images

– Download script– Add software packages and installers (because of license reasons)

– Tweak the script to fit your own needsOR (preferably)

– Run the script, create a local image and then create your own Docker File that takes this image as its starting point

Your OwnDockerfile

Your TweakedImage

43

“Docker” Search results on GitHub

44

For example: build container for Oracle WebLogic

• Clone from GitHub to Docker host– Dockerfile– Shell scripts– Supporting files

• Download RPMs for– JDK 8– WebLogic 12.1.3

• Docker Build• Optionally use second

Docker file on top ofWebLogic image tocreate a WLS Domain

download and

add to build contextStandard

Oracle WebLogicImage,

locally built

45

Build File for WebLogic

Base ImageOraclelinux:7

RUN

COPY

RUN

COPY

COPY

COPY

46

Build File for WebLogic (2)

Base ImageOraclelinux:7

RUN

COPY

RUN

COPY

COPY

COPY

RUN

RUN

RUN

RUN

RUN

47

Base ImageOraclelinux:7

RUN

COPY

RUN

COPY

COPY

COPY

RUN

RUN

RUN

RUN

RUN

Turn container into image

Base ImageOraclelinux:7

RUN

COPY

RUN

COPY

COPY

COPY

RUN

RUN

RUN

RUN

RUN

docker commit <container-id> weblogic:12.1.3-dev

48

Build File for WebLogic Domain

weblogic:12.1.3-dev

Base ImageOraclelinux:

7

RUNCOPY

RUN

COPYCOPYCOPY

RUNRUNRUNRUNRUN

Base ImageWeblogic:12.1.3-

dev

COPY

RUN

Base ImageOraclelinux:7

RUN

COPY

RUN

COPY

COPY

COPY

RUN

RUN

RUN

RUN

RUN

49

Build File for WebLogic Domain (2)

Base ImageWeblogic:12.1.3-

dev

COPY

RUN

RUN

RUN

Base ImageOraclelinux:7

RUN

COPY

RUN

COPY

COPY

COPY

RUN

RUN

RUN

RUN

RUN

50

Tag container as reusable image

Base ImageWeblogic:12.1.3-

dev

COPY

RUN

RUN

RUN

Base ImageOraclelinux:7

RUN

COPY

RUN

COPY

COPY

COPY

RUN

RUN

RUN

RUN

RUN

docker commit <container-id> weblogic-basedomain:12.1.3

oraclelinux:7

weblogic:12.1.3-dev

weblogic-basedomain:12.1.3

51

Build Microservice based on WebLogic plus Custom App

download custom app EAR

add to build context

Microservice: custom app

oraclelinux:7

weblogic:12.1.3-dev

weblogic-basedomain:12.1.3

WLST deploy EARWLST config JMSWLST config data sources Microservice:

database

52

Automated Configuration Management

• Use of Shell Scripting to create | compose | configure environments is not exactly the latest fashion

• Declarative, automated configuration management tools have us specify what we need and then make that happen– No scripting– Cross platform– Parametrized

– Leveraging public catalogs of environment definitions

Container Build process• Regular Docker Build

– From base image– Add Puppet support– Add Puppet Manifests &

Modules

• Start Container– Optional: Map Volume from

host with large files – Run Puppet to apply Manifests– Stop Container, Tag as Image

dockerhostvm

Dockerfile

my-base-container

/files

/puppet

/files

volu

me

1

Very big files

Advanced Docker Container Build using Puppet

2

3

45 4

21

3/puppet Manifests/Modules

5

54

Notes on Using Puppet with Docker

• After applying Puppet – the container can be stopped, tagged and used as base image for next Docker Build– That could EXPOSE, ENV, CMD or ENTRYPOINT

• With some workarounds, Puppet apply can be made to run during Docker Build (with RUN in Dockerfile)– Less control over build context– No Volume mapping from host

• There are Puppet Modules to use for automating the build pipeline of Docker (leveraging the Docker API)– To install Docker, build container, create and ship an image, run container

• What applies to Puppet by and large applies to similar tools such as Chef, Salt and Ansible

• Puppet Modules are available for many Oracle Database & Fusion Middleware configuration management tasks– Oracle Database (EE, SE, XE)– WebLogic, SOA Suite, OSB, BPM Suite, WLST– JDK, Opatch, VirtualBox, GlassFish, Hudson, Maven

55

Continuous Delivery in the Container Era

• What if development team delivers the complete, runnable environment – instead of fine grained artifacts that need to be deployed to an environment?

56

The Image is Delivered

• Source code is built into testable and deployable units• When build and unit test is complete, a Docker Image is built• This image is the unit of delivery

– It is independent of / unimpeeded by the run time platform

57

Automated Workflow

• Source code commit triggers build, test, ship and deploy of Docker Container– For example: Docker Hub can be hooked into GitHub with build trigger

58

Docker – Concepts in Perspective

Dockerfile Image Docker HubRegistry

Internet

Container

buildpush

pull

run commit

Docker Private

Registrypush

Text fileSize = ~ KBInstallation & config instructions

Composed of many read-only layersTypical size = ~ hundred(s) MBCan have multiple versions (akin Git tags)

Shares the image’s read-only layers1 private writeable layer (copy-on-write)Initial size = 0 bytesCan be stopped, started, paused, etc.

Free public hosting

Source: Sébastien Bonami, IT Engineering Student and David Lauzon, Researcher (DevOps 2015)

59

Docker – internal structure

Source: Sébastien Bonami, IT Engineering Student and David Lauzon, Researcher (DevOps 2015)

Docker Daemon Container 1

Host OS Kernel

Docker Storage Backend Container 2 ...

Hardware

Setups & manage the Linux containers.

Stores the image and container’s data layers locally.

60

Quick Docker Recap

61

Image and Container Specifics

Container

Base ImageUbuntu 14.04

COPY

RUN

RUN

Writable Layerrun

commit

Container “state”

diff

history

tagremoveinspect

create save tar load …

pullregistry

62

Image and Container Specifics

Container

Base ImageUbuntu 14.04

COPY

RUN

RUN

Writable Layerrun

startattach(un)pausekillstoprestartremoveinspect

logs

commit

Container “state”

diff

export tarimport

…

Flattened, no image details

63

Container Details & Operations

Container

web

Containerdblink

docker run –d –it

<image-id> /bin/bash

Containerxxx

link

link

808080

/tmp/filesShared Files

/data

/host_files

Docker storage

/softwareShared Files

/repos/repos

1

2

3

4

1

2

3

4

2

4

–p 8080:80

-v /data -v /tmp/files:/host_files

-volumes-from xxx

--name web --link db:db1 –link xxx:web_xxx

64

Run GUI applications in Docker Container

Container

docker run –d –it

-v /tmp/.X11-unix:/tmp/.X11-unix

-e DISPLAY=$DISPLAY

<image-id> /bin/bash

/tmp/.X11-unix

/tmp/.X11-unix

dockerhost

GUI applications

65

Docker and Windows

• Docker sits on Linux Containers– Windows Server 2016 will have containers too – and Docker will sit on those as well– However, today, Docker does not run on Windows

66

Docker cannot run on Windows- directly, without Linux VM

dockerhost

Container

Container

Container

67

Vagrant to the rescue

dockerhost

Container

Container

Container

68

Vagrant to the rescue

• Based on simple declarative definitions…• Vagrant provisions environments through various providers

– VirtualBox, VMware, AWS• Subsequently, provisioning (‘configuration management’) using shell

scripts, Chef, Anisble, Salt or Puppet• Vagrant supports Docker

– Create Docker Host VM, Build | Run | Manage Container• Vagrant makes host-container folder mapping and networking quite easy

dockerhost

Container

69

Vagrant Docker Provisioning

• Vagrantfile defines the Container to run – including name and initial command and also synched folders (i.e. host <=> container mapping)

• Dockerfile contains build recipe for the Container we want to build• DockerHostVagrantfile describes the VM to be used as Docker Host

Vagrantfile

DockerHostVagrantfile

Dockerfile

70

Vagrant Docker Provisioning

dockerhostvm

Vagrantfile

DockerHostVagrantfile

Dockerfilemy-little-container

build process

Docker Hub

ubuntu:14.04

/u01/readme.txt

Vagrant Boxes

ubuntu/trusty64

71

Vagrant with Docker Folder Mapping

dockerhostvm

Vagrantfile

DockerHostVagrantfile

my-little-container

/vagrant

/vagrant

/host_temp

/host_data

/var/lib/docker/docker_generatedId

72

Vagrant on Windows – other options

• Boot2Docker– a lightweight Linux distribution based on Tiny Core Linux made

specifically to run Docker containers. It runs completely from RAM, weighs ~27MB and boots in ~5s

Windows or Mac host

73

Vagrant on Windows – other options

• Docker Toolbox (since August 2015) replaces Boot2Docker– Contains Docker Client for Windows, Kitematic (Docker GUI, alpha release), Docker

Machine, Docker Engine and leverages Oracle VirtualBox– Still uses Boot2Docker Linux Distribution to run containers– Currently no support for Windows host-container folder mapping – No support for GUI in containers

74

Oracle and Docker

• Oracle Linux 6 and 7 Images • WebLogic certified on Docker

– Official “Docker Build-scripts in GitHub to create images”• Solaris Zones as Docker backend• Participate in OCI• Docker on Oracle Cloud??

75

Open Container Initiative

76

Summary

• Build• Ship• Run• Docker – started in 2013• Open Container Initiative

77

Hands On

• Copy workshop resources from USB stick to c:\docker_sig• Install VirtualBox 5.0 and Vagrant 1.7.4• Use vagrant up to build and run a simple Docker Container

– This will make Vagrant create the Linux Docker Host VM• Use vagrant ssh to open a terminal in the Docker Host and start your

Docker adventures– Manage containers and images– Use link, port forwarding and volume configurations

• Use vagrant up to build a Docker Container with Puppet support; then start the container and use Puppet to perform advanced configuration

• Run a container with an Oracle Database inside• Run a container with a WebLogic Domain inside• Bonus:

– Create Linux Desktop Docker Host VM (to run GUI applications in Docker Container)