Build Your Very Own Private Cloud Foundry

BUILD YOUR VERY OWN PRIVATE CLOUD FOUNDRY

MATT STINECOMMUNITY ENGINEER, CLOUD FOUNDRY

MSTINE@GOPIVOTAL.COMHTTP://CLOUDFOUNDRY.ORG

TWITTER: @MSTINE

•CF Architecture - Quick Overview

•What is BOSH? Why BOSH?

•BOSH Architecture

•BOSH Releases

•BOSH Deployments

•CF to BOSH: cf-release

•[:vsphere, :openstack, :aws].each { |iaas| iaas.deploy_cf }

Agenda

Cloud Foundry Architecture(in FIVE minutes or less!)

Cloud Foundry ArchitectureRouter

DEA  Pool    

Apps

Build  Packs

Service  Broker

Service  Nodes

Messaging

User  Authen=ca=on  and  Authoriza=on

Cloud  ControllerHealth  Manager

•Load balancing

•Maintain routing table

•Access logs

Router

PLATFORM

MARKETING ORG FINANCE ORG ENGINEERING ORG

TESTDEV PROD TESTDEV PROD TESTDEV PROD

A S S S A S A S A A A

ROUTER

app1

.mktg

.acme.c

om

app3

.fina

nce.

acm

e.co

m

app2.eng.acme.com

UAA

•Token Server

•ID Server (User management)

•OAuth Scopes (Groups)

•Login Server

• UAA Database

• SAML support (for SSO integration) and Active Directory support with the VMWare SSO Appliance

•Access auditing

Cloud Controller

• App expected state

• Permissions/Auth

• Orgs/Spaces/Users

• Services management

• App placement

• App desired state convergence

• Auditing/Journaling

• Billing events

• Blob storage

Messaging (NATS)

•Non-Persistent messaging

•Pub/Sub

•Queues (app events)

•Directed messages (INBOX)

Droplet Execution Agent (DEA)

•Manage Linux containers (Warden)

• Process

• File system

• Network

• Memory

•Manage app lifecycle

•App log and file streaming

•DEA heartbeats (NATS to CC, HM)

DEA/Warden

Router

App

App

AppApp

DEA/Warden

Health Manager

•Maintains the actual state of apps

•Compares to expected state

•Sends suggestions to make actual match expected

App

App

App

DesiredState

App

App

App

ActualState

HARMONIZER

CloudController

DEA

NATSBulk API

SCHEDULER

NUDGER

NATS

NATS

Observes Observes

Schedule Observation

TriggerHarmonization

HealthManager

Service Broker

•Advertising service catalog

•Makes create/delete/bind/unbind calls to service nodes

•Requests inventory of existing instances and bindings from cloud controller for caching, orphan management

•SaaS marketplace gateway

CF Architecture: Why?

•Component Isolation

•Scalability

•Fault Tolerance

•Pre-provisioned Capacity (Containers / Warden)

How do we manage this thing?

We need a toolchain that can manage a large distributed system through:•Deployment

•Configuration Changes

•Updates/Upgrades (w/ minimal - zero - downtime!)

•Component Failure / Restoration of Service

•Scale Out / Scale In

•Across multiple IaaS providers: vSphere, OpenStack, AWS, and beyond...

What about?

BOSH!(Bosh Outer SHell)

What is BOSH?

Why BOSH?

•Provision services, not machines

•Eliminate bespoke automation on top of configuration management

•Enable continuous delivery of platform services

•Cloud-agnostic view of platform operations

•Holistic toolchain to “rule them all”

•How we manage Cloud Foundry in production!

BOSH Architecture

BOSH Architecture

Director

•Core orchestrating component

•Controls:

• creation/deletion of VMs, Disks, Networks

• software deployment

• lifecycle events for software and services

•Via:

• Cloud Provider Interface

• Director-Agent Interaction

Agent

•Carries out instructions from the Director

•Fetches packages/jobs from Blobstore

•Installs packages/jobs onto VM

•Starts/monitors jobs via Monit

Message Bus (NATS)

•Non-Persistent messaging

•Pub/Sub

•Queues (app events)

•Directed messages (INBOX)

Health Monitor

•Receives from Agent:

• VM health status

• Agent lifecycle events

•Triggers:

• Alerts (Email, CloudWatch, DataDog, PagerDuty, ...)

• Resurrection!

Stemcells

•“Pleuripotent” VMs

• Base VM filesystem image

• Never booted

• Currently ~Ubuntu 10.04.4 LTS

• Embedded BOSH Agent

Blobstore• Stores the content of BOSH Releases

• Uploaded via BOSH CLI

• Stored via Director

• Also stores:

• packages compiled by BOSH

• intermediate location for large payloads

• Options:

• EMC Atmos

• S3

• OpenStack Swift

• Simple/Local

Cloud Provider Interface (CPI)• current_vm_id

• create_stemcell

• delete_stemcell

• create_vm

• delete_vm

• has_vm?

• reboot_vm

• set_vm_metadata

• configure_networks

• create_disk

• delete_disk

• attach_disk

• snapshot_disk

• delete_snapshot

• detach_disk

• get_disks

Great! So how do I deploy my stuff with BOSH?

BOSH Releases

Redis

Hello BOSH Release!

Anatomy of a BOSH Release

•Release Manifest

• Package

•spec

•packaging

• Job

•spec

•monit

•templates

Release Manifest

redis-boshrelease/releases/redis-1.yml

Package

redis-boshrelease/packages/redis/spec

Packages

redis-boshrelease/packages/redis/packaging

Jobs

redis-boshrelease/jobs/redis/spec

Jobs

http://mmonit.com/monit

redis-boshrelease/jobs/redis/monit

Jobs

redis-boshrelease/jobs/redis/templates/bin/redis_ctl

Jobs

redis-boshrelease/jobs/redis/templates/conf/redis.conf.erb

OK. So what happens when I deploy?

BOSH Agent

Director creates VM from Stemcell

Director VM

CPI

Agent Boots

Director VM

Director Pings Agent

Director VMNATS

Director Assigns Job to Agent

Director VMNATS

Agent Fetches Blobs from Blobstore

BlobstoreVMHTTP

Agent Starts Jobs

Director VM

http://mmonit.com/monit

Now let’s go outside-in!

BOSH Deployments

Deployment Manifests

• The mapping of a BOSH release to infrastructure

• Specifies:

• Release

• Compilation VMs

• Update (canary) settings

• Network configuration

• Resource pools (VM “templates”)

• Jobs

• Job Properties

redis-boshrelease/examples/aws-solo.yml

redis-boshrelease/examples/aws-solo.yml

redis-boshrelease/examples/aws-solo.yml

redis-boshrelease/examples/aws-solo.yml

cf-release

Let’s follow one of our components...

• GoRouter

• Release Manifest

• Package

• spec

• packaging

• Job

• spec

• monit

• templates

Release Manifest

cf-release/releases/cf-release-134.yml

Package

cf-release/packages/gorouter/spec

Package

cf-release/packages/gorouter/packaging

Job

cf-release/jobs/gorouter/spec

Jobs

cf-release/jobs/gorouter/monit

http://mmonit.com/monit

Jobs

cf-release/jobs/gorouter/templates/gorouter_ctl.erb

Jobs

cf-release/jobs/gorouter/templates/gorouter_yml.erb

Cloud Foundry on vSphere

vSphere 101

• Minimum Lab HW:

• 6 Cores (12 threads)

• 64 GB RAM

• 4 TB Disk

• vSphere 5.1 (2 ESXi hosts, 1 vCenter)

• can get 30 day evals!

• iSCSI Data Store (http://www.openfiler.com/)

• Available IP addresses = 2X number of VMs

• https://github.com/cloudfoundry-community/vsphere-home-lab

BOSH Bootstrap

•Provide vCenter Credentials (via http://fog.io/)

•Provide Network Settings

•Provide vCenter Settings (Datacenter, Cluster, ...)

•Provide NTP Settings

•Provide MicroBOSH VM Sizing

•Provision MicroBOSH!

BOSH Bootstrap Deploy

BOSH Prepare CF

vSphere Deployment Manifest

vSphere Deployment Manifest

vSphere Deployment Manifest

vSphere Deployment Manifest

http://xip.io

BOSH Create CF

Cloud Foundry on AWS

AWS 101

•Requirements:

• AWS Credentials

• Capacity to provision 8 servers and 3 elastic IPs

• Approximately $0.42/hour you keep it running (us-east-1/us-west-1)

• Manually configured security group for CF

BOSH Bootstrap

•Provide AWS Credentials (via http://fog.io/)

•Provide AWS Region

•Provision MicroBOSH!

BOSH Bootstrap Deploy

BOSH Prepare CF

AWS Deployment Manifest

AWS Deployment Manifest

AWS Deployment Manifest

AWS Deployment Manifest

BOSH Create CF

Create ‘cf’ Security Group

Allocate Elastic IP for CF Router

Cloud Foundry on OpenStack

OpenStack 101

•Requirements:

• OpenStack Credentials

• Capacity to provision 8 servers and 3 floating IPs

• Manually configured security group for CF

BOSH Bootstrap

•Provide OpenStack Credentials (via http://fog.io/)

•Provide OpenStack Region (optional)

•Provision MicroBOSH!

BOSH Bootstrap Deploy

BOSH Prepare CF

OpenStack Deployment Manifest

OpenStack Deployment Manifest

OpenStack Deployment Manifest

OpenStack Deployment Manifest

BOSH Create CF

Create ‘cf’ Security Group

Allocate Floating IP for CF Router

Web Console UI

Edit src/main/resources/styx.properties

mvn clean package

cf push!

THANK YOU!

MATT STINECOMMUNITY ENGINEER, CLOUD FOUNDRY

MSTINE@GOPIVOTAL.COMHTTP://WWW.CLOUDFOUNDRY.COM

TWITTER: @MSTINE