Senlin Clustering Service Deep DiveEthan Lynn, IBM, IRC: elynn

Qiming Teng, IBM, IRC: Qiming

Yanyan Hu, IBM, IRC: yanyanhu

Agenda

• Why did we start senlin project

• Senlin overview

• Senlin status and roadmap

Why did we start senlin

Background

• Why Senlin?• CNCP (IBM research project)

• Heat AutoScaling Evaluation

• Heat AutoScaling Rework

• Customer Engagement

Gap Analysis

Heat AutoScaling ReworkTalk on Atlanta Summit

Evaluation Report

Background

• Why Not Heat?• Heat AutoScaling today is based on

AWS design, with gaps

• Heat's mission• Defined in project governance

• It's NOT about autoscaling, NOT about HA, ...

• It's NOT Heat's job to provide new service(s)

To orchestrate composite cloud applications

using a declarative template format through an

OpenStack-native REST API.

Heat Mission Statement

https://github.com/openstack/governance/blob/

master/reference/projects.yaml

• Proposal:

https://wiki.openstack.org/wiki/Heat/AutoScaling

• API Design: http://docs.heatautoscale.apiary.io/

• Targeting at many use cases not covered by Heat

today

AutoScaling: a new design

Heat as a Compiler

YAML

Template

Heat

Nova

Cinder

Neutron

Keystone

• CREATE

• GET

• UPDATE

• DELETE

Blueprints on Reworking Heat AutoScaling

BP Priority Description

autoscaling-

api-resources

high Heat resources invoking AS APIs

as-api-group-

resource

high ScalingGroup resource wrapping AS API's

group functionality

as-api-policy-

resource

high ScalingPolicy resource wrapping AS API's

policy functionality

as-api-

webhook-res

high Webhook resource wrapping AS API's

execution of webhooks

autoscaling-

api-client

high A python client for Heat to interact with AS

API

autoscaling-

api

- A separate service for the implementation

of autoscaling w/ Heat

as-engine - A separate engine/service for autoscaling

support AS API

as-engine-db - A DB dedicated to autoscaling, using

schema created in as-lib-db

as-lib - A separate module to be used by the AS

service

as-lib-db - A DB for autoscaling bookkeeping

What Do We Really Need?

Scalable

Load-Balanced

Highly-Available

Manageable

......

of any (OpenStack) objects

- What is missing from

OpenStack?

A Clustering Service

- Auto-scaling?

Just one of the usage

scenario of a cluster.

- Auto-Healing (HA)?

Just another usage

scenario.

- We can address the

concerns by making policies

orthogonal

Senlin Overview

Senlin Architecture

REST RPC

Profiles

Policies

Senlin Features

• Profiles: A specification for the objects to be managed

• Policies: Rules to be checked/enforced before/after actions are performed

10

(others)

Senlin

Nova

Docker

Heat

Ironic BareMetal

VMs

Stacks

Containersplacement

deletion

scaling

health

load-balance

affinity

Policies as Plugins Profiles as Plugins Cluster/Nodes Managed

Senlin Server Architecture

openstacksdk

identity

compute

orchestration

network

...

engineengine lock

scheduler

actions

nodecluster

service

registry

receiverparser

drivers

openstack

dummy

(others)

dbapi

rpc client

policies

placement

deletion

scaling

health

load-balance

affinity

receiver

webhoook

MsgQueue

extension points

for external

monitoring

services

extension points

facilitating a

smarter cluster

management

extension points to talk to different

endpoints for object CRUD operations

extension points for interfacing

with different services or clouds

profiles

os.heat.stack

(others)

os.nova.server

senlin-api

WSGI

middleware

apiv1

Senlin Operations (Actions)

• Cluster• CREATE• DELETE• UPDATE• LIST• SHOW• ADD_NODES• DEL_NODES• SCALE_OUT• SCALE_IN• RESIZE• POLICY_ATTACH• POLICY_DETACH• POLICY_UPDATE

12

Node• CREATE• DELETE• UPDATE• LIST• SHOW

Policy• CREATE• UPDATE• DELETE• LIST• SHOW

Profile• CREATE• UPDATE• DELETE• LIST• SHOW

Action• LIST• SHOW

Event• LIST• SHOW

Receiver• CREATE• DELETE• LIST• SHOW

profile/cluster/node operations

13

$ senlin profile-create -s <spec> <name>$ senlin profile-list$ senlin profile-show <profile>$ senlin profile-delete <profile>$ senlin profile-update -n <name> <profile>

$ senlin cluster-create -p <profile> -n <size> <name>$ senlin cluster-delete <cluster>$ senlin cluster-list$ senlin cluster-show <cluster>

$ senlin node-create -p <profile> -c <cluster> <name>$ senlin node-delete <node>$ senlin node-show <node>$ senlin node-join -c <cluster> <node>$ senlin node-leave <node>

$ senlin cluster-node-add -n <nodes> <cluster>$ senlin cluster-node-del -n <nodes> <cluster>$ senlin cluster-node-list <cluster>

Profile CRUD

Cluster

CRUD

Node CRUD

Cluster

Membership

policies operations

14

$ senlin policy-create -s <spec> <name>$ senlin policy-list$ senlin policy-show <policy>$ senlin policy-delete <policy>$ senlin policy-update -n <name> ... <policy>

$ senlin cluster-policy-attach <cluster> <policy>$ senlin cluster-policy-detach <cluster> <policy>$ senlin cluster-policy-list <cluster>$ senlin cluster-policy-show <cluster> <policy>$ senlin cluster-policy-enable <cluster> <policy>$ senlin cluster-policy-disable <cluster> <policy>$ senlin cluster-policy-update <cluster> <policy>

Policy

Management

Cluster-Policy

Bindings

Other Useful Operations

15

$ senlin cluster-scale-in [-c <count>] <cluster>$ senlin cluster-scale-out [-c <count>] <cluster>$ senlin cluster-resize <options> <cluster>

$ senlin profile-type-list$ senlin profile-type-show <type>$ senlin policy-type-list$ senlin policy-type-show <type>

$ senlin action-list$ senlin action-show <action-id>

$ senlin event-list$ senlin event-show <event-id>

Cluster

Resize

Profile Type

Policy Type

Action

Event

senlin cluster-resize

16

usage: senlin cluster-resize [-c <CAPACITY>] [-a <ADJUSTMENT>] [-p <PERCENTAGE>][-t <MIN_STEP>] [-s] [-n MIN] [-m MAX] <CLUSTER>

Positional arguments:<CLUSTER> Name or ID of cluster to operate on.

Optional arguments:-c <CAPACITY>, --capacity <CAPACITY> The desired capacity the cluster.-a <ADJUSTMENT>, --adjustment <ADJUSTMENT>

A positive integer meaning the number of nodes to add,or a negative integer indicating the number of nodesto remove.

-p <PERCENTAGE>, --percentage <PERCENTAGE>A value that is interpreted as the percentage of sizeadjustment. This value can be positive or negative.

-t <MIN_STEP>, --min-step <MIN_STEP>An integer specifying the number of nodes foradjustment when <PERCENTAGE> is specified.

-s, --strict A boolean specifying whether the resize should beperformed on a best-effort basis when the new capacitymay go beyond size constraints.

-n MIN, --min-size MIN New lower bound of cluster size.-m MAX, --max-size MAX New upper bound of cluster size. A value of -1

indicates no upper limit on cluster size.

Adjustment Types

• EXACT CAPACITY

• CAPACITY CHANGE

• CAPACITY PERCENTAGE

Optional Modifiers

• Strict vs. Best-Effort

• Minimum Step

Dealing with Constraints

• MIN_SIZE

• MAX_SIZE

Senlin Roadmap

Senlin Journey

Talk on China

OpenStack/Docker

community joint

meetup

2015-04-25

• IBM internal

announce

• Email to OpenStack

Heat Core team

2015-02-10

Project

accepted to

StackForge

2015-03-16

Community

Announcem

ent

2015-03-26

Git repo

migrate to

github.com

2014-12-25

Initial Git

Repository

inside

CRL

2014-12-10

First weekly

IRC

meeting

2015-06-10

Talk on

OPNFV

meeting

2015-07-08

Start work

with TOSCA

std team

2015-07-09

Talk on 2nd China

OpenStack

Hackathon

2015-08-20

Senlin

dashboard

project

initiated

2015-08-24

Talk on Tokyo

Summit -- Scaling

for Containers

2015-10-29

Senlin

docs

went

online

2015-12-03

Project

accepted

into Big

Tent

2015-11-11

Mitaka

Mid-Cycle

Meetup

2016-01-11

Vancouver

Heat design

summit

2015-05-20

2015

Jan2015

Jul2016

Jan

2015

Apr

2015

Feb

2015

Mar

2015

May

2015

Jun

2015

Oct

2015

Aug

2015

Sep

2015

Nov

2015

Dec

2014

Dec2016

Feb

2016

Mar

2016

Apr

Version

1.0

released

2016-04-07

Senlinclient

0.4.0

released

2016-03-19

Mitaka Features

• API Changes and API Reference

• OpenStackClient Support

• Heat Resources

• Health Management primitive

• User/Developer Documentation

• Senlin Dashboard

Newton Outlook

API micro-version

Health management

Container Cluster

Testing:

API/Scenario/Performance...

Additional Clustering Operations

Queue (e.g. Zaqar) type Receiver

Notifications to other services

Batched Operations

User Defined Actions

Access Control

More communications, more collaborations

Relation To Other Projects

21

Senlin

Ceilometer

Heat

Nova Cinder Neutron Swift Keystone

Horizon

Primitive Data Types

Complex Data Types

struct person {int age;char name[0];

}

person team[10]; // Senlin cluster of Heat stacks

// Senlin cluster of nova servers

// Heat stack containing senlin clusters

Mistral MagnumSaharaMonasca Tacker Zaqar

Senlin provides the array data type for cloud programming

We Are Hiring,

Standard

OpenStack

Customer

IBMTacker

Contributors

Zaqar

Questions?

Thank You!