Deep Dive into the Microsoft OpenStack CI Infrastructure (Alessandro Pilotti)

Premium community conference on Microsoft technologies itcampro@ itcamp14#

Deep dive into the Microsoft OpenStack CI infrastructure

Alessandro Pilotti

Cloudbase Solutions

Hyper-V MVP


Huge thanks to our sponsors & partners!


• OpenStack - what’s this?

• Hyper-V integration

• OpenStack continuous integration

• The Hyper-V CI infrastructure

• Undercloud and Overclouds

Agenda


An open source IaaS project–Apache 2 license

Managed by the OpenStack Foundation–More than 150 companies joined it including:

–AMD, Intel, Canonical, SUSE Linux, Red Hat, Cisco, Dell, HP, IBM, NEC, Vmware

Portable–Mostly used on Linux

Written in Python


• Essex– Apr 2012– Hyper-V removed

• Folsom– October 2012– Cloudbase Solutions involvement - Hyper-V re-included

• Grizzly– Apr 2013

• Havana– October 2013

• Icehouse– Apr 2014

Releases


• Distributed Components– Can be deployed on a single server or multiple servers

• Public RESTful API– Bindings available for various languages (including .Net)

• Queues (AMQP) for private APIs– Great scalability

• Relational DBs for storing the configuration– MySQL, PostgreSQL, SQLite (and SQL server)

• UI– CLI tools– Web dashboard

OpenStack Architecture


Compute (Nova)

Object Storage (Swift)

Block Storage (Cinder)

Image Service (Glance)

Networking (Quantum / Neutron)

Dashboard (Horizon)

Identity (Keystone)

Metering (Ceilometer)

Orchestration (Heat)

Components


Architecture 1


Architecture 2


• AMQP–RabbitMQ

–Apache Qpid

• API–RESTful services

• Database–Any Python SQLAlchemy provider

–Mostly MySQL

– SQL Server required various bug fixes

Component interaction


• FREE edition

– Full Hypervisor

–Minimum OS support

• Minimum impact on security updates, etc

• Windows Server 2012 R2

– Just enable the related role

• Windows 8.1

– For workstation / testing / development usage

Hyper-V Server availability


• Nova Compute driver

• Neutron plugin

• Cinder Volume driver

• Windows Cloud-Init

• Ceilometer Agent

• Heat templates

Hyper-V / Windows Openstack components


• Python Application installed as an application/service on the Hyper-V node.

• OpenStack compute utilizes key features baked into the Hyper-V Virtualization Platform

• Does not require windows clustering services

• Does not require shared storage

Hyper-V Nova Compute


• Hyper-V plugin is part of Quantum since Jan 2013– Project renamed in Neutron since Havana

• Supported network types:– VLAN

– Flat

– Local

• Plugin / agent model– Plugin runs in quantum-server (controller)

– Agent runs on each Hyper-V compute node

Neutron


Typical Neutron setup


• Quantum plugin / agent AMQP RPC protocol is compatible with OpenVSwitch

• You can use the OVS plugin with Hyper-V agents (or vice versa)– Limited to compatible L2 protocols: Flat / VLAN

• You can use the L3 and DHCP agents with the Hyper-V plugin– Using the OVS L2 agent on the networking nodes

• Supports ML2 plugin!

Neutron OVS interop


• We ported OVS to Hyper-V

Open vSwitch Windows porting


• De facto standard for SDN

• Great interoperability– Hyper-V / KVM / etc

• OpenFlow

• Tunnelling–VXLAN

–GRE

• Neutron OVS plugin–No need for a different one

Why OpenVSwitch?


• Great code review platform

–Gerrit

• Strong emphasis on unit tests

–Commits do not pass w/o tests

–Python is a dynamic language

• Syntactical and semantic validation needed

• Great continuous integration effort

– Lots of moving parts

How does OpenStack handle such complexity?


• Originally developed by Google for Android

• Stand-alone patch review system

• Manages:

–Git Repositories

–Code Review

– Event Stream

–Privileges (propose / merge code, manage gitrepositories)

Gerrit


Gerrit workflow


Gerrit UI


• Except special cases every contribution must have either:

–Blueprint

• for new features

• to be approved by core team

–Bug report

• for, ahem, bugs

• http://launchpad.net

Code contribution workflow


• Clone repository, e.g.:

– git clone https://github.com/openstack/nova.git

• Create a short lived branch:

– git checkout -b bug/12345678

– Do changes

– git commit

• Commit message must contain referefence to bug / bp

– git review

• Install with pip install git-review



• Gerrit triggers:– Pep8 / Flake8 tests

• Code conformity über alles!

• Very important with > 850 contributors

– Unit tests

– Integration tests with multiple env (e.g. Nova): • KVM

• Hyper-V

• VMWare

• XenServer

• etc



• Each CI results can vote +1 / -1 or be neutral

–At the moment both errors and failuresaccount for -1

• Hard to distinguish between them

• In case of failures users can issue a “recheck”





• At this stage code can be peer reviewed

• Every user can review the code– Leave comments

– Vote +1 / -1

• Each project (e.g Nova) has a peer elected team with “+2 rights”– Vote +2 / -2

• A commit requires 2x +2 and approval to be merged



• At this point the code is ready to be merged

–Another CI run is required

–Code base is changed since code submission

• If successful, the code will finally merge into the git repository!



Queue monitoring


• Microsoft contributes hardware for CI

–Around 200 servers ATM

–Currently based in Cambridge, MA

• CI support team provided by:

–Microsoft

–Cloudbase Solutions

Hyper-V CI


• Consistency– All projects should act the same way during code

reviews and offer the same interface to the devs

• Repeatability– system should run 100 or 500 times per day

without failing

• Availability– complete automation means that if it stops, the

whole project stops

OpenStack Requirements of automation


• Zuul

• Jenkins

• Undercloud– KVM compute nodes

• Overcloud– Devstack running on KVM VMs

– Hyper-V compute nodes

• Logs HTTP server

• Logstash

Hyper-V CI components


• Open source project

• monitors in-progress jobs for a set of related patches

• will pre-emptively cancel any dependent test jobs that would not succeed due to a failure in a dependent patch

Zuul


Zuul workflow

Gerritregisters commit

Zuul listens Gerrit

stream and records commit

Zuul clones referenced

git repo locally

Predefined Jenkins job(s)

started by Zuul

Zuul collects and pushes upstream

Jenkins job(s) results


pipelines:

- name: check

description: Newly uploaded patchsets enter this pipeline to receive an initial +/-1 Verified vote from Jenkins.

failure-message: Build failed. For rechecking only on the Hyper-V CI, add a review comment with recheck hyper-v

manager: IndependentPipelineManager

precedence: low

trigger:

gerrit:

- event: patchset-created

- event: change-restored

- event: comment-added

comment_filter: (?i)^(Patch Set [0-9]+:\n\n)?\s*recheck(( (?:bug|lp)[\s#:]*(\d+))|( no bug)|( hyper-v))\s*$

success:

gerrit:

verified: 0

failure:

gerrit:

verified: 0

Zuul yaml config 1/2


jobs:- name: ^.*$parameter-function: set_log_url

- name: 01_Reserve_Hyper-V_Node_01voting: truesuccess-pattern:

http://64.119.130.115/{build.parameters[ZUUL_CHANGE]}/{build.parameters[ZUUL_PATCHSET]}/

failure-pattern: http://64.119.130.115/{build.parameters[ZUUL_CHANGE]}/{build.parameters[ZUUL_PATCHSET]}/

success-message: Looks good to mefailure-message: Test run failed.

projects:- name: openstack/novacheck:

- 01_Reserve_Hyper-V_Node_01- name: openstack/neutroncheck:

- 01_Reserve_Hyper-V_Node_01

Zuul yaml config 2/2


• The de facto CI framework–Open Source, written in Java

–Provides a well done UI (CLI as well)

• Lots of available plugins

• Supports git and other SCMs

• Executes jobs:–on demand

– scheduled

–based on repository changes

Jenkins


• Jobs can be executed on

– Jenkins host

– Slaves (Linux, Windows)

• Jobs can be interdependent

– The CI executes multiple ones for each patchset

• Client / Server architecture

– Scales poorly

• Zuul integrates with Jenkins

– Executes jobs and waits for outcome

jenkins


• 5 jobs defined

1. Reserve Hyper-V Node 01

2. Reserve Hyper-V Node 02

3. Initialize Test Environment

4. Run Tempest Test

5. Clean Test Environment

• Return result is the result of executing jobs 1 to 4, result of job 5 is ignored

Jenkins CI jobs


Jenkins UI


• OpenStack’s integration tests suite

• Test types

– Smoke tests

– Positive tests

– Negative tests

– White box tests

• Roughly 2.200 tests executed by the Hyper-V CI

Tempest


Tempest


class MyTest(tempest.test.BaseTestCase):

def setUp(self):

…

def tearDown(self):

…

def do_whatever_action_test(self):

…

def do_whatever_other_action_test(self):

…

Tempest example - regular testcase


• Each time that a patchset is tested, an entire OpenStack cloud needs to be spin up

• To achieve this, we use OpenStack– Dogfooding FTW

• Undercloud– An OpenStack infrastructure

– Spins up IaaS instances for each patchset

• Overcloud– The OpenStack cloud being tests

– runs the tempest tests

Under and Over clouds


• CentOS 6.5 servers

–Deployed with RDO

• Puppet based

–Controller

• APIs and most OpenStack services

–Network

• networking node(s)

• running Neutron + Open vSwitch

–KVM compute nodes

Undercloud


• Ubuntu 12.04 VMs–on undercloud’s VMs

–Running DevStack

–OpenStack development environment

–Code dependencies pulled via git

• Hyper-V compute nodes–Bare metal

– Two per job, to test live migration

–Attached by Jenkins to the DevStack overcloud

Overcloud


Undercloud UI - showing overcloud instances


• Jenkins and tempests logs are copied to a file server

• Served to the user from an HTTP server

– Linked in the Zuul job results

• Logs are quite tricky to parse

– they’re fairly big!

• Enter Logstash!

Logs


• LogStash

–A tool for managing events and logs

• Elasticsearch

–Distributed search and analytics engine

• Kibana

–Web UI

• The Hyper-V CI logstash instance is hosted on Azure

LogStash


LogStash


Q & A

Technology

Deep Dive into the Microsoft OpenStack CI Infrastructure (Alessandro Pilotti)