Premium community conference on Microsoft technologies itcampro@ itcamp14#
Deep dive into the Microsoft OpenStack CI infrastructure
Alessandro Pilotti
Cloudbase Solutions
Hyper-V MVP
Premium community conference on Microsoft technologies itcampro@ itcamp14#
Huge thanks to our sponsors & partners!
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• OpenStack - what’s this?
• Hyper-V integration
• OpenStack continuous integration
• The Hyper-V CI infrastructure
• Undercloud and Overclouds
Agenda
Premium community conference on Microsoft technologies itcampro@ itcamp14#
An open source IaaS project–Apache 2 license
Managed by the OpenStack Foundation–More than 150 companies joined it including:
–AMD, Intel, Canonical, SUSE Linux, Red Hat, Cisco, Dell, HP, IBM, NEC, Vmware
Portable–Mostly used on Linux
Written in Python
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• Essex– Apr 2012– Hyper-V removed
• Folsom– October 2012– Cloudbase Solutions involvement - Hyper-V re-included
• Grizzly– Apr 2013
• Havana– October 2013
• Icehouse– Apr 2014
Releases
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• Distributed Components– Can be deployed on a single server or multiple servers
• Public RESTful API– Bindings available for various languages (including .Net)
• Queues (AMQP) for private APIs– Great scalability
• Relational DBs for storing the configuration– MySQL, PostgreSQL, SQLite (and SQL server)
• UI– CLI tools– Web dashboard
OpenStack Architecture
Premium community conference on Microsoft technologies itcampro@ itcamp14#
Compute (Nova)
Object Storage (Swift)
Block Storage (Cinder)
Image Service (Glance)
Networking (Quantum / Neutron)
Dashboard (Horizon)
Identity (Keystone)
Metering (Ceilometer)
Orchestration (Heat)
Components
Premium community conference on Microsoft technologies itcampro@ itcamp14#
Architecture 1
Premium community conference on Microsoft technologies itcampro@ itcamp14#
Architecture 2
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• AMQP–RabbitMQ
–Apache Qpid
• API–RESTful services
• Database–Any Python SQLAlchemy provider
–Mostly MySQL
– SQL Server required various bug fixes
Component interaction
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• FREE edition
– Full Hypervisor
–Minimum OS support
• Minimum impact on security updates, etc
• Windows Server 2012 R2
– Just enable the related role
• Windows 8.1
– For workstation / testing / development usage
Hyper-V Server availability
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• Nova Compute driver
• Neutron plugin
• Cinder Volume driver
• Windows Cloud-Init
• Ceilometer Agent
• Heat templates
Hyper-V / Windows Openstack components
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• Python Application installed as an application/service on the Hyper-V node.
• OpenStack compute utilizes key features baked into the Hyper-V Virtualization Platform
• Does not require windows clustering services
• Does not require shared storage
Hyper-V Nova Compute
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• Hyper-V plugin is part of Quantum since Jan 2013– Project renamed in Neutron since Havana
• Supported network types:– VLAN
– Flat
– Local
• Plugin / agent model– Plugin runs in quantum-server (controller)
– Agent runs on each Hyper-V compute node
Neutron
Premium community conference on Microsoft technologies itcampro@ itcamp14#
Typical Neutron setup
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• Quantum plugin / agent AMQP RPC protocol is compatible with OpenVSwitch
• You can use the OVS plugin with Hyper-V agents (or vice versa)– Limited to compatible L2 protocols: Flat / VLAN
• You can use the L3 and DHCP agents with the Hyper-V plugin– Using the OVS L2 agent on the networking nodes
• Supports ML2 plugin!
Neutron OVS interop
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• We ported OVS to Hyper-V
Open vSwitch Windows porting
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• De facto standard for SDN
• Great interoperability– Hyper-V / KVM / etc
• OpenFlow
• Tunnelling–VXLAN
–GRE
• Neutron OVS plugin–No need for a different one
Why OpenVSwitch?
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• Great code review platform
–Gerrit
• Strong emphasis on unit tests
–Commits do not pass w/o tests
–Python is a dynamic language
• Syntactical and semantic validation needed
• Great continuous integration effort
– Lots of moving parts
How does OpenStack handle such complexity?
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• Originally developed by Google for Android
• Stand-alone patch review system
• Manages:
–Git Repositories
–Code Review
– Event Stream
–Privileges (propose / merge code, manage gitrepositories)
Gerrit
Premium community conference on Microsoft technologies itcampro@ itcamp14#
Gerrit workflow
Premium community conference on Microsoft technologies itcampro@ itcamp14#
Gerrit UI
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• Except special cases every contribution must have either:
–Blueprint
• for new features
• to be approved by core team
–Bug report
• for, ahem, bugs
• http://launchpad.net
Code contribution workflow
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• Clone repository, e.g.:
– git clone https://github.com/openstack/nova.git
• Create a short lived branch:
– git checkout -b bug/12345678
– Do changes
– git commit
• Commit message must contain referefence to bug / bp
– git review
• Install with pip install git-review
Code contribution workflow
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• Gerrit triggers:– Pep8 / Flake8 tests
• Code conformity über alles!
• Very important with > 850 contributors
– Unit tests
– Integration tests with multiple env (e.g. Nova): • KVM
• Hyper-V
• VMWare
• XenServer
• etc
Code contribution workflow
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• Each CI results can vote +1 / -1 or be neutral
–At the moment both errors and failuresaccount for -1
• Hard to distinguish between them
• In case of failures users can issue a “recheck”
Code contribution workflow
Premium community conference on Microsoft technologies itcampro@ itcamp14#
Code contribution workflow
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• At this stage code can be peer reviewed
• Every user can review the code– Leave comments
– Vote +1 / -1
• Each project (e.g Nova) has a peer elected team with “+2 rights”– Vote +2 / -2
• A commit requires 2x +2 and approval to be merged
Code contribution workflow
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• At this point the code is ready to be merged
–Another CI run is required
–Code base is changed since code submission
• If successful, the code will finally merge into the git repository!
Code contribution workflow
Premium community conference on Microsoft technologies itcampro@ itcamp14#
Queue monitoring
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• Microsoft contributes hardware for CI
–Around 200 servers ATM
–Currently based in Cambridge, MA
• CI support team provided by:
–Microsoft
–Cloudbase Solutions
Hyper-V CI
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• Consistency– All projects should act the same way during code
reviews and offer the same interface to the devs
• Repeatability– system should run 100 or 500 times per day
without failing
• Availability– complete automation means that if it stops, the
whole project stops
OpenStack Requirements of automation
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• Zuul
• Jenkins
• Undercloud– KVM compute nodes
• Overcloud– Devstack running on KVM VMs
– Hyper-V compute nodes
• Logs HTTP server
• Logstash
Hyper-V CI components
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• Open source project
• monitors in-progress jobs for a set of related patches
• will pre-emptively cancel any dependent test jobs that would not succeed due to a failure in a dependent patch
Zuul
Premium community conference on Microsoft technologies itcampro@ itcamp14#
Zuul workflow
Gerritregisters commit
Zuul listens Gerrit
stream and records commit
Zuul clones referenced
git repo locally
Predefined Jenkins job(s)
started by Zuul
Zuul collects and pushes upstream
Jenkins job(s) results
Premium community conference on Microsoft technologies itcampro@ itcamp14#
pipelines:
- name: check
description: Newly uploaded patchsets enter this pipeline to receive an initial +/-1 Verified vote from Jenkins.
failure-message: Build failed. For rechecking only on the Hyper-V CI, add a review comment with recheck hyper-v
manager: IndependentPipelineManager
precedence: low
trigger:
gerrit:
- event: patchset-created
- event: change-restored
- event: comment-added
comment_filter: (?i)^(Patch Set [0-9]+:\n\n)?\s*recheck(( (?:bug|lp)[\s#:]*(\d+))|( no bug)|( hyper-v))\s*$
success:
gerrit:
verified: 0
failure:
gerrit:
verified: 0
Zuul yaml config 1/2
Premium community conference on Microsoft technologies itcampro@ itcamp14#
jobs:- name: ^.*$parameter-function: set_log_url
- name: 01_Reserve_Hyper-V_Node_01voting: truesuccess-pattern:
http://64.119.130.115/{build.parameters[ZUUL_CHANGE]}/{build.parameters[ZUUL_PATCHSET]}/
failure-pattern: http://64.119.130.115/{build.parameters[ZUUL_CHANGE]}/{build.parameters[ZUUL_PATCHSET]}/
success-message: Looks good to mefailure-message: Test run failed.
projects:- name: openstack/novacheck:
- 01_Reserve_Hyper-V_Node_01- name: openstack/neutroncheck:
- 01_Reserve_Hyper-V_Node_01
Zuul yaml config 2/2
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• The de facto CI framework–Open Source, written in Java
–Provides a well done UI (CLI as well)
• Lots of available plugins
• Supports git and other SCMs
• Executes jobs:–on demand
– scheduled
–based on repository changes
Jenkins
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• Jobs can be executed on
– Jenkins host
– Slaves (Linux, Windows)
• Jobs can be interdependent
– The CI executes multiple ones for each patchset
• Client / Server architecture
– Scales poorly
• Zuul integrates with Jenkins
– Executes jobs and waits for outcome
jenkins
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• 5 jobs defined
1. Reserve Hyper-V Node 01
2. Reserve Hyper-V Node 02
3. Initialize Test Environment
4. Run Tempest Test
5. Clean Test Environment
• Return result is the result of executing jobs 1 to 4, result of job 5 is ignored
Jenkins CI jobs
Premium community conference on Microsoft technologies itcampro@ itcamp14#
Jenkins UI
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• OpenStack’s integration tests suite
• Test types
– Smoke tests
– Positive tests
– Negative tests
– White box tests
• Roughly 2.200 tests executed by the Hyper-V CI
Tempest
Premium community conference on Microsoft technologies itcampro@ itcamp14#
Tempest
Premium community conference on Microsoft technologies itcampro@ itcamp14#
class MyTest(tempest.test.BaseTestCase):
def setUp(self):
…
def tearDown(self):
…
def do_whatever_action_test(self):
…
def do_whatever_other_action_test(self):
…
Tempest example - regular testcase
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• Each time that a patchset is tested, an entire OpenStack cloud needs to be spin up
• To achieve this, we use OpenStack– Dogfooding FTW
• Undercloud– An OpenStack infrastructure
– Spins up IaaS instances for each patchset
• Overcloud– The OpenStack cloud being tests
– runs the tempest tests
Under and Over clouds
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• CentOS 6.5 servers
–Deployed with RDO
• Puppet based
–Controller
• APIs and most OpenStack services
–Network
• networking node(s)
• running Neutron + Open vSwitch
–KVM compute nodes
Undercloud
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• Ubuntu 12.04 VMs–on undercloud’s VMs
–Running DevStack
–OpenStack development environment
–Code dependencies pulled via git
• Hyper-V compute nodes–Bare metal
– Two per job, to test live migration
–Attached by Jenkins to the DevStack overcloud
Overcloud
Premium community conference on Microsoft technologies itcampro@ itcamp14#
Undercloud UI - showing overcloud instances
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• Jenkins and tempests logs are copied to a file server
• Served to the user from an HTTP server
– Linked in the Zuul job results
• Logs are quite tricky to parse
– they’re fairly big!
• Enter Logstash!
Logs
Premium community conference on Microsoft technologies itcampro@ itcamp14#
• LogStash
–A tool for managing events and logs
• Elasticsearch
–Distributed search and analytics engine
• Kibana
–Web UI
• The Hyper-V CI logstash instance is hosted on Azure
LogStash
Premium community conference on Microsoft technologies itcampro@ itcamp14#
LogStash
Premium community conference on Microsoft technologies itcampro@ itcamp14#
Q & A