Gidi Samuels CI - DevOps

CI Provisioning with Openstack

WHO AM I?

CI-DevOps Tech Leader

10 years in the field of SW development and CI 5 years @ Liveperson building the CI/CD pipeline

WHO ARE WE

About our CI

1600Releases

~160Micro Services

10KWeekly Builds

1400Git Repos

70BuildAgents

1000TeamCity jobs

30Dev Teams

4CI Members

Agenda

The resource PROBLEM we are trying to solve

FASTER DEVELOPMENT with Openstack

Developing OPENSTACK MAVEN PLUGIN

2

3

4

Using pre-defined servers from POOL5

Openstack Numbers @ Liveperson1

20,000 physical cores

8000 VM’s in 7 DC’s

Upgrading to Kilo in progress

40,000 virtual cores

Web&App tiers 100% on the Openstack

Openstack @ LivePerson

LIVEPERSON DEVELOPMENT TOOLS

Make a change

TeamCity CI Server

Build & Unit Test

GitHub Packaging

Deploy

System Test

Maven lifecycle

Security Analysis

RELEASE & APPROVAL

CI Flow

Make a change

TeamCity CI Server

Build & Unit Test

GitHub Packaging

Deploy

System Test

Maven lifecycle

Security Analysis

RELEASE & APPROVAL

THE PROBLEM

We need ~600 dedicated VM’s for the CI flow

Long time in queue (no free resources)

Dirty and unmanaged VM nodes

Not like Production

The Goal:

Run build in parallel

The Solution:

On-the-fly Openstack provisioning

Extending Maven by developing

Maven ci-provisioning Plugin

Openstack Toolkits

http://www.openstack4j.com/

Identity, Compute, Image, Network, Storage

Open source Java client which allows provisioning and control of an OpenStack system

1.Install maven from https://maven.apache.org/

1.mvn archetype:generate \

-DgroupId=ci.plugins \

-DartifactId=openstack-maven-plugin \

-DarchetypeGroupId=org.apache.maven.archetypes \

-DarchetypeArtifactId=maven-archetype-plugin

Step 1

Add the dependency to your maven pom.xml file

<dependency><groupId>org.pacesys</groupId><artifactId>openstack4j</artifactId><version>2.0.0</version>

</dependency>

Step 2

// authenticate

OSClient authenticate(String tanent) { OSClient os = OSFactory.builder()

.endpoint(OS_AUTH_URL)

.credentials(OS_USERNAME, OS_PASSWORD)

.tenantName(tanent)

.authenticate();return os;

}

Step 3

Develop the Maven plugin following goals:Compute:creates instances on the fly

DeleteInstance:delete instance if build is successful otherwise leave for debugging a period of time

Step 4

TOMCAT APP

TOMCAT APP

How to use it in your pom.xml

<roles><role1>ci-app1</role1><role2>ci-app2</role2> <role3>ci-nginx</role3>

</roles>

Step 4

● 3 Openstack instances will be assigned

● Facter Role will be created on each instance

Now that we can auto provision instances

How do we configure them as a service?

Facter is a cross-platform library for retrieving simple operating system facts, like operating system, linux distribution, or MAC address

PUPPET FACTER

Available during puppet execution

Custom facts located @ /etc/facter/facts.d/

PUPPET FACTER

{"role" : "ci-app1"}

balancermember: node1,node2

JSON style YAML style

role1 => ci-app1role2 => ci-app2role3 => ci-nginxbalancermember => ctor-ci-q10-1,ctor-ci-q11-2

GENERATE CUSTOM FACTS

<web>ci-nginx</web><webdns>ci-web-${build.number}</webdns>

Step 5

Create a unique DNS for the service so it can be accessed from the system test

node /qtvr-ajb4(2|3).*/ inherits default {class{nginx::web':upstream_members => ['ctvr-wap004', 'ctvr-wap005'],

}

if ($role1 == ci-app1) {class{nginx::web':

upstream_membersn => split($balancermember, ',')}

}

FROM STATIC TO DYNAMIC DECLERATION

What are the main

ADVANTAGES?

ADVANTAGES

CI is running in parallelA dedicated provisioned env for each commit

ADVANTAGES

Each instance is freshA new fresh instance created for each commit

ADVANTAGES

Production-like environmentEach instance reflects prod-like env

ADVANTAGES

No resources waste No need to allocated redundant resources

ADVANTAGES

Easy to scale

PROBLEM:CI-provisioning add additional 5 minutes the the CI Process …

SOLUTION:Develop additional Maven goal that created Openstack VM’s pool in the background

NOW WHAT ...

Added to the Maven plugin following goals:

Qmanager: instance Q manager that prepare ready instances

ComputeCIQ: use instances from pull

Step Improvement

THANK YOU

WHO ARE WE - CUSTOMERS

More than 18,000 customers