Optimizing Your CI: Lessons Learned from a Successful Jenkins Rebuild

US East, June 2015

Jonathann Zenou

Footer

#jenkinsconf

Greetings fellow humans

• Hi, my name is Jonathann• Living in Tel Aviv, Israel• Originally from Marseille, France• Leading the DevOps effort at Ravello Systems• @Zenoujohn• jonathann.zenou@ravellosystems.com

#jenkinsconf

Footer

What is Ravello Systems ?

#jenkinsconf

Footer

Ravello Systems

• 30 developers • 7 major projects• Kanban like

#jenkinsconf

Footer

A part of our stack

#jenkinsconf

Footer

What are we going to talk about ?

#jenkinsconf

Footer

Old problems

#jenkinsconf

Footer

Jobs Leftovers

• Causes full disk

• Workspace Cleanup Plugin

• Can pollute your maven repo or npm cache

• Solution needed : a new clean environment for every job

Footer

#jenkinsconf

Better resource distribution

Backend Team Frontend Team

Footer

#jenkinsconf

Better resource distribution

Backend Team Frontend Team

Footer

#jenkinsconf

Better resource distribution

Backend Team Frontend Team

Footer

#jenkinsconf

Better resource distribution

Backend Team Frontend Team

Footer

#jenkinsconf

Better resource distribution

Backend Team Frontend Team

“waiting for next available executor …”

Footer

#jenkinsconf

Better resource distribution

Backend Team/Frontend Team

Footer

#jenkinsconf

Better resource distribution

Backend Team/Frontend Team

Footer

#jenkinsconf

Better resource distribution

Backend Team/Frontend Team

Footer

#jenkinsconf

Frontend Team leader’s reaction

#jenkinsconf

Footer

Better resource distribution

• Solution needed : a rapid and scalable way to spawn dynamic environments

#jenkinsconf

Footer

Better resource distribution

#jenkinsconf

Footer

Using Docker in Jenkins

• Using Jenkins Docker plugin

• Dedicated Docker host for each team

• Hosts machine capabilities are a function of team size and requirements

• All images on all hosts

• Problems solved : jobs leftovers and resource distribution

#jenkinsconf

Footer

Security

• No active directory or LDAP

• Jim uses Dave’s account

• Employee leaves the company

#jenkinsconf

Footer

Security

#jenkinsconf

Footer

Security

#jenkinsconf

Footer

Security

• Solution needed: third party dynamic authentication mechanism

#jenkinsconf

Footer

Security

• Using Jenkins Google Login plugin

• Coupled with Matrix Authorization Strategy plugin

• 2-Factor authentication

• Open our Jenkins to the world

#jenkinsconf

Footer

Too much code in jobs

#jenkinsconf

Footer

Too much code in jobs

• Hard to maintain

• Can be easily broken

• “Who touched this ??”

#jenkinsconf

Footer

Too much code in jobs

• Solution:

• Put your scripts in revision control (Github)

• Using Jenkins Multiple SCM plugin

#jenkinsconf

Footer

Jobs Versioning

#jenkinsconf

Footer

Jobs Versioning• Solution:

• Using Jenkins JobConfigHistory plugin

• Saves old configurations

• Diff between jobs configurations

• Revert job configuration

• Coupled with Jenkins SCM sync configuration plugin

• Saves you whole Jenkins configuration in revision control (Github)

#jenkinsconf

Footer

Jobs data sharing

Backend Job Frontend Job

Parameters:- Backend version: 1.1- Frontend version: stable

Parameters:- Frontend version: 1.1- Backend version: stable

#jenkinsconf

Footer

Jobs data sharing

• Solution needed: an interface/service that will store data that will be accessible from all the jobs

• Using Jenkins Global Variable String Parameter Plugin

• Cannot alter them from a job

• But you can using Groovy …

#jenkinsconf

Footer

Jobs data sharing

#jenkinsconf

Footer

Jobs data sharing

• Keep a global parameter for each of product stable version

• Let all the other jobs use these values

• Change the global parameter value of a product stable version automatically when this product passes a green CI

• Gist link: https://gist.github.com/johnyzed/2af71090419af2b20c5a

#jenkinsconf

Footer

CI at Ravello Systems

build+

unit tests

environment creation

codedeployment

test suitesstop

environments

artifactory

log collection

#jenkinsconf

Footer

CI at Ravello Systems

environment creation

#jenkinsconf

Footer

Before digging further

#jenkinsconf

Footer

The pieces are ready• What Do Developers Want ?

• Blackbox

• No duplication

• C.D.R.S

• Single form, with a lot of triggers

• Using Multijob Plugin

#jenkinsconf

Footer

Jobs sandbox• Improving a job without:

• Breaking it

• Stopping other people using it

• Solution needed: a parallel Jenkins environment almost identical to our production Jenkins environment to test the new changes on

#jenkinsconf

Footer

Jobs sandbox• Jenkins environment blueprint on Ravello Systems

• Using Ansible:

• git clone

• minor changes

• Running regular jobs

• All that orchestrated from a Jenkins job

#jenkinsconf

Footer

Jenkins sandbox flow

changes on Jenkins script

branch

create dynamic Jenkins

environmentrun jobs

stop dynamic environment

#jenkinsconf

Footer

Jobs sandbox

#jenkinsconf

Footer

Comparing Jenkins environment

Old Jenkins New Jenkins

Number of jobs 137 56

Number of slaves 10 0-50Build and test duration 70 43

#jenkinsconf

Footer

Comparing Jenkins environment

#jenkinsconf

Footer

Take-aways

• Ask your developers how to make their life easier• Don’t be afraid to break things• Follow @jenkins_release

#jenkinsconf

Footer

One last thing

• Ravello Repo• http://www.ravellosystems.com/repo/• http://www.ravellosystems.com/repo/blueprints/60391436

#jenkinsconf

Footer

Questions ?

#jenkinsconf

Footer

Please Share Your Feedback

• Did you find this session valuable?• Please share your thoughts in the

Jenkins User Conference Mobile App.• Find the session in the app and click

on the feedback area.

#jenkinsconf

Thank you