Continuous Integration Testing for Plone Using Hudson

Continuous Integration Testing with Hudson

Wednesday, October 27, 2010

Continuous Integration


Continuous Integration

• Extreme Programming principle

• Martin Fowlerhttp://www.martinfowler.com/articles/continuousIntegration.html


http://www.martinfowler.com/articles/continuousIntegration.html




Continuous Integration• Maintain a Source

Repository

• Automate the Build

• Make the Build Self-Testing

• Everyone Commits Every Day

• Every Commit Should Be Built

• Keep the Build Fast

• Test in a Clone of the Production Environment

• Make it Easy to Get the Latest Deliverables

• Everyone Can See the Results of the Latest Build

• Automate Deployment


Essentially

• Commit your code

• Test the crap out of it

• Regressions are bad

• Assign blame

• Shame is a great motivator




Hudson vs Buildbot?


*shrug*


Hudson (Pros)


Easy to install/run

%> java -jar hudson.war


Easy to Configure

• Create

• Modify

• Manage

• All TTW


Frequent Releases~ 1 / Week


Inline Documentation


Plugins


Hudson (Cons)


It’s Java.


It’s Java.1. It eats resources

2. I don’t know how to write plugins


Hudson


Project Layout


Project Layout

• Project/Job


Project Layout

• Project/Job

• Build


Project Layout

• Project

• Build

• Trigger


Triggers

• Other projects

• Remote trigger

• Periodically

• SCM polling


Project Layout

• Project/Job

• Build

• Trigger

• Build Step


Build Steps

• Shell script

• Python script

• Ant

• Maven


Project Layout

• Project/Job

• Build

• Trigger

• Build Step

• Post-Build Action


Post-Build Actions

• Trigger builds

• Archive results

• Generate reports

• Send email


Build Status


Build Status

• Successful

• Unstable

• Failed


Build Stability


Filesystem Layout


Filesystem Layouthudson /

config.xmlhudson.plugins.*.xmljobs /

myjob / builds /

2010-10-21_11-56-402010-10-21_12-20-26

config.xmlworkspace /




myjob / builds /

2010-10-21_11-56-402010-10-21_12-20-26





myjob / builds /

2010-10-21_11-56-402010-10-21_12-20-26





myjob / builds /

2010-10-21_11-56-402010-10-21_12-20-26



Plugins


Plugins

• Green Balls

• Python

• Subversion

• Trac

• Warnings

• Violations

• Cobertura

• Disk Usage

• Chuck Norris


Plugins

• Green Balls

• Python

• Subversion

• Trac

• Warnings

• Violations

• Cobertura

• Disk Usage

• Chuck Norris


Plone Integration


Buildout & mr.developer


buildout.cfg[buildout]...extensions = mr.developer


buildout.cfg[buildout]...extensions = mr.developersources = sources

[sources]fsd.core = svn https:.../weblion/fsd.core/trunkfsd.membrane = svn https:.../weblion/fsd.membrane/trunk


https://weblion.psu.edu/svn/weblion/weblion/fsd.core/trunk


https://weblion.psu.edu/svn/weblion/weblion/fsd.membrane/trunk


mr.developer

• bin/develop checkout

• bin/develop status

• bin/develop update

• bin/develop activate (deactivate)

• bin/develop reset

• bin/develop purge


buildout.cfg[buildout]...extensions = mr.developersources = sourcesauto-checkout = fsd.core fsd.membrane

[sources]fsd.core = svn https:.../weblion/fsd.core/trunkfsd.membrane = svn https:.../weblion/fsd.membrane/trunk






buildoutbin/

buildoutdevelop...

bootstrap.pybuildout.cfg...src/

fsd.corefsd.membrane




Tests


Tests

• bin/test returns text

• Hudson wants JUnit (XML) format

• collective.xmltestreport


hudson.cfg[buildout]extends = buildout.cfg

parts += xmltestrunner

[xmltestrunner]recipe = collective.xmltestreporteggs = ${test:eggs}defaults = ${test:defaults} + ['--xml']











Test Coverage


Test Coverage

[coverage]recipe = zc.recipe.eggeggs = coverageinitialization = sys.argv = sys.argv[:] + ['run', 'bin/xmltestrunner', '-k', '-q', '--xml']


Test Coverage[report]recipe = zc.recipe.eggeggs = coveragescripts = coverage=reportinitialization = eggs = '${buildout:eggs-directory}' bin = '${buildout:directory}/bin' exclude = '--omit=' + ','.join([eggs, sys.prefix, bin]) sys.argv = sys.argv[:] + ['xml', '-i', exclude, include]


Test Coverage






Code Analysis


Code Analysis

• zptlint

• pyflakes

• pylint

• jslint


zptlint


zptlint

• http://pypi.python.org/pypi/zptlint

• Scan page templates for parser and output errors


http://pypi.python.org/pypi/zptlint

http://pypi.python.org/pypi/zptlint

zptlint

[zptlint]recipe = zc.recipe.eggeggs = zptlintentry-points = zptlint=zptlint:run


zptlint

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" xmlns:tal="http://xml.zope.org/namespaces/tal" xmlns:metal="http://xml.zope.org/namespaces/metal" lang="en" metal:use-macro="context/main_template/macros/master"><body> <metal:fill fill-slot="content-core"> <div tal:content="structure view/context/text/output"> <h2>People</h2> <div tal:content="structure context/@@people" /> </metal:fill></body></html>


http://www.w3.org/1999/xhtml


http://xml.zope.org/namespaces/tal


http://xml.zope.org/namespaces/metal


zptlint

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" xmlns:tal="http://xml.zope.org/namespaces/tal" xmlns:metal="http://xml.zope.org/namespaces/metal" lang="en" metal:use-macro="context/main_template/macros/master"><body> <metal:fill fill-slot="content-core"> <div tal:content="structure view/context/text/output"> <h2>People</h2> <div tal:content="structure context/@@people" /> </metal:fill></body></html>








zptlint

%> bin/zptlint src/.../view.pt


zptlint

%> bin/zptlint src/.../view.pt

*** Error in: src/.../view.pt <class 'zope.tal.taldefs.TALError'>: TAL attributes on <div> require explicit </div> , at line 12, column 9


Parsing Log Messages

• Warnings plug-in

• Regular expression

• Groovy script (groovy.codehaus.org)









Priorities

• HIGH

• NORMAL

• LOW


zptlint[zptlint-test]recipe = collective.recipe.templateinput = inline: #!/bin/sh if [ -e zptlint.log ]; then echo "Old zptlint.log file removed" rm zptlint.log fi echo "Running zptlint-test" PACKAGES="${buildout:package-directories}" for pkg in $PACKAGES do find $pkg -regex ".*\.[c|z]?pt" | xargs -r bin/zptlint | perl -p -e 's/\s+$/ /g;s/\s+/ /g;s/\*{3}s?/\n/g' >> zptlint.log doneoutput = ${buildout:directory}/bin/zptlint-testmode = 755










zptlint[buildout]...# A list of package locations to be examined # by Hudsonpackage-directories = src/fsd.core/fsd/core src/fsd.membrane/fsd/membrane




















pyflakes


pyflakes

• Python syntax checking

• Find common errors quickly

• Doesn’t attempt to run Python code


pyflakes

[pyflakes]recipe = zc.recipe.eggeggs = pyflakesentry-points = pyflakes=pyflakes.scripts.pyflakes:main


pyflakes%> bin/pyflakes src/.../person.py


pyflakes%> bin/pyflakes src/.../person.py

src/.../person.py:25: 'ATTRIBUTE_NAME' imported but unusedsrc/.../person.py:25: 'IAttributeUUID' imported but unused


pyflakes[pyflakes-test]recipe = collective.recipe.templateinput = inline: #!/bin/sh if [ -s pyflakes.log ]; then rm pyflakes.log echo "Old pyflakes.log file removed" fi echo "Running pyflakes" PACKAGES="${buildout:package-directories}" for pkg in $PACKAGES do find -L $pkg -regex ".*\.py"|xargs -r bin/pyflakes >> pyflakes.log doneoutput = ${buildout:directory}/bin/pyflakes-testmode = 755










[pyflakes-test]recipe = collective.recipe.templateinput = inline: #!/bin/sh if [ -s pyflakes.log ]; then rm pyflakes.log echo "Old pyflakes.log file removed" fi echo "Running pyflakes" PACKAGES="${buildout:package-directories}" for pkg in $PACKAGES do find -L $pkg -regex ".*\.py"|xargs -r bin/pyflakes >> pyflakes.log doneoutput = ${buildout:directory}/bin/pyflakes-testmode = 755

pyflakes

















pylint


pylint

• Syntax checking

• PEP8 compliance

• Imports code

• You’ve done everything wrong


pylint

[pylint]recipe = zc.recipe.eggeggs = logilab.pylintinstallerextra-paths = ${instance:location}/lib/pythonentry-points = pylint=pylint.lint:Runarguments = [ '--output-format=parseable', '--zope=y', '--reports=y', '--disable-msg=

E0611,F0401,W0232,E1101,C0103,C0111,R0201,W0201,R0911,R0904,F0220, E1103,R0901,E0211,E0213,E1002,W0622',

'--generated-members=objects', ] + sys.argv[1:]


pylint

• disable-msg

• R0911: Too many return statements (%s/%s)

• R0201: Method could be a function

• W0201: Attribute %r defined outside of __init__

• http://pylint-messages.wikidot.com/all-messages


http://pylint-messages.wikidot.com/all-messages

http://pylint-messages.wikidot.com/all-messages

pylint[pylint-test]recipe = collective.recipe.templateinput = inline: #!/bin/sh if [ -s pylint.log ]; then rm pylint.log echo "Old pylint.log file removed" fi echo "Running pylint" PACKAGES="${buildout:package-directories}" for pkg in $PACKAGES do find -L $pkg -regex ".*\.py" | xargs bin/pylint >> pylint.log done echo "Finished"output = ${buildout:directory}/bin/pylint-testmode = 755


pylint


pylint




Selenium


Selenium• Automated web application testing


Selenium1. Open this page

2. Click that thing

3. Fill out that form

4. Click submit

Did it work?


Selenium Testcase

def test_login_overlay(self): self.open("/") self.wait() self.selenium.click('link=Log in') self.waitForElement('form#login_form') self.selenium.type("name=__ac_name", TEST_USER_NAME) self.selenium.type("name=__ac_password", TEST_USER_PASSWORD) self.selenium.click("submit") self.wait() self.failUnless(self.selenium.is_text_present("Log out"))


Selenium Testcaseclass SeleniumLayer(PloneSandboxLayer): defaultBases = (HOST_ADJUSTABLE_ZSERVER_FIXTURE, PLONE_FIXTURE)

# Connection parameters seleniumHost = os.environ.get('SELENIUM_HOST', 'localhost') seleniumPort = os.environ.get('SELENIUM_PORT', '4444') seleniumBrowser = os.environ.get('SELENIUM_BROWSER', '*firefox')

def setUpZope(self, app, configurationContext): ...

def setUpPloneSite(self, portal): ... url = "http://%s:%s/%s" % (self['host'], self['port'], PLONE_SITE_ID) self['selenium'] = selenium.selenium(self.seleniumHost,

self.seleniumPort, self.seleniumBrowser, url) self['selenium'].start()

def tearDownPloneSite(self, portal): self['selenium'].stop() del self['selenium']
















Selenium RC

• Firefox

• Internet Explorer

• Safari

• Chrome

• Opera

• more...


Selenium Grid



“Firefox?”


Thanks!



Selenium Test Runnerfsd/

core/...selenium/

base.pytestSelenium.py

tests/...


Selenium Test Runner

[selenium]recipe = collective.xmltestreporteggs = fsd.core [test] fsd.membrane [test]defaults = ['--tests-pattern', '^f?selenium$', '--xml']



[selenium]recipe = collective.xmltestreporteggs = fsd.core [test] fsd.membrane [test]defaults = ['--tests-pattern', '^f?selenium$', '--xml']



[selenium-firefox]recipe = ${selenium:recipe}eggs = ${selenium:eggs}defaults = ${selenium:defaults} [selenium-safari]recipe = ${selenium:recipe}eggs = ${selenium:eggs}defaults = ${selenium:defaults}



%> bin/selenium-firefox%> bin/selenium-safari



workspace/bin/...parts/

selenium-firefox/testreports/

*.xmlselenium-safari/

testreports/*.xml


~


Sprint!

• Saturday

• QA Sprint

• Selenium testing and more

• Free food!


Multi-Configuration Projects


Multi-Configuration Projects

• Run a build for each set of n x n ( x n x n) variables.



Wish List


Wish List

• Load testing

• Sphinx builds

• Chameleon compatibility checking

• Windmill


Best Practices


Check in Your Configs

• Configuration is stored in XML files

• Easy to back up to SVN


Check in Your Configshudson /


myjob / builds /

2010-10-21_11-56-402010-10-21_12-20-26



hudson /config.xmlhudson.plugins.*.xmljobs /

myjob / builds /

2010-10-21_11-56-402010-10-21_12-20-26






Watch Your Disk Space

• “Hudson disk-usage” plugin







Use an Egg Cache

• Set up a default.cfg for your hudson account.

[buildout]eggs-directory =

/home/hudson/.buildout/eggsdownload-cache =

/home/hudson/.buildout/downloads


Pin unittest2

• (If you’re using it)

• plone.app.testing

• 3x faster buildout


Split Big Builds

• How long does the build take?

• How often does each part need to be run?

• Tests?

• Coverage?


Split Big Builds

• Build

• Test

• Syntax Checking

• Coverage

• Selenium


Split Big Builds1. “Build other projects” post-build action



2. Custom workspace



2. Custom workspace

3. Build Fingerprinting


Fingerprinting

• “These builds are related”

• Archive an md5 hash of a file to act as an identifier


FingerprintingIn parent build:

In parent and child builds:


Reporting


Hudson Report Views


Email


Chuck


iPhone


Extreme Feedback Devices


hudson.plone.org

• Currently builds Plone 4.0.x

• Core developers have access

• Add your core & collective projects


Thanks

• Martin Aspeli

• Timo Stollenwerk

• Hanno Schlichting


QA Sprint!

• Saturday

• QA Sprint

• Selenium testing and more

• Free food!


Technology

Continuous Integration Testing for Plone Using Hudson