1.How to build quality in a talefrom the trenchesPeter Antman2013-05-23

2. Who am I?Peter Antman 2Background Developer since 1995 Linux, Open Source andEnterprise Java Leader and manager forsoftware developer since 2000 Head of Product Development,Polopoly Atex 2007 - 2011 Media business Drive: Help organizations andpeople fully realize theirpotential (doing softwaredevelopment)Peter Antman0760 140 150peter.antman@crisp.se@peterantmanwww.crisp.se/konsulter/peter.antmanblog.crisp.se/peterantman, antman.se 3. Crisp is an employee owned companyknown for agile courses with internationallyrenowned teachers and experienced agiledevelopers and coaches.h"p://blog.crisp.se Persistent improvements 4. Polopoly Enterprise WebCMS International take of 2008 Started Agile transformation 2007 Existed for 16 year Large code base Large user base Thousands of editors, millions of users, billions ofpage viewsDevelopment Manager @ PolopolyPeter Antman 4 5. Organized around leanprinciples 6. Cult of quality 7. http://commons.wikimedia.org/wiki/File:1924_Non-Stop_Shuttle_Change_Toyoda_Automatic_Loom,_Type_G_2.jpgThe Type-G Toyoda Automatic Loom, the worlds rst automaticloom with a non-stop shuttle-change motion, was invented bySakichi Toyoda in 1924. This loom automatically stopped whenit detected a problem such as thread breakage. 8. Time Boxed releases 9. No junk on trunk9RELTeamDemoedTeam test branch (convinience)Story branchAllways releaseable!Test everything! 10. http://www.makeve.com/categories/news-business/business/top-5-items-i-sold-on-ebayContinous integration 11. 1.5 million lines of code 10 000 tests PEAR PAF Upgrade, and more configurations Five database vendors * multiple versions Four browsers * multiple versions Three web containers * multiple versions Two JDK:s * multiple versions Two core OS:es * multiple versions Two EJB containers * multiple versions 12. Plattform to validateLinux Windows Mysql Oracle Msql Postgres Derby Sun JDK IBM JDKTomcat JBossWeb WS WebJBoss WebsphereChrome Firefox Safari IE1.5 million 10 000 tests MultiplecongurationsMultiple versions 13. Thats a lot of tests 14. Had to build our own test cloud Started with RedHat kickstart on oldmachines under our tables Fragile, hard to upgrade, electricity out takes Went to static installs on blades Lots of sys admin. One machine = one setup Tried VMWare Clock issues (went backwards) Started using Amazon To slow to upload our builds Static Xen on local blades One machine = one setup Eucalytus (kvm) on local blades, elasticcloud Very buggy (more than 6 month to stabilize)History 15. 596 570For example when merging a bugfix to 9.10 16. Number of tests (november2011) 17. Our version of the automaticloom 18. http://geekandpoke.typepad.com/.a/6a00d8341d3df553ef014e8adc7838970d-pi 19. Compile classTest classCompile moduleTest moduleBuild productTest productAcceptance test productTest on specic plattformTest with specic congurationAt what stage is it possibleto continue working even ifits broken?Incremental compileSoftware development 20. http://xedgear.se/forum/viewtopic.php?f=1&t=2650&start=20Urban decay 21. http://www.codinghorror.com/blog/2005/06/the-broken-window-theory.htmlBroken WindowsA sociological theory from 1982 22. http://huntington.patch.com/articles/volunteers-clean-up-in-the-station#photo-5852615The New York Theorem:Fighting big crime by pickinglitter 23. http://www.makeve.com/categories/news-business/business/top-5-items-i-sold-on-ebayBroken builds are broken windows 24. The line must be stopedmanually 25. ToolsCulture 26. Everything must be tested Examine each failed test, always New bugs are, either fixed when standing on them handled as a fix next sprint (first in line) put into drawerStop the line and fix it 27. Bugs discovered by non automated tests arefixed in next sprint Philosophy: We strive not to produce softwarewith errors Policy: If it had been covered by a test it wouldhave been fixed immediately Practically: Old bugs tends to never get fixed(adapt to capacity)FixNextSprint never let awindows be broken 28. Integrate first on ticket branch, then team test branch, then team branch, then branch Hide known bugs (KnownBugs) Run only when changed (600 000 -> 100 000) Nightly test check responsibility (rotate between teams) Daily summary mail on test faults Tickets for test faults Indexed in Solr Annotate in JenkinsTools and policies are necesary 29. Analyzing Test Faults 30. Clean up GoGreen(adapt to capacity)FixNextSprint(Stop-the-line) 31. Its a never ending work 32. Heres a song to singI keep a close watch on these tests of mineI keep my Jenkins open all the timeI see a defect coming down the lineBecuse youre mine, I stop the line