NA-MIC, 2008 June Workshop, IHK Akademie Westerham
Engineering Methodology
Sebastien BARRE, Ph.D. - Kitware, Inc.
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 2
Outline
• Software Process, Extreme Programming• The components of the NA-MIC kit software engineering tools:
• CMake• CPack• CTest• CDash• KWStyle
June 16-17, 2008NA-MIC, 2008 June Workshop, IHK Akademie Westerham 2
NA-MIC, 2008 June Workshop, IHK Akademie Westerham
Software Process, Extreme Programming
Sebastien BARRE, Ph.D. - Kitware, Inc.
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 4
Challenges
• Software is larger and more complex than ever• Complex software requires teams; multi-site, multi-disciplinary,
multi-layered, academic and commercial• Complex software demands long-term maintenance• Testing is vital to insure quality• Documentation needs are greater• Managing change to the system is critical to capture bug fixes• Abundance of computer configurations (hard/software, OS)
means addressing cross-platform issues
June 16-17, 2008NA-MIC, 2008 June Workshop, IHK Akademie Westerham 4
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 5
Our solution
• A methodology developed over several years in support of large-scale open-source projects
• A methodology based on the principles of agile-programming or extreme-programming (late 90´s)
• Ultimate Goal:• A production software release every day
June 16-17, 2008NA-MIC, 2008 June Workshop, IHK Akademie Westerham 5
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 6
Extreme Programming
The idea is that the standard development tasks:• requirements generation,• software design,• managing source code versions and updates,• configuring projects for specific platforms,• compilation and linking,• testing the code at run time,• verifying the validity of output,• documenting the code,• tracking and repairing bugs
are performed continuously rather than in a waterfall fashion.
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 7
Extreme Programming (cont.)
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 8
Heart and Soul of the Process
• Testing is the mantra• Every night• Every time changes are made• Every time new computing platforms are introduced• “If is isn’t tested, it doesn’t work”
• The results are posted to a web page for everyone to monitor• All developers have the right and encouragement to repair
errors
• A feedback loop is created between the community and the automated processes that measure software quality
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 9
Heart and Soul of the Process (cont.)
CVS/SVN
Results posted on web(the dashboard)
Source coderepository Testing client compiles
source code, runs tests
Developers review results
Developers check-in code
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 10
Extreme Programming (cont.)
The community owns the code
• Although the identity of the original author is kept, other developers are free to correct defects and enhance each other's code
• In the end, all of the software should appear as though one author wrote it
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 11
Extreme Programming (cont.)
Release early, release often
• Although developers are tempted to keep their code under wraps until it is perfect, the process encourages them to release their code as soon as it passes some minimum tests
• The longer the code is visible to the community, the better integrated it will be
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 12
Extreme Programming (cont.)
Continuous integration
• There is no scheduled porting to computer platforms• All new software builds on supported platforms every evening
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 13
Extreme Programming (cont.)
All developers agree to keep the software defect free
• Although everyone is encouraged to submit their code early, the code must compile and pass tests nightly
• A continuous build process sends e-mails to developers who check in code that does not compile
• More effectively, the community enforces the commitment though peer pressure
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 14
Software Process Tools
• Communication and Documentation• Revision Control• Build Management • Testing• Release process
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 15
Communication and Documentation
• Mailing lists• Wiki• Online documentation (Doxygen)• Issue/Bug Tracker• IM, TCON, VCON
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 16
Mailing Lists
• Mailman list manager• http://www.gnu.org/software/mailman/• Two lists per project
• developer – discussion about implementation details• user – discussion by users about how to use it
• Provides a searchable archive• Dynamic, quick exchange of information
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 17
Wiki
• Powerful collaborative websites or intranets• http://www.mediawiki.org/wiki/MediaWiki• Easy to setup• Easy to edit• Don´t forget to backup
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 18
Doxygen
• Literate programming concepts• combine source and documentation
• http://www.stack.nl/~dimitri/doxygen/docblocks.html • Run every night• Simple markup for C++/C code
/** * ... text ... */ /*! * ... text ... */
/*! \brief Brief description. * Brief description continued. * Detailed description starts here. */
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 19
Doxygen (cont.)
From vtkActor.h:
// Description:
// This causes the actor to be rendered. It in turn will render the actor's
// property, texture map and then mapper. If a property hasn't been
// assigned, then the actor will create one automatically. Note that a side
// effect of this method is that the pipeline will be updated.
virtual void Render(vtkRenderer *, vtkMapper *) {}
http://www.vtk.org/doc/nightly/html/classvtkActor.html
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 20
Issue Tracker
• Web based issue tracker is required to keep track of bugs and fixes
• Tracker of choice: Mantis http://www.mantisbt.org/• Requires periodic “triage” (usually before a major release)
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 21
Revision Control
• Projects evolve over time• Who changed what? When? Why?• SCM tools like CVS or SVN maintain a central repository of
source code in which developers retrieve, modify and commit files and changes to files.
<html><body>Hello, World!</body></html>
sample.html<html><head><title>Simple Example</title></head><body>Hello, World!</body></html>
sample.html
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 22
Revision Control (cont.)
• Commit• Date/time of change• Log message describing change• Person making the change• Versions before and after change
• Commits can conflict, but the repository will not lock• Commits can be referenced in bug trackers
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 23
Revision Control (cont.)
• Traditionally, history is always a tree• Each development path is a branch
trunk
release-1_2
release-1_0
= Version/Revision/State= Commit (new state and pointer to old)= Series of Commits
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 24
Revision Control (cont.)
Repository
Working Tree
Working Tree
Working Tree
Working Tree
Working Tree
= flow of changes
W.T.
Repo
RepositoryW.T.
Repo
W.T.
Repo
W.T.
Repo
W.T.
Repo
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 25
Build Management
• Often overlooked• CMake is the (Kitware) answer, more about it later…
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 26
Testing
• Smoke test • does it compile• does it run and not crash
• Regression testing• If something fails track, find the change and fix it• harder to implement
• CTest/Cdash/DART are used; more about it later...
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 27
Release Process
• One person freezes and maintains the release branch• Two types of release
• Binary• Source
• CPack – tool to create release packages; more about it later
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 28
Software Process Tools Summary
• CMake – cross platform building• Dashboards, CTest, CDash, DART – continuous testing• Mailman Mailing List – Developer communication• Wiki – Store developer communication• Doxygen – Online documentation• Mantis Bug Tracker – keep track of bugs and feature requests• Revision control – cvs, svn• CPack – release management
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 29
Software Process Tools Summary
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 30
Conclusion
• A set of tools, a set of rules and practices that creates a feedback loop resulting in higher quality code
• But it does require some discipline• Paying attention to the dashboard• Picking enforcers• Creating tests
June 16-17, 2008NA-MIC, 2008 June Workshop, IHK Akademie Westerham 30
NA-MIC, 2008 June Workshop, IHK Akademie Westerham
CMake
Sebastien BARRE, Ph.D. - Kitware, Inc.
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 32
• 1999 Start of the Insight Segmentation and Registration Toolkit (ITK) funded by NLM as part of the Visible Human Project www.itk.org www.itk.org
• Kitware tasked with build environment• VTK current build system
• autoconf on UNIX• pcmaker on Windows• Very specific to VTK and hard to maintain
• CMake born out of necessity • 2006 KDE desktop adopted CMake
Why CMake
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 33
CMake Features
• Single input format for all platforms• Generate native build files
• Visual Studio 6/7/8/9• Nmake, borland make, unix make• Xcode• KDevelop• Eclipse
• Platform inspection tool can search for• Programs• Libraries• Include paths• Determine hardware specifics like byte order
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 34
CMake Features (cont.)
• Linux, SunOS, HPUX, IRIX, Windows, Mac OSX, QNX, etc.• Simple, easy to use• Support complex, large build environments (KDE, ParaView,
SecondLife)
• Do not require any software other than CMake to be installed. • Need only a C++ compiler (same as ITK itself)
• Full cross platform install system• Comes with a GUI layer for easy edition of input variables.
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 35
Windows GUI
Unix GUI
CMakeLists.txt
MSVCProject
.NET Solution
Makefile
Userselects
CMake Example Usage
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 36
CMake GUI Process Flow
Configure
Write CMakeCache.txt
Did Cache
Change?
Yes
No
GenerateMakefile or
Project
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 37
A very simple project (see how easy it is!)
Files needed:• Hello.c • CMakeLists.txt
The CMakeLists.txt file would contain two lines: • PROJECT(Hello) • ADD_EXECUTABLE(Hello Hello.c)
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 38
Basic CMake Syntax
• COMMAND_NAME(space separated arguments) • command can be upper or lower case• SCREAM_COMMAND or quiet_command
• Variables• SET(VAR value)• Semi-colon separated lists
• SET(VAR “a b c”) - VAR holds one thing• SET(VAR a;b;c) or SET(VAR a b c) – VAR holds three
things• LIST or FOREACH will get access to each thing
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 39
Flow control (IF)
IF(VAR) codeENDIF(VAR)
IF(NOT VAR)IF(VAR AND VAR2)IF(VAR OR VAR2)IF(VAR MATCHES regular_expression)IF(COMMAND command)IF(EXISTS file)IF(VAR LESS VAR2)
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 40
Flow control (FOREACH WHILE)
foreach(F a b c) message(${F})endforeach(F)
while(VAR) message(${VAR}) set(VAR FALSE)endwhile(VAR)
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 41
FIND_PACKAGE
• Easy for smaller projects to use larger projects
project(SampleProject)
find_package(VTK REQUIRED)
include(${VTK_USE_FILE})
set(SOURCES mySource1.cxx mySource2.cxx)
add_executable(myExecutable ${SOURCES})
target_link_libraries(myExecutable vtkRendering)
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 42
CMake Advanced Features
• Shared library and DLL building• Support code generation at build time (VTK wrappers, Qt’s moc)• Create configured .h files
• Automatic dependency generation• Parallel builds • Out of source builds• Command line support• Create OSX library frameworks and application bundles • Cross compiling• Integrated testing system called Ctest• Integrated packaging system called Cpack• Easy integration with CDash and Dart dashboard servers• And more...
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 43June 16-17, 2008NA-MIC, 2008 June Workshop, IHK Akademie Westerham 43
Find more about CMake…
• Online documentation• http://cmake.org• Wiki• FAQ• Mailing list
• Books• Mastering Cmake
NA-MIC, 2008 June Workshop, IHK Akademie Westerham
CPack
Sebastien BARRE, Ph.D. - Kitware, Inc.
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 45
CPack
• Cross platform packaging and distribution tool• Package binary into native package format• Distributed with CMake
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 46
CPack Features
• Support both CMake and Non-CMake projects• Unix:
• TGZ, TZ, TBZ2, • STGZ (Self extract TGZ), • RPM• DEB
• Windows:• NullSoft Scriptable Install System (NSIS), • ZIP
• Mac OSX:• PackageMaker• OSXX11 (Mac OSX X11 Bundle)
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 47
Using CPack
• On Windows install command line ZIP program, and NSIS• Setup your project to work with CPack
• Get make install to work• install(…)• make sure you executables work with relative paths and
can work from any directory• set CPack option variables if needed (license, version, etc.)• include(CPack)
• Run CPack• make package (create all packages)• make package_source (create source package)• cpack -C CPackConfig.cmake -G NSIS• cpack -C CPackConfig.cmake -G ZIP• cpack -C CPackSourceConfig.cmake -G ZIP
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 48
Using CPack Simple Example
project(CoolStuff)add_executable(coolstuff coolstuff.cxx)install(TARGETS coolstuff RUNTIME DESTINATION bin)SET(CPACK_PACKAGE_EXECUTABLES "coolstuff" "The Cool Stuff") # used by NSISinclude(CPack)
$ make package[100%] Built target coolstuffRun CPack packaging tool...CPack: Create package using NSISCPack: Install projectsCPack: - Run preinstall target for: CoolStuffCPack: - Install project: CoolStuffCPack: Compress packageCPack: Finalize packageCPack: Package C:/CoolStuffExample/CoolStuff-0.1.1-win32.exe generated.CPack: Create package using ZIPCPack: Install projectsCPack: - Run preinstall target for: CoolStuffCPack: - Install project: CoolStuffCPack: Compress packageCPack: Finalize packageCPack: Package C:/CoolStuffExample/CoolStuff-0.1.1-win32.zip generated.
June 16-17, 2008 NA-MIC, 2008 June Workshop, IHK Akademie Westerham 49June 16-17, 2008NA-MIC, 2008 June Workshop, IHK Akademie Westerham 49
Find more about CPack…
• Online documentation• http://cmake.org• Wiki
• http://www.cmake.org/Wiki/CMake:Packaging_With_CPack• http://www.cmake.org/Wiki/CMake:CPackConfiguration• http://www.cmake.org/Wiki/CMake:CPackPackageGenerators
• Books• Mastering CMake