Build Audit and Processes
A line of sight from DEV to PROD
Alec Clews
http://alecthegeek.wordpress.com/http://twitter.com/alecthegeekhttp://github.com/alecthegeek
Voga Consultinghttp://voga.com.au/
Licensed under Creative Commons Attribution-Share Alike 2.5
Australia License
Summary
Build software in an audited and managed environment as part of
Configuration Management (CM)
Using Open Source Tools (Make and Git) as a framework for a
controlled build process
Agenda
The different types of build and the place of the system
build
Auditing and logging requirements for system build
Implementing a CM System Build FrameworkThe Makefile
The Driver script
The output
Motivation
This paper takes a commercial view on the process of software
change management
Desire to provide a business value quickly
Communicate effectively between team members
Reduce the amount of unsatisfying project workrework, report
writing,etc...
The Change Lifecyle
Document and agree the change
Implement the change (hack the code and desktop
build/testing)
Integration build and unit test
System Build in a controlled environment
System Testing (or UAT, or beta, or ...)
Release for distribution or production
Continuous Integration vs. System Builds
Continuous Integration (c.f. Martin Fowler)Builds rapidly and
frequently
Runs unit tests
Sets up for unit testing framework
Happens automagically
May only build small pieces
May include code instrumentation and additional debugging
options
System Builds
Build for production, or pre-production Compiler optimisation,
different configurations etc.
Runs on a different schedule
Needs documentation
May have large scope
How might the system build be done?
Fetch the sources and build them
Ask the developer for already built software!
Issues with these approaches. How can we answer such questions
as?Which source versions were used
Under what configuration was this built?Compiler switches, which
libraries and tools (the build configuration)
Can we re-create the build?
re-create the build is a myth
After problems often asked to re-create the environment for
forensic testing
But if we re-run the build process that does not assure that we
will get the same resultVersion No, dates and time change: trivial
difference masks possibly other major changes in our files
We usually don't have a record of the previous build
configuration and can't re-create it anyway
Result: we can't reliably get the same result twice
Consistent Build Process
It is possible to define a consistent build process by using
tools such as Make, Ant or CONS and some additional scaffolding
Must be done in a controlled environment
Must be done with pre-defined scripts and documented process
Scripts should be under version control
A record should be keep of all system builds
Baseline build results
Use the consistent process
After the system build save the build directory tree (source
code, makefiles, build log, built files) into an archive (consider
burning a CD)This is our built baseline. We can always take this
off the shelf and see what software we used to re-create the
build.
However it's often hard to identify source versions and what
build environment was used
Manage Build Environments
Consider building on virtual hosts
The host image can be archived as changes are made
There is a better way
As we build new files identifySource files and their version
Tools and their settings
Configuration used (platforms, packages etc.)
At the completion of a successful build we can preserve the
outputs in Version ControlIncluding a complete audit trail of the
environment, the build log and the input files
The result
In our version control system we have a complete audit trail of
each system build
We can extract our built files at any time now we can re-create
the build just by fetching files from version control
If our change ticket system is integrated into version control
we can identify which changes when into a build
Why do we want to do this
We want to close the change management loopA Line of sight from
change request to deployment
Improved complianceN.B. May be a legal requirement to keep the
external auditors happy. Sometimes process auditors can shut down
the business!
Our KPI may depend on SLAs and process outcomes
Outcomes
Better forensics what is really running in production?
Solid documentation
Reduced rework
Example Implementation
There are a variety of ways to this
Following example usesA simple C project (5 source files)
Make
Git distributed version control tool
Commercial products also available.e.g. Atlassian Bamboo,
Electric Cloud
Git
Now a major player in Version Control market
Baselines are created using tags. A tag is a label attached to a
commit
Some concepts and words
Build Tags. Tags created for each build we perform. c.f. Build
Baseline
A Build Configuration. Documentation of the environment in which
the build is being done (e.g. Platform, compiler, library
versions...)
The Makefile
To create the audit trail we must know ALL dependenciesi.e. All
the source files used to build a target file
Peter Miller already did the hard work for meHis paper explains
an effective way to implement makefiles with a complete dynamic
dependencies
A side affect is that your makefile is more efficient!
Dependencies in C
A list of input files that an output file depends on. e.g.a.o:
a.c.a.h b.h
main: main.o a.o b.o
Dependencies can change any time we edit the source code,
including editing any header files
We must re-calculate the dependencies at make time if the source
files or header files change
Dynamic dependency
# Generate a dependency file
$(DEP_DIR)/%.d: $(SRC_DIR)/%.c$(CC) $(CFLAGS) -MM -MG -MT
$(OBJ_DIR)/$*.o -MT \
$(DEP_DIR)/$*.d -MF $@ $ b/$(@F).bfl echo $(CC) $(CFLAGS) | \git
hash-object --stdin >> b/$(@F).bfl echo "BLD_ENV:
$(BUILDCONFIG)" >> b/$(@F).bfl for i in $^ $(MAKEFILE_LIST);
do \ echo -n "$$i:" >> b/$(@F).bfl \ git hash-object $$i
>> b/$(@F).bfl \done
Example BFL
BLD_FLAGS [cc -ansi] \
a275bd53898b96ff56682992cfef600c8d9172b6BLD_ENV
8b137891791fe96927ad78e64b0aad7bded08bdcsrc/a.c
fbb2a1faee180bd3c81d6d00bb39d1bb68b51725h/a.h
8ed2304c36b99ef1fd9a1f22a41ee2afe4eb995emakefile
34e0b0adb203123c8ebfdf2c4fb059ff0c942e27d/main.d
3045b0da533f900bf1ab044de22725475d4d171cd/a.d
8bbf937c5cc911242a3b3fdc531c5d1a5cf7f37cd/b.d
bba81a7805d40e267b139a01f69c15c676d0a82d
Storing the target & BFL
Every time we commit a new built target to the repositoryAdd the
BFL to the repository
Add the new target to the repository
git add b/$(@F).bflgit add $(@)
Driver script sets up
Build ID derived from sha1 + timestamp
The Build Configuration. A simple string representing the build
environment
host name of build machine usually not important unless
problem
DEMO
Further work
Better Error Checking!!
Save and re-use built intermediate files(but see constraints
later)
Update tickets with Fixed In Build information
Further work contd.
Integrate with meta make toolse.g. cmake, autoconf/automake,
...
Automate the production of release notes and other reports from
the BFL and other version control information
How to re-use targets in later Builds
Validate sha1 values on source files on original BFL against
source files in Build areas must match
Check build configuration for target on build tag against
current environment must match
Check command line switches must match
We don't care about dates and time
Outstanding Questions
Does this scale?More complex directory structures
Multiple languages (e.g. grammar files & assembler)
What about languages and tools with hard to calculate
dependencies (e.g. Java, code generators,...)
Outstanding Questions contd
Which other version control tools workSVN version implemented in
2006
Will this work across platformsdifferences in tools
problems because text files change? (tool dependent)
....
Further reading
Example scripts written for this paper
http://github.com/alecthegeek/gitbuildingRecursive Make
Considered Harmful by Peter Miller
http://www.canb.auug.org.au/~millerp/rmch/recu-make-cons-harm.html
Continuous Integration by Martin Fowler
http://www.martinfowler.com/articles/continuousIntegration.html
Click to edit the outline text formatSecond Outline LevelThird
Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline
LevelSeventh Outline LevelEighth Outline LevelNinth Outline
Level
