A Scalable Software Build Accelerator

A Scalable Software Build Accelerator: Break the Build Bottleneck with Faster, More Accurate Builds

John Ousterhout Founder and Chairman

John Graham-Cumming Founder

Executive Summary

For organizations that depend on software innovation, a slow software

build process can be a bottleneck for the entire company. Slow build

times not only impact engineering efficiency, they also affect product

quality and company agility. Furthermore, diagnosing build problems is

difficult or impossible: cryptic output in Make log files can be hard to

decipher and is difficult to relate to the individual build steps in a large

build. Electric Cloud's core products, ElectricAccelerator and

ElectricInsight, solve these problems by reducing software build times

dramatically and providing graphical insight into the performance and

structure of builds at a level impossible with existing tools.

The solution to the build speed problem is at a first glance simple: create

a distributed version of industry standard build tools (such as GNU Make

or Microsoft NMAKE) that distributes individual job steps in parallel to a

cluster of inexpensive servers. Over the years, many attempts have been

made to create “parallel” or “distributed” build systems. However, none

are in widespread use due to dependency issues and distributed

computing problems that lead to broken builds.

Electric Cloud, Inc. has developed an automated dependency

management system that makes parallel builds safe, scalable, and

efficient. ElectricAccelerator is a software build accelerator that

significantly reduces software build times by distributing the build over a

large cluster of inexpensive servers. ElectricAccelerator uses its patented

dependency management system to identify and fix problems in real time

that would break traditional parallel builds. ElectricAccelerator plugs

seamlessly into existing Make- or Visual Studio-based infrastructures, and

includes Web-based reporting and management tools.

ElectricAccelerator has been proven in some of the most demanding

software organizations and against large open-source projects.

In one organization, a product took four and a half hours to build on a single system. It now builds 20x faster on a 30-node ElectricAccelerator cluster, finishing in less than 13 minutes.

2. v2007.06

whitepaper ©Electric Cloud, Inc. All rights reserved.

In another organization, a product took 3 hours and 12 minutes to build on a single system. It now builds 16x faster on a 30-node ElectricAccelerator cluster, finishing in less than 12 minutes.

The open-source Samba file and print server takes 16 minutes to build on a single processor, but builds 16x faster, in 58 seconds, on a 20-node ElectricAccelerator cluster.

The open-source MySQL database takes over 23 minutes to build on a single processor. It builds 12x faster, in 1 minute 54 seconds, on a 20-node ElectricAccelerator cluster.

ElectricAccelerator improves the software development process by

reducing build times, so development teams can reduce costs, shorten

time-to-market, and improve quality and customer satisfaction.

ElectricInsight is Electric Cloud’s build visualization tool. A companion

tool to ElectricAccelerator, it takes advantage of structural information

recorded by ElectricAccelerator and provides a graphical display of this

information that makes it easy to understand the behavior of builds, tune

performance, and quickly debug broken builds.

3. v2007.06


The Real Impact of Slow Builds

Every software engineer has experienced the frustration and delay caused

by slow software builds. Every VP of Engineering has dreamed of rapid,

accurate builds that meet the needs of overextended QA teams, enable

rapid innovation to acquire new customers and allow timely turnaround

for critical bug fixes.

For most organizations such builds are quite simply dreams.

Although the widespread Make-based build infrastructure used by many

large software organizations has been around for over 20 years, it has

lagged behind advanced IDEs, new languages, and template libraries that

churn out more and more lines of code and result in slower and slower

builds.

As these projects grow, they are typically partitioned into a deep

hierarchy of directories with dependencies hidden in the arcane language

of Make. Trying to untangle a complex Makefile is a black art; it creates a

vast legacy of code that any solution must support without change.

Additionally, these deeply recursive Makes lead to brittleness as

dependencies between directories are often implicitly defined by the order

in which jobs are run without taking advantage of Make’s dependency

mechanism.

We have met with hundreds of commercial software development teams

and very few have production build times less than two hours. More than

half of the projects had build times in the 5-10 hour range, and a few

organizations reported that build times had reached 40 hours or more at

some point. Furthermore, the build issues are compounded since most

organizations must simultaneously support multiple platforms and product

versions.

A large tangible cost is engineering team productivity: the time engineers

spend waiting for their builds to complete. Most developers spend at

least two hours per week waiting for builds to complete, and in some

organizations developers spend as many as 10 hours per week waiting for

builds during some development phases. Engineers are often forced to

4. v2007.06


switch back to a bug fix checked in the day before because an overnight

build has finally shown that there was a problem.

Another cost is those times in the software development process where

the team is constrained by the build process. Typically, this is during the

“integration storm” phase of a release. Integration storms are periods of

instability that occur several times during a release cycle when developers

synchronize their changes into the main code line. Inevitably there are

interactions between the changes made by different developers, causing

broken builds and incorrect product behavior. It can take anywhere from

several days to several weeks to iron out all the problems; during this

period virtually the entire engineering organization is tied up fixing

problems or waiting for the code line to stabilize. If builds take overnight,

the organization may not be able to fix more than one or two problems

per day.

Long builds can also impact product quality. If builds take too long,

developers don’t have the resources to quickly do a complete re-build of

their product before they check in. It’s not uncommon for their changes

to break the build, but that problem is not discovered until the nightly

build runs. If that build was intended for the QA team, then after the

problem has been identified and fixed, the team is forced to wait for the

next nightly build. This means one less day testing that build. Since the

period between QA drop builds is often fixed, there is not enough time to

execute all of the scenarios on that build. If this happens near the end of

a release cycle, it’s possible for a bug to make it out to the field, where

customers encounter it.

Traditional Approaches to Improving Build Performance

There have been numerous attempts to improve the performance of Make

over the last two decades. They fall into two general classes: “faster”

approaches that execute pieces of the build in parallel, and “smarter”

approaches that avoid work entirely.

5. v2007.06


SMP Hardware One solution to build speed is to buy a large multiprocessor machine and

use GNU Make’s -j switch to force it to run multiple jobs in parallel on the

same machine.

Although this approach gives some speedup (typically 2-4x), it does not

scale well because of the high per-CPU cost in a multiprocessor machine

and because incomplete dependencies (especially in hierarchical Makes)

become the build’s Achilles’ Heel. With incomplete dependencies, the

parallel build tends to reorder build steps in ways that break the build,

leading to unpredictable and inaccurate builds.

ElectricAccelerator uses its patented dependency management system to

identify and fix problems in real time that would break traditional parallel

builds. With this perfect information, ElectricAccelerator can achieve

speedups of up to 20x.

In addition, builds using GNU Make’s -j switch produce log output that

differs with each run, which makes it difficult to verify and debug builds.

Electric Make ensures that the build log is written in the same order every

time.

Distributed Builds A variation of the parallel build approach is distributed builds, where

builds are run in parallel using a cluster of machines instead of a

multiprocessor.

In addition to all of the build ordering problems of parallel builds

described in the previous section, this approach is fraught with difficulties.

The clocks on the remote machines must be synchronized to ensure that

Make’s time stamp-based dependencies work correctly. All of the

machines must be mounted on a reliable shared file system. Any failure

on an individual node will cause the build to fail. Incomplete dependency

information can still cause inaccurate build results. Furthermore, the time

taken invoking a job (e.g. with ‘rsh’) can be high in traditional

approaches, limiting the performance benefits.

The ElectricAccelerator architecture eliminates several distributed-systems

issues that threaten the correctness or robustness of distributed builds.

6. v2007.06


Electric Make manages timestamps centrally to avoid clock

synchronization problems. It communicates with the nodes providing all

files through a reliable protocol that can self-heal if a node fails, thus

eliminating the need for a mounted file system and ensuring build

accuracy every time regardless of hardware or operating system failure.

Additionally, Electric Make uses a fast binary protocol to send jobs to

nodes to reduce overhead and help achieve massively distributed builds.

Manually Partition Makefiles Some organizations have taken the extreme step of manually breaking a

build up into a small number of steps that are run in parallel on different

machines. This difficult and error-prone task requires detailed knowledge

of Makefile internals and typically yields only small speedups as

partitioning the build into smaller and smaller steps requires enormous

effort to ensure correct results.

ElectricAccelerator completely automates the parallelization of builds at

the lowest level possible: individual job steps. It merges hierarchical

builds into a single build and uses a multi-threaded architecture to run as

many jobs as possible at the same time.

Build Avoidance Another approach for improving build performance is to reduce the

amount of work that must be done, either by doing better incremental

builds or by sharing results between independent builds. This is typically

done by trying to rely on incremental builds rather than complete builds.

Very few build organizations are willing to do incremental builds for their

production software; instead they rely on complete builds for QA and

release. The risk of a broken build and complexity of ensuring that a

build is accurate across multiple product components leads most

organizations to rely on full clean builds.

Summary In summary, each of the approaches described above offers the potential

for speeding up builds, but each makes the build process more brittle by

increasing the risk that a build will fail or that it will be inconsistent with

the sources.

7. v2007.06


None of the organizations we have talked with has been able to achieve

more than a 6x speedup reliable enough for production builds, and only a

very few have achieved even a 3x speedup after significant investments

of time and resources. Most organizations run their builds completely

sequentially or with only a small speedup, in order to keep the process as

reliable as possible.

ElectricAccelerator: a Highly Scalable Solution

ElectricAccelerator is a software build accelerator that takes advantage of

the abundant parallelism available in builds and capitalizes on recent

technology improvements in inexpensive servers and fast networks.

Instead of running a build sequentially on a single processor,

ElectricAccelerator executes pieces of the build in parallel on a large

cluster of inexpensive servers. (see Figure 1). ElectricAccelerator has

four main software components:

Electric Make, a new version of Make that reads Makefiles, analyzes

dependencies, and coordinates activities on the nodes. Electric Make also

acts as a file server for the nodes in the build cluster.

Electric File System, a special-purpose file system driver that runs on the

nodes in the cluster. It monitors every file access to provide the complete

dependency information that allows ElectricAccelerator to automatically

detect and correct out-of-order build steps.

Electric Agent, a user-level component that runs on the nodes serves as

an intermediary between Electric Make and Electric File System, and runs

jobs at Electric Make's request.

Cluster Manager, a Web server that allocates nodes for individual builds

and provides reporting and management tools.

To a user, ElectricAccelerator appears identical to other versions of Make

or Visual Studio. Electric Make can be invoked anywhere that other

versions of Make might be invoked, such as engineer workstations or

dedicated build machines. Electric Make can be invoked interactively or

8. v2007.06


as part of build scripts. The use of a cluster for the builds is invisible to

the Electric Make user, except that the builds run much faster.

Node

Agent

Electric File System

Node

Agent


Node

Agent


Cluster

Build Machine

Electric MakeSchedulerFile Server

TCP/IP

Cluster Manager

Web Server

DB

HTTP

HTTP

Node

User LevelKernel

Agent


Figure 1: The ElectricAccelerator Architecture

Massively Distributed Builds To achieve build speedups of up to 20x, ElectricAccelerator couples a

cluster of servers running as many jobs in parallel as possible with the

kernel-level Electric File System. The Electric File System monitors every

file access to compute dependencies automatically and ensure that build

results are perfect every time.

When a build starts, Electric Make reads existing Makefiles and

determines the list of jobs that need to be executed. Electric Make then

communicates with the Cluster Manager component. Cluster Manager

controls access to the cluster of nodes and allocates a collection of nodes

to Electric Make.

The Cluster Manager controls access, but also adjudicates between

competing builds. If multiple builds are requested simultaneously on a

build cluster, the Cluster Manager is able to fairly allocate cluster nodes to

the builds taking into account the available build resources, the

requirements of a specific build and build priorities and related policies set

in the Cluster Manager.

9. v2007.06


For example, the Cluster Manager might be configured to allow general

use of the cluster with automatic sharing and allow a build manager to

take over the entire cluster when a build must be rapidly produced (for

example, for address a critical bug fix for a customer). Cluster Manager

will ensure that a build manager is able to use the entire cluster without

terminating other running builds. Low priority builds would be placed in

a wait state until the top priority build was completed and then would be

automatically continued. After continuing, normal sharing of the cluster

would resume.

Electric Make then instructs each node to perform jobs on its behalf. A

node running a job reads and writes a variety of files (such as source and

object files) which are passed dynamically across the network via Electric

Make using a fast binary protocol developed by Electric Cloud. When each

job completes, it sends its results (such as files written and log output) to

Electric Make for final storage on disk. File data is cached on nodes for

the life of a build, in order to minimize network traffic for files that are

reused.

In addition the Electric File System running on the nodes captures every

single file access performed by jobs and provides that information back to

Electric Make. Using that information, coupled with the Makefiles’

dependency information, Electric Make is able to determine the exact

relationship between jobs and files and fix any missing dependencies in

real time.

If Electric Make detects that two jobs were performed in the wrong order

because missing Makefile dependency information made them appear

independent, it will automatically reschedule them for completion in the

correct order and make a note of the missing dependency.

Electric Make saves this missing dependency information to increase the

performance of subsequent runs. Successive builds use the additional

dependency information to run the build steps in the correct order, further

improving performance. Electric Make updates this missing dependency

information after every build, so that even as Makefiles evolve,

performance is automatically maintained. Other approaches to parallel

builds require a large on-going investment in Makefile maintenance to

10. v2007.06


keep the build from breaking, and to ensure that parallel performance is

preserved.

At the end of the build, Electric Make communicates build results to the

Cluster Manager where they are made available through a web-based

interface for reporting and management.

ElectricAccelerator’s Cluster Manager

Accurate Incremental Builds ElectricAccelerator uses its perfect information about dependencies for

more than just safe parallel builds. It also uses the dependency

information in a feature called eDepend, which enables accurate

incremental builds. Historically, incremental builds have been unreliable:

they only work if perfect dependency information is available, so that

Make knows which subset of files must be regenerated after a particular

file is changed. Since dependencies were not perfect, incremental builds

would sometimes fail to regenerate files affected by a change, so the only

safe approach was to do a complete rebuild.

Electric Make uses dependency information collected during previous

builds to decide what to regenerate during incremental builds. This

makes incremental builds accurate and reliable, and eliminates the need

for clumsy, slow ‘make depend’ steps. eDepend is superior to

dependency searching techniques (such as the free tool makedepend)

because it is completely language agnostic. eDepend detects

11. v2007.06


dependencies at the file level without the need to search within files and

understand individual languages. This language independence means

that eDepend also detects dependencies between object files (for

example, eDepend can automatically detect a dependency between an

executable and a library that the executable is linked with).

Hierarchical Builds Another patented technique enables Electric Make to achieve massively

distributed builds by flattening deep hierarchies of nested Makes. In a

typical recursive build, a top-level Make executes a set of Makes in

subdirectories to make individual components.

Electric Make treats this hierarchy as a single Make by merging

dependency information from the complete hierarchy and identifying jobs

that can be safely run in parallel. In this way it achieves massive

concurrency with the largest number of jobs running at the same time,

even from different Makes.

To ensure that the build is 100% accurate, Electric Make's eDepend

feature also analyzes file usage information from the Electric File System

to automatically detect dependencies between recursive Makes that were

never specified in the Makefiles.

Plug Compatible Electric Cloud’s replacement for the Make program, known as Electric

Make, is compatible with GNU Make, Microsoft NMAKE and Visual Studio.

It understands the complete GNU Make and Microsoft NMAKE languages

and has identical command-line options.

Starting to use Electric Make is a simple matter of changing invocations of

gmake or nmake to eMake with an appropriate command-line option

specifying the emulation mode. For the ultimate in slot-in deployment, if

the eMake program is renamed to gmake or nmake it automatically

determines whether it is implementing GNU Make or Microsoft NMAKE.

Because Electric Make requires no Makefile changes it’s easy to deploy

and it’s even easier to verify. In addition to reading standard Makefiles

and accepting standard command-line options, Electric Make produces

identical log file output. For example, verifying that Electric Make is doing

12. v2007.06


the same work as an existing Make is a simple matter of running the diff

program. Electric Make even produces identical error messages in the

event of a broken build step.

The only difference is that Electric Make runs the build as much as 20x

faster.

Because Electric Make looks just like Make, existing build scripts (such as

Perl wrappers) can be run without change. Electric Make plugs right into

the existing build system.

Robust ElectricAccelerator’s three user-level components (Electric Make, Cluster

Manager and Electric Agent) are in constant communication ensuring that

all are operating correctly so that failures are detected rapidly and fixed in

real time.

As a build is running, Electric Make keeps track of the files and jobs

running on each machine on the cluster. In the event that a cluster node

fails, Electric Make automatically detects the failure and reruns the

incomplete job on another node.

Electric Make communicates node failures to the Cluster Manager for

reporting and management purposes, and continues the build maintaining

high speed and build accuracy.

As well as being able to handle cluster failures, ElectricAccelerator’s

unique Electric File System automatically makes up for deficiencies in

Makefiles where missing dependencies can cause traditional parallelization

methods to create incorrect builds.

Multi-Platform Although some organizations have the luxury of working on a single

platform, many face the realities of a heterogeneous world. Electric Cloud

is no different. All of the ElectricAccelerator components work on

Microsoft Windows, Sun Solaris and Linux. Electric Make emulates the

popular GNU Make and Microsoft NMAKE programs.

13. v2007.06


Real World Performance

ElectricAccelerator has been tested in some of the most demanding

enterprise software organizations and against large open-source projects.

In one organization, a product took four and a half hours to build on a

single system. It now builds 20x faster on a 30-node ElectricAccelerator

cluster, finishing is less than 13 minutes.

In another organization, a product took 3 hours and 12 minutes to build

on a single system. It now builds 16x faster on a 30-node

ElectricAccelerator cluster, finishing is less than 12 minutes.

he open-source Samba file and print server takes 16 minutes to build on

he open-source MySQL database takes over 23 minutes to build on a

T

a single processor, but it builds 16x faster, in 58 seconds, on a 20-node

ElectricAccelerator cluster.

T

single processor. It builds 12x faster, in 1 minute 54 seconds, on a 20-

node ElectricAccelerator cluster.

14. v2007.06


15. v2007.06


Build Visualization

Complementing ElectricAccelerator is El

mines extensive build information gene

Electric Make to provide unprecedented in

a large build.

With ElectricInsight it's easy to get an

build. This screen shot shows an exampl

h 13m) and consists of 1,457 jobs. In the terminology of ElectricInsight,

job is an individual step from a Makefile such as a compilation or link.

this case 'node0') and a horizontal bar chart. Each bar represents a single job and

the actual jobs in detail.

ectricInsight, a graphical tool that

rated when a build is run with

formation about the structure of

overview of the running of a long

e build that last 4,380s (about

1

a

The ElectricInsight display shows the name of the machine running the build (in

the bars length is proportional to that job's running time. The bars are ordered from left to right in the same order as the jobs were executed by make.

Just glancing at the display gives instant information about the build. In

addition to the figures displayed in the bottom right hand corner (where

you can find the total number of jobs executed and running time in

seconds), the bar chart shows that:

• There are a number of very long running jobs. Very early on in

the build there's a single job (shown as a large patch of blue)

which takes up about 15% of the build.

• Areas of black are when many small jobs are running in

succession with tight packing. Zooming on the display will reveal

• There's a gap about half way through the build when no job is

running. That's clearly a waste of time that needs investigatin

g.

5mass/debug/a2a5mass.so and

spent 219s (or about 5% of the total time) on it.

One job

Investig l tool like ElectricInsight.

Firs it'

over the

he name of the binary being built is revealed:

build/bin/output/binaryserver/ debug/a1binaryserver.so. And it's

taking 639.21s; over 10 minutes to compile! ElectricInsight shows that

this one job consumes 14.59% of the entire build.

If the name of the binary isn't enough to track down in the Makefile the

job that's running, a double click on the blue bar brings up an additional

window containing detailed information on the job. Here are

a1binaryserver.so's details.

• One of the large jobs is highlighted (by hovering the cursor over

it) in pink and the bottom half of the ElectricInsight display shows

the details for the job. In this case the job built

build/motor/output/sharea

in the example build is taking 15% of the total build time.

ating the job is trivial with a graphica

t, s obvious which jobs are taking up most of the time, and hovering

longest one reveals information about it.

T

16. v2007.06


The Output pane shows the actual output in the Make log that's

associated with this one job. It reveals that the job consists of deleting

four files using the rm command, followed by compiling the object using

mpile. All the arguments and options for each command can be clicking

(in this case simply Makefile) and the line

co

the Show Commands check box.

The Job Details display also helps narrow down the job even further by

giving the name of the Makefile

17. v2007.06


number within the Makefile (in this case 1953) where the rule for this job

is defined.

ElectricInsight also provides valuable information when a build is run in

parallel with the ElectricAccelerator system. The same example build

when run against an ElectricAccelerator build cluster of ten nodes is

shown here.

With ElectricAccelerator, this build has dropped to running in 20m. If the

build manager wants to reduce the build time even more, the

ElectricInsight display shows areas for further optimization:

• The same long running jobs (a1binaryserver.so, a2motor.so,

ect. The build runs initially for about 50% of the time with

jobs on every node consuming the CPU resources. After that, a

d by explicit dependency information in the Makefile (i.e. the

etc.) still dominate the build and optimizing them would bring

down the overall time.

• There are a number of large gaps meaning that parallelism isn't

perf

number of large jobs block the rest of the build. This blocking

effect occurs because the jobs that run at the end of the build are

waiting for the large jobs to complete. Typically the blockage is

cause

final jobs are not permitted to run until specific objects have been

built).

In summary, optimizing the long jobs will make this build faster (and

much faster in the parallel case).

After optimizing the longest running jobs (perhaps by adding more

memory to the machines they are running on, or getting developer help in

breaking the project apart), ElectricInsight can be used to visualize the

result.

18. v2007.06


From the ElectricInsight diagram it's clear that all the nodes in the

Conclusion

ElectricAccelerator cluster are being used fully to run the build, leading to

a much reduced build time.

Fast, reliable builds are now a reality.

Electric Cloud's solution provides enterprise-class software that

accelerates the time consuming and costly software build process by as

much as 20x and removes the guess work from keeping builds running

accurately and quickly. ElectricAccelerator transforms inexpensive

servers into highly scalable clusters so that eight-hour builds can finish in

30 minutes. ElectricInsight provides never-before-seen information about

the structure, timing and dependencies in every build. And eDepend

means the end of long dependency generation steps and perfect

incremental builds every time.

19. v2007.06


With Electric Cloud, development teams can reduce costs, shorten time-

to-market, and improve overall product quality.

About Electric Cloud

Electric Cloud is the leading provider of software production management

solutions that automate, accelerate, and analyze the software

development tasks that follow the check-in of new code. These include

the software build, package, test and deploy processes. The company's

patented and award-winning solutions improve productivity in the face of

increasing product complexity and time-to-market pressures for software

delivery. In addition to ElectricAccelerator and ElectricInsight, Electric

Cloud offers the only enterprise-class build and release management

solution, called ElectricCommander.

Leading companies such as Qualcomm, Intuit, and Expedia rely on

Electric Cloud's Software Production Management solutions to change

ftware production from a liability to a competitive advantage. For

lectric Cloud, ElectricInsight, ElectricAccelerator, ElectricCommander and

Electric Make are trademarks of Electric Cloud. Other company and

product names may be trademarks of their respective owners.

so

customer inquiries please contact Electric Cloud at (650) 968-2950 or

www.electric-cloud.com.

E

20. v2007.06


Technology

A Scalable Software Build Accelerator