The Top 10 Blunders in Integration and Testing - And How to Avoid Them - Rob Cheng (Borland)

8/4/2019 The Top 10 Blunders in Integration and Testing - And How to Avoid Them - Rob Cheng (Borland)

http://slidepdf.com/reader/full/the-top-10-blunders-in-integration-and-testing-and-how-to-avoid-them-rob 1/14

The Top 10 Blunders In Integration And

Testing ... And How To Avoid Them

Rob Cheng, Borland Software Corp.,

Burke Cox, Stelligent Inc.,

and John Minnihan, Freepository Corp.

W h i t e P a p e r

M a r c h 2 0 0 7



The Top 10 Blunders in Integration and Testing ... and How to Avoid Them

2

Contents

Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3

The Bumpy Road to Software Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4

The Top 10 Blunders in Integration and Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5

Blunder # 1: Building Infrequently

The Solution: Continuous Integration Enabled by Build Automation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5

Blunder #2: Not Validating Changes Against Target Environments

The Solution: Server-Based Builds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6

Blunder #3: Broken Builds

The Solution: Sandboxing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7

Blunder #4: Not Testing the Build

The Solution: Automated Build Validation Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7

Blunder #5: Infrequent Check-Ins

The Solution: Frequent Check-Ins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8

Blunder #6: Not Publishing Builds

The Solution: Build and Distribution Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9

Blunder #7: Interpreting Code Coverage in Isolation

The Solution: Relative Coverage and Trend Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9

Blunder #8: Lack of Visibility

The Solution: Visibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10

Blunder #9: Relying Only on Unit Tests

The Solution: Comprehensive Build Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11

Blunder #10: Ignoring History

The Solution: Capture and Mine Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11

Using Software to Automate and Monitor Best Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12

About the Authors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14




3

Executive Summary

Software is the engine of the enterprise and one of the most strategic areas of investment for companies

today. Most senior executives consider software a primary vehicle through which they can achieve

competitive advantage. So the fact that it remains the most inefficient and unpredictable area within IT

today is a serious problem. As illustrated in Figure 1, most software projects are either delivered late, over

budget, don’t meet customer needs, or are cancelled altogether.1 And software failures across all industries

– from car manufacturers to airlines – cost the American economy billions of dollars annually.2

Figure 1: Common Problems in Software Development

What’s the source of the problem? In many ways, software development remains a relatively immature

industry. Most development teams continue to create software the way cars were initially built – by hand,

one at a time. This approach allows for individual artistry and creativity, which is essential to developing

innovative, quality software. But to address today’s challenges, this approach must be balanced by a more

structured, automated approach to development that optimizes production and supports continual

process and product improvement.

Integration and testing – the focus of this paper – is just one of many steps in the software development

process that can benefit tremendously from a more structured, automated process. Failures in integration

and testing not only impact product quality and delivery deadlines, but also have a trickle-down effect

that can bring the activities of those responsible for testing, documenting, marketing, and selling

products to a grinding halt. The end result is lost time to market, fewer sales, and lower customer

satisfaction and retention.

This paper helps you understand both the business and technical impacts of most common integration

and testing blunders and provides specific recommendations on how to address them. You’ll learn about

industry best practices that you can begin to implement today, as well as recommended technology

investments that can vastly simplify and accelerate the best practice implementation process.

0%

100%

Cancelled

544666

90%0%

DeliveredOver

Budget

NotConsideredSuccessful

DeliveredLate

300

Source: 2004 Standish CHAOS Report

1 The CHAOS Report, The Standish Group International, Inc.,2004.2 According to the 2002 American National Institute of Standards (NIST), software failures cost the United States economy $59 billion




4

The Bumpy Road to Software Development

Software runs businesses today, and for many organizations, it is their primary source of competitive advantage. Executivesat all levels understand that right software product deployed at the right time can help them automate manual, costly

processes; cut operational costs and boost efficiency; accelerate innovation internally and with partners; and differentiate

their business by improving product and service quality.

But software development remains a bumpy road for most software engineering teams. Many companies – particularly those

for whom software development is not their core business – lack the right tools and best practices needed to simplify and

automate build and test activities. Furthermore, developers continue to be trained to focus on coding individual features, not

on testing or the assembly of software components into a finished product. This emphasis is usually reinforced during the

development process, as management teams tend to provide developers with lots of positive feedback about a cool new

feature – not about how well they have integrated new code. So it’s not surprising that development teams can forget that the

end goal is a functional, reliable, and easy to maintain final product.

As a result, development teams can find it immensely difficult to deliver quality software on a regular basis. They are often

caught off guard by unexpected defects and other quality control issues. Repeated delays and expensive rework are the norm

rather than the exception. Development cycles are often slow due to lengthy integration and testing phases, during which no

one can add new features or customer enhancements. And rising support costs – caused by high defect rates – cut into the

bottom line.

Complicating matters is the fact that most managers lack the visibility needed to properly oversee the development of a

quality product or enforce best practices for integration and testing. Without complete visibility into the development

process, how can they be sure of the stability and performance of their software? How they coordinate the activities of their

internal and offshore teams in ways that ensure adequate code coverage, eliminate code redundancy, and enforce consistent

use of best practices? How can they determine how much more work will be needed before an application is ready to ship?Without answers to these questions – and ample time to address problems that are identified, it’s extremely difficult to

deliver quality software in a reliable, predictable manner. This not only frustrates management across the organization –

particularly those responsible for testing, documenting, selling, and marketing the product – but also represents a significant

source of risk for your business in the form of lost time and revenues, damaged brands, and even legal liability.

Do these challenges sound familiar? As a software development manager, you are likely flooded with new requirements for

your projects and are just trying to keep up. And as a result, you have little time to think about continuous process and

quality improvement as it pertains to building, testing, and delivering software. The good news is that you can address the

root causes of these challenges by implementing the right tools and best practices. By taking the time to examine your

organization’s existing integration and testing practices – and making strategic changes – you’ll actually save far more time

for actual development and turn those efficiencies into competitive advantage.




5

The Top 10 Blunders in Integration and Testing

To help you assess the current state of your software development processes, Borland has compiled a list of the top ten

integration and testing blunders. Compiled through consultation with industry experts and our own work with countless

software organizations worldwide, the top ten list can help you identify areas of weakness that are holding your business

back from accelerating the development of reliable software products.

Building Infrequently

According to Steve McConnell, author of Code Complete, only 25 to 30 percent of software development

organizations are scheduling daily build events.3 And this means that the majority of software development teams

are building far too infrequently. Why? Because most developers are trained primarily on coding features (algorithms, data

structures, etc.) and few university curriculums spend much time on collaborative software construction. Developers

graduate with little or no exposure to build automation software or source code management systems, which are essential to

working efficiently and effectively with others on a software project. And finally, putting pieces of code together is simply nota problem that interests engineers, who enjoy focusing on solving problems.

As a result, building frequently is often a low priority. It’s not uncommon for organizations to schedule a massive integration

effort after weeks of developer work. And the result is usually the same: a painful software build process that takes days and

even weeks to complete and results in structural misalignment across the software. In addition to breaking the build, these

types of speculative, big-bang integrations can generate a host of problems, including the following:

• Progress comes to a grinding halt as developers scramble to fix the build.

• Costs can skyrocket as developers sift through massive amounts of code to find and fix errors.

• QA teams may waste time identifying bugs that are really the result of developer conflicts that should have been resolved before

testing began.

• It’s difficult to plan a project around such high-risk events. For example, they can’t predict how long it will take to integrate the various

pieces of code, address errors, or finalize the product for release to customers.

• It’s impossible to gauge the quality of the software. Big-bang integrations only provide development teams with quantitative information

about the build (for example, did the code compile or not?) – not qualitative information regarding code coverage, compliance with

coding standards, and completion of unit tests.

The Solution: Continuous Integration Enabled by Build Automation

Most of these issues can be avoided by building frequently – preferably continuously, but at the very least daily. Building

software is a critical validation step because it’s hard to know if something is broken until you try putting the software

together and running it. Building frequently gives you greater confidence that you know the true state of your softwaredevelopment project at any given point in time.

With continuous integration, the work completed by your developers is checked in every few hours and added to the main

line repository. Ideally, developers first perform automated builds on their own development machines by pulling down a

working copy of the main line code, integrating their changes, and fixing problems until they achieve a good build. When the

3 Code Complete. Steve McConnell, Microsoft Press. June 2004.




6

build is successful, developers can start to commit their changes to main line repository. As a rule, if the mainline build fails

– which is often caused by conflicts with recent changes made by other developers – it must be fixed immediately so that all

developers on the team can continue to work from a stable base of code.

You can optimize the efficiency of your development team by using software to automate the build process, as it can be a

time consuming and potentially error-prone process. As a best practice, you should include everything in the automated

build – from compiling to loading schemas onto databases – so that anyone can check the sources out of the version control

library and use a single command to have a running system on their machine.

Continuous integration allows everyone on your team to work from a shared, up-to-date, and stable base that’s bug-free and

works the way it’s supposed to. Ideally, the build automation software should also help you enforce best practices around testing

and integration by, for example, automatically performing unit tests – tasks that are often overlooked during manual builds.

Not Validating Changes Against Target Environments

When customers have a problem with your software, the last thing they want to hear are the words, “Well, it worked

on my machine.” Developers often assume that their local environment is an acceptable proxy for the “real world.”

And this oversight is the hidden cause of many problems that ultimately lead to high support costs, low customer

satisfaction, and lost customers. It’s unrealistic to expect that developers will have the right setup on their desktops to fully

validate their changes, especially when they need to build and test on multiple platforms or have test environments that

require external resources.

The Solution: Server-Based Builds

As a best practice, you should validate code changes regularly against all of the environments that your product is intended

to support (operating systems, application servers, databases, etc.). Each target environment should be maintained on

separate servers and duplicate customer realities in as many ways as possible. It’s equally important to set up an automated

process that publishes mainline code to the different servers modeling each environment. Automated build and test toolsmake the validation process more efficient by speeding the build process, ensuring codewise integration across a build, and

making the build process structured and predictable, rather than ad hoc. This predictability allows you to, for example,

regularly schedule events for publishing the most current mainline code to various servers. Your team can then run multi-

day tests on each stack, validating assertions that a particular build runs properly on different target environments. And if it

doesn’t run properly, your team has time to fix problems as they come up – rather than just before you go to market – which

reduces development costs and ensures that you can release the product on time.

Equally important, you should be able to reproduce the target build environments in their entirety when deploying each new

build – from reloading the database to making detailed configurations. This is vital to developing a more reliable build

process. It’s not uncommon for developers testing code on a different environment to make small tweaks to it in order to

have the build run properly. If these tweaks get lost in the rush to code the next feature – and there can be many tweaks over

time – future builds won’t run. And when tweaks are not reproduced in the build each time, they are quickly forgotten by

those who first made them.




7

Broken Builds

How many times have you – or one of your developers – been stuck waiting for someone to fix the build before you

could continue your work? For all the time developers save with visual editors and other coding productivity tools,

ten times as much is wasted due to broken builds. All it takes is one rogue developer who makes a large commit on a Friday

afternoon to destabilize the build – and prevent others from resuming work on Monday.

For companies in the business of developing software, the costs associated with broken builds are particularly high because

projects are so large and complex and so many employees are impacted by delays. The cascade effect includes not just

software engineers, but also quality assurance (because they can’t test if the software doesn’t run), documentation teams

(because they can’t document software that doesn’t work), marketing (because they aren’t sure what features will make it

into the software and need to be communicated), sales people (who can’t sell until the product’s launched), and so on.

Without a code drop, the entire software development and launch process comes to a grinding halt.

To avoid these delays, some companies actually cue up developers to perform an integration build on a separate machinebefore scheduling a manual build event. Developers must stay by the server, fixing problems until their code integrates and

the build is successful, and then the next person adds their code and repeats the process. These types of processes can do

more to prevent bad code from sitting in the repository than many continuous integration servers, the majority of which

don’t prevent error-laden code being committed. But there are far more efficient and effective ways to minimize – and even

prevent – broken builds.

The Solution: Sandboxing

You can eliminate broken builds by adopting the best practice of successfully building code before it is integrated into the

main code line, where it can affect others. Ideally, your developers should still regularly check work-in-progress into version

control, but it should be kept isolated in a virtual “sandbox” – a branch off the mainline code – until it builds successfully

and passes a battery of predetermined tests. By quarantining problematic code until it’s fixed by the responsible developer,

you can eliminate instances of broken builds and limit the impact of code defects on other developers. To return to our prior

example, if a rogue developer makes a large commit on a Friday afternoon, he or she will be the only person impacted when

the team resumes work. All others can continue working without interruption.

Not Testing the Build

Many organizations fail to take advantage of the build process as an opportunity to test earlier, more consistently,

and for a wider range of potential problems. They’re satisfied when code passes unit tests and compiles successfully.

Unit-level tests are certainly vital to the development process, but they don’t catch certain types of bugs or serious issues

relating to functional integration, for example; it is entirely possible to have developers writing perfect code that does

nothing to solve problems or further the project functionally. At the same time, you need other types of build-level tests to

help your developers pay more attention to issues such as coding standards, code duplication, code coverage, dependency

analysis, and complexity monitoring. All of these areas can impact the quality of your products, as well as the maintainability of the product’s code over time.

The Solution: Automated Build Validation Tests

One of the best ways to address these issues is to include comprehensive, automated build validation tests in the build

process. You can create self-testing code that automatically tests the majority of the code base for you. Automated software

inspection and build validation tests can detect regressions and other problems early, when they are easier to find and

cheaper to fix (not least of all because the changes in question will be recent enough for developers to remember what they




8

were thinking when they wrote that piece of code). When defects live for a long time in the code, lots of other code becomes

dependent on that defect. So, when the defect is finally found and fixed, other dependent code breaks – and development

costs go up.

You should be able to initiate the self-testing code process using a single command, and failure of any single test should

cause the build to fail. Plan ahead for which tests should be performed as part of every build and which should be performed

on a nightly or weekly basis. Tests that can be performed quickly should be run as part of every build event. Extended tests

that may take an hour or more – and would significantly delay build processes for developers – can be scheduled for the

nightly build, when they are least disruptive to your team.

It’s important to note that 1) you can’t count on tests to find every bug and 2) automated build validation tests are only as

effective as the unit tests that your developers create and the code inspection tools that they choose to deploy. The value of

automation is that it helps you enforce build-level testing as a best practice, rather than leaving it as an option for developers

rushing to complete the next new unit of code.

Infrequent Check-Ins

Making large, infrequent check-ins is not just a recipe for losing lots of work when you accidentally corrupt your

hard drive or your laptop battery explodes. It also limits the value of other best practices that you may implement,

such as frequently executing and testing the build. Continuous integration is only as continuous as the version control

activity driving it. If your developers only commit your changes once a day, there’s little point in doing anything more

frequent than daily builds.

If you check in your changes less than once a day, you significantly increase the likelihood of code conflicts with other

developers. And the more time that goes by, the greater the risk and the costs associated with fixing problems. You also lose

visibility into how code is being developed over time. For example, when a large chunk of code is checked in by a developer,

it’s hard for you as the manager to know whether that employee just stayed up all night and accomplished a lot at once, stolethe code from an unapproved library, or included open-source code that will significantly change your licensing requirements.

The Solution: Frequent Check-Ins

As a best practice, commit single code changes as they are completed and manage each one in isolation. Checking in smaller

revisions more often makes it easier to keep track of your work and track down the cause of any developer conflicts that are

discovered. It’s not unusual for developers to avoid frequent check-ins because they are worried about breaking the build.

You can remove this fear by implementing best practices such as sandboxing, whereby defective code is quarantined until

defects are fixed. Sandboxing can even be used to encourage more senior developers to perform optimistic check-ins, during

which they perform only minimal validation on their local environment before committing changes to the server. As long as

these developers are disciplined about fixing defects immediately once they are detected by automated unit and build

validation tests, this approach can save time and increase efficiency.




9

Not Publishing Builds

Collaboration is the key to successful software projects – and working code is the best artifact around which to

collaborate. But less mature development teams often fail to publish builds in a central location so that, at a

minimum, everyone can access the latest executable version and run it. Instead, they may rely upon emailed versions sent by

the person who made the most recent changes. This is a recipe for disaster, resulting in version control problems that can be

costly and time consuming to fix. You may have one version being used as a demo for by the sales force, different versions

being tested by QA staff, and several others being updated by different developers simultaneously.

The Solution: Build and Distribution Management

Giving other developers and QA engineers easy access to the latest builds helps keep everyone in sync. And best of all, it’s

easy to do. Simply determine a well-known location, define a self-describing numbering schema for files, and publish all

builds there so everyone can access them. Ideally, you want your build system to automatically publish successful builds so

you don’t have to deal with managing distributions manually. And be sure to also archive your build configurations so you

can reliably reproduce customer issues on older versions of your software.

Interpreting Code Coverage in Isolation

The most frequently cited code coverage-related blunder is probably “wasting time trying to get all the way to 100

percent code coverage” or “believing that very high coverage means your code works.” Code coverage is actually a

metric that tells you how much of the source code has been tested and gives you an indication about how much confidence

you should have in the quality of the product being produced; as developers write code, they should be creating new tests to

prove assertions that the code will build successfully and meet functional and other requirements. It’s not uncommon for

development teams to hide their code coverage figures – usually because they know they are not as high as they need to be.

Generally, if code coverage is staying the same relative to the growth of a project, that’s okay. If it’s going up, it’s even better.

If it’s going down, then you have a problem. And if you have spikes, you should be able to correlate to them to events in the

project – such as a large introduction of new, untested source code developed by outside firm that requires extensive testing

internally.

But high code coverage does not guarantee quality, as the quality of the tests can vary, and developers simply can’t test for

everything. From a management perspective, you also need to look at relative code coverage – a commonly overlooked issue.

If you can’t compare coverage between different projects, then you won’t know where scarce testing resources will be best

spent. At the same time, if you don’t know (or can’t remember) what your code coverage was last week, last month, or on

the previous version of the software, it’s hard to evaluate today’s number. Code coverage is far more useful when you and

your team can view it in context.

The Solution: Relative Coverage and Trend Analysis

As a best practice, define a target for code coverage that’s properly aligned with 1) the complexity of the product being

developed and 2) how often that code is accessed by the application. It doesn’t do you much good to increase code coverage

for simple code (e.g., with a cyclomatic complexity of less than three) because the chances of something being wrong is quite

low. But as complexity increases – or as code is accessed more frequently – set higher expectations for code coverage. For

example, plan for one test for when code complexity falls between three and ten, two tests when it falls between eleven and

twenty, and so on.




10

In addition, you can create a positive feedback loop by giving developers real-time insight into code coverage (for example,

via a live web dashboard – not just weekly reports) and providing incentives for increasing coverage (such as a catered lunch

when the target it hit). Ideally, your build automation software should monitor code coverage for you in real time and

display code coverage data for all developers to see.

Lack of Visibility

While it’s very common to see businesses adopt open source software today, it’s rare to see those businesses adopt

any of the practices that help make open source software successful. One key characteristic of open source

development projects is visibility – everyone can see what everyone else is doing, fixing, or breaking. Visibility encourages

both peer accountability and recognition of individual contributions, which is especially important when working with

distributed development teams (i.e., the right hand often doesn’t know what the left hand is doing because it’s asleep when

the other one’s working). At the same time, visibility allows all project members to learn from one another and improve their

skills over time.

But few development teams have the visibility they need today. Typical managers, for example, still use three variables to

manage software development projects: 1) requirements (what do we need to create?), 2) a calendar (how much time will it

take to complete?), and 3) a defects list (how many defects are in the code?). To estimate how much more time is needed to

complete a project or phase, they must rely on ballpark estimates provided by project leads – essentially hunches that are

based on experience, not fact.

The Solution: Visibility

But imagine if you had access to data that provided accurate, detailed insight into the development process – for example,

you could see that code complexity is increasing and code coverage is decreasing. Armed with this insight, you could better

assess where things really stand and work with project leads to proactively address problems. You would also know when

teams are working well and projects are increasing in terms of stability, quality, and timing. Armed with this knowledge, you

could create a positive team environment that celebrates individual and team accomplishments.

Your software development organization would be well served by adopting practices and technologies that help increase

visibility. This could involve, for example, publishing build and test results in a central location for everyone to see, or

sending out automated notifications when certain thresholds are reached (either positive or negative).

4 Software Maintenance Costs. Jussi Koskinen, Information Technology Research Institute. September 2003.




11

Relying Only on Unit Tests

For development teams that have historically decoupled development from testing – or waste time manually

performing tests – automating unit tests is a good start to improving the development process. But if you limit

yourself to only doing unit testing, you’re missing an opportunity to catch a much broader range of potential problems.

Since up to 80 percent of the cost of code is tied up in maintenance, it pays off to take a closer look at your compiled code.4

Don’t make the mistake of thinking that just because code compiles successfully that it is quality code. Compiling code is like

using spell check on a document – you can use it to detect incorrectly spelled words, but you won’t catch glaring grammar

problems or instances of repeat paragraphs. Similarly, unless you run a broad range of build-level tests that break the build

when problems are detected, your executable binary could be full of hidden problems.

The Solution: Comprehensive Build Validation

Valuable tools ranging from static analysis tools (such as FindBugs or NDepend) to enterprise products for security analysis,

license compliance auditing, and functional testing can help you detect a variety of issues early. They are all forms of build-level testing that can give you the qualitative, “big picture” status of a project early on.

Because some of these tests can take considerable time to perform, you may not want to run all of them continuously. But as

a best practice, they should be performed regularly and consistently. The best way to ensure that they happen as required is

to automate the testing process. Start by making your unit tests part of the source code repository and automating them so

that are part of every build event. Then start adding (and automating) other tests as well, such as component tests that allow

you to see how units interact together, functional tests that allow you to see how the components interact together, and so

on. The goal is to test as much as you can and move towards a more efficient assembly line process for software

development.

Ignoring History

If you regularly build and test your software but don’t store the results in a database, you’re missing the opportunity to gain valuable insights into your development process and software projects. There’s a wealth of information in

the form of build reports, test output, and other artifacts just waiting to be mined if only they were stored in a convenient

way. Even relatively simple queries could uncover previously unknown relationships or dependencies. For example, finding

out that half of your build failures occurred after someone touched an obscure configuration file would give you a fast,

simple way to optimize your build process.

The Solution: Capture and Mine Data

As a best practice, implement software that helps you turn data into actionable information. You won’t gain much from

analytics software that simply captures and reports on granular detail or provides isolated data points. Choose a product that

helps you see trends relating to complexity, code coverage, duplications, check-in frequency, and more so you can see how

your project is getting better or worse over time across key metrics. This kind of insight helps you change the behavior of

your engineering team – and improve your software development process over time.




12

Using Software to Automate and Monitor Best Practices

If you identified one or more weaknesses in your software development process, the next step is to implement best practicesthat automate key build and test processes and provide real-time visibility and metrics so you can identify and fix quality

issues early in the development process.

Borland Gauntlet™ is the first comprehensive build-and-test automation system that improves visibility, software quality, and

developer productivity by proactively building and testing code, isolating defects, and monitoring key development metrics

across the entire delivery lifecycle. These and other capabilities enable Gauntlet to play a vital role in helping you apply a

customer-centric approach to transforming software delivery into a managed business process.

Gauntlet helps you significantly reduce the risk of releasing unreliable applications by providing continuous integration and

testing capabilities that identify problems early so they can be tracked and fixed immediately, while also helping you reduce

time to market and boost developer productivity. And as illustrated in Figure 2, it also provides you with early visibility into

key trends so you can identify project risks in time to effectively adjust scope or increase resources.

Figure 2: Gauntlet Provides Early Visibility into Key Trends




13

Gauntlet works transparently with your existing version control systems to continuously analyze changes, promoting code to

the main line only if it builds successfully and passes all gating tests. This process is supported by key functions designed to

proactively address the most common blunders shared by software development teams, such as the following:

• Server-based build and test automation

Gauntlet will automate the execution of your build scripts, including any unit tests or other tasks that you have defined, whenever

you check changes into the version control system (or at scheduled intervals).

• Sandboxing

Gauntlet can build and test your changes in sandboxes and automatically integrate them into the main code line only if they build

successfully and pass all gating tests. Changes that fail to do so will remain in the sandbox until fixed by the responsible developer.

• Reporting and trending

Gauntlet can automatically parse the output of unit tests, code coverage tools, and many other static and run-time analysis tools,

and associate the results with revisions in the version control system. The software’s datamart and web dashboard enable you to chart,

filter, and compare these data (alongside other metrics such as build performance) to give you a better sense of the stability and

quality of your software projects.

• Fine-grained correlation

Gauntlet builds and tests individual changes made by each developer separately, allowing for more granular analysis and faster root

cause isolation. In addition, because Gauntlet measures and tracks results over time, it can give you insight into hidden dependencies

that can be otherwise be difficult to find and fix.

• Publishing

Gauntlet provides a convenient web interface to access current and previous distributions, along with their associated artifacts and

reports. You can even rebuild your entire project at any past revision with the push of a button.

• Replay

Gauntlet can import data from an existing CVS, StarTeam, or Subversion repository, allowing you to build old versions of your software

with new tests or dependencies. Replay can also be configured to run continuously, which allows you to use Gauntlet alongside your

current SCM system without having to migrate.

• Modular plug-in architecture

Gauntlet offers an open API for writing custom plug-ins, making it easy to add additional tests, analyzers, or other tools. Plug-ins for

many popular tools are already available.

Gauntlet does all of its work on the server, so it’s entirely non-intrusive to the development environment – your developers

can continue to use their existing version control clients and favorite IDEs.

Since Gauntlet works on various platforms and integrates with other tools you may already be using, it’s also a great example

of Borland’s Open ALM promise. Open ALM is the antidote to proprietary ALM solutions. Rather than locking you into

expensive, inflexible single-vendor solutions, Borland integrates your existing processes, technologies and platforms acrossthe software delivery lifecycle — empowering you to deliver software on your terms.

For more information, please visit www.borland.com.




14

About the Authors

Rob Cheng Director of Development Intelligence Solutions, Borland Software Corporation

Rob Cheng is the Director of Development Intelligence Solutions at Borland, responsible for strategy and evangelism in the

emerging field of applying business intelligence disciplines to software development. He has recently focused on the launch

of Borland Gauntlet, a continuous build and test automation system that provides real-time visibility and software quality

metrics across the entire delivery lifecycle.

Prior to Borland, Rob spent several years at Oracle Corporation, holding various product management, developer

evangelism and technology marketing roles. He also served as the Marketing Chair of the Web Services-Interoperability

organization during its inaugural year. Rob has a Bachelor of Science degree in Electrical Engineering and Computer

Science from the University of California at Berkeley and also studied Computer Science and Telecommunications Policy in

the Technology and Policy Program at MIT.

Burke Cox

CEO, Stelligent Incorporated

Burke Cox is a thought leader in accelerated software delivery and CEO of Stelligent Incorporated. He has assembled the

premier team of experts on optimizing software production, who have authored the upcoming book from Addison-Wesley,

Continuous Integration: Improving Software Quality and Reducing Risk (http://www.stelligent.com/go?cibook). Burke is a

nationally recognized speaker, editorial contributor and commentator on important topics facing developers and the

software industry.

He earned a Bachelor of Computer Engineering from Auburn University and is on the Industry Advisory Board for the

Auburn University Department of Computer Science and Software Engineering, who named him its Outstanding Alumnus

for 2003. He is also a member of the advisory board for the George Mason University College of Information Technology and Engineering.

John Minnihan

Founder and CEO, Freepository Corporation

John Minnihan is Founder & CEO of Freepository (www.freepository.com), a provider of on-demand software development

infrastructure such as source control, defect tracking and release management. With a career in business computing that

began with component-level hardware repairs and has progressed through the provision of strategic consulting at Fortune

100 companies, John has experienced first-hand the value of well-engineered software development tools and processes.

Freepository customers include universities, financial institutions, and governmental agencies across the globe.

About BorlandBorland is the leading vendor of Open Application Lifecycle Management (ALM) solutions - open to customers' processes, tools

and platforms - providing the flexibility to manage, measure and improve the software delivery process.

www.borland.com

Copyright © 2007 Borland Software Corporation. All Borland brand and product names are service marks, trademarks, or

Documents

The Top 10 Blunders in Integration and Testing - And How to Avoid Them - Rob Cheng (Borland)