Integrating Automated Testing into DevOps

TE AM Tutorial

10/14/2014 8:30:00 AM

"Integrating Automated Testing into

DevOps"

Presented by:

Jeff Payne

Coveros, Inc.

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Jeff Payne

Coveros, Inc.

Jeff Payne is CEO and founder of Coveros, Inc., a software company that builds secure software applications using agile methods. Since its inception in 2008, Coveros has become a market leader in secure agile principles and was recognized by Inc. magazine as one of the fastest growing private US companies. Prior to founding Coveros, Jeff was chairman of the board, CEO, and cofounder of Cigital, Inc., a market leader in software security consulting. Jeff has published more than thirty papers on software development and testing, and testified before Congress on issues of national importance, including intellectual property rights, cyber terrorism, and software quality. Follow Jeff on Twitter @jefferyepayne.

1 © Copyright 2014 Coveros Corporation. All rights reserved.

Integrating Automated Testing

into DevOps

Jeffery Payne

@jefferyepayne

2 © Copyright 2014 Coveros, Inc.. All rights reserved.

Bio

Jeffery Payne [email protected]

@jefferyepayne

Jeffery Payne is CEO and founder of Coveros, Inc., a software company that

helps organizations accelerate the delivery of secure, reliable software. Coveros

uses agile development methods and a proven software assurance framework to

build security and quality into software from the ground up. Prior to founding

Coveros, Jeffery was Chairman of the Board, CEO, and co-founder of Cigital, Inc.

Under his direction, Cigital became a leader in software security and software

quality solutions, helping clients mitigate the risk of software failure. Jeffery is a

recognized software expert and popular speaker at both business and technology

conferences on a variety of software quality, security, and agile development

topics. He has also testified before Congress on issues of national importance,

including intellectual property rights, cyber-terrorism, software research funding,

and software quality.


Coveros helps organizations accelerate the delivery of secure, reliable software

Our consulting services: – Agile software development

– Agile testing & automation

– Application security

– DevOps automation

– Agile training & coaching

Our key markets: – Financial services

– Healthcare

– Defense

– Critical Infrastructure

Areas of Expertise

About Coveros


Agenda

What is DevOps? – Definition(s) – Common terminology – Common architecture

Automated CI Testing

Automated CD Testing

Demos of various types of automated testing


What is DevOps?


What is DevOps?

Common Definition

DevOps is a software development method that stresses communication, collaboration and integration between software developers, QA, and information technology (IT) professionals. DevOps is a response to the interdependence of software development and IT operations. It aims to help an organization rapidly produce software products and services.

DevOps is not a tool

DevOps is not a methodology

DevOps is a philosophy for how people in the software development supply chain interact


What is DevOps?

DevOps practices include processes and tools that streamline

software delivery by improving feedback from production to

development and automating delivery activities.

○ Processes

■ Alignment of responsibilities with Artifacts

■ Well defined (and automated!) pipeline from code development to

production environments (ideally)

■ Implementation within agile iterations/sprints further support agile

principles

○ Tools

■ Tools to automate and manage builds

■ Tools to automate testing and results reporting

■ Tools to provision environments (physical and virtual, privide and cloud)

■ Tools to orchestrate continuous delivery of applications


What is DevOps?

Development vs. Operations - The Essential Conflict

● Development

○ Need for Change - Desire for rapid rollout to provide value to Customers.

Customers often demand change, not weighing risk

■ New Features

■ Feature Changes

■ Bug Fixes

● Operations

○ Fear of Change - Desire for stable conditions for production systems

■ Server Uptime, Reliability, Availability,

■ Capacity, response times

■ Security

■ SLAs (contracts)

■ This conflict results in quality and agility problems


It compiles, therefore it works!

Big bang integration

Throwing software over the fence to late lifecycle QA

“It works on my machine”

“We’ll get that QA environment stood up for you by … 2016”

DevOps helps solve common quality/agility problems

What is DevOps?


What is DevOps?

DevOps Terminology

Build automation – Automating the compilation, linking, and packaging of software applications

Continuous integration – Leveraging build automation capabilities to integrate and test software on a regular basis to thwart integration and quality issues earlier in the process

Check-in builds – Builds performed whenever code is changed. Often includes unit tests and code quality checks.

Nightly builds – Integration builds performed nightly to assure nothing changed during the day breaks existing functionality. Regression testing included. Platform TBD.


What is DevOps?

DevOps Terminology

Continuous delivery – A series of practices to assure code can be rapidly and safely deployed to downstream environments (QA, staging, production, etc.) by delivering every change to downstream environments while ensuring functionality works as expected through automated testing.

Continuous deployment – A series of practices to assure code can be rapidly and safely deployed to production by delivering every change to a production (or production-like) environment while ensuring functionality works as expected through automated testing.


When to Test

● Testing practices should be broken up into different

environments ○ Development env

■ Unit Testing

■ Acceptance Tests

■ Sanity Testing

○ Continuous Integration env ■ Unit Testing

■ Acceptance Tests

■ Incremental integration testing

○ QA env ■ Smoke testing

■ Integration testing

■ Functional Testing

■ Usability testing

■ Compatibility testing

■ Install/uninstall testing

■ Regression testing

○ Staging env

■ Performance Testing

■ Stress Testing

■ Load testing

■ End-to-end testing

■ System Testing


Environments

● While continuously testing and building software is

important for a robust final product, if it difficult to

continuously release software

● Having multiple environments allows for continuous

development and low level testing to continue, while

having more static versions of the software available for

testers

● Ideally, a set of 4 or 5 environments will exist ○ Development

○ Dev Integration (CI)

○ QA

○ Staging

○ Production


Environments (cont.)

● Development ○ Should be updated continuously, always with the most up to date code

that passed all static analysis, unit tests, and coverage bars

● QA ○ Should be updated nightly, always with the last code to pass all bars

○ Updating more frequently will cause manual and longer regression testing

issues

● Staging ○ Somewhere with stable code, mimicking production

○ Used for end-to-end testing and non-functional testing that must be

performed on production-like environments

● Production ○ Should be updated with releases only, always with stable fully testing

code


What is DevOps?

Simple Example DevOps architecture

Dev Int QA Staging Production


Automating CI Tests


Continuous Integration

● Continuous Integration will enable the integration of

multiple software, data, and tests from multiple teams

into one robust, cohesive application

● CI combines the use of automated build tools in

conjunction with automated testing and deployment tools.

● The continuous integration effort will promote and

coordinate functionality between developers rapidly,

while also delivering an effective automated defect

identification and resolution.


Continuous Integration (cont.)

● Each level of testing will be dependent on previous levels

succeeding to keep out bad code, and new successful

builds being processed at a high rate.

● All users involved in this effort, should be able to see the

iterative functionality, giving a window into development

processes and quick knowledge on the results of each

piece of functionality.

● Monitoring build will provide quick and easy access to

failures, which will shorten the timeline to fix issues and

increase overall quality.


Developer commits code to a branch CI server detects changes and fires off a build request

– Static analysis is performed on code – Unit tests are execute – Code coverage is calculated on units

If build/testing is successful, branch merged to master Appropriate applications are built and put into repository

– Smoke testing is performed on new version in QA if possible

When the build breaks (either code does not complete or does not pass testing/quality checks), team is obligated to FIX THE BUILD.

Do nightly builds go here?

Typical CI Process

Automating CI Tests

20 © Copyright 2014 Coveros, Inc.. All rights reserved. © Copyright 2014 Coveros, Inc. All rights reserved.

Common Tools


Source Control

● Source control and tight integration with build automation

tools is a key element of a successful CI implementation

● The ability to use source/version control technology is

fundamental to ○ Ability for teams to work on a code base with out clobbering each others

changes

○ Meet rapid build and deployment objectives

○ Assure that identified defect identification are tied to correct code

version

○ Proper rollbacks when mistakes are made and problems cannot be

debugged quickly

● Code should be checked in regularly for a CI process to

be most effective


● Git ○ Decentralized, and aims to be

fast, flexible, and robust

● Subversion ○ Open source versioning control

system inspired by CVS

● CVS ○ Built on RCS, with emphasis put

on security and source code

correctness

● Vesta ○ Build system with a versioning

file system and support for

distributed repositories

Source Control - Examples

Git Flow Example within CI


Build Automation

● Build Automation is the scripting of common tasks

performed by the software developers for: ○ Extracting the appropriate version of the source code from a source-

code repository;

○ Assuring the necessary compilation dependencies are in place;

○ Compiling source code into binary objects while referencing the

compilation dependencies;

○ Packaging binary objects;

○ Assuring the necessary unit testing dependencies are in place;

○ Executing tests while referencing the testing dependencies;

○ Managing the artifacts created during the process.

● All software developers perform these operations to

some degree or as part of their daily tasks for the

modules to which s/he is contributing.

● Tools: Ant, Anthill, Maven, make


CI Servers

● CI Servers have become increasingly popular to help

manage the continuous integration process

● They provide a central location for all code to be

examined, compiled, and executed

● Most of these tools provide a web interface for ○ Viewing the status of any of the modules on the project

○ Maintaining a history of each build outcome

○ A detailed account of what happened: These details usually lead to the

cause of the failure.

● While these tools provide information about the health of

the code, they are still no replacement for developer run

acceptance tests


Build Tools - Examples

● Jenkins / (fka Hudson) ○ Open source continuous integration

tool written in Java

● CruiseControl ○ Open source continuous integration

tool written for Java-based

frameworks

● Bamboo ○ Bamboo is a continuous integration

server from Atlassian

● TeamCity ○ continuous integration and build

management


Test Tools

● The goal of test automation within CI is to identify coding

and subsystem errors as close to their introduction as

possible

● Supports the notion that code is not “done” until it is

tested

● The amount (and levels) of testing performed during CI

depends upon the environments you have control over

and the amount of time it takes to build and run your

application.


Test Tools - Static Analysis

● Test automation begins with static analysis

● Before any code is compiled or tested, it needs to meet

specific requirements

● Each time code is checked in, a style checker should run

over the code to checking for ○ layout issues

○ comments

○ documentation

● A rating will be assigned to each piece of new code, and

code failing to meet a certain bar should be rejected


Test Tools - Static Analysis (cont.)

● Again, before the code is even compiled, a code checker

should be run over the source code to check for common

programming flaws.

● This can catch things such as

○ Unused variables

○ Empty catch blocks

○ Unnecessary object creation

○ Duplicate code


Test Tools - Static Analysis (cont.)

● A bug checker should be run over the un-compiled code

to check for common bugs

○ This will provide an extremely quick check for potential issues and most

tools rank them in severity

○ If no level 1, 2 or 3 level issues are found, the application and test code

should finally be compiled.

● After the code is compiled, design quality metric tools can

traverse the compiled directories to generate design

quality metrics

● Depending on the tool, reports can display

○ extensibility, reusability, and maintainability

○ dependencies, and cycles for each analyzed package


Static Analysis Tools - Examples

● Ruby Stack ○ rubocop

○ Rails best practices

○ Cane

○ Reek

● Java Stack ○ Checkstyle

○ PMD

○ Findbugs

○ JDepend

○ .Net Stack ○ StyleCop

○ SourceMonitor

○ Clone Detective

○ NDepend


Test Tools - Unit Tests

● Unit tests are simple and each should ○ Look at each singular piece of functionality

○ Have a fixed set of datum

○ Expect deterministic results

● A unit test provides a strict, written contract that the piece

of code must satisfy.

● All developers should be responsible to provide unit tests

to test and prove the completeness and soundness of

their checked-in code

● All unit tests should be expected to pass for the

application to move onto the next phase in testing


Test Tools - Unit Tests (cont.)

● The goal of unit testing is to isolate each part of the

program and show that the individual parts are correct

● Unit Testing affords several benefits

○ Finds problems early

○ Facilitates change

○ Simplifies integration

○ Provides documentation

● Unit testing only tests the functionality of the units

themselves, and will not catch integration errors or

broader system-level errors.


Unit Testing Tools - Examples

● JUnit ○ A popular framework for writing unit tests in Java/J2EE environment

● TestNG ○ Similar to jUnit, but provides dependencies and expanded annotations

● NUnit ○ includes GUI, command line, integrates into Visual Studio

● MSTest ○ A command-line tool for executing Visual Studio created unit tests

outside of the Visual Studio IDE

○ Not really a testing framework as it is a part of the Visual Studio Unit

Testing Framework

● RSpec ○ Behaviour-driven development framework


CI Test Tools - Code Coverage

● The last stage of testing at the code level will be to run a

coverage tools to examine the previous tests run

● The amount of code accessed by the unit tests will be

calculated, and based on the result, the completeness of

the unit tests will be determined

● Code lacking proper coverage will fail this test, and the

code be sent back for further development ○ Usually this means developers will need to add more unit tests, not fix

their code

○ Code Coverage Tools: Cobertura (Java), NCover (.NET),

Simplecov (Ruby)


CI Testing Tools - Reporting

● Multiple tools exist outside of build tools to provide further

analysis on the application

● Many of these tools, such as SonarQube or HP Quality

Center, allow the centralization of all metrics into one

area


Demo

Demo of CI Environment


CI Demo


Automating CD Tests


Automating CD Tests

Tests

Provisioning

Deployment tests

Integration smoke tests

Security tests

Others?


Continuous delivery

● Continuous Delivery enables integration and testing on downstream

environments on a continuous basis to assure software works

properly in these environments

● CD is typically orchestrated by a CI Server and a Deployment

Manager

● The continuous delivery effort will promote and coordinate

functionality between developers, QA, and operations rapidly to make

sure the software works in all environments

● Also supports automated setup, teardown, and use of testing

environments if possible


New build has been promoted to repository after having passed smoke tests

Application is deployed onto stable QA environment for testing. If possible, QA environment is automatically provisioned either locally or in the cloud

– Automated and manual functional testing is performed – Regression run as well to assure quality – Non-functional testing is performed as appropriate – At appropriate times based upon environment, test environments

are torn down

Application is deployed onto production-like Staging environment. If possible Stating is automatically provisioned either locally or in the cloud

– Tests are performed that need to be run on a production-like env

Application is provisioned and deployed in production

Typical CD Process

Automating CD Tests


Common Tools


Artifact Repository’s and Repository Manager’s

● Artifact Repository’s provide a structured destination for

artifact’s produce during the software build process

● Repository Manager’s provide a centralized mechanism

for accessing artifacts within both public and private

repository’s, increasing speed and efficiency of building

and accessing artifacts

● Artifact Repository’s – Maven, Alfresco, Yum

● Repository Manager’s – Apache Archive, Artifactory,

Sonatype Nexus


CI Servers in CD

● CI Servers typically control CD activities now as well

● Server Scripts are written to push new releases into

downstream environments

● These scripts can be launched manually with a “button

push” or tied together into an automated delivery pipeline

that moves new versions of applications through

environments automatically as tests are passed

● Without effective testing, Continuous Delivery (and also

DevOps in general) cannot be optimized! ● Bad automated tests used to result in bad test results faster

● Not bad automated tests result in bad code being deployed!


IT Automation Tools allow teams to define environment, system, and application configurations and parameters such that downstream environments can be automatically provisions and any conflicts between expected application parameters can be identified

Significantly reduces the amount of operational IT support needed for environment setup, application installation/configuration, and teardown necessary to move applications from development to production

80-90% decrease in ops support with accelerated delivery

Common Tools: Puppet, Chef, Ansible

IT Automation Tools


Test Tools for CD

● Test tools used within CD are typically the transitional

subsystem, integration, and system test tools used in

traditional processes.

● Difference is that these tests are run much more

frequently than in the past

● There is also more emphasis on testing of functionality

below the UI so integration tests can be run earlier in the

development process and are less brittle

● Common Tools: Selenium, FIT & FITness, Cucumber,

QTP


Demo

Demo of CD Environment


Wrap-Up


Questions?

Contact Information:

[email protected]

703.431.2920

@jefferyepayne

http://www.coveros.com

Technology

Integrating Automated Testing into DevOps