DevOps 2017 Conf: evolving from automated to continuous

Evolving from Automated to Continuous Testing for Agile and DevOps

DevOps Remote Conference 2017Arthur “Code Curmudgeon” Hicken

IntroArthur Hicken is Chief Evangelist at Parasoft where he has been involved in automating various software development and testing practices for over 20 years. He has worked on projects including cybersecurity, database development, the software development lifecycle, web publishing and monitoring, and integration with legacy systems and maintains the IoT Hall-of-Shame http://bit.ly/iotshame

Follow him @codecurmudgeonBlog: http://codecurmudgeon.comWeb: http://parasoft.com

2© 2017 Parasoft Corporation Perfecting Software

Agenda

Today’s challenges

What IS Continuous?

Barriers

Best practices

Role of automation

Role of static analysis!?

3© 2017 Parasoft CorporationPerfecting Software

Normal Situation

• Pressure to deliver software quickly• Pressure to reduce defect rate• Late stage “bug-hunting”• No time for error prevention• Quality can be rushed• Security is forgotten


How to you know a build is successful?

• When it compiles?• When all unit-tests have run?• When the right coverage goal is achieved?• When the right % of test failures occurs?• When it’s deployed?


Continuous

• Build, Integration, Testing, Release, Delivery, Deployment• Builds on and enables DevOps• High dependency on automation:

• Processes• Assessment• Decisions


Feature Release

Business Stakeholder Developer Customer Support

SoftwareFeature

Continuous measurements mean continuous refinement of the process

Real-time feedback from

Objective assessment and go/no go Defects are eliminated at the point

that they are easiest to fix

Quality gates: Organizations can automatically promote software through quality gates when business expectations have been met


Continuous Testing is Beyond Automation

Challenges of Continuous

• Tests must produce binary decision go/no-go• Reuse unit test and functional tests from dev to QA• High level of automation• Requires disciplined mature process• Testing must automatically answer

• Is it stable• Will it do what it’s supposed to do


Importance of Testing

• How do you know when you're done? • How do you know if a fix for a minor bug broke a major function of the

system? • How can the system evolve into something more than is currently

envisioned? • Testing, both unit and functional, needs to be an integrated part of the

development process.


Continuous Testing re-positions the question from “are you done testing?” to “is the level of risk understood and accepted?”

ContinuousTesting

RiskAssessment

PolicyAnalysis

RequirementsTraceability

TestEnvironment

Access

TestOptimization

AdvancedAnalysis

Ensure access to complete test environments

Automate defect prevention andPolicy measurement

Expand test coverage and measure test effectiveness

Define actionablepractices

Connect functional with non-functional requirements

Process improvementopportunities


Elements of Continuous Testing

Prerequisites

Tools

• Version control• Build server• Deployment server• CI tools• Automation tools (test, etc)

Process

• Commit/update often (each change)• Always create tests (pass and fail)• Test regularly• Run regularly


Continuous requires:Proper infrastructure

A binary definition of “done”

Reliable quality gates

Extreme automation


Infrastructure

• Real requirements system• Flexible developer level project/scrum management system• Automated build/testing system• Data collection that can answer

• Is it done?• When will it BE done?• Is it good enough to release


Defining Done

• Are requirements coded?• Do tests exist for the requirements and code?• Are the tests passing

• To your satisfaction (often not 100%)

• Will the tests be enough?


Defining Quality

• Consistent• Repeatable• Meaningful

Automated objective measure

• Unit tests• Functional tests• Regressions tests• Static analysis

Made up of


Effective Continuous Testing

• Avoid manual end-to-end testing• Invest more into automated tests• Follow testing pyramid rules• Measure:

• test results• test effectiveness

• Determine:• Risk


How to do it better ?

• Give absolute priority to automated tests• Invest time into designing interfaces (API)• Use API (service) testing tools to cover interfaces • Measure the quality of the test


How Can Static Analysis Help?

Coding standards

Quality gate (bug detection)

Prevention


Bug Gates

• Binary Decisions• No “bugs”• No “critical” static analysis findings


Quotable Quotes

• An ounce of prevention is worth a pound of cure. – Benjamin Franklin• Cease dependence on inspection to achieve quality. Eliminate the need for

inspection on a mass basis by building quality into the product in the first place. – Deming

• Simply finding bugs with static analysis isn’t enough.• Avoid risky behavior


Static Analysis Continuous Feedback Loop

Identify Error

Isolate Root Cause

FixNew Rule to Prevent

Monitor

§ Code review

§ Regression

§ QA

§ Field bugs

§ Post-mortem


Main Points for Today

• We must define “all” quality expectations upfront

• We must change the primary goal of testing

• We must accept that testing is only a task in the quality process

• We must be able to measure trust of the process

• We must automate breaking the application


Speeding up the conveyor belt does not yield better results…

Modern DevOps must embrace systematic process improvement—Focused on testing

DevOps is Continuous Improvement


How to Achieve Quality @ Speed• Business risks drive quality activities• Teams collaborate on risk definition• All team members trained on risks• Acceptance criteria visible and

measured• DevTest team activities prioritized per

risk definition• Results of quality activities are visible

and translated for all levels of business


Why ContinuousTesting?

Systematic, centralized decision making to factor business risk into SDLC

Safety net to allow developers to bring innovations to market faster

…and automation to break the application

A feedback system for better trade-off decisions between release scope, time

and quality

Continuous Testing Mitigates Business Risk


Enabling Technologies

Stubs

ServiceVirtualization

APItesting

Testdatamanagement

Environmentmanagement

Self-servicetestenvironments


How to do it better?

• Use service virtualization to improve testing automation• Isolate at the message layer• Simulate functional scenarios and performance conditions


The Test Environment ChallengeTest environment access is outside the control of development and test leaving gaps in the process

Parallel development delays… Need simple, realistic access to dependent components…

Too much time waiting for access…Need reliable test data

Need a realistic testenvironment easy to maintain

Scheduling

Configuration

Access Limits

DependentApplications

Staged Assets

3rd Party Assets Virtual Environments

HyperVisor

App App App

IT Operations


• The Challenge• Multiple teams using the same test database• Teams not respecting data integrity & others test data records• Regression tests consistently failing. Takes >1 hour to determine

that it was due to “data changes”. • “Real problems” were getting lost in the noise

SharedDatabase

Eliminated 83% of configuration time for a major telecom company

Test data management for complex transactions

✔

✖

✖29© 2017 Parasoft Corporation

Perfecting Software

• The Solution• Setup Virtual Assets to model the SQL queries and use API

testing tool to manage automated nightly regressions against both virtual assets and live systems

• The Business Benefit• Test teams able to focus on ‘real regressions’ and separate out

data integrity issues from functional test failure

Virtual Asset

Virtual Asset

Virtual Asset

✔

✔

✔


Eliminated 83% of configuration time for a major telecom company

Summary

• Continuous * requires mature infrastructure and process• Quality gates must produce a binary answer• Automation is everything• Policy is critical, because automation is critical• Static analysis gets you to the gate


Thank you!

32

Continuous Testing Book: http://alm.parasoft.com/continuoustestingbook

© 2017 Parasoft Corporation Perfecting Software