AW15 Agile Development Concurrent Session 11/12/2014 4:15 PM

"Putting Quality in the Driver’s Seat with DevOps and ATDD"

Presented by:

Adam Auerbach Captial One

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Adam Auerbach is the Technology Director for Advanced Testing and Release services for Capital One Financial Corporation, a diversified bank with 65-million customer accounts worldwide and more than 900 branch locations. Adam is responsible for Capital One’s enterprise performance and automated testing departments as well as enterprise release management. Since joining Capital One, he has provided leadership for the agile transformation of their quality assurance group and led the enterprise adoption of DevOps and ATDD. Before joining Capital One, Adam was with Chase and other financial and insurance companies, in various leadership positions focusing on quality and agile practices.

Putting Quality in the Driver seat with DevOps and ATDD Presented by Adam Auerbach

About Me

• Technology Director at Capital One – @Bugman31 | Adam.Auerbach@capitalone.com

• Responsible for Capital One’s Advanced Testing and

Release Services Teams, which include: – Performance Testing

– Automated Testing

– Release Management

• Provides leadership for the enterprise adoption of DevOps and ATDD.

• Before joining Capital One in 2013 Mr. Auerbach was with Chase and other Financial and Insurance companies, in various leadership positions, with a focus on Quality and Agile practices.

We are more than a Credit Card Company

Capital One’s Journey

Agile DevOps ATDD Waterfall

As we adopted Agile, we recognized that we still had opportunities for improvements

Development Sprints

Testing Sprints

Integration and Hardening

Key Observations: • Working features delivered every couple months

• Integration of code happened in later sprints

• Environment availability was top impediment

• Automated Tests built and maintained by dedicated team

• Stories took several sprints to complete

The Scaled Agile Framework (SAFe) helped provide a framework for DevOps adoption

Sample System Team Kanban board

To drive adoption, we created a team to own the implementation

LOB System Teams

LOB Accountable

Execs

Enterprise SWAT Team

Some lessons learned from creating these new teams

1. Treat as a new Agile Team – All team members are properly trained – Team members are fully dedicated – Ensure access to all tools and systems needed

2. Leverage a sprint 0, to focus on grooming backlog and understanding

of platform supporting

1. Focus on early outcomes to develop credibility and gain momentum

With System Teams in place, we still needed more…

Develop Build Test

Develop Build Test

Hardening

Key Opportunities: • Teams still working in silos (PO, Dev, Testers)

• Stories too large to get done in single sprints

• Automation occurs outside of feature teams, technical debt was piling up

• No improvements on multiple monthly lead time

• Industry practice in which whole team collaborates on Acceptance criteria and the definition of done

• Automate acceptance tests while production code is being developed – Automation completes before development

• Developer focuses on making acceptance tests pass • Tests become part of build pipeline and are run throughout the sprint

Acceptance Test Driven Development (ATDD)

• Acceptance Tests are customer tests that demonstrate that the business intent of the system are being met • Users point of view • External view of the system

• Examine externally visible effects • Inputs and outputs • State changes, Business rules • External Interfaces

• Implementation independent • Guide development to design cleaner code

What are Acceptance Tests?

The overall ATDD process incorporates BDD and TDD principles

Product Owner/Business Analyst

Product Owner/Developer/Tester

Developer

Tester

Tester/Developer

User Story

Expand Scenarios/

Steps Cucumber

Gherkin

Pair “Show Me”

AUT

Acceptance Test

Feature Files Cucumber

Gherkin(BDD)

Develop

User Acceptance

Automate Tests

Step Definition Ruby Code Libraries

1. Author the tests • Customer (PO/BSA) with tester, developer together • Tester and developer can create more and have it reviewed by

customer

2. Connect tests to the system • Developer or technical testers by writing short bits of code

3. Run the tests

• Developer, then tester • Manual or Automated (however Automated acceptance test allows

them to be run as regression tests to ensure new features do not interfere with previously developed features)

Who does what?

• Tighter cross-functional Team integration – Because the PO, BSA, Dev, Testers are involved in Acceptance Test creation there is tighter cross-functional integration

• Workflows working first time the system was brought up, Getting business rules right prevents rework

• Crisp Visible Story completion criteria – visibly demonstrate that the story is complete

• Not a line a code is written that does not provide business value

• Acceptance tests can be used as Regression tests at the end of the sprint

• Run suite daily, CI/CD practices (find out what's broken faster)

Here are some benefits we observed from teams using ATDD

Success Criteria for implementing ATDD

Criteria Success Condition

Training All Training for BSA's, Developers and testers is complete including installation, hands on practice, development, execution and refactoring

Gherkin Approved Gherkin format to be used by all BSA's Train all BSA's and commitment from BSA Review and approved actual Gherkins/stories

Competency BSA are able to write Gherkin features to the specified standard Functional Testers able to develop and execute overall 80% of scripts by End of XX sprints or release/PSI

Software Availability of all required Software in EAPL, Ruby 2.x, Cucumber, Ruby Mine, Selenium etc..

Environment/Infrastructure Stable Environment for continous testing, Infrastructure for parallel and distributed execution using GRID including support for multi browser and mainframe/Terminal Emulator (windows environment)

End-End flow Should be able to automate an end-end flow, including all frontend, backend and middleware validations with E2E integrations

ATDD Workflow

BSA, PO - write stories and give basic Gherkin features Tester/Developer - Elaborate scenarios/Acceptance tests Tester/Developer - Step definitions 3 Amigo's Tester - Ruby Automation scripts Exeucte Automation Script Developer - Buld/TDD-refactor until Acceptance test passes Refactor Acceptance tests Integration Test

TDM - Test Data TDM team should be able to read gherkin and provide test data aligned with the gherkin CI/CD Integration Invoke/Execute/Report after every code drop

Governance Model Established governance model for code reviews, peer reviews, guidelines, executed and signed off

Release Management

Quality Assurance, Test Automation &

Performance Validation

Quality Driven Delivery puts Quality in the driver’s seat

After implemented Quality Driven Delivery, we were able to see significant benefits

Before After Features implemented every couple months

Working features delivered to production more frequently

Integration of code happened several sprints after initial development

Integration happens multiple times a day

Environment availability was top impediment

Environment issues have limited impact on our teams

Automated Tests built and maintained by dedicated team

Automation is owned by teams

Stories took several sprints to complete Stories are small and completed each sprint.

www.capitalonecareers.com

How Capital One put Quality in the Driver seat thru DevOps and other best practices Presented by Adam Auerbach