Dealing with Auditors: Helping Them Understand Agile

AT12 Agile Development Concurrent Session 11/13/2014 1:30 PM

"Dealing with Auditors: Helping Them Understand Agile"

Presented by:

Steve Nunziata Independent Consultant

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Steve Nunziata (CSM, PMP, ACP, SAFe SPC) has more than twenty-five years in IT project management, using waterfall and agile methodologies—and numerous hybrids in between. Steve’s industry experience ranges from health care, sporting goods, transportation, and insurance. For the past ten years, he has focused on agile practices and teams, fulfilling roles such as ScrumMaster, Product Owner, agile coach, project manager, and quality assurance advisor―sometimes in the same day! Steve is very active in the San Antonio agile community, facilitating monthly meet-ups and education events. In his spare time, he enjoys playing in his classic rock band and being with his wonderful family.

Dealing with Auditors: Helping them Understand Agile

CHAOS, CONSISTENCY, CREATIVITY: A JOURNEY THROUGH AGILE AUDITABILITY

Steve Nunziata, PMP, PMI-ACP, CSM, SAFe SPC November 13th, 2014

About Steve…

PMP, ACP, CSM, SAFe SPC

EDS, Nike, Adidas, USAA

Agile Trainer & Coach

New Jersey / Oregon

Bassist Extraordinaire

Alamo Agilistas / PMI

So… Why Are We Here? Opportunity:

Educate internal auditors to evolve away from formal artifacts and accept Agile tenets of visibility and transparency to demonstrate adherence to defined Quality standards.

We will collaborate on an approach to define an Agile Risk & Control framework that can start you on your journey.

How Would You Like: A 50% - or more – reduction in project ‘paperwork’ to demonstrate adherence to compliance processes?

WATERFALL AGILE

59

30

PROJECT COMPLIANCE

ARTIFACTS

A framework for consistent application of Agile practices and ceremonies across a large – and growing – organization?

Background: My Story

Zero to Sixty (Days): Chaos to Consistency

Agenda Ch

aos Failings of

Today’s Risk Management Processes

Cons

isten

cy

Why Audit Execution Models Need to Evolve Cr

eativ

ity

Creating an Agile Auditable Framework

Managing Risk – How Important is it?

The primary goal of a business is to… stay in business.

It is therefore necessary to continually evaluate, monitor, and address threats to retain market share. Otherwise, what would happen?

Managing Risk – The Risk Management Process

Risk Identification

Risk Assessment

Risk Response Risk Review

Managing Risk – ISO 9001 Summary Part 4 – The Company must establish, document, and maintain a Quality Management System (QMS)

Part 5 – Management commitment in evidence for the QMS

Part 6 – Necessary resources must be determined & provisioned

Part 7 – Plan & Develop processes for product realization. The processes must produce documents that can be (1) reviewed for acceptance; and (2) used as proof of conformance

Part 8 – All reports of non-conformances, both of the product or the process, shall be reported upon, analyzed and lead to corrective action

Managing Risk – Risk & Control Compliance Framework

Risk Controls

Control Tests

Reporting & Review

Operational Risks

Incomplete Requirements Ineffective or Incomplete

Software Solution Poor User Experience Poor Project Execution

Plan

Formal Requirements Baseline Process

Project Execution Schedule Review

Code Peer Reviews

Evidence of Formal Signoffs

Published Meeting Minutes

Documented Decisions / Logs

Formal results of Audit published for review; opportunities for improvements noted

Auditors

Are Risk Management Processes Inherently anti-Agile?

Source: http://www.devballs.com/wp-content/uploads/2010/02/agilemanifesto.gif

SDLC & Process Audit Execution Models: Challenges

While Agile adoption and evolution has continued unabated over the past several years, traditional process audits have largely been unable to keep pace. Why might this be?

SDLC & Process Audit Execution Models

Req’s Analysis Design Build Test Deploy

Systems Development Life Cycle – Linear View

SDLC & Process Audit Execution Models

Source: http://julianeverett.wordpress.com/

Red Dotted Line: Waterfall

Blue Dotted Line: Agile

RISK

TIME Project Risk Profile – Agile & Waterfall

SDLC & Process Audit Execution Models

Daily

24 H

ours

Iteration

2-4 Weeks

Release

~3 Months

Closure

~9-12 Months

SDLC Execution – Waterfall, Incremental, & Agile

SDLC & Process Audit Execution Models

Process Audit vs. SDLC Execution Gap Analysis

Closure

~9-12 Months

Release

~3 Months

Iteration

2-4 Weeks

Daily

24 H

ours

SDLC & Process Audit Execution Models

Daily Iteration

2-4 Weeks

Release

~3 Months

Closure

SDLC and Process Audit Execution: Optimal Quality State

5 Steps to Establishing an Agile Auditable Framework

Risk Validation

Inventory Agile Practices

Create Acceptable Parameters

Determine Method of Control

Establish Operational Parameters

1

2

3

4

5

5 Steps to Evolving an Agile Auditable Framework

Risk Validation

Review and Validate the current Risk & Control Framework, ensuring traceability from Risks to Controls to Control Tests.

Operational Risk: Risk Control: Control Test:

Failure to Manage Project Risks

Risk Management Process

Evidence of a Periodic Risk Review (Risk Log)

Issue Management Process

Formal, Complete Issues Log

1

5 Steps to Evolving an Agile Auditable Framework

Inventory Agile Practices

Inventory the Agile Practices supported by the organization. Scrum practices and ceremonies provide a good start.

Match the Agile ceremonies to the list of Risks in the current

Risk & Control Framework. Can a Ceremony or Practice provide an acceptable substitute? How / Why?

2

5 Steps to Evolving an Agile Auditable Framework

Inventory Agile Practices

Introduce the Agile Practice as a Control. Could it work? Could it be effective? What would be the value of the current control set – should anything remain, or can they be dismissed?

Operational Risk: Risk Control: Control Test:

Failure to Manage Project Risks

Risk Management Process

Evidence of a Periodic Risk Review

Agile Daily Standup

2

5 Steps to Evolving an Agile Auditable Framework

Create Acceptable Parameters

Research Industry standard ‘best practices’ for the ceremonies or practices you plan on using as a Control (mitigation strategy) for the Risk. A great example is Version One’s The Agile Checklist

Create a matrix defining minimally acceptable behaviors, along with anti-patterns, and radiate the desired outcomes in a common area

3

5 Steps to Evolving an Agile Auditable Framework

Create Acceptable Parameters

Agile Ceremony: Daily Standup

Best Practice Acceptable Partial Unacceptable

Occurs 5 Days per Week

Occurs 4 Days per Week

Occurs 3 Days per Week

Occurs <3 Days per Week

3 Core Questions Addressed

3 Core Questions Addressed

<3 Core Questions Addressed

<3 Core Questions Addressed

…Your Organization?

…Your Organization?

…Your Organization?

….Your Organization?

3

5 Steps to Evolving an Agile Auditable Framework

Determine Method of Control

Does the new Control Test require someone observe an Agile Ceremony, or is there a consistent formal artifact from an Agile practice that can be viewed?

4

5 Steps to Evolving an Agile Auditable Framework

Establish Operational Parameters

Review the total number of Control Tests. How many require observation from an Auditor?

Establish the Audit cycle & reporting time (Weekly? Sprint Level? Release Level? Other..?)

Train and deploy Audit resources

Execute an Audit cycle… and report to Risk Owners Learn… and continue to evolve!

5

5 Steps to Evolving… Creativity Host a Retrospective Ceremony with some of the Agile teams to uncover:

What may be challenging teams in conforming to minimal standards?

What opportunities can they recommend to evolve to controls?

Are the audits providing value in holding roles accountable for their deliverables?

Finally – when minimal standards are easily achieved – it’s time to take the next steps in maturity, and shift the pattern.

5 Steps to Evolving - Going Beyond... Challenge: can you evolve traditional, formal artifacts into a

more Agile framework? How can you continuously improve?

Picture Source: http://agile101.wordpress.com/2009/07/27/ agile-risk-management-assessing-risks-step-2-of-4/

Positive Outcomes Better alignment of Controls and Tests to the project execution model

Real time, actionable feedback & reporting to teams and Risk owners

Scalable for future methodologies & practices

Continual quality assessments; a project can have multiple reviews

Sets a benchmark for Agile maturity across an Organization

‘Humanizes’ the Audit (not ‘check the box’) – gives teams a voice

Experience – 50% reduction in Controls… while doubling Quality

Leading – NOT lagging – metric; address problems before they manifest

Opportunity for two-way communication and learnings

Challenges Optimal model is labor intensive

Inherent subjectivity in assessments (‘Auditor Bias’)

Potential for teams to feel ‘over controlled’

Oversight and administration of the process

Communication and support for changes

Determining boundaries of adherence vs. non-adherence, and appropriate remedies

Ever-evolving process; can feel like an ‘arms race’

Common Questions Does this model Scale?

How much time per week would this require?

Isn’t this just the Scrum Master’s… or (insert role here) – job?

Could we use Pair Programming as a Control?

What is the future of Agile Quality Assurance?

Objectives Met?

Source: http://www.devballs.com/wp-content/uploads/2010/02/agilemanifesto.gif

Remember: Auditors are the Board of Health!

Questions?

Thank You!