Maturing Agile SDLC

Notes for next meeting

Where We Left OffMy Opinion: • Current workflows in JIRA capture neither what we actually do nor

what we should do• Reviews and Showcases are not baked in to development workflow;

often are skipped because of that – This leads to increased defects and rework– Putting these in the workflow is easy and increases compliance

• Integration QA testing during sprint (more than unit tests) need to be the norm

• First step to ensure we do reviews and showcases is to put them in the JIRA workflow and GreenHopper– You will get better compliance without ineffective nagging if Reviews and

Showcases are on the Card Wall

Where We Left Off (2)

• There are many opinions of what Agile is and isn‘t– There are many flavors of Agile and many

misconceptions– Team interactions suffer when we all have our own

definitions and preconceptions• We need a clear and consistent definition of the

large scale agile approach – Need to define what the team is free to determine

and what eE needs them to do

What To Do

• Upgrade to JIRA– Upgrade scheduled for March 29– Not in critical path but important

• Improve Workflows– DEVTEST project has a rich set of example workflows

• Improve GreenHopper Card Walls• Create Filters and Dashboards for useful metrics • Define Roles and Responsibilities

What to Do (2)

• Create Consistency across all teams– Allows for mgmt roll-ups across groups– All teams should follow the selected successful

patterns• Teams can self organize and determine many details,

but not all– Addresses issue from SEs that nonstandard

workflows in different JIRA projects cause confusion when developers are assigned work on different teams

Maturing Agile SDLC

• Goal: Team gets to self-organize and work most efficiently, but there are other considerations

• Work for team is seconds a day, compliance is convenient, manageable and mandatory– They should be doing this now, so there are no

reasonable objections here…– Manage compliance, do not assume it won’t happen– Get the team’s commitment; people will mostly

honor their commitments

7

Task examples from Scrum Alliance SCM Training

8

Agile SDLC Concepts – Alt 1

Themes• A Theme is a top-level objective that may span projects and products. Themes sometimes may be

broken down into sub-themes, which are more likely to be product-specific scope for a Theme.• Themes can be used at both Program and Project Level to drive strategic alignment and

communicate a clear direction.Epics• An Agile Epic is a group of related User Stories. You would be unlikely to introduce an Epic into a

sprint without first breaking it down into it’s component User Stories so to reduce uncertainty.• Epics can also be used at a both Program and Project LevelUser Stories• A User story is an Independent, Negotiable, Valuable, Estimatable, Small, Testable requirement

(“INVEST Acronym”). User stories are clustered into Epics on a Product Roadmap.• User Stories are great for Development Teams and Product Managers as they are easy to

understand, discuss and prioritize – they are used at Sprint-level. User Stories will often be broken down into Tasks during the Sprint Planning Process.

Tasks• Tasks define the work needed to implement the User Story and can assigned and tracked separately • Tasks are organized in JIRA issues types by the workflow that should be followed to complete the

task (e.g. Development Work, QA Work, Defect , etc.)• Tasks are estimated in Hours with increments such as 2, 4, 8, 12 16, 20 & 24• Raise any disconnects in original estimates and total task estimates

http://agile101.net/2009/08/10/the-difference-between-agile-themes-epics-and-user-stories/

9

Agile SLDC Concepts – Definitions (1)

• Theme: Describe collections of Epics. Themes tend to identify larger, strategic organizational goals.– Can be issues in JIRA; certainly documented in Wiki.– Often are Products and product functionality is grouped into Themes and usually span

many groups/projects and Releases. – The Executive and Program Management Office should own Themes– Examples: Cart ,Checkout, Payments, Payments-PayPal, In-Store Pickup.

10

Agile SLDC Concepts – Definitions (2)

• Epic: Describe collections of User Stories. – Epics are often 1 to 3 months development duration. Epics can, but usually do not,

span releases. – All user stores that decompose the Epic into specific tasks are linked back to the Epic

in JIRA – Epics often span teams. – The Product Owners own Epics– Examples: Process PayPal Express Authorization, Process Credit Card return, Accept

Store Value cards as a payment type

11

Agile SLDC Concepts – Definitions (3)

• User Story: The smallest increment of delivered value. – They are thin slices of value that is a complete implementation across the full vertical

of the product. – User Stories usually are defined with scope so that the development can be completed

in an iteration. – Use the INVEST approach and guidelines (Independent, Negotiable, Valuable,

Estimable, Small, Testable)– The User Story sentence (see examples) is only part of a fully defined User Story– The Acceptance Testing Procedures (acceptance criteria) or Narrative of the User

Story defines in detail the Requirements, screen shots , acceptance criteria, Out of Scope, Notes, Q&A, UxD tasks and Design Notes• Example: Show Orders having mixed bag items in My Account (see Appendix also)

– The Component management team (Project Manager, Technical Manager) owns User Stories.

– Examples: • As a consumer, I want to see the next 5 closest stores whenever there are no stores within my

specified zip and radius on the Find in Stores page so that I can choose a store for in-Store Pick up from results

• As the store owner, I want to apply existing cart limits for an “ISPU only cart” and “mixed cart” so that consumers do not exceed the existing cart limits even with ISPU item(s)

12

Agile SLDC Concepts – Definitions (4)

• Tasks: Specific, independent and assignable work – In JIRA, these are “sub-tasks” under a User Story task– Tasks define work that can be assigned and tracked separately – Tasks equate to indented tasks in MS Project and be worked in parallel (Start-Start

dependency) or sequentially (Finish-Start dependency)– Examples:, Development Work, QA Work, Defect/Bug and less commonly

Requirements Definition and Design if needed– Example: For the User Story “As a consumer, I want to see the next 5 closest stores whenever

there are no stores within my specified zip and radius on the Find in Stores page so that I can choose a store for in-Store Pick up from results”• There should always be at least two tasks, Development Work and QA Work. • There could be spikes, additional Development Work tasks, Design Work or other

tasks as the team determines is needed to implement the story– Raise any disconnects in original estimates and task estimates and actual

work early

13

Tasks – Objections and Answers

• Some people object to decomposing User Stories into Tasks– This is an Agile best practice for almost all Agile approaches

• Objection: team member that picks up the story must complete the story– Agile / Scrum misconception; agile does not demand this, in fact it promotes the

opposite if it makes sense– There can be as many tasks as needed completed by whomever is best suited to

complete the work– Development can be broken down as needed; Testing should be separated from

development.– Many teams have dedicated Testers; Testing and Development are specialized skills.

Testers do not have to develop or vice-versa– The team or management can certainly make this decision base don the team

makeup

14

Tasks – Objections and Answers

• Tasks increases Agility– Agility (according to Geiser) is the effective decoupling of work.– Tasks allow teams to decouple and complete work for a User Story in parallel. Forcing

someone to do all the work is not agile nor the best approach• Other examples

– It could be that work for a User Story is tasked as follows• Java POJOs and services: java coding (including UI rough-in if needed)• Testing: test script development and test execution• UI finish work: Getting a UI-Guy to do finish work and JS is often more efficient,

producers fewer defects and increased standards compliance than having the less proficient java services dude do work they are not proficient

• Performance: If the work hits a key piece of functionality and the performance after the change needs to be compared against an established baseline, this specialized work is best delegated

• SQL and Stored Procs: Java rockstars are usually not as good at writing SQL as they think they are. If they can do the work, great! If there is alot of complex SQL work in areas that have to scale and be performant, get the right person to do the work to minimize defects and maximize productivity and Agility

15

Task examples from Scrum Alliance SCM Training

16

Quick Example

• Theme– Determined and owned by the Portfolio Office, Release Management, and Product Owner– Example: In-Store Pickup

• Epic– Product Owner, architect, and development participants chart out the Epics and rough size for scoping Releases– Best to use one estimation technique (Tee-Shirt sizes, etc) across teams– Example: Find-A-Store page

• User Story– The many User Stories of the Epic are described and estimated (Planning Poker session) before iterations. – Ensure this more precise sizing of effort is in-line with original Epic estimation; raise any discrepancies in

estimations immediately.– Best to use one estimation technique (story points, ideal days, etc) across teams– Best if Acceptance Criteria and all/only needed details (wireframes and implementation details, if any, such as

architectural guidance that the team needs to follow for any reason) are “mostly” completed (~75% to 90%) before the sprint the User Story is deemed available to be picked up

– Maintain a backlog of available User Stories in the range of 120% to 200% of sprint velocity– In Iteration Planning Meeting, select stories for iteration, (other IPM meeting activities)– Example:

• Tasks– Best to use one fine grained estimation technique (hours, idea days) across teams– SE and QA must review and refine Acceptance Criteria (clear up issues, remove improper implementation

directives, etc) and determine the tasks and effort for tasks – Ensure this more precise sizing of effort is in-line with original User Story estimation. Raise any discrepancies in

estimations immediately; this is a better estimate.

17

Teams Are Still Agile

• For team level tasks, teams still should self organize and control their work, but there are other teams

• Teams need to follow standard workflows just like they follow coding standards and the same definition of done all the teams follow– It’s like the Federal Government and the states; some laws are federally mandated but

the states have some leeway in how they are implemented and can establish their own (non-conflicting) laws

• Team members need to log time and keep their work updated on the card wall– This really takes seconds a day; compliance is not optional and teams are expected to

do this already; suck it up, and just freaking do it!• The team still controls in almost the entirely the tasks created, how the tasks

get picked up, etc– Can teams skip doing tasks? No; they can’t skip doing unit tests, reviews or

showcases either– Will there be the rare case has only Testing stories or development work? Probably. I

can’t see how we’d have Dev work but not QA work.

18

Objections: But that’s not Agile!

Yes it is!• Agile is NOT one strict rigid set of practices; blindly adhering to some random

website guideline is an agile anti-pattern!• The Agile SDLC should evolve to fit the needs of the organization• Development can be agile while upstream (Program Release Management)

and downstream (customer) segments of business could be more waterfall-ish (ideal if all are agile, but realistically…)

19

Objections: But that’s not Agile!

• Agile is often misunderstood or redefined to suit what people don‘t like doing:– The “entire team” does not need to be limited to one small group; but breaking down to small

groups of 7 +/-2 is ideal and teams tend to self organize on these scales anyway– SOS (Scrum of Scrums) meetings with leaders/Scrum Masters coordinate status of teams for the

entire project – Not every decision has to be, or should be, left up to the team

• Developers do not define requirements or architecture but need to understand both completely for the User Story they pick up– Thinking all aspects of the User Story must be developed by the team during a sprint is not Agile;

it’s “tiny waterfall”• Agile is about decoupling steps in the SDLC and doing them when they make the most sense; which may be weeks or minutes

before development and testing– In an Enterprise with multiple releases and production support, documentation is vital; Agile is not

an excuse to not document• The Agile Manifesto does not say “don’t documentation” it says: “Working software over comprehensive documentation” and “That

is, while there is value in the items on the right, we value the items on the left more.” Notice use of the term “comprehensive” documentation

• The right level of documentation is the right info for the teams or others to do their work but not more than is needed for that purpose

– In reality, there are more roles than Product Owner, Team Member and ScrumMaster– The small overhead from updating JIRA allows for greater agility, not less.

• The trick is to ensure that we do not get process heavy or have too little process that people do not know what’s going on with the project

– Information in JIRA is not Standup material and needs to be communicated to people who have no business being in the standup anyway because they will make us less effective but they need info to help us be effective

20

Objections: But that’s not Agile!

• Agile is NOT an excuse to stop producing documentation: It IS a reason to examine why you are producing the document to determine if it’s truly useful and valuable or if it’s simply what you’ve always done. In Agile we question the need for the document, and eliminate producing anything that doesn’t make sense, isn’t valuable, or isn’t useful. The team does not always get to make that decision.

• Agile is NOT an opportunity to eliminate planning: It IS an opportunity to institute rolling wave planning, a practice that’s documented and accepted in all Agile methodologies. In Agile projects, we don’t stop planning; instead we plan all the time, with the appropriate amount of detail for the time horizon that we’re planning for. Our project plan is a high-level vision with several key features, our release plans are focused on more detailed product features, our iteration plans get down to the tasks required to implement these features, and our daily plans coordinate activities, raise issues, and identify roadblocks. We can and do provide projected dates of completion and cost, using top-down planning and gross-level estimation techniques.

• Agile is NOT open season on scope creep: It IS an invitation to the customer to collaborate with the team. It IS an opening for the customer to have a way to change the requirements (in light of the consequences) and for the team to react accordingly. It IS a way to prioritize valuable features and work them through to completion, because we realize that eventually we’ll run out of time or money, or both.

• Agile is NOT about blindly following a set of “best” practices, whether or not they’re best for your project: Agile IS about doing what makes sense, based on the agile philosophy and the given situation.

21

Aligning with the Agile Manifesto

http://www.agilemanifesto.org/ © 2001

We are uncovering better ways of developing software by doing it and helping others do it.

Through this work we have come to value:• Individuals and interactions over processes and tools• Working software over comprehensive documentation• Customer collaboration over contract negotiation• Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

22

Aligning with the Agile Manifesto

Individuals and interactions over processes and tools• How we work in 2013 differs from how we worked in 2001• Similar example: the Design Patterns book by the GoF was originally published in 1995 and is in the

39th edition• Face-to-face interactions are great, but some interactions need to be captured for people who are

not there! • Tools like JIRA are the best way to maximize interactions; not everyone can/should be in the team

roomWorking software over comprehensive documentation• The right level of documentation is important, too little documentation is just as bad as too much; it

reduces agility and reduces qualityCustomer collaboration over contract negotiation• The goal is to completely meet the needs of the stakeholder, this will mean we do not get all the

requirements 100% correct before the project starts but 0% is just a badResponding to change over following a plan• Change still has to be controlled, but it is a normal part of any realistic development project; not

recognizing that is encouraging failure

23

Aligning with the Agile Manifesto – 12 Agile Principles

We follow these principles: • Our highest priority is to satisfy the customer through early and continuous delivery of valuable

software. • Welcome changing requirements, even late in development. Agile processes harness change for the

customer's competitive advantage. • Deliver working software frequently, from a couple of weeks to a couple of months, with a preference

to the shorter timescale. • Business people and developers must work together daily throughout the project. • Build projects around motivated individuals. Give them the environment and support they need, and

trust them to get the job done. • The most efficient and effective method of conveying information to and within a development team

is face-to-face conversation. • Working software is the primary measure of progress. • Agile processes promote sustainable development. The sponsors, developers, and users should be

able to maintain a constant pace indefinitely. • Continuous attention to technical excellence and good design enhances agility. • Simplicity--the art of maximizing the amount of work not done--is essential. • The best architectures, requirements, and designs emerge from self-organizing teams. • At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its

behavior accordingly.

24

Aligning with the Agile Manifesto – 12 Agile Principles

We follow these principles: • Our highest priority is to satisfy the customer through early and continuous delivery of valuable

software. [Because it is the best way to build software]• Welcome changing requirements, even late in development. Agile processes harness change for the

customer's competitive advantage. [Change control and impact of changes must be considered]• Deliver working software frequently, from a couple of weeks to a couple of months, with a preference

to the shorter timescale. [This is a strategy that gives us the best chance of success]• Business people and developers must work together daily [when required] throughout the project.

[But that is not the ONLY way to work and face-to-face is preferred, not required]• Build projects around motivated individuals. Give them the environment and support they need, and

trust them to get the job done. [Recognize the individuals areas of expertise and align accordingly]• The most efficient and effective method of conveying information to and within a development team

is face-to-face conversation. [but information needs to be captured and available for many reasons]• Working software is the primary measure of progress. [And we see these in Showcases]• Agile processes promote sustainable development. The sponsors, developers, and users should be

able to maintain a constant pace indefinitely. [Quality products are not made working 80 hr weeks]• Continuous attention to technical excellence and good design enhances agility. [Sometimes that

means the team doesn’t have a say in the design]• Simplicity--the art of maximizing the amount of work not done--is essential. [Do only what is asked]• The best architectures, requirements, and designs emerge from self-organizing teams. [Reviews are

performed by ad hoc teams]• At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its

behavior accordingly. [This is why we have retrospectives]

Workflows Changes

• Improved workflows• GreenHopper Card Wall Templates that use

improved workflows

Development Workflow

• Simplified version reflects Card Wall options– Dev Work shows up only after it is made Available by TM

or PM– Assigned and In Development Status show queued and in

progress work– In Review non-optional step and on Card Wall– Ready For Showcase on Wall

• TM/PM own the Close• Probably done in bulk at end of iteration but can be done from

wall as a one task move

• Other Transitions available from the full JIRA Issue page

Dev Workflow “Backbone”

• Yellow Statuses on Card Wall

• Available = ToDo• Assigned = ToDo

but someone specific is tagged

• In Dev = In Progress

• In Review = needed, not done now religously

• Showcase – IBID• Closed = Done

Complete Dev Workflow

• Non-Card Wall transitions available on full Issue view page in JIRA

• Managers generally do these functions

Create Issue

Open

Start Work

Assigned

In Development In Review

Start Work

Completed

Return to Development

Clarification

ClarificationNeeded

Return to Define

Requirements

ClarificationNeeded

Return to Assigned

Close

Closed

Sign-off Review

Reopen

Blocked

Block Resolved

Blocked

ClarificationNeeded

Stop Work

Sign-off Showcase

Ready for Showcase

ClarificationNeeded

Define Requirements

Create Requirements

Close

Assign Work

AvailableMake

Available

Make Available

Make Available

GreenHopper Changes

• Card Wall modified to show tracks used by Teams

• Assumption is Card Wall is how most transitions for these status will be done (OK even if not valid assumption)

• Filter for Card Wall shows • Transitions usually do not have screens;

assignee set to currentUser() by default

GreenHopper Card Wall

JIRA Upgrade

• In the GreenHopper version paired with JIRA, you can have multiple card wall templates per project– Ex: DEV and QA can have different card walls to

match the workflows for their tasks• The Card wall is no longer buggy as hell;

transitions actually work as expected– In the Card Wall in the prior slide, some transitions

do not work…GreenHopper is flakey

Filters and Dashboards

• Additional Statuses for Dashboards and filters that improve management and transparency

• Status easy to see by people who really should not be in a standup or SOS meetings

Assessing Done-ness

Column order is wrong in this example but column order is settable

Transparency: Anyone can at any time can assess status independently of project• Statuses serve as milestones • User Stories move from Defined Requirements to Available to In

Development to In Review to Ready For Showcase to Closed• Progress is based on actual status of work and earned value is clear• This info goes on a progress board. • Mgmt is able to assess Capacity and Demand (Available vs In Dev), limit

WIP and balance workloads across staff

Create

Open

In Development CompletedAwaiting QA Deployment

Deployed to QA

Ready for QA

QA in Progress

Start QA

QA Failed

QA Passed

QA Passed

Closed

Close

Stop QAQA Failed

Assign Test Developmer

Return to QA

QA Domain

Management Domain

Define Requirements

Start Work

Close

Available

Make Available

Make Available

QA Workflow “Backbone”

Complete QA Workflow

Appendix material

37

What We’ve Done in the Past

• Workflows were often not “designed”– Groups took JIRA default examples and made multiple rounds of changes– Sometimes our processes were changed to match simplified default example

workflows in JIRA – Workflows should be changed to match optimized processes– There are workflows that do have good thinking behind them– People who are not workflows experts made changes that may not have achieved the

desired goals and there was no assessments if the changes were beneficial – Many changes were done by one team without understanding the needs or processes

of the others teams involved– Some changes were often reactionary and not planned

• Workflows that affected many groups were implemented by one group without inclusion of others

• No or little training or support materials (such as Workflow diagrams and written descriptions of main and alternate flow statuses and transitions

• All projects are free to use any workflows they want– Feedback form SEs is that this is confusing when you work issues form other projects– Prevents unified metrics across all projects

38

What We Can and Should Do

• Standardize workflows across projects– We should do the same process the same way on all teams; all teams use the best

practices and follow the most successful patterns– Will allow for better and more comprehensive metrics across projects

• Create workflows that match what we should be doing– Many good examples to follow from in and outside GSI

• Manage change to workflows better– Someone with Workflow, Process Management and Lean Six Sigma background

should own the processes and changes• The sooner we improve the process, the sooner we benefit• Minimize disruption by showing impact to teams in limited in new workflows to

color coded “domain” (see workflow diagrams)• Changing to a new workflow set is not as disruptive as it is often thought to be

– Plan the change to minimize impact (buy-in, training, etc.)– Make one major change and tweak on feedback

• Many incremental sustainable improvements build into big impact very quickly

39

Guiding Principals of Workflows Design

• Good Workflows:– Map out processes; institutionalize value add steps and eliminate non-value add steps– Model what we should be doing so the process is repeatable– Reduce learning curves by guiding people through processes– Are as simple as possible but not so simple that they do not capture the needed

process• JIRA workflow Statuses and Transitions should be as self documenting as

possible.– Ex Statuses: “In Triage”, “In Development”, “Closed”, “Review Needed”, “Return to

Development”– Ex Transitions: “Start Work” , “Stop Work” “Make Available”, “Close”

• Refactor workflows if needed– Small changes that are mostly self-documenting need little or no training and are not

disruptive– Sustainable process improvements are easy to roll out and their effectiveness can

usually be measured