Upload
techwellpresentations
View
280
Download
0
Embed Size (px)
Citation preview
TR PM Tutorial
10/14/2014 1:00:00 PM
"Test Automation Strategies for the
Agile World"
Presented by:
Bob Galen
RGalen Counsulting Group, LLC
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
Bob Galen
Velocity Partners An agile methodologist, practitioner, and coach based in Cary, NC, Bob Galen helps guide companies in their adoption of Scrum and other agile methodologies and practices. Bob is a principal agile evangelist at Velocity Partners, a leading agile nearshore development partner; president of RGCG; and frequent speaker on software development, project management, software testing, and team leadership at conferences and professional groups. He is a Certified Scrum Coach, Certified Scrum Product Owner, and an active member of the Agile and Scrum Alliances. In 2013 Bob published Scrum Product Ownership–Balancing Value from the Inside Out. Reach him at [email protected].
1
Test Automation Strategies for the
Agile World
Bob GalenPresident & Principal Consultant
RGCG, LLC
Copyright © 2014 RGCG, LLC 2
Introduction
Bob Galen� Independent Agile Coach (CSC) at RGCG, LLC
� Principle Agile Evangelist at Velocity Partners
� Somewhere ‘north’ of 30 years overall experience ☺� Wide variety of technical stacks and business domains
� Developer first, then Project Management / Leadership, then Testing
� Senior/Executive software development leadership for 20 years
� Practicing formal agility since 2000
� XP, Lean, Scrum, and Kanban experience
� From Cary, North Carolina
� Connect w/ me via LinkedIn and Twitter @bobgalen
Bias Disclaimer:
Agile is THE BEST Methodology
for Software Development�
However, NOT a Silver Bullet!
2
Copyright © 2014 RGCG, LLC 3
Outline
� Traditional Automation – Business Case & ROI
� 3-Pillars
� Agile Test Automation Pyramid
� Agile Automation – Business Case & ROI
� Implementation Strategy
� Communication
� Wrap-up
Copyright © 2014 RGCG, LLC 44
3
Let’s start with…
Traditional Automation Strategy
� What are your current strategies towards:
� Test Automation
� Frameworks
� Tooling
� Maintenance
� ROI
� Team structure
� Get together in “pairs” and chat about this for 20
minutes. List your approaches and tactics.
� Then leverage SWOT analysis to capture your current
position. Then we’ll gather your results@
Copyright © 2014 RGCG, LLC 55
Let’s start with…
Traditional Automation Strategy
� SWOT – Often used to analyze strategic direction or
positions. List the “position”, then identify aspects of:
� Strengths
� Weaknesses
� Opportunities
� Threats
Often in Quadrants
� Force Field Analysis – again, list a position
� Forces (FOR) �
� Forces (AGAINST)
Copyright © 2014 RGCG, LLC 66
4
Traditional Automation
Business Case & ROI
Copyright © 2014 RGCG, LLC 7
It’s simple really…
� Manual testing is:
� Slow
� Labor intensive
� Not intellectually stimulating
� Error prone
� Iterative
� Did I say slow?
� And we’re testing software@right?
� So we ought to be able to automate the testing.
Copyright © 2014 RGCG, LLC 88
5
It’s simple really…
� Automated tests are:
� Fast
� Low/no errors
� Free after initial development
� Run continuously
� Did I say fast?
� Did I say ‘free’?
� And we’re testing software@right?
� So we ought to be able to automate ALL of the testing.
Copyright © 2014 RGCG, LLC 99
10Copyright © 2014 RGCG, LLC 10
� Capture size of the team
� Cost per tester – usually hourly
� # of Test cases to be run
� Time to run them
� Cyclical nature of the testing cycles (regression, integration,
system, etc.)
� Configurations
� Number of tests run per release
� Number of releases per year
� Do the math to come up with iterative testing run-time
investments@and costs
Test Automation
ROI Sample Points – Manual Testing
6
11Copyright © 2014 RGCG, LLC 11
� Then contrast that against the cost for automation:
� Time / cost to establish automation infrastructure
� Time / cost to establish automated tests
� Time / cost for ongoing maintenance and results analysis
� Cost for automation tooling – depends on ‘type’, ex: Keyword
Driven
� Cost per automation developer vs. manual tester
� Automation replacement target for manual test cases
� Do the simple, discrete math to come up with cost & time
to replace manual testing run-time and ROI savings
Test Automation
ROI Sample Points – Automation Development
12Copyright © 2014 RGCG, LLC 12
� Run more tests, faster and with fewer people (lower
costs)
� VALUE being driven by: Running tests!
� http://www.aspiresys.com/testautomationroi/index.php?action=__
reset
� Did I say with fewer people (testers)?
� We can hire more developers with the cost savings! And get
MORE done@
� We can even reduce the size of the bathrooms ☺
Traditional
Business Case
7
Copyright © 2014 RGCG, LLC 13
http://www.aspiresys.com/testautomationroi/index.php#ROI
Copyright © 2014 RGCG, LLC 14
http://www.aspiresys.com/testautomationroi/index.php#ROI
8
But often there is underestimation of risks…
So the ROI was always a ‘fuzzy’ estimate
� Underestimating the nature of a software project
� Architecture & design, complexity, risks, unknowns, ambiguity,
etc.
� Underestimating parallel software projects (product +
automation) complexity
� Integration, dependencies, coordination
� Underestimating ongoing support & maintenance
� Repairs & updates
� Designing and adding new tests
� Impact of new product technologies
� Underestimating skill levels for automation development
Copyright © 2014 RGCG, LLC 1515
It’s also very dangerous!
Will Robinson…
� It’s commoditizing your testing activity.
Could you imagine –� Measuring developer value by LOC@and doing ROI?
� Measuring architectural integrity by complexity analysis@and doing
ROI?
� Measuring product owner value by number of backlog elements@and
doing ROI?
Based on some sort of automation introduction? Of
course not. It’s insane.
� Then why do we do it with tests@and testers?
Copyright © 2014 RGCG, LLC 1616
9
17Copyright © 2014 RGCG, LLC 17
� We’re all using them, so they must be good and the
approaches must be ‘correct’@right?
� However, there are challenges—
� Training
� Size
� UI-centric
� COST
� Can’t test everything (Back-end database code)
� One-size fits all strategy
� Limited developer / whole team usage
Traditional Test Automation Tools
Just a quick stop…
Typical Deployment
Level of effort
Copyright © 2014 RGCG, LLC 18
Initial design
& Architecture
Initial Test
Case Automation
Technology Disruption
Ongoing Maintenance
MORE
Typical Time & Budget
Investment
LESS
�Level of Effort -
Investment
10
Now it may sound…
� Like I’m opposed to test automation.
Truly, I’m not. I think automation
rocks! Or like I’m opposed to gaining
a ‘return’, absolutely not.
� Automation is clearly a foundation
element for increased quality and going
faster – releasing more often
� But, this traditional model or
approach of focusing on ROI@is
sort of insane
Copyright © 2014 RGCG, LLC 19
3-Pillars of Agile
Quality & Testing
Copyright © 2014 RGCG, LLC 20
11
Copyright © 2014 RGCG, LLC 2121
3-Pillars
Genesis
� I’ve learned that “Balance” is important
� A sad tale of:
� Thousands of ATDD testing; Gherkin run amok
� All of them are working; continuously testing; increasing
“coverage’ and life is Good!
� BUT
� These same teams couldn’t write a cohesive User Story to save
their life
� So, where were the Acceptance Tests coming from?
3-Pillars of Agile Quality
Copyright © 2014 RGCG, LLC
22
Development & Test
Automation
• Pyramid-based Strategy:
(Unit + Cucumber +
Selenium)
• Continuous Integration
• Attack technical
infrastructure in the Backlog
• Visual Feedback –
Dashboards
• Actively practice ATDD and
BDD
Software Testing
• Risk-based testing:
Functional & Non-Functional
• Test planning @ Release &
Sprint levels
• Exploratory Testing
• Standards – checklists,
templates, repositories
• Balance across manual,
exploratory & automation
Cross-Functional Team
Practices
• Team-based Pairing
• Stop-the-Line Mindset
• Code Reviews & Standards
• Active Done-Ness
• Aggressive Refactoring of
Technical Debt
• User Stories, “3 Amigo”
based Conversations
• Whole Team Ownership of “Quality”
• Knowing the Right Thing to Build; And Building it Right
• Healthy – Agile Centric Metrics
• Steering via: Center of Excellence or Community of Practice
• Strategic balance across 3 Pillars; Assessment, Recalibration, and Continuous Improvement
12
Foundation of the 3-Pillars
Copyright © 2014 RGCG, LLC
23
• Whole Team Ownership of
“Quality”
• Knowing the “Right” thing to
Build AND Building it “Right”
• Healthy – Agile Centric Metrics
• Steering Required – CoE or CoP
• Strategic balance across 3
Pillars; Assessment,
Recalibration, and Continuous
Improvement
• Whole team view includes building it right,
everyone tests, everyone demo’s, etc.
• Focus on features/stories, confirmation,
conversation, and getting them staged
properly OVER testing
• 4-tier metrics: Quality, Value, Prediction, Team
• Agile strategies need light-handed “steering”;
establish a CoE (heavier weight) or a CoP
(lightweight)
• Consider finding an assessment framework
and then tying it to your strategy
measurement, recalibration, and continuous
improvement.
• Make the foundation visible thru information
radiators and metrics
3-Pillars of Agile Quality
Copyright © 2014 RGCG, LLC
24
Development &
Test Automation
• Pyramid-based
Strategy: (Unit +
Cucumber + Selenium)
• Continuous Integration
• Attack technical
infrastructure in the
Backlog
• Visual Feedback –
Dashboards
• Actively practice ATDD
and BDD
A central part of agile adoption is focusing on CI, 3-
tiered Automation development, and Dashboards to
begin incrementally building coverage for faster
feedback on changes.
100% automation is NOT the Goal!
In the interim, Hardening or Stabilization Sprints and
having a risk-based Release Train concept help
It’s important that Test or QA not ‘own’ the tooling or
all of the automation efforts. The strategy can come
from QA, but the tactical automation development is
best left to the team.
Mature teams invest in Automation, Tooling, and
Technical Debt reduction as part of Done-ness and
continually add it to their backlogs
13
3-Pillars of Agile Quality
Copyright © 2014 RGCG, LLC
25
Software Testing
• Risk-based testing:
Functional & Non-
Functional
• Test planning @
Release & Sprint levels
• Exploratory Testing
• Standards – checklists,
templates, repositories
• Balance across
manual, exploratory &
automation
Exploratory Testing (SBET with pairing) can be an
incredibly effective way to establish a whole-team,
collaborative view towards quality and testing. It also
emerges new tests.
Leverage ‘plans’ as a whole-team collaboration-
conversation mechanism; at Sprint and Release
levels.
Do not measure testing or tester progress; instead,
measure throughput, output, sprint outcomes, and
done-ness escapes at a team level.
You need a balanced test team; not everyone needs
to be able to program. But everyone needs to be
passionately skilled testers with curiosity.
Agile testing is a Risk-Based play in every Sprint and
across a release sequence.
3-Pillars of Agile Quality
Copyright © 2014 RGCG, LLC
26
Cross-Functional
Team Practices
• Team-based Pairing
• Stop-the-Line Mindset
• Code Reviews &
Standards
• Active Done-Ness
• Aggressive Refactoring
of Technical Debt
• User Stories – 3 Amigo
based Conversations
One of the hardest areas to get ‘right’ culturally. It
needs leadership alignment from Quality/Testing to
Product to Development and a consistent voice of
whole-team approaches.
This is where LEAN Thinking lives, where whole-
team collaboration happens, where professionalism
and craftsmanship are held dear.
I like the view of testers becoming the VOC,
champions of quality, and consistent questioners of
what is being build. Are we solving the right
problems@as simply as possible. Notions of Minimal
Viable Product / Feature help with focus.
And yes Virginia, there ARE standards, templates,
and a focus on x-team consistency!
14
Agile Test Automation
Pyramid
Copyright © 2014 RGCG, LLC 27
Copyright © 2014 RGCG, LLC 28
Agile Test Automation
Pyramid
� Coined by Mike Cohn, ~ 2005
� Establishes the validity of turning Traditional Automation
– upside down
� Invests:
� Most effort (%) in Unit Tests
� Moderate effort (%) in Middle-tier tests (business logic, API,
component, feature)
� Least effort (%) in UI-centric, top down tests
� Granularity of the tests is part of the strategy
15
Copyright © 2014 RGCG, LLC 29
Agile Test Automation
Pyramid
Often:
� Tied to Definition-of-Done from a maintenance
perspective
� You break it, you fix it
� Percentages vary; intent does not
� Whole-team ownership of automation
� Although it skews at either end of the pyramid
� Run as much as possible
� Check-in, Overnight, Cyclically
� Prime Directive = Real-time Feedback
Agile Test Automation
Pyramid
Copyright © 2014 RGCG, LLC 30
+80%, UI Centric Automation
0-10%, Service or Middle Tier Automation
0-10%, Unit level Automation
0-10%, UI Centric Automation
20-40%, Service or Middle Tier Automation
50-60%, Unit level Automation
Traditional
Agile
16
Agile Test Automation PyramidMike Cohn; Lisa Crispin & Janet Gregory
http://behaviordrivendevelopment.wikispaces.com/Testing
Copyright © 2014 RGCG, LLC 31
Brainstorm…
Agile, Multi-tiered Automation
� Get together in small groups of 4-6 to discuss
� Take a few minutes and think about your current
automation approaches:
� Tooling, approaches & strategies, strengths, weaknesses,
opportunities, maintenance challenges, future technology, etc.
� What sorts of adjustments would you need to make to
take this approach?
� What would be the largest challenges in taking this
approach? How might you overcome them?
� Do you “buy” the whole-team view to automation?
Copyright © 2014 RGCG, LLC 32
17
Agile Test Automation
ROI
Copyright © 2014 RGCG, LLC 33
What agile has exposed to me…
wrt/Automation & Testing
� Testers vs. a Whole Team view� Build teams in ratios – developers vs. testers
� Instead invest in cross-functional teams composed of the skills to design,
construct, test, and deliver excellent products
� Only testers write test automation
� The entire team writes test automation and, oh by the way, can test as
well
� Only testers test the application
� The whole team participates in writing / running automation, functional
testing, exploratory testing, regression testing, etc.
Copyright © 2014 RGCG, LLC 34
18
What agile has exposed to me…
wrt/Automation & Testing
� Traditional vs. Open Source
Tools� Instead of leveraging singular, UI-centric tools
� Leverage multiple tools at all-tiers of your application stack
� Higher cost
� Lower cost and a sense of “Community”
� Developers won’t use them; costs
� Everyone is leveraging the same tool-sets; AND integrating them in
Continuous Integration and Continuous Deployment
Copyright © 2014 RGCG, LLC 35
What agile has exposed to me…
wrt/Automation & Testing
� People are your primary value
proposition� Instead of looking at testers as a commodity
� Consider good testers to be as valuable as any other team member
� Instead of thinking that testing is a by-rote exercise with little intellectual
pursuit
� Testing is a profession. Testers are valuable craftspeople. Testing is an
intellectual exercise.
� Instead of thinking the value is in the test cases
� The value is in the testers (people) continuously reevaluating the right
things to test at the right time@for valuable feedback
Copyright © 2014 RGCG, LLC 36
19
Copyright © 2014 RGCG, LLC 37
http://outrage.typepad.com/crisisanalysis/2010/11/the-tyranny-of-roi.html
The NEW Test Automation
Business Case
� So, don’t
� Count pennies
� Count heads
� Count cycles
� Work as part of a TEAM
� Focus in on testing continuously – just the right things at just the
right time
� Providing feedback, transparency, and reaction
� Allowing your teams time to learn and improve
� Solving your customers’ problems; delivering value
Copyright © 2014 RGCG, LLC 38
20
39Copyright © 2014 RGCG, LLC 39
� Run more tests, faster, unattended, continuous
feedback, learning & adjustments
� Yes, saving time. Now what to do with it???
� Drive your new VALUE by:
� Early feedback for adjustments—risk avoidance, learning, solving
real problems
� Having more time to continuously refine your test cases
(automated and not)
� Having more time to focus on building Quality into your products
� Creatively testing the application (Exploratory Testing, Test Idea
Generation, Test Design)
The NEW Test Automation
Business Case
So, with the additional time…
INVEST in your testers & quality
� In building In Quality
� In attacking the customers REAL problems
� In better testing
� In more balanced automation
� In team training; in slack time
� In better collaboration
� In doing more than thought possible
� In creative solutions
� In adapting to new technologies
� In having FUN
Copyright © 2014 RGCG, LLC 40
21
How do we “Sell”
this change?
� You don’t sell it from a testing
perspective. Nor from a
bean-counting perspective.
� You allow the results to begin speaking for themselves
as we begin focusing on and talking about—
� How one builds truly innovative and creative software teams
� That understand and solve customer problems
� With simple solutions that fundamentally work
� And do so iteratively & quickly, while nimbly adjusting to
customer feedback
Copyright © 2014 RGCG, LLC 41
But Bob…
� That doesn't work in the “Real World”!
� My boss would never go for it without ROI justification
and hard measures
� Testing IS a commodity
� We’ve done automation perfectly – seeing none of the
issues/risks you’ve pointed out
Copyright © 2014 RGCG, LLC 42
22
But Bob…
� Think about what I’m saying; taking
the time and@
� Invest in people
� Invest in better quality & testing
� Invest in better product solutions to customer problems
� Invest in improved workflow and delivering high customer value
WOW – how crazy is that?
� Meet in small groups of 2-3 (pro / con) and chat for 10
minutes about the “sides” and the possibility for change?
� Then we’ll debrief@
Copyright © 2014 RGCG, LLC 43
Agile Test Automation
Implementation Strategy
Copyright © 2014 RGCG, LLC 44
Automation is one of the foundational investments of effective Agile
Adoptions. It’s not a choice, it’s an imperative! It’s a cost of doing business.
23
Implementation Strategy
Typically a 5-Phased Strategy
1. Strategy – This session: 3 Pillars + Pyramid + Team
2. Team Formation
3. Tools Selection
4. Training & Infrastructure Development
5. Automation Development
6. Care & Feeding
Copyright © 2014 RGCG, LLC 45
Phase 2
Team Formation
� GOAL@whole team ownership
� Skill-set discussion; SDET vs. Test Professional?
� May influence
� Development focus on Unit Testing
� Automation team to “gain momentum”
� Agile execution (Scrum, Kanban, transparency, demo, etc)
� Connect the dots architecturally
� Bias: I like to lead automation with a Test Automation
Architect
� Rather than with someone from the development team
Copyright © 2014 RGCG, LLC 46
24
Phase 3
Tools Selection
� Open Source vs. Commercial
� Singular, all-purpose vs. Tool-kit
� Cobble together or Integrate disparate tools
� Always perform a “Bake off” within your teams
� If Open Source, consider support implications
� Connect to development tooling
� Including CI and CD
� Additional Considerations: Test data, virtualization,
DevOps
� Be prepared to iterate, learn, and possibly fail
Copyright © 2014 RGCG, LLC 47
Phase 4
Training & Infrastructure
� Design, coding, and documentation standards
� Templates
� Community of Practice
� Example code, standards, forum, collaborative groups
� Models:
� Keyword, Data-driven, Table-driven, DSL, Model-driven, others
� Test Suite; Hierarchy, Naming
� Commercial: invest in training and/or consulting
� Open Source: hire and/or acquire direct expertise
� Also can build the expertise
Copyright © 2014 RGCG, LLC 48
25
Phase 5
Automation Development
� Move it to your teams ASAP
� Make it a part of Definition-of-Done
� It should become a natural outcome and not a choice or question
� Complete it within each sprint
� SAFe as a goal model for completing work in the same sprint!
� Salesforce as well
� It needs to be a negotiated part of your Product Backlog
� Have % targets;
� For every story:
� The TEAM decides the requisite automated tests at each level of
the pyramid?
Copyright © 2014 RGCG, LLC 49
Phase 6
Care & Feeding
� Automation evolution roadmap
� Merge with your architectural look-ahead
� Merge with your Product Roadmap
� Factor in ongoing automation investments
� Refactoring & Repairs as part of your Technical Debt
handling
� Consider it at the same level of investment as your
application code
� Stop-the-Line
� Commitment
Copyright © 2014 RGCG, LLC 50
26
Communication
� Leadership Vision
� High-level Plans, Strategic sharing with your teams
� Leadership collaboration (Development + QA)
� Intent to doggedly pursue automation
� Information Radiators
� Roadmaps & Product Backlogs
� Sprint & Release Reviews
� Even though it’s hard to show “working software”
� Let the “process” transparently communicate your progress,
impediments, plans, challenges
Copyright © 2014 RGCG, LLC 51
Let’s end with…
Agile Automation Strategy
� Based on what you’ve heard, discussed, learned@
� Get together in “pairs” and chat about this for 20
minutes. List your strategic approaches. Then leverage
FFA to capture your driving forces FOR and AGAINST
moving towards Agile Automation
� Most importantly, focus on strategies to overcome your
obstacles.
� Then we’ll gather your results@
Copyright © 2014 RGCG, LLC 5252
27
Copyright © 2014 RGCG, LLC 5353
Copyright © 2014 RGCG, LLC
Wrap-up
THANK YOU!
� I hope I’ve added to your previous
assumptions and understanding
� Inspired you to change your view towards Automation
ROI and investment
� Introduced you to models for
attacking Agile Automation Strategy
and ROI
� Final questions or discussion?
5454
28
Contact Info
Bob GalenPrincipal Consultant,
RGalen Consulting Group, L.L.C.
Experience-driven agile focused training, coaching & consulting
Cell: (919) 272-0719
[email protected] www.rgalen.com
[email protected] www.velocitypartners.net
BlogsProject Times - http://www.projecttimes.com/robert-galen/
BA Times - http://www.batimes.com/robert-galen/
Podcast on all things ‘agile’ - http://www.meta-cast.com/
55Copyright © 2014 RGCG, LLC 55
References
Copyright © 2014 RGCG, LLC 56
29
Links
� http://www.agileengineeringdesign.com/2012/01/7-
deadly-sins-of-automated-software-testing/
� Link to PDF of Pillar-1 chapter
� Links to “old” STP 3-part automation series
� Link to Mary’s – Agile Test Manager article
Copyright © 2014 RGCG, LLC 57
Open Source Tools
� Unit Level
� x-Unit
� Junit, nunit, phpunit,
� ATDD
� FitNesse, Robot Framework
� BDD
� Cucumber, JBehave
� UI
� Selenium, Watir, HP-UTF
Copyright © 2014 RGCG, LLC 58
30
…DD
much from the XP space
� TDD - Test Driven Development
� Test-first; Unit Tests as a design aid
� Debate around creating the unit tests “first”
� ATDD - Acceptance Test Driven Development
� User Story – Acceptance Tests
� Executable Requirements
� BDD - Behavior Driven Development
� jBehave, Cucumber; Gherkin – Given, When, Then
� More useful phrasing
� Tests as functional documentation
� Product Owners, Customer – writing tests
Copyright © 2014 RGCG, LLC 59
Agile Testing QuadrantsBrian Marick; Lisa Crispin & Janet Gregory
Copyright © 2014 RGCG, LLC 60
Exploratory testing
Scenarios
Usability testing
UAT
Alpha / Beta
Unit tests
Component tests
API tests
Functional tests
Story tests
Examples
Prototypes
Simulations
Performance testing
Load testing
Security testing
Non-functional requirements
Q1
Q2 Q3
Q4
Automated &
Manual
Automated &
Manual
Manual
Automation,
Tools, and
Manual
Business Facing
Technology Facing
Supporting the Team
Critiq
ue the Product