Mark meninger-feb-2010

An Examination of Test Automation Use Cases(and the factors behind the examination)

January 19, 2010

1

Mark Meninger

Confidential McAfee Internal Use Only

An Examination of Test Automation Use Cases

Contents

• The Speaker

• Presentation Objectives

Objectives and Perspective

• Objectives of test automation

• GUI vs non-GUI

• A look at 2 products

Test Automation Philosophy

Test Automation Use Cases

How does this all fit in?

January 20, 2010

2


January 20, 2010

3



January 20, 2010

4

The Speaker ([email protected])


• Current role @ McAfee: Driving test automation on Consumer side

• Functional GUI automation (support for L10n) across 3 sites

• Functional non-GUI automation (development focused)

• Automated Performance testing

• Previously @ RIM

• As test automation manager for end-user BlackBerry testing

• RIM Test Automation Conference Chair

• A student of test automation and testing

• Worked (and work) with some very capable individuals

• Made some good mistakes

• Learned from my mistakes

• Little experience with Agile

• Presentations @ StarEast, QAI, KWSQA

• My Blog (as of this Friday): http://automatedtestingblog.blogspot.com


January 20, 2010

5

Presentation Objectives


1. Talk about my learnings and philosophy of test automation

2. Hear your perspective on what I’m talking about

3. Evaluate test automation use cases/examples (group-

exercise)

a) Interface complexity

b) Speed of test automation

c) Speed of execution

d) Derived value

e) How this fits into the testing cycle

4. In the context of Agile, I would…


January 20, 2010

6

Mark’s Test Automation Philosophy



January 20, 2010

7

Manual Testing - A path in the trees


• Strategy/Plan (resourcing, infrastructure, approach)

• Need to learn the SUT:

• User-stories

• Use Cases

• Tools

• Technologies

• Testing types & cycles (regression, defect, smoke)

• Sprints

• Apply common techniques

• Scripted test cases (with different methodologies),

• Exploratory testing

• Then testing can begin

• I’ve seen most testing organizations do this well to varying degrees


January 20, 2010

8

Automated testing – Find a path in the forest


• Test automation is less of a cookie-cutter approach than manual testing

• Common tasks in test automation

• Interface, framework, integration

• Very specific considerations:

• The technology of the system under test (SUT)

• The technologies used to test the SUT

• Impact on the development/testing schedule

• These details impact the success of the

implementation, the derived value, the

time to delivery, availability etc

• This makes each test automation

implementation unique and challenging

• Not all orgs do test automation well


January 20, 2010

9

Evolution of the Approach to Test Automation


In the beginning:

Here is a test automation tool… now automate! (seen it)


January 20, 2010

10



And Then:

Evaluate tools, choose one and automate!

(done it)


January 20, 2010

11



After much learning (now do it!) :Examine my test automation requirements

a) Look at

• Objectives

• SUT technology, roadmap

• Available budget, resources

• Available tool technologies, capabilities and constraints

• SDLC methodology (Agile, Waterfall etc.)

• Development culture

• Testing culture

• Relationships

b) Put together my business case

c) Start with a careful plan


January 20, 2010

12

Objectives of test automation

1. Executed successfully

relatively early within the

development/testing cycle

2. Reduction in manual testing

time


�Test functionality sooner

rather than later in the cycle

�Interface used is available

and stable

�Automated execution as soon

as a build is released:

� Evenings, Weekends

�Automated execution run in

parallel

� Execution across platforms and

test suites

� Limited by availability of

infrastructure


January 20, 2010

13

Test Automation Objectives

3. Provides reliable results on

1st run

4. The solution is scalable &

maintainable


�Tests can be executed repeatedly

with minimal false negatives

�Regular testing of interface and

framework essential

�Changing interface not breaking

tests

�We can increase coverage of

manual test cases without excessive

framework growth

�Grow test case numbers with

confidence:

� 10’s -> 100’s

� 100’s -> 1000’s


January 20, 2010

14

Test Automation Objectives

5. Agile Environment


�Out-of-the-box, light-weight test

automation supports planned sprints

�Test automation team structured,

organized to support agile testing

requirements

�Solution, approach is flexible and

expandable


January 20, 2010

15

GUI vs non-GUI Technology Considerations


GUINon-GUI(some API)


January 20, 2010

16

Test Automation – The Big Question


Re

UI-Logic Interface

UI

And Another

The Basement

Another Abstraction

Do we start here?

Or here?

Or here?

Where?Where?Where?Where?

SoftwareAbstractions

First Business Logic Layer


January 20, 2010

17

Test Automation - GUI Fat Client (Generalities)


• Interface

• Typically complex - more points of failure

• Changes more frequent

• Increased maintenance

• Automation - dependent upon being stable

• Technology

• Vendor tools provide better support

• Scripting technology usually has limited support (i.e. VBScript)

• Performance

• Management for unresponsive GUI

• GUI adds performance hit each time accessed

• Framework can add considerable overhead

GUI - Fat Client• Skill-set

• Requires solid technical skill-set with good design and programming abilities

• Deeper test automation skill-set and experience required

• More expensive

• Localization

• Running identical functional tests across localized builds

• Language independence support is essential to support L10n automation


January 20, 2010

18

Test Automation – GUI Web (Generalities)


• Interface

• Typically simpler – fewer complexities to manage

• Changes frequently over time

• Automation – dependent upon being stable

• Technology

• Good open source (Selenium, Watir)

• Performance

• Fewer if no performance issues

• Good available open source frameworks

• Skill-set

• Better use of ‘scripters’ rather than developers

• Less investment required in more experienced test automation skill-sets

• Less expensive

• Localization

• Language independent testing can be supported

GUI - Web


January 20, 2010

19

Test Automation – non-GUI (Generalities)


• Interface

• Stable and reliable earlier in the dev/testing cycle

• Less changes over time

• Fewer points of failure

• High-level testing can be supported

• Usually needs to be modified for increased testability over time (fits Agile well IMO)

• Technology

• Choose scripting technology with robust library support

• Integrate into development cycle

Non-GUI (API) Automation

• Performance

• No GUI impact - much faster

• Choose integrated framework with little overhead

• Skill-set

• Requires knowledge of underlying business logic, application architecture and design

• More expensive

• Localization

• Issues around language independence (localized strings integrated into GUI abstractions)


January 20, 2010

20

GUI Automation


• Automation Implication

• Coverage

• Good top-to-bottom testing solution

• Broader

• # of automated test cases dependent upon GUI comlexity

• Development (GUI complexity)

• Higher maintenance costs

• More effort to write tests

• Longer development times

• Execution

• Dependent upon stable GUI

• Slower execution times

• More expensive (resources, equipment and license cost)

• Value-add later in dev/testing cycle for products with major GUI changes

GUI


January 20, 2010

21

Non-GUI Automation


• Automation Implication

• Coverage

• Not top-to-bottom

• Deeper testing

• Larger # of automated test cases

• Development

• Easier to write tests

• Shorter development times

• Lower maintenance costs

• Execution

• Ready when the API are ready

• Test much sooner in the dev/test

• Test results faster; test more often

• Gate before release to testing

• Cheaper (resources, no licenses)

• Add quality paradigm to development organization (code for testability)

• Value-add sooner in dev/testing cycle

Non-GUI Automation


January 20, 201022

RTQA Build Release Cycle

API Automation

Quality AssuranceStarts

Automated

API Testing

Manual

Testing

TestsPass?

Automated

API Testing

Automated GUI

Testing & Manual

Testing

TestsPass?

!!!??

Manual TestingGUI Automation

Start End

No No

YesYes


January 20, 2010

23

Automated Testing Coverage

Cost

Complex GUI - Cost for Coverage

• Complex GUI applications will incur an increasing cost to

automate larger #’s of test cases

• Cost: Effort, Infrastructure, Dollars, Time for Execution

Automation Ceiling

Costs become prohibitive


GUIGUIGUIGUI


January 20, 2010

24

Automated Testing Coverage

Cost

Automation Philosophy Review – Cost for Coverage

• Non-GUI Automation has lower cost for coverage

• Cost: Effort, Infrastructure, Dollars, Time for Execution

Automation Ceiling

Less expensive


GUIGUIGUIGUI

NonNonNonNon----GUIGUIGUIGUI


r

Consumer Test Automation Framework (CTAF)/QTP/MAGI

MAGI FrameworkServices

Core Functions

CTAF Extensions

HP Quick Test Pro

GUI HTMLControl

ID HTMLControl

ID HTMLControl

ID

CTAF External LibrariesGUI/Table

LanguageDependentAssert

VBScript Test Script

CTAF Internal Libraries

Unit Test

Helper

GlobalMPF MVS

Lots of librariesto build(and support)

Complex Framework(yet very helpful)

Our ownextensions


Python py.test/nose(?) Test Automation Framework

Python

Libraries

COM InterfaceImplement IDispatch

High-Level Class

High-Level Class

High-Level Class

Python Test Script

CTAF API Python

Libraries

Example API Test Automation – Using Python

Python hasnumerousrich and diverse libraries

Framework exists which nicely integrates with Python


January 20, 2010

27


A look at some interfaces

• Google Search

• McAfee Security Center (2010 release)

Disclaimer

• These thoughts are my own and are not necessary correct

• Part of the process would be for me to understand the

technology and determine the best approach

(Finding a path through the forest I find myself in)



Points to Evaluate

• What approach would you take?

• What are the risks of doing it this way?

• What are the costs/investment requirements of doing this? (be

specific)

• What are the gains of doing this?

• Why would you do this?

• How would you integrate this into a testing cycle?

• Why would you integrate this into the testing cycle this way?

• How could this approach be fit into an Agile/Lean

methodology?

• What type of buy-in would you need to achieve this?

• What relationships would you need to establish to be

successful?

January 20, 2010

28



What I will focus on

• Interface complexity

• Speed of test automation

• Speed of execution

• Derived value

• How this fits into the testing cycle

January 20, 2010

29


January 20, 2010

30


Test Automation Use Cases – Google Search

•Google Search

– From: http://en.wikipedia.org/wiki/Google_search

– PageRank Logic

– synonyms, weather forecasts, time zones, stock quotes,

maps, earthquake data, movie showtimes, airports, home

listings, and sports scores

– prices, temperatures, money/unit conversions ("10.5 cm in

inches"), calculations ( 3*4+sqrt(6)-pi/2 ), package tracking,

patents, area codes,[6] and rudimentary language

translation of displayed pages

– ‘I’m feeling lucky’

– Privatization logic (Switzerland)

– Ajax code

– ….

January 20, 2010

31



Our job to automate Search logic • Core functionality

• Page rank is an algorithm that evaluates an index using

supposedly over 200 factors

January 20, 2010

32



•What I’d do– Back-end (non-gui) automation

– Focus just on the engine and it’s data

– Evaluate the probability and weighting of each factor

– Generate a list of results and probably would fit into

some level of confidence of accuracy

– Rendering of data could be easily verified manually

January 20, 2010

33



•How• Work with developers to build a testing engine that could handle

‘plug-ins’ of new testing factors

• A common testing pattern would be for each factor

• Determine how each factor would return results on its own or in

interaction with another factor

• Integrate this all into the testing engine

• Establish a common mechanism for executing tests, gathering

expected results and comparing actual results

• Utilize the common testing pattern

• Utilize the same pattern for evaluation in the testing engine

• Drive a common testing interface across those adding new ranking

functionality to the google search engine

January 20, 2010

34



Interface complexity

• Interface would be simple – an API

• The combination of algorithms (factors) would make this

complex

Speed of test automation delivery

• Fast

– Could start writing tests for each factor

– Could start building in some relationships for each factor

• Slower

– Integration of everything together

– This would also include the test engine

January 20, 2010

35



Speed of execution

• Very fast

• Could run very often

Derived value

• Substantial

• Could fully automate the algorithm

• If the integration of factors together could be done

successfully, this would be substantial

How this fits into the testing cycle

• Immediately

• Write code, test code

• No throw-over to SV&V/QA

January 20, 2010

36


January 20, 2010

37


January 20, 2010

38

Cialis - erectile dysfunction drug

Radical Prostatectomy - is an operation to

remove the prostate gland and some of

the tissue around it Very fast


January 20, 2010

39


Test Automation Use Cases – McAfee Security

Center

•McAfee Consumer Product (2010)

– Security Center

– Firewall, AntiVirus, AntiSpam, Parental Controls…

– Focus on AntiVirus

– Verify GUI and Functional behaviour

January 20, 2010

40



• McAfee Consumer Product (2010)

– GUI – thick GUI


January 20, 2010

41



• How

• Vendor tool

January 20, 2010

42






January 20, 2010

43






January 20, 2010

44



Center

•What I’d do & How– Get a good GUI automation tool

– Find a manageable way to integrate with the GUI DOM

• Re-usability

• Maintainability

– First go: limit my automation to easiest test cases

– Find a good framework to integrate with my GUI

automation tool

• Automate the automation

– Aim would be reduce manual testing with an eye to

reduce maintenance

January 20, 2010

45



Center


• Interface is very complex

– Lots of objects to manage (lots of points of failure)

– How we work with the interface is complex under the

covers (constrained by tool)


• Slow!!

– Most off-the-shelf have limited library support

– Have common set of libraries across the products

– Need to build libraries for each product at the GUI & functional

level

– Integration into the framework could take even longer

January 20, 2010

46



Center

Speed of execution

• Slow!

• Fat client GUIs are usually slower

• Hooks from application driving the GUI adds overhead

• Integrating a framework that drives all this adds additional

time

Derived value

• Some!

• New GUI; some changes – later in the cycle

• Same GUI; little changes – earlier in the cycle

• Need more infrastructure to support

January 20, 2010

47



Center


• If no GUI changes - > Right Away

• GUI changes - > Later

• Testing cycles will take longer

• Slower to execute

January 20, 2010

48


January 20, 2010

49

Test Automation – The Big Question

Test Automation Use Cases – McAfee Security Center

Re

UI-Logic Interface

UI

And Another

The Basement

First Business Logic Layer

Another Abstraction

Do we start here?

Or here?

SoftwareAbstractions



Center

•What I’d do & How– Examine the layers below the GUI

– Can I use any of them to automate testing?

– Work to drive this idea within the organization

– Most likely some developer has thought the same thing

– That is my foothold and I build on that

January 20, 2010

50



Center


• Interface is less complex

– New technology learning curve

– Easier to call some API than manage GUI objects

– Less change; more stability


– Fast!!

– Available frameworks (don’t need to build your own) (i.e. py.test)

– Access to rich scripting environments & libraries (i.e. Python) – don’t

need to build your own

– Less points of failure to manage

January 20, 2010

51



Center

Speed of execution

• Fast!! No GUI overhead

• Integrated framework will also add little overhead (i.e.

TestNG, Py.test)

Derived value

• Higher value


• Almost always test earlier in the cycle

• Test more frequently

• Integrate into the development cycle rather than QA

– Quality moving upstream

January 20, 2010

52


January 20, 2010

53


January 20, 2010

54



• Have your test automation specialist begin to develop a methodology that

will fit your agile development cycle

• Tell her/him what your requirements are and ask for a solution that will

meet this

• Optimize your test automation execution for Agile!

• This will most likely be an out-of-the-box approach to test automation

• All pay-offs should be well understood:

• Light-weight, quick to execute, easy to develop, not-too-deep solution

• Heavier, complex, longer-to execute, harder to develop, deeper test

automation solution


January 20, 2010

55



• They should have a good understanding of the available technologies to

use and what the trade-offs are for each

• What solution will best meet the above requirements?

• Let that test automation specialist know what’s needed and why. This will

hopefully inspire them to build the solution that meets these needs

• Integrate automated testing into your testing cycles

• Fit the automated testing for a given sprint/release

• Each automated testing approach will have a given set of benefits and

costs

• Choose the automated testing items that make most sense


Consumer Applications QATest Automation Strategy – Detailed Review

January 20, 2010

56

[email protected]

As of this Friday

http://automatedtestingblog.blogspot.com

Documents

Mark meninger-feb-2010