Failure and Success in Software Automation

Comparing 7 Projects

Oleksandr Reminnyi

Pre word

▪ The only identification of project being successful is the fact that your customer is happy.

PMbook

Understanding the goals

We want to save money and time later on manual testingWe have a corporate direction…But the main is…Automation is a BUZZ word

Average Automation Reality

Automation AND Skill Set

HOKKU

«Let`s automate everything!», the client said. And Mount Fuji trembled.

Customer skills

Should customer be mature in automation?

This is definitely not a prerequisite.

But it can influence…

ХОККУ

Читаю чужі тести і торкається моїх скронь сивина…То мозок проливається

Automation Dev. Team Skills

Does mature mean good automation here?

Not always.

Practices mean more.

Automation AND Time line

Automation and SDLC

When the automation might be successful?

Waterfall OR Agile?

Depends on automation layer;

Post release period!

Implementation OF one TC

Depends on complexityDepends on number of steps

Depends on tested application instrumentation

Average 4 to 20 man hours. Happens 50

Automation Toolset

Automation and Backend

Unit TestsReally? Why?

Component/integration testingGood job!

Service layer automationDefinite place to automate!

Automation And TOOLS

Can tool influence the success of automation?YES!

Automation of Mobility

Can mobile platform automation be buried before birth?

YES!Silk Mobile -

$5000 per dev. per year,

20-30% failure rate only because of the tool issues

MonkeyTalk, Ranorex, etc.Build in the agent, then find issues

Appium maybe light in the end of the tunnel. But later

Consider out of device testing!

Documentation

Can automation exist without regression test Cases suite?

YES!

At least they say it’s successful…

Different types of manual test cases being Automated

- Functional- Regression- Acceptance- …

Benefit is brought at once by

Smoke ones

Automation Metrics

HOKKU

Sacura`s leaves are destined to fall down.Once again we`ll see the Test Report in red.Not only leaves are down.

What is usually Calculated?

ROI? Nothing!

Sometimes:

Failure Rate Suite Run Duration

Given architecture…

SUT UI Other SUT facilities that can be hooked but not needed for automation tool accessServices …

Layer Test scriptsTest scriptsFunctional Libraries

Layer Test scriptsTest scriptsTest scripts

Fr

amew

ork

utiliti

esBa

se c

lass

esRe

porti

ng

Load

ing

confi

gura

tion

Logg

ing

Etc.

Layer Test scriptsTest scriptsUI Maps – facilities to access UI (or other services) through automation tools API

Som

etim

es jo

ined

Layer Automation Tool

DB

Hel

pers

Faci

lities

to h

ook

into

SU

T th

roug

h AF

Mul

ti - l

ayer

ed a

rchi

tect

ure

Gave us possibility to collect More metrics

Something obvious

Questions?

“I see the testing standards that do not tell me how to create a test worthy of running." 

DeMarco