PushToTest TestMaker 6.5 Open Source Test Design Document

TestMaker 6.5 PlanFrank Cohen, CEO(408) [email protected] 8, 2011

Collaborative Results, Resource Repository, UI Enhancements

Intellectual Property of PushToTest. (c) 2011 All rights reserved.Note: All trademarks mentioned in this document are the trademarks of their respective holders.

Open Source Test Automation

Intellectual Property of PushToTest. (c) 2011 All rights reserved.

• Improved Test Operation Speed

• Jasper Business Intelligence Reporting

• Test Object Designer

• Record/Playback

• Firefox, IE, Safari, Chrome, Opera

• Open Source Alternative toHP QTP

• Adds Sahi, Flex

• Data and Object Enabled

• HtmlUnit Viewer

Product RoadmapTestMaker 6.1 TestMaker 6.5 TestMaker 72011 2011 2012

• TestCards Phase 2

• Web Reports

• Web Controller

• Test Management

• Rally

• TestLink

• HP QC

• Zephyr

• QMetry

TestMaker 6.02011

• Easier Test Controller

• Reports

• Collaborative Charts

• Live Charts for Drill Downs

• Test Object Repository

• Share Test Resources

• Search

• Access Control

• LDAP/Rally

• TestCards 1

• Tabular Results Charts

• Flex 4 In-Browser Recording

• TestMaker JAT for Oracle Applications

• Citrix Applications

• TestCards Phase 1

• Rally Integration

• Web Reports

• Web Controller

• Bug Fixes



TestMaker 6.5‣Resource Storage in the Repository

‣Access Control for users and LDAP integration

‣Search

‣Reports

‣Live Reports

‣Collaborative Reports

‣Easier Test Controller

‣Streaming Results using Snapshot Protocol

3



Needs Research for 6.5‣TestMaker to restore Windows and IE proxy on quit

‣Console Connects To TestNodes (Cloud) Through Proxy

‣URL to Web-based TestMaker Test Results Automatically Saved To Rally Test Case

‣L7 and HPOpenView reports gathered automatically

4



Not In TestMaker 6.5‣Full Web Interface for TestMaker‣Test Management‣Rerun Tests of Failed Previous Test Operations‣TestLink, QMetry, Zephyr, Quality Center Integration‣Colossus DPL (aka Unique DPL v2)‣Dynamic “Push” Bundle Deployment, Use Resources Instead‣Remote TestNode Sahi Proxy SSL Certificate Registration in

Community‣Open Designer, see Editor too‣No dashboard view “How Are We Doing”‣Each Controller has a custom status display (icon and text) settable

by API from a test step context‣Graphical view to edit and manage RDBMS for DPL content

5



Resource Repository‣File System or Repository

‣Editor, TestMaker Object Designer, Console

‣Open, Save, Save As

6

Local Repository

QA Unit 1 Repository

Open Test Resource

Repositories

OpenCancelShow Versions

/

Project X

3rd Party Connector Project

TestScenarios

Project Y

Project Z

Load Test 103 Scenario

mcohenJune 12, 2011 6:05 pm






Last Modified Creator

Local Repository


Save Test Resource

Repositories


SaveCancelShow Versions

/

Project X


TestScenarios

Project Y

Project Z











Resource Repository Search‣Search File Names

‣Search TestMaker Object Designer Meta Data

‣Name, Creator, Links, Version, Types, Operations, Goals

‣Object Repository objects: Story, Form, Object

‣Object name, Type, Accessor, Comment

‣Search Editor Meta Data

‣Author Name, Creator, Links, Versions, Categories

7



Access Control‣TestCard Viewable by Anonymous, Role, User Name

‣TestScenario View, Change, Run, Stop, Pause, Create, Delete by Anonymous, Role, User Name

‣Repository Read, Change, Create DB, Create Contents, Delete by Anonymous, Role, User Name

‣Role Create, Change, Delete by Anonymous, Role, User Name

‣Reports Create, Change, Delete by Anonymous, Role, User Name

8



SSO Option with Rally‣Optionally Uses Rally LDAP for User Authentication

9

PushToTest TestCards - Best Buy

help

TestMaker Enterprise 6.0.1

Licensed for 50 virtual users

and 10 service monitors

Welcome To TestMaker TestCards

fcohenUser ID

•••••••••Password

Sign-in

Sign-in Using Rally

Options



Enhanced Controller‣Web (for TCS 1) and SWING

‣Reports Button -> New Reports

‣Tabs Expand To New Window

10

TestNode Status

51340 Seconds Remaining

513 Success

65 Failed

Ajax Load Test

Executing iteration: 1 Users * 1

Instant Results

LogsReports Configuration

1 TPS2.5 TPS 4.6 TPS

Real Time Scalability Index Chart

6

5

4

3

2

1

0

Local TestNode Log Local TestNode Error Log

12.67 % Failure

1 Users * 1

5 Failed

.02% Failure

213 Success

2 Users * 1

35 Failed

7.36% Failure

475 Success

4 Users * 1

65 Failed

12.67% Failure

513 Success

Next let’s talk about



Reports Design

‣Resource Plan

‣User Interface Design

‣Required Changes to TestMaker 6.0

11



Reports Summary‣User Interface Design (Tomas)

‣Reports - A report is a collection of charts

‣Charts - A graphical view of the test results repository. For example, the Transaction Distribution chart

‣Live Charts - A user interface to see custom step logging data

‣ScriptRunner (EOP) Enhancements (Luis Carlos)

‣DesignerScript - Options to store screenshots, DOM after executing each command in a .ds file

‣soapUI - Options to save each TestCase steps request & response

‣Java - Options to save request, response, stack trace

12



Required Changes

13

‣PTTStepListener and Repository Schema

‣StepRequest (String, may contain CR&LF characters)

‣StepResponse (Blob, may contain CR&LF characters)

‣StepStackTrace (String, up to 10K characters)

‣StepExceptionName (String)

‣Screenshot (Blob)

‣TestScenario Schema Changes

‣Define Report, Charts, Drill Downs



Optional Enhancements‣Caching of Jasper generated charts to avoid reprocessing time

14



Run A Report

15

PushToTest TestMaker 6 - Best Buy

help




TestMaker Results Repository

Run Report Edit Report Delete Report New Report

Reports

simg

Weekly Scalability ReportTSH Working Team 2

simg

Load Test for Mods To 3rd Party GovPerformance after changes

simg

Functional Test for Geek Squad RegTesting new registration process



Report Appears In Window

16

PushToTest TestMaker 6 - Weekly Scalability Report - Created on September 10, 2011

Weekly Scalability ReportCreated on September 10, 2011

Transaction Distribution Chart



New/Edit A Report

17


help





New Report

NextCancel

Report Name:



Add Charts To The Report

18


help





Report Settings

SaveCancel

Weekly Scalability ReportReport Name:

Weekly Scalability ReportHeader:

M Cohen - [email protected] x3817Footer:

Scalability Index

Transactions by Avg Duration


Charts

!

!

Report Details

New Chart Edit Chart Delete Chart



Chart Types

19

‣Scalability Chart

‣Bar Chart

‣Grouped Data Points

‣Transaction Chart

‣Line and Data Point Chart

‣Shows Individual Points of Data

‣Resource Chart

‣Bar Chart

‣Resource (CPU, Net, Memory) Usage Percentage

‣Data Chart

‣Grid Chart

‣Transaction and Step Data



Scalability Chart Settings‣Vertical

‣Transactions Per Second

‣Count of All Transactions

‣Count of Failed Transactions

‣Count of Successful Transactions

‣Avg, Min, Max Transaction Time

‣Horizontal: VU Iteration

‣Secondary Axis (%)

‣Failure Percentage

‣Success Percentage

‣Resource Monitoring (Choose from CPU, Net, Memory)

‣Option: Display List of Step Names

20



Scalability Chart Settings

21


help





Scalability Chart Settings

Cancel Save

Transactions Per Second (TPS)

Count of All Transactions

Count of Successful Transactions

Count of Failed Transactions

Avg, Min, Max Transaction Time

Secondary AxisFailure Percentage

Success Percentage

Resource Monitoring CPU

Average

Option: Display List of Step Names



Transaction Chart Settings‣Vertical

‣Transaction Duration Time

‣Vertical Options

‣Successful

‣Failed

‣Duration: Avg, Min, Max

‣Horizontal

‣Time Offset Into Test

‣Specific Start/End Time

22



Transaction Chart Settings

23


help





Transaction Chart Settings

Cancel Save

Transaction Duration Time

Successful

Failed

Duration Average

Horizontal Axis

Show standard date/time



Resource Chart Settings‣Vertical

‣Percentage used

‣Grouped by CPU, Net, Memory

‣Horizontal

‣Time Offset Into Test

‣Specific Start/End Time

24



Resource Chart Settings

25


help





Resource Chart Settings

Cancel Save

Percentage Used

Grouped by CPU

Horizontal Axis

Show standard date/time



Data Chart Settings‣Successful Transactions

‣Failed Transactions

‣Option: Show Steps

26



Data Chart

27


help





Data Chart Settings

Cancel Save

Successful Transactions

Failed Transactions

Show Steps



Live Charts for Drill Downs

28


help





Live Charts

Scalability Index

Transactions by Avg Duration


Charts

New Chart Edit Chart Delete ChartView Chart




help






29




help






30




help






31




help





Click To See Steps

32

Request 2

Groovy Script - Validate

Request 1

Steps In Selected Transaction - soapUI TestSuite

175

3,201

643



Additional Requirements

33

‣Keyboard left-right arrows move selected point

‣User entered Notes

‣Steps need labeling



Click Step To See Detail

34

CustomErrorResponseRequest

Request 2

Groovy Script - Validate

Request 1

Steps In Selected Transaction - soapUI TestSuite

<soapenv:Envelope xmlns:soapenv="http:// <soapenv:Header/> <soapenv:Body> <exam:Change_Price> <ProductNumber>1033</ProductNumber> <Type>add_product</Type> <Quantity>55</Quantity> <Location>2</Location>

175

3,201

643



For Java, DS Steps

35

Product Catalog Search

Buy Products

Log-in To Application

Steps In Selected Transaction - DesignerScript

3,201

175

675

ScreenshotErrorResponseRequest

Command: _clickLocator: textbox("company")Value: "PushToTest"



For Java, DS Steps

36


Row1_ColumnB: 38178Row2_ColumnB: 98123Row3_ColumnB: 18177


Unable To Find Flight Number Exception

java.lang.Throwable atboo.hoo.StackTrace$FirstNested$SecondNested. at boo.hoo.StackTrace$FirstNested.<init>(St at boo.hoo.StackTrace.<init>(StackTrace.j



Add A Step As A Chart

37

Product Catalog Search

Buy Products

Log-in To Application

Steps In Selected Transaction - DesignerScript

3,201

175

675

Add This Transaction To Report As New Chart

Add This Step To Report As New Chart



Add More To The Report

38

PushToTest TestMaker 6 - Weekly Scalability Report - Created on September 10, 2011

Weekly Scalability ReportCreated on September 10, 2011


Command: _clickLocator: textbox("company")Value: "PushToTest"

Step Detail Chart - Transaction 18372 - Step Log-in To Application

Unable To Find Flight Number Exception

java.lang.Throwable atboo.hoo.StackTrace$FirstNested$SecondNested. at boo.hoo.StackTrace$FirstNested.<init>(St at boo.hoo.StackTrace.<init>(StackTrace.j

Request

Error

Row1_ColumnB: 38178Row2_ColumnB: 98123Row3_ColumnB: 18177

Response

Screenshot

Next let’s talk about



Resource Repository Design

‣Resource Plan

‣User Interface Design

‣Required Changes to TestMaker 6.0

39



Repository Client Access‣Designer

‣Editor

‣Controller

‣EOPs

40



Resource Repository - Open

41

Local Repository


Open Test Resource

Repositories

OpenCancelShow Versions

/

Project X


TestScenarios

Project Y

Project Z











Resource Repository - Save

42

Local Repository


Save Test Resource

Repositories


SaveCancelShow Versions

/

Project X


TestScenarios

Project Y

Project Z











Resource Repository

43

Local Repository


Preferences

Repositories

Add

Repositories TestCard Server Notifications

jdbc:mysql//localhost:1527/testmaker

Local Repository

Type MySQL

Location

testmakerUser

testmakerPassword

Save

Remove



Learn More‣http://www.pushtotest.com

‣Post your feedback, comments, ideas, designs to PushToTest Forums

‣Attend the Open Source Test Workshop

‣http://workshop.pushtotest.com

‣Send Feedback to [email protected]

44

Technology

PushToTest TestMaker 6.5 Open Source Test Design Document