20110812 CyberTAN presentation

QualiSystems Proprietary & Confidential1

TestShell FrameworkIntroduction

Wednesday, April 12, 2023

Standards of E

xcellence

Richard Hsu

9001:2000 90003:2004

Global Software CompanySanta Clara, USALondon, UKMunich, GermanyTel-Aviv, Israel APAC

• Singapore , APAC HQ • Shanghai, China • Taipei, Taiwan• Seoul, Korea

• APAC Channels, Australia, Japan , India

Founded 2004

Flag product – TestShellTM Test-Automation Framework

“Customer First” Culture

QualiSystems – Company Overview


What we do - TestShell Framework


End-to-End framework for Test Automation that supports the entire life cycle of a test for components, devices, or systems

TestShell automates and simplifies the process of testing, allows you to focus on pure testing, and thus substantially:

Improve Product Quality

Shorten Time to Market

Reduce Expenditures

Lengthy test processes that can take up to several daysThe wide array of benchmark applications/Scripts include Hardware and SoftwareThe variety of vendors and models requiring diverse specificationsThe need to control and manage many distributed test stations Time-consuming test result collation and analysisCompound aggregation of extensive amounts of test data

Testing Challenges without Automation


Time Lost

Quality Impacted

Expense Overhead

Resources Unexploited

Significantly reduce Time-To-Market Full regression cycle 64 hours → 24 hours at USI (環隆電氣 )Single test cycle 15 days → 1.5 day at Cameo (友勁科技 )

Significantly reduce Operational ExpendituresResources involved in test creation down by 50% at Deutsche TelekomSingle test cycle effort 2 Days → 5 hours at Telstra

Managing and Improving Quality ControlDaily Build and regression at Ceragon NetworksCustomer service centers at CrowSaving X10 on RMA at Syneron

Proven ROI within the first year of use



Our Customers – across the value chain

http://www.kepri.re.kr/index.php

http://ceoworld.biz/ceo/wp-content/uploads/2009/05/telstra_logo.jpg

http://www.mrv.com/

End-to-End Test Automation Framework

7 QualiSystems Proprietary & Confidential

Write Tests & Regressions

Create reusable functions

Control Test Assets

Coordinate Test Execution

Launch Test Stations

Run Tests & Regressions

Collect Logs & Results

Monitor Real-time

Aggregate Results

Customized Reports

Customized Dashboards

Create

ExecuteSchedule

Analyze

CentralManagement

Tests & Regressions

Assets

Equipment

Test Results

TestShell Platform Details




Create

ExecuteSchedule

Analyze

CentralManagement

• Messages, Instructions• Alerts, Remarks• Watchers, Graphs• etc.

Dialogs

• Variables• Cases, Loops, Delays• Parallelization, events• etc.

Logic Modules

• Traffic generators• Network analyzers• Spectrum, scope • Generators, etc.

T&M Modules

• Symbian• Windows Mobile• iPhone, Blackberry• Android

Mobile Modules

• Mail & SMS, File Engine• GUI Automation, Attributes• Requirements Management,

QTP ,LR , Etc..

Utilities

• Telnet, SSH, TCP, UDP• SNMP (& Traps), FTP• Traffic Generators, Switches• etc.

Networking

• Web GUI• Web Services• etc.

Web Modules

• Exe, DLLs, CLI, VIs• Script-Engines• TCL, Perl, Python, VB, Java• MatLab, OCR

Scripts & Software

TestShell SystemInterfaces to Software and hardware compnents (Any many more..)


Testing in the All IP World - Example Scenario

Typical test flow for validating :

Allocate all testing resources

Configure the Device Under Test

Configure Traffic Generator(s)

Applying traffic to the DUT and

sniffing /getting statistics

Analyzing

Typical tests:Traffic blasting, protocol emulation, load & stress,

conformance , acceptance, interoperability, etc

Typical validations:

End-to-End: Path Recovery, Clock Recovery,

Delay, Latency, Throughput, Loss Ratio,

Information-leakage, routing/switching performance, etc

Start

Setup Topology

Run Traffic

Analyze Results

End

Configure Traffic Generator (ex. IXIA

, SPIRENT)

Configure DUT



Multiple Network Elements from different vendors such as:

Ethernet SwitchOSSWireless devices (Access Points)WiMAX/4G (STB)Multi Service RouterHigh Speed RouterBBRARDSLAMGatewayModem

Start

Setup Topology

Run Traffic

Analyze Results

Store & Report Data

End

Configure Traffic

Configure DUT



Various network protocols and interfaces

Start

Setup Topology

Run Traffic

Analyze Results

Store & Report Data

End

Configure Traffic

Configure DUT

TelnetSSH

SFTP

TCP

Serial

UDP

SNMP

SNMP Traps

GPIB

HTTP

VISA

SOAP

SMTP

FTP

TFTP



Multiple test equipment typesfrom different vendors

Traffic GeneratorsNetwork EmulatorsNetwork SniffersNetwork Analyzers Baseband EmulatorsLogic analyzersSignal analyzersOscilloscopesSignal generators

Start

Setup Topology

Run Traffic

Analyze Results

Store & Report Data

End

Configure Traffic

Configure DUT



DEMO

TestShell Common Scenario for Network


DEMO 1: Router Testing via Traffic Generator- Configure DUT by Telnet

Check aliveCommand line

Configure Router/Switch

TelnetRun TrafficDriver

Analyze ResultsDriver

Traffic Sender

Traffic Receiver

DataData

Rou

ter

DEMO 1: Router Testing via Traffic Generator (cont.)Code Free Test Creation as easy as Visio

Studio FlowVisio Flow


DEMO 2: AP Router Testing via Traffic Generator- Configure DUT by Website(GUI)

Check aliveCommand line

Configure RouterWebsite(GUI)

Run TrafficDriver

Analyze ResultsDriver

Traffic Sender

Traffic Receiver

DataData

Special features for Wireless Device:1. Use different configuration files to complete

diverse testing, ex. different mode(802.11n,b,g), different channel(1~11) or different encryption type(WEP, TKIP).

2. Use excel file to maintain the path of configuration files and import data of excel file by “File Engine” on Testshell.

3. TestShell’s GUI Automator can provide “Inspect” and “Valuate” feature to check states of web screen.

4. TestShell can get snapshoot of testing result and export to report automatically.

DEMO 2: AP Router Testing via Traffic Generator (cont.)

- Configure DUT by Website(GUI)


Drag & Drop Test Creator – non programmers can join the automationReusable Building BlocksRecord and Replay ActionsIntegration with both Traffic Generators and T&MSupported interfaces:Network Protocols (Telnet, SSH, Serial, TCP, UDP, SNMP, etc)GUI – Web, Java, Win appLogic and DialogIntegration with external files (configuration and results)Resource Asset Management & DeploymentDocumentationScriptsExecutablesDlls

Tests Creation - Benefits



Create

ExecuteSchedule

Analyze

CentralManagement


Scheduler Snapshot

Tests scheduling – overnight & weekend

Resource management & Equipment utilization

Automatic scheduling according to time, station and priority

Test scheduling across single or multiple stations

Advanced "Outlook-like" interface

Virtual station locking

Fully customizable execution rules

Automatic real-time execution conflict handling

Tests Scheduling – Scheduler



Create

ExecuteSchedule

Analyze

CentralManagement

Execution planningSelect Test or RegressionSelect Unit Under Test and TopologyRecord Admin informationSerial, Version, Batch etc… (Dynamic)

RunComplete RunStep by StepAutomatic troubleshooting and action executionReal-time data tracking

Standardize data collectionData transfer guarantee

Tests Execution – Runner



Create

ExecuteSchedule

Analyze

CentralManagement

Melbourne

3,3 V

12 mA

51 kb/s81,9101,3

MHz3E-07128 ms

21°C69%11,9 mA

Test Analysis - InSightCentral data collection

5,3Canbara3,2 V

57 kb/sSydney

21,6°C19 W102,1

MHz

Multi-Site Repository

Test Lab 1 Test Lab nTest Lab 2

Data from all test labs and stations is collected and stored in a central server

Data is then ready for comprehensive analysis, using TestShell InSight

www

Test Analysis - InSightOnline customizable display

The analayzed data is presented via online Dasboards:• Providing real time analysis• Reviewed any time anywhere

www

The dashboards are completely customizable according to user preferences, supporting multiple analysis & display options, including OLAP



Create

ExecuteSchedule

Analyze

CentralManagement

Test Management modules - Snapshot

QualiSystems Proprietary & Confidential31 QualiSystems Proprietary & Confidential

Central repository across the organization

Tests and Regression management

DUT management

Fully scalable and open frameworkTests version control Permission and user control

TestShell Components


TestShell Applications


TestShell Studio

TestShell Runner

TestShell InSight

TestShell Server

Create

Execute

Analyze

ManageTestShell SchedulerTestShell Controller


Thank you

To Schedule a Live Demonstration go to: www.qualisystems.com/demo

Technology

20110812 CyberTAN presentation