Tariq M. King

NORTH DAKOTA STATE UNIVERSITY DEPARTMENT OF COMPUTER SCIENCE

© NDSU SOFTWARE TESTING RESEARCH GROUP

Tariq M. King

STITC 2011Co-located with

Towards a Self-Testing Cloud Computing Infrastructure

Markham, Ontario, Canada

© NDSU SOFTWARE TESTING RESEARCH GROUP 2

Outline Introduction Research Questions Cloud Testability Test Support as-a-Service Vision: Self-Testing in the Cloud Summary Open Discussion


Cloud provides ubiquitous, on-demand access to computing resources “as services” via the Internet

Operational Perspective Availability, Scalability, Efficiency, …

Development Perspective Reusable Components, Info. Hiding Highly Complex Local Infrastructure

Testing Perspective?

Introduction

3

CLOUD


Cloud brings new testing challenges and benefits Software Testing In the Cloud (STITC)

STITC 2010 Workshop focused on three areas: Testing IN the Cloud – leveraging the cloud for

test execution and testing in a virtualized environment

Testing OF the Cloud – validating applications that are hosted and deployed cloud

Testing TO the Cloud – moving the testing process and other assets to the cloud

Introduction

4

STITC


Services in cloud computing typically fall intothree categories:

Software-as-a-Service | Google Docs, Office Web Apps

Platform-as-a-Service | Google App Engine, Windows Azure

Infrastructure-as-a-Service | Amazon EC2, Rackspace

Introduction CLOUD SERVICES


Many cloud platforms offer Virtualization support Creating abstract computing resources from

more powerful physical resources

Hosted Architecture

Introduction VIRTUALIZATION

Hypervisor Architecture


Introduction Cloud Computing shares characteristics with

Autonomic Computing (AC), Grid, Utility, …

Central theme of AC is self-management to reduce burden of human administration and integration

King et. al 2008 introduce animplicit self-test characteristic

Regardless of self-* featuresimplemented, system shouldvalidate adaptations at runtime

AUTONOMIC COMPUTING

Self-Configure

Self-Optimize

Self-Heal

Self-Protect

Self-Test


“Migrating Autonomic Self-Testing to the Cloud” Tariq M. King and Annaji S. Ganti | STITC 2010 High-level position on how a self-testing

approach for autonomic software can be applied to the cloud

In today’s talk, we highlight our recent progress and developments from the above perspective Extended version of King et al. | CASCON 2011 Summarizes contents of book chapter submitted

to STITC: Perspectives on an Emerging Discipline

Introduction

8

ABOUT THIS TALK


How does cloud computing affect our ability to test applications that are either built or deployed in a cloud environment, or depend on cloud services?

Research Question #1

Testability

Investigates impact of cloud development

model on software testability

(Positive, Negative)


How can the test artifacts and tools used to validate a cloud application service during development, be reused to help test derivative applications?


Test Reuse

Investigates ways to reuse outputs of pre-deployment

testing effort

TEST (Artifacts, Tooling)


How can built-in cloud computing technologies such as virtualization be used to improve the process of testing cloud applications?


Cloud Leverage

Investigates ways in whichthe cloud infrastructure

itself can be harnessed for Testing OF the Cloud

(Power, Virtualization)


Cloud Testability: The Negative

Complexity

Remoteness

InformationHiding

Autonomy Adaptiveness

Statefulness

High Performance

Criteria

High Dependabili

tyCriteria

(Paradigm Infancy)


Computational Power

Storage

Testing requires lots of resources and the cloud is certainly powerful enough to handle it

Virtualization

Cloud Testability: The Positive


As a step towards Research Questions 2 and 3, we investigate a novel approach to software testing of the cloud, in the cloud | King et al., CASCON ‘11

Scenario: One host (Provider B) is developing an application service that will extend functionality of a service offered by another host (Provider A)

Testing of the Cloud, In the Cloud


Even with its limited scope, our scenario reveals many cloud integration testing challenges: Difficult for Provider B to Setup Some Tests

for his application since he does not have full control and observation of Service A.

Developing Accurate Stubs is Hard since Provider B has no knowledge of Service A’s implementation

Provider B Limited to Black Box Testing of those parts of his application implemented by Service A

Service B can Change w/o Provider B’s Knowledge i.e., Update to Service A Need to Retest Service B

Integration Testing Challenges


Our proposed solution introduces the notion of Test Support as-a-Service (TSaaS) in the cloud

Prior to the deployment of Service A, Provider A would have tested it, and created: Test Artifacts – Test Drivers, Test Stubs, Test

Data Local Test Bed – Hardware/Software

Environment

TSaaS seeks to reuse existing test artifacts, tools and configurations of the local test bed, to aid the rapid development and delivery of test support services for collaborating partners

Proposed Solution


Provider B has full access to a test copy of Service A Test hooks, instrumentation, and scaffolding

reuse Test data (non-confidential) can be made

available Testing does not interrupt service in

production

Concerns: Maintenance, Scalability, Security

Test Support as-a-Service (TSaaS)

Provider A exposes TSaaS to Provider B


Hypervisor-Based Virtualization and Secure Access

TSaaS Architecture


TSaaS Control Flow

Control Flow When Realizing Test Support Services


Using Existing Test Automation to Guide Development

TSaaS Development Method


Designed with goal of making its components easily deployable to Windows Azure®

Web & Worker Roles (ASP.NET, IIS7, WCF Services)

VM Role (Windows 2008 R2 Server Image)

Application Domain: Credit Reporting/Banking

TSaaS Prototype DETAILED DESIGN


We were able to develop and expose a cross section of test support operations via the TSaaS API Test Specification – user defines fixture

operations

Test Execution – runs specified fixture operations

Code Coverage – analyzes branch and lines covered

Server-Side Diagnostics – facilitates configuring web server, including performing a reset after failures

Runtime Virtualization – create, start, stop, delete, take snapshot, and update VHD configuration

TSaaS PrototypePROGRAMMING INTERFACE


TSaaS PrototypeGRAPHICAL USER INTERFACE


Vision: Self-Testing In The Cloud

Service A (Production Env.)

Monitor

Analyze Plan

Execute

Knowledge

Autonomic Test Manager

TSaaS

SensorProvider A

Provider B

Test Harness

Service B(Test Environment)

Provider A builds a

sensor for Service A

Provider B monitors Service A

for changes

If a change occurs, Provider B generates a plan for regression testing of Service BProvider B reruns

tests on latest build of Service B, which is set to use new

version of Service A via TSaaS


Test Support as-a-Service Improved Testability and Confidence

(Partner) Control/Observation, White Box Testing,

Metrics

Improved Product (Provider) Partners test your component, bug

reporting/fixes

Virtualization Benefits Snapshots: Speed Up, Bug

Replication/Debugging Scalability, Ease of Maintenance, Hardware

Savings, and Environment Testing (OS, Applications)

Foreseen Benefits


Self-Testing in the Cloud via TSaaS Automatic Regression Testing of changes

due to: Software maintenance and evolution Dynamic adaptation and updating

Improved Autonomic Decision-Making Use of validation to determine which service

may be the best candidate to select during adaptation

As a proactive form of self-healing (fault detection through formalized testing techniques)

Foreseen Benefits (cont.)


Standardization Providing standardized interfaces and

guidelines for performing testing in the cloud, and developing test support infrastructures

Evaluation Providing convincing evidence that these

approaches are not only feasible to implement but can have real value to practitioners

Validation Showing virtualization is consistent with

production

Key Challenges


Described our progress towards our grand vision of migrating autonomic self-testing in the cloud Proposed an approach to Test Support as-a-

Service,which uses the cloud to tackle testing of the cloud

Described the design and implementation of a a prototype of TSaaS

Discussed benefits and key challenges associated with TSaaS and use for self-testing OF/IN the cloud

In terms of our progress, we have barely scratched the surface of the research problem under investigation

Summary


Acknowledgements Annaji Ganti | PhD Student,

NDSU/Microsoft

Neha Kale | MS Student, NDSU

David Froslie | Test Architect, Microsoft

Alisson Sol | Knowledge Engineer, Microsoft

Dr. Scott Tilley | Professor, Florida Tech

Participants and Reviewers STITC 2010 CASCON 2011


Thank You! Questions?

¿Preguntas?問題Sawwalвопросы質問domandeερωτήσεις

Documents

Tariq M. King