IMPROVING TEST EFFECTIVNESS Some techniques and their application with Visual Studio 2010 N. Sudhakar Senior Development Engineer Microsoft India R&D [email protected]

IMPROVING TEST EFFECTIVNESS

Some techniques and their application with Visual Studio 2010

N. SudhakarSenior Development Engineer

Microsoft India R&[email protected]

Motivations• Quality has been a longstanding challenge in software• Poor quality can lead to

– Project failure– Significant damage in production– Loss of credibility

• Lot of times, individual engineers get blamed for quality problems

– E.g. “How could you make such a mistake?”, “Why wasn’t this bug found earlier?”, “Why should testing this take so much time?”

• Understand realities– This is a short-sighted approach. Issue is more to do with

engineering than with engineers.– Software is complex. Tackle complexity smartly.– New processes and practices evolving. Understand and leverage.

• Way forward: sound engineering practices and tools to support them.

Aspects of Quality

Culture of testing- Testing integral part of

all efforts- Validate Modules with

unit tests- Scalability with load

tests- Use-cases with end-to-

end tests

Processes and practices-Automated processes make testing repeatable- Makes for efficiency. Automation is in some sense ‘codified knowledge’.- e.g. Continuous Integration

Tools- Supports the processes – makes them automatic. - Sustains the culture – testing doesn’t a become a

burden.- Frees up time for engineers to look deeper,

harder

Quality

Continuous Integration

• One-time integrations are problematic• Software, lot of times, get built as separate

components and modules, and assembled together late in the cycle

• Many problems:– Components built in isolation don’t integrate well.– User scenario cannot be tested until integration– Cannot show product to customers to get feedback– Cannot show meaningful progress to sponsors– To give an analogy, it is like painting a picture by

piecing together a jigsaw puzzle!


• Much better way is to keep product always integrated.• All major parts of the application are always there, right

from beginning. No special integrations.– Build all your code, integrate all your components, install

the whole product, everyday– What gets added incrementally are the scenarios, use-cases– Lot of benefits: user scenarios be tested sooner, can be

validated with customer, sponsor can get a sense of progress, etc

• To ensure that this healthy baseline is always maintained, you need a daily process. This process ensures that all components are building, integrate well and work together as a product– Catch Build breaks almost immediately– Use-case regressions caught on same day

• Repetitive - automated process is very important here


• Continuous Integration with VS 2010

Build application

Provision ‘clean’

environment

Deploy application

Run tests


• DEMO

Test environment provisioning and configuration

• A ‘test bed’ or a ‘test environment’ is needed to test your application– If you are testing a 3 tier web-app, you would have an environment with 3 machines– Different environment for different efforts: CI, Reliability, Manual, Pre-production

• Significant effort to create the environment– Each machine of the environment needs to have OS, Platform (IIS/SQL), Application code, Test tools

• Manual environment preparation can take away significant portion of (scarce) testing time.


• And this is needed everyday – Environment becomes ‘stale’ after installing and testing one build– Risky to re-use same environment for next build– All in all, all your time goes in installing OS & platforms. Very little

left for testing• Automation is the way forward, but has challenges• Organizations may have automated OS provisioning systems to

build physical machines for office use. Testers sometimes use this to build test environments

• This is problematic– Large OS image transfers, several 10s of GBs in size– Long time for install– Corporate network is clogged– Testers don’t have control of images to create composite images (OS

+ SQL, etc.)

• VS 2010 Lab Management to the rescue• Use virtual machines instead of physical

– Self-serviced: No need to ask for new hardware for new environments

• ‘Library’ of Generalized VM Templates used to build the environment– Admin just puts basic set of OS Templates. E.g. Win7, Win2k8, etc.– Testers can enrich Library with new Templates as part of their

work. E.g. Win2k8 + SQL, Win2k8 + IIS– A collaborative, self-serviced effort



• One time creation of Environment. – Just create it once for lifetime of the project. No need to do

it daily.

• Get to clean state with Environment Snapshots– VM Snapshots are one of the best things offered by

Virtualization– Snapshot is a point-in-time copy of a VM managed by

Hypervisor– Can go back to a point in time in matter of minutes (as

opposed to hours needed to rebuild a fresh OS)– Lab Management provides you Environment snapshots

using VM snapshots underneath

• As a final step, to deploy applications, Continuous Integration workflow can be reused– Just fire the CI workflow and go to lunch. By the time you

come back, everything is ready for testing!


• DEMO

Testing conflict-prone applications with Network

Isolation• Problem: Some intranet applications can prove conflict-

prone during testing. – Hard to test on corporate network– Multiple testing efforts going on at same time can collide

• Typical in Intranet applications that rely on Active Directory (AD)– Access decisions based on user designations, roles,

management hierarchy in AD– Security through domain based authentication,

authorization, group policy enforcement, etc.


Isolation• Example: Content management application that stores

confidential files. Access is based on designation. Say, ‘top secret’ patents accessible only to CEO.

• Can you add a CEO object in your corporate AD for testing? No way! – Testing with the corporate AD is ruled out– You need to setup your own AD. – AD relies on other infrastructure systems like DNS. This will

force you to setup your own network.

• Lets say you did create a network with its own AD, and not rely on corporate network. Can all your test efforts use that network? Unfortunately, No.


Isolation• Can two tests both try to create CEO objects in the

directory at same time? ‒ A company can have only one CEO! So, no.‒ Two parallel test efforts cannot use the same AD

• In summary, for every test effort (CI, Reliability, Pre-production, etc. ), you will need a separate networks with its own AD

• Not good‒ Lot of effort to setup, and to manage‒ Test lab gets fragmented with several networks‒ Hard to move data into and out of such networks

• Avoiding it is also not a good option either


Isolation• Network Isolation to the rescue

– Network Isolation specifically designed to test these type of applications

• Each environment has its own ‘internal’ network with its own Active Directory. – Application doesn’t rely on corporate network or directory– Application works purely based on the internal network IP

addresses and names


Isolation• Every test machine also connected to the external

network. No connectivity problems– Every test machine gets an external IP, and a unique

external name– You can RDP into your test machine from anywhere– You can access build shares and other corporate resources

from your test machine

• Clonable! Easy to create new environments. Each tester can clone and have his own self-contained environment. – Easy to kickstart multiple test efforts


Isolation

Network Isolation

• VIDEO

ON CLOSING

• Summary– Use Continuous Integration to keep your product

healthy and working, everyday– Leverage Lab Management features to get clean

environments with latest builds for testing– Use Network Isolation to isolate and test conflict-

prone applications

• Q&A

© 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.

The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and

Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Documents

IMPROVING TEST EFFECTIVNESS Some techniques and their application with Visual Studio 2010 N. Sudhakar Senior Development Engineer Microsoft India R&D [email protected]