MTLM Visual Studio 2010 ALM - day2

DAY 2 MTLM

Start Topics

9:00 Recap 9:15 Create Basic CodedUI, from Test Case and Manual 9:45 UIMap and customize the code for optimization 10:15 Data driven Tests and assertions 10:45 break

11:00 Troubleshoot CodedUI, Common Practices and questions 11:30 Configure the build to execute the CodedUI tests, test settings from VS2010, Execute

from VS2010 with test agents and analyze the results 11:30 Associate test automation with test case, execute automated test case from MTM,

configure the build to run from the build within MTM and analyze the results 12:30 break

13:15 MTLM, Scrum and other project Methodologies 13:45 Lab Management - Virtualized Test Environment and execution Demo 14:15 Test info in the OLAP Cube 14:30 Extensibility - test Scribe, TCMIMport 14:45 Create your own Data Diagnostic Adapter 15:15 MTLM usages with Azure projects 16:00 Closing

Test Activities Planning…

RECAP

Test Case Management RECAP

Test Case Specification

RECAP

Test Case Execution

RECAP


Test Environments

RECAP

"by special request"

F5

1 Open the employee application Set the intellitrace options Set a breakpoint at …

2 When on the breakpoint, save the trace file on your desktop

3 Close Visual Studio and double click the intellitrace file

4 In the intellitrace summary, open the main threat …

5 … and search for The values that where entered during the previous run.

6 And now from Microsoft Test Manager

6 Set the diagnostic adapter

7 Configure the build… it needs a drop location for the symbols (pdb files)

. Make test case with the parameter 1 and 1, crash again… See lab 7 yesterday

8 Run the test case and see it crash… file a bug.

9 See the itrace file in the bug report...

10

Close MTM, open VS (with or without the employee application). Got to the bug and click the itrace file… See the exception and start debugging…

11

Go back in time

11

Go back in time … and find the entered values

Start Topics






Regression Test Candidates

See also on www.tmap.net: Completion Phase

http://www.tmap.net/

http://eng.tmap.net/Home/TMap/The_4_essentials/Structured_Test_Process/Acceptance_and_system_tests/Completion_Phase.jsp




Test Automation

Test automation Where? When? How? Who? Why? What?

Where? When? How? Who? Why? What?



VS2010 Test Tools UI Test Automation Capabilities

Fast Forward CodedUI

Test Infrastructure

1

2

3

No investment in any kind of automation. Just click and test.

Clean up of the Action log maybe re-execute, for more smooth Fast Forward.

Basic CodedUI. Only use the default generation, add your own assertion.

Advanced CodedUI. Customize the CodedUI and UIMap for optimization.

0

Test Automation Investment Levels

1 Clean up of the Action

2

VS2010 ALM MTLM usages patterns for Windows Azure hosted application development

Basic CodedUI

http://www.clemensreijnen.nl/post/2011/03/29/VS2010-ALM-MTLM-usages-patterns-for-Windows-Azure-hosted-application-development.aspx



3 Advanced CodedUI

Fast Forward

Cleaning the action recording When: executing the same test case on multiple environments or when using test data iteration or … and don’t forget the shared steps

CodedUI

CodedUI from existing action recording from Manual Tester



1

3

4

2

CodedUI Create a new action recording by Developer

Lab 12

Create a CodedUI from Test Case Create a CodedUI by using the CodedUI Builder

Timebox: 10 minuts

Start Topics






UIMap

UIMap

Lab 13

Tune the CodedUI with the UIMap…

Timebox: 10 minuts

Start Topics






Data-Driven Coded UI Test

Lab 14

Make CodedUI parameterized from an Excel sheet

Timebox: 10 minuts

Start Topics






BREAK TILL:

11:00

(extra time to finish the labs :)

Start Topics






CodedUI Common Questions

• In what version what functionality?

• How to debug?

• Where can I find test run results for a test run?

• Where do I see these assertions are passed or failed?

• For which properties can I make an assertion?

• …

CodedUI Advanced Questions

• How does CodedUI work?

• Can I use CodedUI for performance testing?

• How to maintain CodedUI tests? • How to execute in the build?

• …

CodedUI Challenges

• The playback failed to find the control with the given search properties.

• Identifying controls with no name search criteria or changing ID’s

• Selecting correct Search Configurations for a control

• How to select the particular value from combo box

• …find controls

Start Topics




configure the build to run from the build within MTM and analyze the results 12:30 Lunch


Execute CodedUI During the build with Visual Studio 2010

For the next 1,5 hour, you must have: - A successful running codedui - Test agent and controller - A build - Testplan and a test case

C

Flavor C: Execution during build with Test controller…

Configurations for automated test execution:

Part of Build Verification Tests on multiple test agents (triggered by: build) Configure Test Controller (don’t register it with a project collection ) http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestControllers Configure Test Agents on clients (interactive mode) http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestAgents Configure *.Testsettings file in solution to use Test Controller and Test Agents Configure the build to run the tests (see B)

Pro: Test run on test environments

Tests run on multiple environments Test Results in Build result

Con: No collection of test results in TFS Harder to configure Need for specific test client environments Test Settings from VS

Preferred configuration above flavor B

http://msdn.microsoft.com/en-us/library/dd286580.aspx










1 Configure Test Controller.

2 Add new Test Settings file Set remote execution and test controller.

3 Create a new build definition Set the Automated Test Properties

4 Queue a new Build Watch the Test Agent and Test Environment

Lab 15

Run the automated test from the build. Look for the test results

Timebox: 10 minuts

Start Topics






Execute CodedUI Within a test case as an associated automation manualy

D

Flavor D: Execution from Microsoft Test Manager…


Part of Regression Tests (triggered by: MTM user, right mouse click on test case, run) Configure Test Controller (register it with a project collection ) Configure Test Agents on clients (interactive mode, can be the same as MTM) Configure Lab Center in MTM to use test controller and create test ‘agent’ environment. http://msdn.microsoft.com/en-us/library/ee390842.aspx http://msdn.microsoft.com/en-us/library/dd293551.aspx Associate CodedUI test with WI Test Case from VS. http://www.richard-banks.org/2010/11/how-to-use-codedui-tests-watin-and-mtm.html


Tests run on multiple environments Test Result in MTM and TFS

Test Settings from MTM

Con: Manually triggered by Tester Hard to configure Need for specific test client (same as MTM?)

Other type of test than BVT



http://msdn.microsoft.com/en-us/library/ee390842.aspx







http://www.richard-banks.org/2010/11/how-to-use-codedui-tests-watin-and-mtm.html


















1 Configure Test Controller.

2 Associate the CodedUI with a Test Case.

2 Create Test Environment

3 Set test Settings

4 Associate the CodedUI with a Test Case.

5 Run Automated Tests from Test Manager

Lab 16

Run the automated test from MTM. Look for the test results

Timebox: 10 minuts

Execute CodedUI Within a test case as an associated automation from the build

E

Flavor E: Execution from MTM during Build…


Part of BVT (triggered by: Build) Configure Test Controller (register it with a project collection ) Configure Test Agents on clients (interactive mode, can be the same as MTM) Configure Lab Center in MTM to use test controller and create test ‘agent’ environment. Associate CodedUI test with WI Test Case from VS. Create Build task to run MTC or MSTEST task for Test Plan http://blogs.microsoft.co.il/blogs/shair/archive/2010/10/30/how-to-run-coded-ui-tests-from-command-line.aspx


Tests run on multiple environments Test Result in MTM and TFS

Triggered by build Test Settings from MTM

Con: Hard to configure Need for specific test client (same as MTM?)

Preferred configuration above flavor C Flavor D and E can be configured together



http://blogs.microsoft.co.il/blogs/shair/archive/2010/10/30/how-to-run-coded-ui-tests-from-command-line.aspx



















Execute automated test cases on different environments

0 Same steps as previous lab.

TFS 2010

Physical Test Client

MTLM Client

Physical Test Client

Win 7 IE8 configuration


1 Configure additional environments in Test Plan properties.



TC

1 Get Test Plan ID Get Test Suite Get Test Configurations with TCM.exe

2 Create command line for Test Run for the different Test Configurations

"run /create /title:""Run on Host for specific Test Point"" /planid:16 /querytext:""SELECT * FROM TestPoint WHERE ConfigurationId=13"" /settingsname:""Automated Test on Windows 7 Environment IE9"" /testenvironment:""Win 7 IE9"" /collection:http://clemenes-pc:8080/tfs/defaultcollection /teamproject:EmployeeApplication"

"run /create /title:""Run on test env for specific Test Point"" /planid:16 /querytext:""SELECT * FROM TestPoint WHERE ConfigurationId=12"" /settingsname:""Automated Test on Windows 7 Environment IE8"" /testenvironment:""Win 7 IE8"" /collection:http://clemenes-pc:8080/tfs/defaultcollection /teamproject:EmployeeApplication"

3 Add build commands

6 Queue a new Build Watch the Test Agent and Test Environment

Start Topics






LUNCH TILL:

13:15


Start Topics






VS2010 Process Support

TMap, ALM and process templates

# Planning the project

# Product Backlog

# Release planning

meeting Estimating and prioritizing the Product Backlog

Evolving User Needs

Owned by Product Owner Prio by Business Value

Test during the Release Planning meeting.

# Planning the project

# Planning the sprint

‘only execute tasks who make an addition to the product are worthy to execute’

# Sprint Backlog

# Sprint planning

meeting

Add PBI to sprint Decomposition Commitment

Tasks Related to PBI’s

Hours remaining

Test during the Sprint Planning meeting

# Planning the sprint

‘only execute tasks who make an addition to the product are worthy to execute’

# Running the

sprint

# Daily scrum

Track

# Sprint Review # Sprint

Retrospective Demo the value

Agile Practices – Improvement Test early and often Build and deploy continuously Acceptance Test Driven Development Emergent Architectures Refactor Test Driven Development Agile Database Development Pair Programming

Undone End to End Testing

http://www.scrum.org/scrumguides/

Start Topics






Build application

Provision ‘clean’ env

Deploy application

Run tests

• Reproduce the exact conditions of a bug or other development issue.

• Build, deploy, and test applications automatically in a clean environment.

• Reduce the time required to create and configure machines for testing an application.

• Run multiple copies of a test or development at the same time.

• Enable members of a team to create and manage virtual environments without requiring system administrator privileges.

VS2010 Lab Management

System Center Virtual Machine Manager (SCVMM)

TFS 2010

Hyper-V Host

VM

VM

Library Shares

share

share

MTLM Client

VM

Lab Agent

TFS 2010

Visual Studio 2010 LAB Management Guide – Infrastructure Visual Studio ALM Rangers 2011-03-23

Prerequisites 1. TFS Application Tier (AT)

• Windows Server 2003 SP2, 2003 R2, 2003 R2, 2008 SP2, 2008 R2 • Windows Vista SP2, 7 • Ports used by services 8443 team web access

8080 server

2. TFS Data Tier (DT) • Windows Server 2003 SP2, 2003 R2, 2003 R2, 2008 SP2, 2008 R2 • Windows Vista SP2, 7 • Ports used by services 1433 SQL

3. TFS Build Controller • Windows Server 2003 SP2, 2003 R2, 2003 R2 SP2, 2008 SP2, 2008 R2 • Windows XP SP3, Vista SP2, 7 • Ports used by services 9191 controller

4. TFS Clients • Windows XP SP3, Vista SP2, 7 • Windows Server 2003 SP2, 2003 R2, 2008 SP2, 2008 R2

5. VM Factory • Windows Server 2003 SP2, 2003 R2, 2008 SP2, 2008 R2 • Windows XP SP2, Vista, 7 • Ports used by services 3389 RDP

137 UDP 139 TCP

• For more information, refer to the quick reference poster: Visual Studio 2010 LAB Management Guide – VM Factory Checklist

6. VMM Server • Windows Server 2008 or 2008 R2. R2 is recommended. • Ports used by services 8100 VMM Agent

135 WMI (49152-65535) 3389 RDP

7. VMM Hyper-V Host • Windows Server 2008, 2008R2. R2 is recommended.

8. Test Controller • Windows Server 2003 SP2, 2003 R2, 2003 R2 SP2, 2008 SP2, 2008 R2 • Windows XP SP3, Vista SP2, 7 • Ports used by services 6901 .NET Remoting

9050 Lab Agent

9. VMM Library • Windows Server 2008 or 2008 R2. R2 is recommended. • Ports used by services 443 BITS

80 WinRm

CORP INFRASTRUCTURE

TEAM FOUNDATION SERVER

VM FACTORIES LAB MANAGEMENT

INTERNET LAB INFRASTRUCTURE

Lab

Man

agem

ent

W

alkt

hro

ugh

VH

D

External Users

Internet NLB

CORP DOMAIN

LAB DOMAIN (could be same as CORP DOMAIN)

TFS Application Tier (AT)

TFS Data Tier (DT)

VS/MTM Clients

TFS Build Servers

Primary Factory

Secondary Factory

Domain Controller Network Storage

Domain Controller

SAN

VMM Server

VMM Library

Test Controller

VMM Hyper-V Host

Build Golden Images

Build VS/MTM Client Machines

Deploy and Host VMs

?

1

2

3

4

5

6

7

8

9

Create LabMan template from golden image

References • Visual Studio 2010 Lab Management Guide

• http://go.microsoft.com/fwlink/?LinkID=206935&clcid=0x409 • Visual Studio ALM Rangers VM Factory Guidance

• http://rangersvsvmfactory.codeplex.com/ • Capacity Planning Quick Reference Poster

Team Project Collection Planning Quick Reference Poster • http://vs2010quickref.codeplex.com/

http://go.microsoft.com/fwlink/?LinkID=206935&clcid=0x409

http://go.microsoft.com/fwlink/?LinkID=206935&clcid=0x409

http://rangersvsvmfactory.codeplex.com/



http://vs2010quickref.codeplex.com/



Testing Test Cases

Test results

Debugging Corrections

Cannot reproduce

Suspicions, add more tests

Visual Studio 2010 Test Architecture

References Visual Studio ALM Rangers, Copyright © 2009 Microsoft Corporation

Visual Studio MSDN Site http://msdn.microsoft.com Visual Studio ALM Rangers http://msdn.microsoft.com/en-us/teamsystem/ee358786.aspx

System

Validation

Integration

Unit

The final integration

testing the solution.

In the world of NASA

this could be the

final countdown,

doing a complete

system and

environment test

before the launch.

Design validation

focused testing,

which validates

solution deliverables

with requirements. It

is especially

important when

testing interfaces

which are shared

and are the basis of

some or all of the

building blocks.

Also known as user

acceptance testing,

the validation testing

validates the solution

against test

requirements, which

are based on business

requirements.

Unit of work focused

testing, at the core of the

solution, typically done by

the developer and

automated testing. Unit

tests focus on topics such

as initialization, logic and

error handling at a code level.

• Generalist Tester

• Minimal Footprint

Included in Ultimate Manage virtual environments for

use in application development,

deployment, and testing.

Visual Studio Products

Building upon each

other and

supporting the

testing strategy.

Co

mm

on

Te

st P

has

es

No more “No Repro”!

Ultimate Premium Test

Professional Lab

Management Test Load Agent

http://msdn.microsoft.com/

http://msdn.microsoft.com/en-us/teamsystem/ee358786.aspx



Start Topics





13:15 MTLM, Scrum and other project Methodologies 13:45 Lab Management - Virtualized Test Environment and execution Demo 14:15 Test info in the OLAP Cube Break 14:30 Test Scribe, TCMImport 14:45 Create your own Data Diagnostic Adapter 15:15 MTLM usages with Azure projects 16:00 Closing

BREAK TILL:

14:30


Lab 17

Install Test Scribe and create a report

Timebox: 10 minuts

Start Topics






Start Topics






A u t o m a t e d A L M f o r A z u r e C l o u d D e v e l o p m e n t ALM 4 Azure

127

Start Topics





13:15 MTLM, Scrum and other project Methodologies 13:45 Lab Management - Virtualized Test Environment and execution Demo 14:15 Test info in the OLAP Cube Extensibility - test Scribe, TCMIMport 14:30 break 14:45 Create your own Data Diagnostic Adapter 15:15 MTLM usages with Azure projects 16:00 Closing

Test Activities Planning…

RECAP


Test Case Specification

RECAP

Test Case Execution

RECAP


Test Environments

RECAP

Testing Practices with ALM tools

Know the value of MTM and know how to use its capabilities

GOAL:

staat voor resultaat

Technology

MTLM Visual Studio 2010 ALM - day2