Upload
softwarecentral
View
2.414
Download
2
Tags:
Embed Size (px)
Citation preview
Testing Best Practices with VSTS
Shahidul Islam, MPH
QA Manager
Emerging Health Information Technology12/4/08
Agenda
Visual Studio Test Edition for Software Testers (VSTEST) Manual Tests Web Tests Load Tests TFS Work Item Tracking TFS Reporting
Sources for Test Cases by Test Type
Business ReqsUser Reqs
System ReqsTechnical Architecture
Hardware SpecsSoftware Specs
Detailed DesignDev Standards
Produce Code
Unit Testing
System Testing
Integration Testing
Acceptance Testing
Validates
Validates
ValidatesBusiness
Focus
Technical Focus
Regression Testing
Reference: Learning Tree International Course 316 “Software Testing and Inspection Methods
Validates
VSTSWebTests
Test PlanDocuments project-specific strategy for all test types.
Master Test Plan
Acceptance Test Plan
System Test Plan
Integration Test Plan
Unit Test Plan
Manual test case scenarios
Automated test case scenarios
Load test scenarios
GENERATES
Defects log
Test logs
VSTEST TFS
VSTEST
Visual Studio Test Edition for Software Testers
Author Tests
Run Tests
Publish and Share the results
Reporting with TFS
VSTEST Basics
Test Projects are Visual Studio Projects Compile/deploy like any assembly in .NET Visual Studio options under “Test Tools” The solution should be stored within source control
(like any other piece of code)
Tool set integrated in VS allows for easy adoption from developers.
Testers are more integrated with the development process
Requirements Management
Manual Tests
VSTS Manual Tests—Excel
File>Properties
Custom TestType: “Manual Test” TestID: “Get a GUID from VSTS>Tools>Create a GUID” Save As .mht file
Web Testing
Uses a recorder for test creation: Records web requests using Internet Explorer Analyzes the HTML traffic– each response can be validated
for expected behavior Ordered series of HTTP requests against a target web
applications Integrates with Visual Studio Available only via VSTS Team Edition for Software Testers Runs with Test Manager Integrated with Team Build
Web Testing
Web Tests are of two types: Recorded Tests
Launch the browser Record session steps Playback recorded session Convertible to (Coded Tests) C# or VB code to handle complex
scenarios and flow control Coded Tests
Scripted Better, adapted from recorded tests Playback
Supports: ASP.NET, HTTP based, Web Page, Web Services Compatible with HTTPS
Demo then proceed to slide 43
Web Test Demo
Recording a Web Test
Record a browser session using the Web Test Recorder
Analyze the http requests being recorded
Can pause in between requests
Recording can be a good baseline for coded Web tests
Recording a Web Test
Steps: Test..New Test Choose Web Test Choose Default Visual C#
test project Create
A new Internet Explorer instance opens up
Recording a Web Test
Two windows open: A new Internet Explorer
instance Web Test Recorder on the
left pane of the IE instance Each request or page navigated
is displayed Options available to stop or
pause recording Can add comments during
recording
Recording a Web Test
Once stopped, recorded test can be run from Test Window
Editing a Web Test
Web Test Editor available once recording is stopped
Features: Toolbar for basic commands Expandable Request Tree
view of requests in the execution order
Can expand to see Query String parameters or Form Post parameters
Properties Window Properties and Settings of
the Web Test
Editing a Web Test
Adding a New Request: Right click on request to
add new request
Clean up unnecessary recorded steps
Remove extraneous or wrong user actions
Modify the order of requests Be conscious of
dependent requests
Editing A Web Test: Properties
Properties: Description: Text for describing the test Pre-authenticate:
True: Every request includes authentication header (default)
False: Authentication header sent when requested
User Name and Password: Values set using the Set Credentials button on Web Test Editor for sites using
Basic authentication or Integrated Windows authentication Credentials can be edited directly on
properties window or Can bind to data source for multiple runs
with different credentials – (recommended)
Editing A Web Test: Requests
Requests: HTTP request steps which are executed in given order and whose response is processed
Request Tree in Web Test Editor consolidates child requests to be executed in order for main request
Properties of a Request: Cache Control Encoding Follow Redirects Method Parse Dependent Request Record Results Response Time Goal Think Time Timeout URL Version
Editing A Web Test: Requests
Cache Control: True/False directive to simulate browser caching
Encoding: Text format for requests - usually UTF-8 for HTTP traffic
Follow Redirects: True/False to honor status codes 300-307 (redirect) to final page or stop and treat the redirect as final page
Method: GET or POST
Parse Dependent Request: True/False to process dependent requests such as loading image files. True will get standard behavior (load images)
Editing A Web Test: Requests
Record Results: Used only with Load Testing to determine if timing and performance data need to be included
Response Time Goal: Used with Load Test reporting to record how long the request took to process
Think Time: Delay (in seconds) to pause on page before processing next request
Timeout: Maximum time (in seconds) to wait for response before timing out and logging a failure
URL: URL of the request
Version: Versions of HTTP to use – default 1.1
Editing a Web Test: Request Sub Items Request Sub-items: Simple requests
can be enhanced with sub-items like validation rules etc.
Dependent Requests: A normal HTTP request which
has a parent request Example: Load image on a page Handled by Parse Dependent
Requests property Can have own dependent
requests Can be added explicitly –
example, extraction or validate rule
Editing a Web Test: Request Sub Items
Query String Parameters: Name=Value pairs set in the Property window Show Separate Request Results:
Used to group results in Load Testing Result data is grouped if False (default); if True,
displays separate grouping for data (response time etc)
URL Encode: Determines if Name/Value will be URL encoded
Form Post Parameters: If the request method is POST, enables two form POST parameters:
Common Form Post Parameter File Upload Parameter
Headers: Name=Value pair to be included in HTTP header. Ex: Referer=www.microsoft.com
Validation Rules: Add validation rules to verify if response from the request is expected one
Extraction Rules: Capture data from response
Editing a Web Test : Transactions
Allow grouping of requests as a unit to
Document Move/Copy Capture timing information
To insert: Right click at the request
where the transaction is to be inserted
Complete the Add Transaction dialog
Requests can be moved between transactions using drag and drop.
Editing a Web Test: Comments, Context Parameters
Comments: Tests can be documented with comments
Context Parameters: Replace static values with
dynamic values for property, value or parameter
Variables expand to real values at runtime to make the test flexible and dynamic
Usually done through Parameterizing Web Servers
Editing a Web Test: Parameterizing Web Servers
Select web test, and right-click
Select Parameterize Web Servers… Allows changing URLs from static
to context-parameterized variables to re-target the test
Parameterize Web Servers dialog lists web servers used with a default Context Parameter Name
Can be changed using the Change button to change
Web Server Local ASP.NET Development
Web Server
Editing a Web Test: Parameterizing Web Servers
After changing, the Context Parameters node appears in the web test
Values can be edited via right click on Properties of newly added
parametername=value
Context Parameter can then be used in requests using
{{..}} notation
Example: {{WebServer6}}/live/search/search.xml
Editing Web Tests: Extraction Rules Data from POST/GET requests can be extracted for
Subsequent requests Providing debugging information
Extraction rules verify that a Web application is working correctly by
extracting data from the responses to Web requests
storing results in the test context as name value pairs.
Extraction rules can extract form fields Text Attributes Headers regular expressions hidden fields.
Editing Web Tests: Extraction Rules
To Extract, right-click on a request and choose Add Extraction Rule
Extraction rules define what data is to be extracted from an HTTP response and store the data in context parameters
Dynamic data generated via requests can be passed down to later requests.
Editing Web Tests: Extraction Rules
Editing Web Tests: Extraction Rules
To add Extraction Rule: Right-click on request Choose Add Extraction
Rule Modify/add parameters Set Context Parameter to
a unique name
Annoying Error Messages!!
RequestFailed: Context parameter '$HIDDEN1.__VIEWSTATE' not found in test context”.
Add Extract Hidden Fields
Annoying Error Messages!!
Web Test ran flawlessly right after recording but Next Morning?
500 Server Error
Search for parameters you think may be dynamic (Session Variable)
Add Extract Text Rule
Example: Context Parameter Name = ControlID, Ends With = &, HTMLDecode = True, Ignore Case = False, Index = 0, Required = True, Starts With = ControlID =, Use Regular Expression = false.
Editing Web Tests: Validation Rules
Validation rules validate existence of
Text tags, or attributes
on the page returned by a request.
Validation rules also verify the amount of time it takes a request to finish, and the existence of form fields and their values.
Editing Web Tests: Validation Rules
Editing Web Tests: Validation Rules To add a validation rule:
Select a request Right-click Add Validation rules in
dialog Validation Level:
Used only during a load test
Higher validation levels effect speed, hence slower tests
Does not mean priority Determines when the
validation rule is used in a load test.
Setting to High means that the rule is executed only when the load test validation level is set to high.
Editing Web Tests: Validation Rules Validation Levels:
Used only during a load test Higher validation levels
means slower tests Does not mean priority Determines when the
validation rule is used in a load test.
Setting to High means that the rule is executed only when the load test validation level is set to high.
Running Web Tests
Settings can be configured from .testrunconfig file
Choose Web Test from dialog: Number of iterations Browser Type Network Type Simulate Think Times
These settings are overridden when run under a Load Test
Running Web Tests:
From the tool bar: Run Test
From the command line Invoke MSTEST
Interpreting Web Test Results Results Viewer:
Results Panel: Request: The full URL requested HTTP Status: HTTP status returned from the server – typically
200 OK Response Time: Time web server took to respond Size: Size of response
Interpreting Web Test Results Details Panel:
Web Browser Tab: Response rendered as browser page
Request: Contents of the selected request
Response: Contents of the response to selected request
Context: Current context parameters, useful for debugging extraction rules
Details: Results of extraction and validation rules for selected request.
Coded Web Tests
Written in C# or Visual Basic
Incorporates flow of control as opposed to sequential recorded tests
Use Recorded tests for baseline and then convert:
Select web test Right-click Generate Code
Why Coded Web Tests?
Can read expected results from a data source
Flexibility of conditionals and looping
Copy set of request from one test to another
Factor out sections of requests and other operation into methods that any test can call
AJAX in Web Tests
VSTS 2008 now records AJAX requests
Freeware Fiddler need not be used, although it’s still useful for diagnosing
See: http://www.fiddlertool.com/fiddler/
Test Case Management
Test List Editor New Test List per release/project/cycle Properties>Associated Work Items
Run Test Run one or multiple tests Debug tests
Test Results Window
Run/Debug
Pause/Stop Run
Import/Export Results
Group results by
LOAD TESTS
Step 1: Identify Objectives.
Step 2: Identify Key Scenarios.
Step 3: Identify Workload.
Step 4: Identify Metrics.
Step 5: Create Test Cases.
Step 6: Simulate Load.
Step 7: Analyze the Results.
Understanding Load Test
Primary goal of a Load Test is to simulate many users accessing server at the same time
By adding Web Tests to a Load Test you can,
Simulate multiple users opening simultaneous connections to a server
Make multiple HTTP requests
Set other properties that apply to individual Web Tests
Understanding Load Test Continues..
Load Tests are used in several different types of testing:
Smoke: How your application performs under light loads for short duration
Stress: To determine if the application will run successfully for a sustained duration under heavy load
Performance: To determine how responsive your application is
Capacity: To determine how many users and/or transactions a given system will support and still meet performance goals
Identify Objectives
Write down the performance objectives for your application. Response time. For example, the product catalog must be
displayed in less than 3 seconds. Throughput. For example, the system must support 100 requests
per second. Resource utilization. A frequently overlooked aspect is how
much resource your application is consuming, in terms of CPU, memory, disk I/O, and network I/O.
Maximum User Load. Determining how many maximum users can run on a specific hardware configuration
Identify Key Scenarios
What are scenarios? Scenarios are anticipated user paths that generally incorporate
multiple application activities.
How do you identify scenarios? Key scenarios are those for which you have specific performance
goals or those that have a significant performance impact.
For example: Login to the site. Browse albums. Register to the site.
Identify the workload
Identify the distribution IIS Logs (Exist). Research (Not exist).
Identify the user loads What are the max expected
concurrent users?
53
Identify Metrics
Metrics provide information about how close your application is to your performance objectives.
Metrics help you identify problem areas and bottlenecks.
Network-specific metrics.
System-related metrics.
Platform-specific metrics.
Application-specific metrics.
Create Load Test Cases
What is a Load test case? An group of activities involved in a scenario. For example: Place an Order.
The test cases are created based on the scenarios and the profile mix identified in the previous steps.
Each test case should mention the expected results in such a way that each test case can be marked as a pass or fail after execution.
Performance Test Cases Load Test Case example:
Expected Results example:
VS Performance Test Load patterns and Real world test analogy:
Constant pattern Load test, Stress test
Step pattern Compatibility test, Performance test, Benchmarks
Goal based pattern Stress test , Capacity test
Load Test Life Cycle http://blogs.vertigosoftware.com/teamsystem/archive/2006/02/06/VSTS_Load_Testing_Lab_Setup.aspx
Demo
1. Add a blank web test
2. Call web test from another web test (Login and LinkToLos)
3. Data binding-csv-LoginPage
4. Run WebTest once then edit run settings to run per row
5. Validation rule passing
6. Show Load Test setup and run
7. Analyze LoadTest Result
Control Load Modeling
Load Tests now offer additional load-modeling options that enable us to create load tests that more accurately model the expected real-world usage of an application:
Number of tests run Determines which test is run when a virtual user starts a
iteration Follow this model when you are basing test mix on
transaction percentages in an IIS log or production data
Control Load Modeling continues..
The amount of time spent on each test Determines the percentage of virtual users who will run a
particular test Follow this model when you are basing the test mix on the
percentage of users running particular test
The pace/speed at which the users run the tests Each test is run the specified number of times per user per
hour Follow this model when you want virtual users to run tests at
a certain pace throughout the load test
How should the test mix be modeled
Improved Load Test Analyzer Views
New Summary View with key indicators in a single page
Four new built-in graphs display key information at the same time
Easily accessed table view
Load Tests Summary
Test Run Information Name, Description, Start/End times, Controller
Overall Results Number of request/sec Number of failed requests Average response time Average page time
Load Test Summary Continues…
Key Statistics: Top 5 Slowest Pages URL (Can click the link to see details)
and average page load time
Key Statistics: Top 5 Slowest Tests Test name (Can click the link to see details)
Average test time
Key Statistics: Top 5 Slowest SQL operation Name of the operation (Can click the link to see details)
Duration in Microseconds
Load Tests Summary Continues..
Test Results List of all tests and scenarios Number of time each scenario ran Number of times it failed Average test time
Load Tests Summary Continues..
Page Results List of all web pages in the load test Total count of requests made for the web page Average page time
Load Test Summary Continues..
Transaction Results
NOT A DATABASE TRANSACTION Web Test Transaction List of all the transactions in the Load Test Response Time Elapsed Time Total Counts
Load Test Summary Continues…
Controller and Agent Resources Contains list of computers that are used to run the test % processor time Available memory at the end of test run
Load Test Summary Continues…
System Under Test Resources List of computers that are the set of target computers for
which load is being generated Includes any computer from which you collect counter sets
other than Agent or Controller.
Load Test Summary Continues…
Errors List of all errors
Type i.e. HttpEerror Sub Type i.e. LoadTestException
Error count Number of each error type
Error message i.e. 404-NotFound
Graphs View
Table View
Errors-Displays a list of errors that occurred during the load test run
Pages-Displays a list of pages accessed during a load test run
Requests-Displays details for individual requests issued during a load test
SQL Trace-Displays the results of SQL tracing only if SQL tracing was enabled with right connection string
Tests-Displays details for individual tests run during a load test
Thresholds-Displays a list of threshold rule violations that occurred during a load
Transactions-Displays a list of transactions that occurred during a load test run
Load Test Result Repository Management
Information gathered during a Load Test run is stored in the Result Repository
Repository contains performance counter data and error data
You can manage your load result from the Load test editor Open Import Export Remove You do not have to go the dbase to clean up the repository
Defects Tracking Using TFS
Defects Tracking Using TFS Continues..
Reporting Using TFS—Remaining Work Items
Reporting Using TFS—Bug Rates
References
http://blogs.msdn.com/slumley/pages/how-to-debug-a-web-test.aspx
http://blogs.msdn.com/slumley/pages/web-test-correlation-helper-feature-in-orcas.aspx
http://msdn.microsoft.com/en-us/library/aa730850(vs.80).aspx