PA Department of Health Project Test Guide Documents/DOH Test Proces… · PA Department of Health Project Test Guide Version: ... DOH Project Test Guide v1.0 Page 2 of 15 Project

Bureau of Informatics and Information Technology Project Management Office

PA Department of Health Project Test Guide

Version: 1.0

Bureau of Informatics and

Information Technology

DOH Project Test Guide v1.0 Page 2 of 15 Project Management Office

Document History

Version Date Author Status Revision Descriptions

0.1 02/16/2017 DOH PMO Initial Draft

1.0 03/08/2017 DOH PMO First Published




Table of Contents

1 INTRODUCTION .................................................................................................................. 4

1.1 PURPOSE OF THIS DOCUMENT .................................................................................... 4 1.2 GLOSSARY AND ACRONYMS ........................................................................................ 4

2 TEST APPROACHES .......................................................................................................... 6

2.1 WATERFALL TESTING .................................................................................................. 6 2.2 AGILE TESTING ........................................................................................................... 6

3 WATERFALL TESTING ....................................................................................................... 7

3.1 PLANNING ACTIVITIES ................................................................................................. 7 3.1.1 Roles & Responsibilities ................................................................................ 7 3.1.2 Test Plan ....................................................................................................... 8

3.2 TEST MANAGEMENT .................................................................................................. 10 3.2.1 Test Cases .................................................................................................. 10 3.2.2 Change Control/Traceability ........................................................................ 12 3.2.3 Test Issues .................................................................................................. 12 3.2.4 Testing Approval ......................................................................................... 13

3.3 DOH TEMPLATES AND TOOLS ................................................................................... 13

4 AGILE TESTING ................................................................................................................ 14

4.1 PLANNING ACTIVITIES ............................................................................................... 14 4.1.1 Roles & Responsibilities .............................................................................. 14 4.1.2 Test Planning .............................................................................................. 14

4.2 PRODUCT DEMONSTRATION ...................................................................................... 15 4.3 USE CASES/USER STORIES ....................................................................................... 15




1 Introduction

1.1 Purpose of This Document

The PA Department of Health (DOH) Bureau of Informatics and Information Technology (BIIT) Project Management Office (PMO) developed this document to provide guidance for all DOH Business Analysts (BAs). As DOH further matures the Department’s project methodology and processes, this document provides guidelines for the testing process and documentation to apply for all DOH projects. This document was written with reference to a traditional environment.

1.2 Glossary and Acronyms

Table: Terms and Acronyms Used in This Document

Acronym/Term Definition

BIIT Bureau of Informatics & Information Technology, bureau within DOH

DOH Department of Health

IIBA International Institute of Business Analysis – a professional organization for business analysis which sets standards and best practices for the industry. The standards and best practices are compiled in the Business Analysis Body of Knowledge (BABOK), along with special focuses on agile, government, etc.

MTM Microsoft Test Manager – Microsoft Test Manager is used to help test the application is being built. MTM stores the test plans and results on Team Foundation Server (TFS).

PMI Project Management Institute - a professional organization for project managers and business analysts which sets standards and best practices for the industry. The standards and best practices are compiled in the Project Management Body of Knowledge (PMBOK), along with additional standards and guides for business analysis, program management, portfolio management, etc.

PMO Project Management Office – Department within DOH responsible for managing the project governance process, providing oversight for DOH projects, and providing mentoring and guidance to department project managers (PM) and business analysts (BA).

RTM Requirements Traceability Matrix – a process of documenting the links between the requirements and the work products developed to implement and verify those requirements. The RTM captures all requirements and their traceability through the project lifecycle in a single document; typically, waterfall methodology centric.

SDLC Software development lifecycle – A software development lifecycle is essentially a series of steps, or phases, that provide a model for the development and lifecycle management of an application or piece of software.

SDLC Agile Methodology

The Agile methodology model is a combination of iterative and incremental process models with focus on process adaptability and customer satisfaction by rapid delivery of working software product. The Agile methods break the product into small incremental builds. These builds are provided in iterations. Each iteration typically lasts from about one to three weeks. Every iteration involves cross functional teams working simultaneously on various areas like planning, requirements analysis, design, coding, unit testing and acceptance testing. At the end of the iteration a working product is displayed to the customer and important stakeholders.




Acronym/Term Definition

SDLC Waterfall Methodology

The Waterfall methodology was the first Process model to be introduced. It is also referred to as a linear-sequential life cycle model. It is very simple to understand and use. In a waterfall model, each phase must be completed before the next phase can begin and there is no overlapping in the phases.

Test Cases A set of conditions under which a tester will determine whether an application is working correctly or not.

Test Data In order to test a software application, you need to enter some data for testing most of the features. Any such specifically identified data which is used in tests is known as test data.

Test Log A test log is a listing of the test cases that were executed, in what order, who executed the test case and the status of the test case (pass/fail).

Test Strategy An outline that describes the testing portion of the software development cycle. It is created to inform PM, testers and developers about some key issues of the testing process. This includes the testing objectives, method of testing, total time and resources required for the project and the testing environments.

Test Suite A collection of test cases that are used to test a software program to show that it has some specified set of behaviors. A test suite often contains detailed instructions and information for each collection of test cases on the system configuration to be used during testing. Test suites are used to group similar test cases together.

UAT User Acceptance Testing – This is the last phase of the software testing process. During UAT, actual software users test the software to make sure it can handle required tasks in real-world scenarios, according to specifications.

Additional DOH acronyms can be found by clicking here.

https://pagov.sharepoint.com/sites/DOH-Intranet/IITServices/Pages/Acronyms.aspx.




2 Test Approaches The approach to testing varies based on the project’s methodology. At a high level, there is a choice between waterfall (e.g. predictive) and agile (e.g. adaptive). For more on these topics, please refer to the applicable IIBA Business Analysis Body of Knowledge (BABOK) and Project Management Institute (PMI) sections and guides.

2.1 Waterfall Testing

The waterfall model is a sequential (non-iterative) design process, used in software development processes, in which progress is seen as flowing steadily downwards (like a waterfall) through the phases of conception, initiation, analysis, design, construction, testing, production/implementation and maintenance. In some cases, the next phase may begin before the previous phase is completed; however, the prior phase must always complete before the next phase can be completed. Baselines are established by obtaining sponsor and stakeholder approvals to confirm accuracy, completeness, and agreement on the project deliverables. For the purpose of the testing phase, requirements and design have been completed and signed off by the sponsor/stakeholders. Development of the solution is completed and the solution is then moved into a testing phase to validate the development and confirm it successfully meets the requirements prior to implementing the solution into a production environment. Most of the earlier phases’ activities, such as design, coding, and unit testing, are associated with the software development team. Testing activities typically include the full project team and may include a subset of stakeholders and/or end users.

2.2 Agile Testing

Unlike a waterfall approach, the requirements in an agile project are often not clearly defined or known early in the project. Requirements are defined in sufficient detail for the solution developers to begin coding. A mantra for agile is to test early and often; development is not completed before testing commences. Through an iterative approach, development is completed and tested with the product owner for review and feedback. Through each cycle, stakeholder feedback further clarifies the requirements and adjustments are made to the solution. Testing and coding are done incrementally and interactively, building up each feature until it provides enough value to release to production. A solution may be implemented into a production environment as soon as there is sufficient functionality available to make it viable; additional enhancements will continue to be made to the product/solution during its lifecycle.




3 Waterfall Testing

3.1 Planning Activities

3.1.1 Roles & Responsibilities

The responsibilities for testing will be defined for the specific project and team members, with ultimate responsibility for delivery with the Project Manager. The test plan should clearly define roles and responsibilities for the team members during testing. Common roles and responsibilities are outlined below:

Role Responsibilities

Project Manager (PM)

The project manager is responsible for overall execution in accordance with the project plan and project design.

Exercise a general supervision over the activities of the test lead and developers, and insure that testing is thorough and complete.

Insure that the standards are complete and unambiguous for development and QA, and that they conform to the user’s standards.

Escalate any non-resolved issues to the project sponsor.

Review problems with users in order to determine whether the issue is a bug or a potential enhancement.

System Analyst/ Developers

The system analyst may be an application developer, the DBA, a documentation developer, or any other member of the technical/development project team.

Develop the work product in accordance with specifications and standards.

Test that work product to confirm adherence to those specifications and standards.

Perform test activities detailed in the project’s Test Plan.

Correct any deficiencies and/or problems, whether or not they are obvious to others involved in the testing and review process.

Seek assistance from QA Lead, Project Manager, and/or Project Sponsor as necessary to complete responsibilities.

Quality Assurance (QA)/ Test Lead

The test lead is responsible for overall quality assurance and test coordination across the team. These responsibilities include assisting team members that are having difficulty with their assignment, and hands-on correction as needed.

Verifying each deliverable has been tested in accordance with the procedures in the project’s test plan.

Review the user acceptance test cases for traceability to requirements and functional specifications.

Maintain a log of testing results and assign/track problems to completion.

Work with the other team members to resolve any deficiencies and/or problems.

Work with users to coordinate and review results of UAT and with team members to get problems and modules re-submitted for re-testing.

Review of test documentation, including detailed user test plans, test data, and testing results.

Plan, coordinate, and monitor overall test progress and adherence to schedules; assign schedules to individual module developers.

Escalate any non-resolved issues to the project manager.

User/Tester

The tester is responsible for assisting in creation and execution of acceptance testing in accordance with the project and testing plans.

Assist in creation of test scripts and cases based on requirements of system and process features and functions.

Execute test scripts and cases, including documenting the test results and any identified bugs/defects.





Work with project team members to resolve deficiencies and issues.

Signoff at the completion of acceptance testing confirming all testing meets the expected results and may be promoted to production.

3.1.2 Test Plan

A test plan is written for each project describing the scope, approach, resources, and high level schedule for the testing activities. In addition, the test plan defines the exit criteria and structure for a go/no go decision to complete testing and move to implementation of the solution. The plan serves as a communication tool and record of the test planning process. The test plan is signed off by the sponsor and key stakeholders to ensure common understanding and consensus. Planning activities should begin after the requirements have been baselined and be complete by the end of the design phase. Environments To avoid impacting current production processes and users, there are typically multiple environments created for a product/solution – including hardware and software components. The environments can range from subsets/components of the full product to a complete replication of the production version. The number and structure of the environments will vary by product/solution. The test plan should specifically identify what environment(s) apply to the project’s testing activities. Some common environments/structures are outlined below.

Environment Description

Development This environment is typically restricted to the application development/technical teams.

Usually, the environment starts with a ‘copy’ of the production application and all new coding is completed in this environment.

In addition to the actual development, the technical team will frequently complete unit/module testing in this environment.

Test This environment is typically restricted to the project team members involved in development and/or testing.

This environment has a copy of the ‘enhanced’ product/solution with all the enhancements/revisions included.

There may be multiple test environments and at least 1 environment typically provides a production-like copy of the hardware and software.

Training This environment is typically available to a subset of product users and/or a training team.

This environment typically provides a production-like copy of hardware and software.

Training materials and/or classes will utilize this environment and it may co-exist with a test environment.

Production This is the ‘live’ version of the product/solution.

All product users will have access and almost exclusive use of this environment.




Phases/Types Just as the project has multiple phases, testing activities will often be grouped into phases, also referred to as types of testing. In general, testing becomes progressively more elaborate and encompassing as it moves from one phase to the next. Initial testing activities may focus on a single module or step in the product and expand to include the full end-to-end activities of the product. Some common test phases are outlined below. The test plan should specifically identify what phases apply to the project’s testing activities and the condition criteria required to move from one phase to the next.

Type Audience Definition

Unit

Development/ technical team

Validates the behavior of components/modules of a product to ensure alignment to the design.

Integration


Modules are combined and tested as a group to verify they work together as expected. Test components are typically code modules, individual applications, client and server applications on a network, etc.

System


A complete, integrated system to evaluate compliance with the specified design. System testing can include any or all of the following:

Functional – validates the end-to-end application/system performs as expected from a function/process perspective for all user types, as outlined in the design specifications.

Load/Stress/Performance – validates the application meets expected performance standards during normal and peak volume periods; it can also be used to verify/measure the application’s scalability.

Compatibility - validates compatibility of an application or website with different browsers, operating systems, and hardware platforms.

Security/Penetration – validates the application meets security standards

Data Mapping/Validation – validates a conversion or migration of data from one system to another completes as expected from the content, format/structure, and historical perspectives.

Acceptance Project team and end users

Validates a product meets customer-specified requirements. A customer or representative user community usually performs this type of testing. It may also include data validation related to expected process or output for the data.

Regression (a.k.a. Test for Production)

Development/ technical team, Project team and end users

Ensure the new code does not negatively impact any existing code in the application and/or other applications used with the system. Development & technical team members verify all technical aspects can co-exist; project team and end users ensure existing functionality continues to work as expected with no negative impact to current processes. It can also be referred to as a ‘dress-rehearsal’ for production implementation.




Best Practices When creating the test plan, keep the following best practices in mind: Make the plan concise; avoid redundancy. If you think you do not need a

section in the template, delete that section in your test plan. Be specific. For example, when you specify an operating system (OS) as a

property of a test environment, mention the OS edition/version as well, not just the OS name.

Make use of lists and tables wherever possible; avoid lengthy paragraphs. Review the test plan a number of times prior to baselining it or sending it for

approval. The quality of the test plan reflects the quality of the testing the team is going to perform.

Update the plan as and when necessary; an outdated and unused document is worse than not having the document in the first place.

3.2 Test Management

3.2.1 Test Cases

Test cases outline the conditions and/or steps necessary to validate the application/solution produces the expected results. Test cases can be created for each type/phase of testing to be completed on the project. Specific content, structure, and expected results will vary based on the type/phase of testing. (Refer

to phase information in section 3.1.2)

Content

Test cases may be written manually in an Excel file or on a SharePoint list. There are also automated tools, such as Microsoft Test Manager (MTM). Regardless of how they are written and tracked, good test cases should contain the following elements:

Field Description

Number A unique tracking number for each test case. Each pass/fail condition should have a number. This number is referenced on the RTM for traceability purposes.

Title Identify the scenario with a brief, descriptive title

Priority Identify priority (example: high, medium, low); may be incorporated as part of the exit criteria

Pre-Requisite Identify any pre-existing conditions and/or data required to execute or validate the test case

Action/Steps Identify the steps necessary for the validator to execute the test case

Expected Result Identify what the validator should expect to see/occur after completing the action/steps outlined

Comments General comments/information about the test case for the validator

Requirement ID Associated requirement(s) which will be validated by this test case

Tracking




During testing, the results and status should be tracked for each test case. At a minimum, the following items should be included:

Field Description

Validated by Name of person who validated the test case

Result Result of test case indicating whether or not the expected results were achieved; typically pass or fail, but other values may be used

Date Date the test case was validated

Issue # If the test case failed, the issue must be reported and tracked on the issue log to resolution. If using an automated tool, this may be system generated.

Comments General comments/information about the test case from the validator

Metrics can also be used as a tool to report status and progress through the testing phases/cycles. The metrics could also be used in defining the exit criteria. The test plan should clearly define the exit criteria agreed upon between the project team and the sponsor/stakeholders.

Example

Number Title Priority Pre-

Requisite Action/Steps Expected Result

1.0 System Log on

Medium Internet access

Open Internet Explorer

Able to successfully open browser and access the Internet

1.1

A valid user ID

In the address bar, type the application URL

The log-on screen appears

1.2 Type your user name in the username field

User ID retained in the field after moving to the password field

1.3 Type your password in the password field

Password retained in the field

1.4 Click “log on” A pop up window stating that the webpage is trying to close the window appears.

1.5 Click “yes” on the dialog window to allow the pop up.

Welcome screen displayed; log in complete

Best Practices




The following outlines common best practices associated with writing and managing test cases.

Structure:

Write in correct sequence and clearly map to expected results

Write to the purpose/focus of the user, e.g. a system test script would be written for the developer’s perspective; an acceptance test script would be written for the end user’s perspective

Identify any dependencies or pre-requisites

Do not overlap functions/features or unnecessarily complicate test cases

Consider positive and negative scenarios, e.g. verify only letters cannot be entered into a phone number field

Language:

Write in simple and easy to understand language; clear and without ambiguity

Use active voice verbs: do this, do that.

Use exact and consistent names (of forms, fields, etc.). Characteristics:

Accurate

Economical, no unnecessary steps or words

Traceable, traced to a requirement in the RTM

Repeatable, can be replicated

Reusable, can be reused for regression if applicable

3.2.2 Change Control/Traceability

The test plan and all test cases must be included in the change control process. To ensure each requirement is tested and validated, the test cases should be mapped to the applicable requirement(s) on the RTM. All change requests submitted should include assessment for possible impacts to the test plan. In addition, all test cases must be updated to reflect any requirement and/or design changes.

3.2.3 Test Issues

Any errors or unexpected results experienced during test validation are referred to as defects or ‘bugs’. All defects must be reported and tracked through resolution by the project team. Defects may be tracked manually on a log or through an automated tool. Regardless of how the issues are tracked, the following components should be included:

Summarize the issue, including expected and actual result

Provide supporting information that may assist in troubleshooting, such as browser type and version for web-based applications

Screen prints and/or error message text




Prioritization of the issue o Determine impact to the overall test case and the test case’s

priority o Determine if it is truly an issue or a changed/missed requirement o Determine if there is a workaround available and the feasibility of

the workaround

Determine when/how the fix can be validated and if any pre-requisite activities must occur

The test plan should clearly define the issue reporting and tracking process to be used for a project. It should also outline expectations for issue response/resolution timeframes and escalation paths.

3.2.4 Testing Approval

When testing activities have completed, the project team should evaluate if the exit criteria has been successfully met. If the criteria have been met, the sign off and go/no go approval should be received from the sponsor/stakeholders, as outlined in the test plan. If the criteria have not been met, the project team should conduct an assessment of options and risks to be presented to the sponsor/stakeholder for review and decision. Implementation to the production environment should not occur until the sponsor and/or designated stakeholder(s) have provided approval to proceed.

3.3 DOH Templates and Tools

For DOH projects, SharePoint team sites are typically created for document repository and project management tool purposes. In addition to any automated tool used by IT or program areas, templates and/or SharePoint logs are available for the following deliverables:

Test Plan

Test Case

Test Issue Log

Requirements Traceability Matrix (RTM)

https://sharepoint.health.pa.gov/sites/collab/PMM/Template%20Library/Test%20Plan%20Template.docx

https://sharepoint.health.pa.gov/sites/collab/PMM/Template%20Library/Test%20Case%20Template%20w%20Metrics.xlsx

https://sharepoint.health.pa.gov/sites/collab/PMM/Template%20Library/Test%20Issue%20Log%20Template.xlsx

https://sharepoint.health.pa.gov/sites/collab/PMM/Template%20Library/Requirements%20Traceability%20Matrix.xlsx




4 Agile Testing There are several different agile methodologies; the specific activities and terminology used varies by methodology. This section will outline high-level and common activities associated with testing for an agile project.

4.1 Planning Activities

4.1.1 Roles & Responsibilities


Product Owner Defines features and release dates

Responsible for Return on Investment

Prioritizes features by business value

Accepts or rejects work

Team Lead (a.k.a. Scrum Master)

Ensures team is functioning and productive

Removes barriers/impediments for the team

Shields team from external interference

Ensures the process is followed

Facilitates planning

Team Member Typical teams include 5 to 9 members

Has cross functional responsibilities which may include modeling, programming, testing, release activities and more

Self-directed; organizes and completes tasks with minimal to no direct supervision

Committed to the project iteration/sprint including demos to the product owner

Customer Defines the minimal useable feature set

Provides specific feedback at each iteration

Willing to spend time with the team

Willing to receive value iteratively

Focused on simplicity

4.1.2 Test Planning

While traditional testing is often associated with the end of the development cycle, testing occurs more frequently and throughout the process with iterative or adaptive project cycles. Systems-level testing is incorporated with the actual development, testing-as-you-go approach. Acceptance-level testing may occur at the completion of a segment (e.g. iteration or sprint) or when a user story is completed. Based on agile’s integrated approach, test planning is typically defined/outlined as part of the overall iteration and release planning. Evaluation (e.g. testing) criteria can be defined early in the process and provide input to developing user stories/use cases, strategies, plans, and test cases.




4.2 Product Demonstration

A review/demonstration is held at the completion of an iteration and/or prior to a product release. The demonstration is a light and quick version of acceptance testing, generally limited to ½ day or less. During the demonstration,

the project team demonstrates how the code meets the acceptance criteria,

the product owner determines which items on the features list/backlog have been completed and which items need further refinement/modifications

new requests/enhancements are added to the backlog list for review, prioritization, and planning

based on the pre-determined release schedule, approved code is implemented

The team will conduct a retrospective/lessons learned and proceed with planning the next iteration/sprint and/or release. The product owner has full authority in determining whether or not a user story/feature is complete.

4.3 Use Cases/User Stories

In an agile approach, specific test cases may be written or the use cases/user stories from the requirements definition may be reused. The product owner and project team should agree on the approach selected. Refer to section 3.2.1 for more information on test cases; refer to the DOH Requirements Guide, section 4.3 for more information on user stories.