View
215
Download
0
Category
Tags:
Preview:
Citation preview
CBIIT Quality Assurance Process
Preston WoodNCI CBIIT Government Quality Representative (GQR)
January 2014
RS
CBIIT Quality Assurance (QA) Process (1)
Agenda
• Goals for Quality Assurance• Quality Assurance Process• QA Artifacts• Support for Iterative Development• QA Process Innovations
2
CBIIT Quality Assurance (QA) Process (2)
Goals for Quality Assurance
• Requirements are verified• Test Approach is appropriate• Test coverage is adequate• Cost and schedule is understood• Testing can be completed on time• Audit capability is ensured• Confidence in releasing the application
3
CBIIT Quality Assurance (QA) Process (3)
4
CBIIT Quality Assurance (QA) Process (4)
QA ArtifactsShould be specified in early stage of the project.
Test Plan• Defines the test approach • Outlines scope of tests • Determines what time frame the project can be
completed– Schedule in P6 to determine cost/time impact to project
• Identifies risks going into the project, and mitigation plan• Requires sign-off and for any modification.
5
CBIIT Quality Assurance (QA) Process (5)
Test Cases
• Step by step directions for the tester• Contains verifications• Part of comprehensive test approach• Are living documents and updated.
6
CBIIT Quality Assurance (QA) Process (6)
Requirements Traceability Matrix (RTM)
• Identifies each requirement specifically• Identifies and matches test cases to requirements• Shows level of planned test coverage for identified
requirements.• Is a living and updated document.
7
CBIIT Quality Assurance (QA) Process (7)
QA Entrance Checklist• Vital to a repeatable QA process• Ensures not missing components to conduct testing:
– Tier is provisioned and code is operational– List of features implemented, and known issues– Unit test report from Development– Deployment documentation– Installation guide – Test case review– RTM is complete– Tag accepted by QA (smoke test prior to formal tests)– QA Test Plan reviewed and signed-off
Note: “No” is a possibility for an individual item, but has to be approved
8
CBIIT Quality Assurance (QA) Process (8)
Test Report
• Conveys what was tested• Contains variances on tests conducted• Announces what new issues were found, fixed, or still
exist• Categorizes defects to show problem areas of the app• Identifies risks to the project going forward, and
mitigation plan• Requires sign-off and is final.
9
CBIIT Quality Assurance (QA) Process (9)
Updated Requirements Traceability Matrix (RTM)
• Includes additional test cases• Shows completion of tests• Shows final level of test coverage for identified
requirements.
10
CBIIT Quality Assurance (QA) Process (10)
QA Exit Checklist•Vital to a repeatable QA process•Ensures not missing components to complete the tier:
• Source code tag received• List of defects fixed, features implemented• Test cases and execution reporting• Deployment documentation• Installation guide• Test cases list for Appscan• Appscan and 508 compliance passed
Note: “No” is a possibility for an individual item, but has to be approved
11
CBIIT Quality Assurance (QA) Process (11)
Audit and Confidence
The QA artifacts give us confidence in releasing the software to the next tier or Production because we have full audit capability to know:
– What was tested– How it was tested– Why it was tested– Where it was tested– When it was tested– Who tested it
12
CBIIT Quality Assurance (QA) Process (12)
And we know,
• Issues that are fixed,• Issues that still exist, and• What we are going to do about them.
13
CBIIT Quality Assurance (QA) Process (13)
The most important aspect of this process is that it is repeatable.
14
CBIIT Quality Assurance (QA) Process (14)
Grand Assumptions being made:
•All requirements are known from the start•No new functionality is introduced after testing begins.
What if new functionality is introduced while testing? – Not known before Test Plan written – Not a phased approach
15
CBIIT Quality Assurance (QA) Process (15)
Challenges for QA:• Test approach is now incomplete and unable to fully
address the new functionality.• Scope of testing may increase• Project may exceed schedule or cost• Project risks change and/or increase• Risk mitigation plan is obsolete• Project could be open to failure• Audit capability compromised• New project expectations may exceed QA capability
16
CBIIT Quality Assurance (QA) Process (16)
A ripple effect will occur in the community if any challenge goes unanswered.
Q: How do we address these challenges to reestablish confidence in releasing the application to Production?
A: Implement support for IterativeDevelopment
17
CBIIT Quality Assurance (QA) Process (17)
18
QA Support for Iterative Development
Iterative Testability Assessment
1. New functional requirements are identified
2. QA conducts requirements testability assessment
3. QA determines if updates to test plan needed:• Test Approach change needed?• Test Scope and/or Environment change needed?• Test Schedule and Milestones change needed?
CBIIT Quality Assurance (QA) Process (18)
If all questions are “no” (typically bug fixes only), then
• New test cases are created• RTM is updated as usual• Testing resumes without changes to the Test Plan.
19
CBIIT Quality Assurance (QA) Process (19)
If any question is “yes”, QA advises project group of the impact to the project.
The project determines a cost assessment of the impact.
Should the impact and cost be deemed acceptable,
1. The test plan is edited accordingly to account for the new requirements,
2. RTM is updated with new requirements and test cases
3. GQR, GPM, and GS give sign-off on the edits.
4. Then the new tag is accepted, and testing resumes for the new iteration.
20
CBIIT Quality Assurance (QA) Process (20)
21
CBIIT Quality Assurance (QA) Process (21)
Benefits of the Iterative Testability Assessment
•Requirements are properly verified•Audit capability is ensured•Confidence in releasing the application is restored
because…
22
CBIIT Quality Assurance (QA) Process (22)
Benefits of the Iterative Testability Assessment
•New requirements were analyzed and documented.•Test Approach was updated and comprehensive.•New timeline and cost for QA was assessed and accounted for.•Correct environment to test was confirmed.•Test coverage was appropriate.
So testing can be completed on time and on budget.
23
CBIIT Quality Assurance (QA) Process (23)
Challenges to QA Support for Iterative Development
•Iterative development can be cost intensive– Restarting QA multiple times.– New functionality may invalidate previous verifications.– Completed tests may need to be re-run.
•Finding the balance point, cost analysis– More iteration/costs vs features to the community– Streamlining the sign-off process
• new JIRA Workflow Process
– Tracking the project in P6
24
CBIIT Quality Assurance (QA) Process (24)
QA Innovations and Focus on Individual Sub-Project Needs
• QA Process Modification – Sub-Projects deploying to Production tier before end of
FY’13. – Critical fix deployment
• Funding running out• Test Plan waived
– assume previous test approach would be adequate
• Form Builder 4.0.3.1 critical fix– deployed from Dev to Production tier in two weeks
25
CBIIT Quality Assurance (QA) Process (25)
QA Innovations and Focus on Individual Sub-Project Needs
• Implemented QA support for iterative development processes– QA can properly support sub-projects with multiple
iterations containing new requirements– EPLC process can be maintained
26
CBIIT Quality Assurance (QA) Process (26)
QA Innovations and Focus on Individual Sub-Project Needs
• QA Process Improvement– Review and grooming of QA artifact templates– One test plan– Research for a new appscan policy proposal– Improve workflow, transparency of process
• Adopt new JIRA workflow to track projects
27
CBIIT Quality Assurance (QA) Process (27)
Recent QA Process Innovations:
• Review and Approval of Test Cases in HP Quality Center– Review, comments, approval
• Electronically approve QA Artifact PDF– For projects that request that level of tracking
• QA Document Generation by HP Quality Center– Test Case spreadsheet (alt. workflow for test case review)– RTM (needs requirement tracking in HP-QC)
28
CBIIT Quality Assurance (QA) Process (28)
29
CBIIT Quality Assurance (QA) Process (29)
Reference material:
CBIIT QA wiki site
https://wiki.nci.nih.gov/display/CBIITQA
CBIIT Engineering wiki site
https://wiki.nci.nih.gov/display/CBIITeng/CBIIT+Engineering+SOPs
30
CBIIT Quality Assurance (QA) Process (30)
Questions
31
Recommended