1PBI_SAS_08_Tech_Shull September 2008MAC-T IVV-08-162
Dr. Forrest Shull, FCMDKurt Woodham, L-3 Communications
OSMA SAS 08
Infusion of Perspective-Based Inspections for
NASA IV&V
(Technical Briefing)
2PBI_SAS_08_Tech_Shull September 2008MAC-T IVV-08-162
Introduction to Perspective-Based Inspection (PBI)
Tailoring of PBI for the IV&V Context
Evaluation Plan
Outline
3PBI_SAS_08_Tech_Shull September 2008MAC-T IVV-08-162
A long history of research & application shows that structured human inspection is one of the most cost-effective practices for achieving quality software “Cost savings rule” – Cost to find & fix software defects is about 100x
more expensive after delivery than in early lifecycle phases, for certain types of defects.
IBM: 117:1 between code and use Toshiba: 137:1 between pre- and post-shipment Data Analysis Center for Software: 100:1
“Inspection effectiveness rule” – Reviews and inspections find over 50% of the defects in an artifact, regardless of the lifecycle phase applied.
50-70% across many companies (Laitenberger) 64% on large projects at Harris GCSD (Elliott) 60% in PSP design/code reviews (Roy) 50-95%, rising with increased discipline (O’Neill) … many others
Software Inspection
4PBI_SAS_08_Tech_Shull September 2008MAC-T IVV-08-162
Planning
Preparation
DefectReportForm
Meeting
Follow-through
SoftwareArtifact
PlanningForm
Defect Correction
Form
1
2
3
4
organizer
inspector
moderatorinspectorsauthor
author
CorrectedSoftwareArtifactSoftware
Inspection
DefectCollection
Form
Roles
Activities
Products
Improving Inspections
Software Reading Techniques: Procedural guidance incorporating tested best practices for individual preparation
5PBI_SAS_08_Tech_Shull September 2008MAC-T IVV-08-162
A procedural technique incorporating three best practices:1. Giving each reviewer a particular and unique focus (or
perspective) on the document under review Base perspectives on document stakeholders to help
understand necessary participants, relate inspection tasks to development skills
2. Making individual review of a document an active (rather than passive) undertaking
3. Articulating the quality aspects of interest
Perspective-Based Inspection (PBI)
6PBI_SAS_08_Tech_Shull September 2008MAC-T IVV-08-162
History 2001-2003: SARP research funding
Refined basic approach, implemented on NASA projects and in classrooms, measured results
2004: Tech infusion with Flight Software Branch / GSFC Goal: Produce new Branch standards Success metric: Based on experiences on pilot(s), how much of the recommended
process is adopted by Branch?
2004: Tech infusion with United Space Alliance / JSC Goal: Reduce defect slippage over current increment Success metric: Show reduced defect slippage on PB-inspected artifacts
2007-2009: SARP funding for improved general inspection planning
Other Results and Benefits Improved defect detection effectiveness substantially (NASA & elsewhere) Helped re-invigorate inspection practices Forms the basis of industrial and academic training
Perspective-Based Inspection (PBI)
7PBI_SAS_08_Tech_Shull September 2008MAC-T IVV-08-162
Pathfinder: Infuse PBI into the IV&V practices being applied to JWST JWST Project/Developer artifact review JWST IV&V System Reference Model (SRM) validation
Technology Provider… Adapts PBI to IV&V domain and JWST-specific context Administers training Facilitates initial reviews Monitors efficacy of PBI and documents results/recommendations
JWST IV&V Team… Receives training Conducts reviews in-line with IV&V execution Provides ancillary feedback to Technology Provider (minimal disruption)
IV&V Facility Research… Monitors progress and determines next-step expansion based on JWST
pathfinder
Adopting PBI for IV&V
8PBI_SAS_08_Tech_Shull September 2008MAC-T IVV-08-162
JWST System Overview
Integrated Instruments• Mid-Infrared Instrument (MIRI) • Near-Infrared Spectrograph (NIRSpec)• Near-Infrared Camera (NIRCam)• Fine Guidance Sensor (FGS)
• Launch in 2013• Collaboration between NASA, ESA, CSA• Managed by GSFC• NGST – prime contractor• STScI – operator after launch
9PBI_SAS_08_Tech_Shull September 2008MAC-T IVV-08-162
JWST Observatory Diagram
Image credit: http://www.jwst.nasa.gov/observatory.html
10PBI_SAS_08_Tech_Shull September 2008MAC-T IVV-08-162
JWST IV&V Overview IV&V Scope
Observatory and Ground System segments
SRM-based approach starting with top-level user needs flowed systematically down through segment, elements, subsystem, and component levels Specific focus areas driven by IV&V criticality/risk analysis with close
coordination with JWST project
WBS driven SLP 09-1 Rev J (http://www.nasa.gov/centers/ivv/ims/slps/index.html) Defines validation and verification tasking to be performed at various
levels of system decomposition• Emphasis on early validation of requirements with respect to system
needs, goals, and objectives in light of three validation questions• Validated SRM plays pivotal role in validation and verification
activities throughout IV&V life-cycle
11PBI_SAS_08_Tech_Shull September 2008MAC-T IVV-08-162
Motivation: Focus the responsibilities of each inspector Minimize overlap among inspector responsibilities Maximize union of defects found
Document with defects
High overlap Low coverage(Ad hoc)
Low overlapHigh coverage(PBI)
Adopting PBI for IV&V: Perspectives
12PBI_SAS_08_Tech_Shull September 2008MAC-T IVV-08-162
Tailoring required for new domain, new issues, new artifacts
Adopting PBI for IV&V: Perspectives
Needs, goals, objectives elucidation
Analysis val./ver. target
Model construction
System Ref Model
Fanny May : Loan Arranger
Borrower : Borrower
A Lender : Specified Lender
Loan : Loan
verify_report()
new_loan(lender, borrowers)
new_
look_for_a_lender(lender)
look_for_a_loan(loan)
look_for_a_
update_loan(lender, borrower)
update_
lender :
new_lender(name,contact, phone_number)
update(lender)
monthly_report(lender, loans, borrowers)
identify_report_format()
Receive Monthly Report
Fanny May : Loan Arranger
Borrower : Borrower
A Lender : Specified Lender
Loan : Loan
verify_report()
new_loan(lender, borrowers)
new_
look_for_a_lender(lender)
look_for_a_loan(loan)
look_for_a_
update_loan(lender, borrower)
update_
lender :
new_lender(name,contact, phone_number)
update(lender)
monthly_report(lender, loans, borrowers)
identify_report_format()
Receive Monthly Report
Fanny May : Loan Arranger
Borrower : Borrower
A Lender : Specified Lender
Loan : Loan
verify_report()
new_loan(lender, borrowers)
new_
look_for_a_lender(lender)
look_for_a_loan(loan)
look_for_a_
update_loan(lender, borrower)
update_
lender :
new_lender(name,contact, phone_number)
update(lender)
monthly_report(lender, loans, borrowers)
identify_report_format()
Receive Monthly Report
Model validation
Do sources adequately define user needs?
Is model complete, correct, consistent, and testable?
Do behaviors defined in val./ver. targets map to equivalent definition in model?
13PBI_SAS_08_Tech_Shull September 2008MAC-T IVV-08-162
Interpretation of “defect”
Avoid focusing on correctness as the one and only quality property
Any issue that would cause a problem for a stakeholder downstream.
Defect Type Application to System Reference Model
Completeness One or more diagrams or use cases are missing a representation for a key concept.
Correctness/ Compliance
A diagram or use case contains a misrepresentation of a concept intrinsic to the domain described in the general requirements.
Consistency A representation of a concept in one diagram or use case disagrees with a representation of the same concept in either the same or another diagram / use case.
Clarity A representation of a concept is unclear, and could cause a user of the document (developer, low-level designer, etc.) to misinterpret or misunderstand the meaning of the concept.
Redundancy The diagram or use case includes information that, while perhaps true, does not apply to this domain and should not be included.
Adopting PBI for IV&V: Defect Types
14PBI_SAS_08_Tech_Shull September 2008MAC-T IVV-08-162
Adopting PBI for IV&V: Defect Types
Other sources of defect categories: Checklists and defect categories from previous work Experienced inspectors
What is it that the most experienced people are looking for?Can get from:
• Asking them!
• Noting recurring questions at inspection meetings
Defect/discrepancy report databases What types of problems seem to be common?What types of problems take the longest to close?
15PBI_SAS_08_Tech_Shull September 2008MAC-T IVV-08-162
Reading through a document one page at a time is not the most effective approach.
In PBI every inspector receives a scenario to guide his/her work.
Based on your perspective…
Imagine you are creating some other artifact you need as part of your job…
E.g. for tester perspective, think about building a test plan
Then read through the document as you normally would to get the necessary information.
Inspectors have questions about quality to answer while they follow the scenario.
Adopting PBI for IV&V: Active Review
16PBI_SAS_08_Tech_Shull September 2008MAC-T IVV-08-162
Initial perspectives for this domain: Modeler
Uses requirements & existing models to create the next iteration Quality foci: Appropriate level of detail in requirements; completeness
and clarity of requirements & models; consistency and correctness of models
Domain expert Assesses whether model accurately and usefully captures domain Quality foci: Correctness of requirements (independent of models);
identification of stakeholders and evaluation of usability from their POV; checking flow of control & use of reusable components
Quality assurance Assesses whether models and system can be validated properly Quality foci: Handling of exception cases / unexpected system conditions;
robustness of system; testability
Adopting PBI for IV&V: Summary
17PBI_SAS_08_Tech_Shull September 2008MAC-T IVV-08-162
Infusion Plan Understand team-specific quality concerns Understand likely perspectives for the team Refine set of perspectives and define
procedures for each. E.g., What quality concerns map to which
perspectives What is a feasible/effective process for checking
those concerns
Finalize procedures and provide training Analyze ongoing inspection results (and
update procedures if necessary) Monitor downstream defect profile Write final report
Status: Reviewed sample models & processes;
Constructed draft PBI techniques;
Created draft training course
18PBI_SAS_08_Tech_Shull September 2008MAC-T IVV-08-162
Evaluation Success metric: Effectiveness of PB inspection in comparison to past
inspections
Current JWST approach Project documents
Issues written against missing/incorrect/incomplete/inconsistent content in source artifacts
SRM Reviews IV&V team maintains SRM change logs, action item lists, and other sources for the
infusion team to evaluate relative effectiveness IV&V team documents discrepancy when comparing SRM against val/ver target
Strategy is to maintain current approach (issue reports, review records), but to track how PBI assisted in identifying the items Likely to involve follow-up conversation/interviews to assess results and
collect ancillary observations from the IV&V team Will balance this with objective to minimize impact to IV&V activities
Infusion Plan: Path Forward
19PBI_SAS_08_Tech_Shull September 2008MAC-T IVV-08-162
Use measurement results to further optimize, e.g., Consistently missing a certain type of defect?
Add a new defect type and associated questions to relevant perspectives
Missing a whole set of defects?Consider whether a new perspective should be added.
Do we never find defects of type x? Are there any perspectives that don’t seem to add much that the other perspectives didn’t find?
Consider deleting the relevant defect type or perspective. Do we consistently face a lot of problems of type x?
Add more reviewers using relevant perspectives
Future Considerations
20PBI_SAS_08_Tech_Shull September 2008MAC-T IVV-08-162
Forrest Shull
301-403-8970
Contact information
Kurt Woodham
757-644-5807