November 2004Mariya Shapran

Popkin Software:Software Quality Assurance

www.popkin.com2

Content TableContent Table

1. Popkin Software1. who we are2. what we do

2. Our Development Life Cycle1. what, when, why, who and where..2. Quality Assurance vs. Quality Control3. Quality assurance as part of the product development lifecycle4. Standards

3. Test Life Cycle1. SA Metamodel2. IDEF0/Black Box/White Box Test Approaches3. Best practices4. Test Approaches5. Merging industry standards into our test methodology

4. Defects Life Cycle1. Critical path2. Non-critical path

5. Questions and Answers

The Enterprise Architecture CompanyPopkin Software

www.popkin.com4

Popkin Software OverviewPopkin Software Overview

• The Enterprise Architecture Company• Founded in 1986• Leading Provider of Enterprise Architecture Tools

– Flagship Product is System Architect®

• Vision:– We develop, globally market, sell, and support a diverse but integrated set of Enterprise

Architecture, Business Analysis, and Systems Design tools.– Our tools enable clients to visualize, analyze, implement, and communicate their

Enterprise Architecture. This enables clients to maximize the efficiency of their organization and improve their strategic decision making capabilities.

• World Wide Presence: – Corporate Headquarters in New York City – European Headquarters in United Kingdom– Other Major Offices in Washington DC and The Netherlands

• Distributors throughout the World

www.popkin.com5

Company OverviewCompany Overview• Products:

– System Architect® – Flagship Product– System Architect DoDAF– iRMA™– SA Information Web Publisher™– SA Compare™– SA Simulator™– Training – Consulting

• Users:– More than 80,000 users and 40,000 licenses at 7,000 sites worldwide

• User groups in the US, UK, and Australia

www.popkin.com6

Partial List of ClientsPartial List of Clients• IBM• Lockheed Martin• Publix • Paychex• McGraw Hill• Scholastic• CSC• JPMorganChase• Virgin Mobile• DuPont Corporation• Borders• Vodafone• JB Hunt• Siemens

• British Airways• Samsung • Verizon• AT&T• Glaxo-Smithkline• MCI Worldcom• Bell Canada• Time Warner Telecom• BT• Ericsson• Nokia• T-Mobile • Telia Sonera

• Department of Defense

• US Air Force• Department of

Treasury• Department of

Agriculture• Veterans

Administration• GSA• Department of

Homeland Security• Intelligence Agencies

System Architect

www.popkin.com8

System ArchitectSystem Architect

• Leading Enterprise Architecture Tool on Market

• Integrated Support for Modeling Disciplines:

– Business Modeling

• Latest Standards: BPMN, BPEL4WS, etc

• Simulation of Process Models

– UML for Systems Modeling

• Synchronization with code such as Java, VB.NET, C++, etc

– Relational Data Modeling

• Implement/Reverse Engineer DBMS’s such as Oracle, DB2, and SQL Server

– XML Design

– Network Modeling

– Communicate Models by Publishing to Websites

– Repository based with Networking support

www.popkin.com9

XML MS Office HTML XML SVG HTML OLE

Automation CSV XML HTML MS Office XML

Reporting/Publishing Browsing Interfaces Matrices

Customizable Repository Metamodel

RepositoryExplorer Diagram

Analyze Business:•Enterprise Direction • Processes (BPMN)• Organization• Functions• Technology Infra• IDEF0/IDEF3• Simulation (IDEF3, BPMN, Process Charts)

UMLDesign Applications:• Use Cases• Object Interactions • Classes• Components• State Machines• Java• VB .NET

Design Databases:• ERD Model • Physical Model • IDEF1X• DB Synchronize™

•SQL Server•Oracle 8, 9i

Analyze Legacy Systems:• Gane/Sarson• Ward/Mellor• Yourdon/DeMarco• SSADM

Design XML Schemas:• DTDs• BizTalk• Instance Docs• Test Data• UML Integration• Data Modeling

Integration

XML DesignStructuredMethods

Data Modeling

Business Modeling

Shared Definitions

FrameworkManager

Microsoft SQL Server

SA Guidebooks

System Architect V10System Architect V10

SA InfoWeb Pub

SA Compare

www.popkin.com10

SA Compare Backup Reporting Merge/Extract

Configuration Management

Customizable Repository Metamodel

Repository

Customizable Repository Metamodel

Repository

Customizable Repository Metamodel

Repository

Catalog

FunctionsUsersEncyclopediasNT Authorization

Role-BasedAccess Control

System Architect V10System Architect V10

www.popkin.com11

Entity Relationships

UMLUML

IDEF BPMNCatalyst

Frameworks

Vendor-driven

FEA BPEL4WS

Popkin’s View of StandardsPopkin’s View of Standards

www.popkin.com12

Enterprise ArchitectureEnterprise Architecture

Make ReservationOver Internet

Make Reservation

Reject CustomerBecause ofBad Credit

Check Customer Credit

Customer

<<extend>>

Check Customer Credit

<<extend>>

<<include>>

ICAMSICAMS

Enterprise view of dataEnterprise view of data

XML

Information Information PublisherPublisher

Automated Reports on Information in

Repository

16+ years of 16+ years of enterprise enterprise

architecture architecture and and

modelingmodeling

MethodsMethodsUMLUML

BPMBPM

E/R DataE/R Data

FrameworksFrameworks

XSLT

ReservationPrimary Key

code [PK1]ID [PK2] [FK]Property_Code [PK3] [FK]Unit_Number [PK4] [FK]ID2 [PK5] [FK]Non-Key Attributes

statusdatedurationroomTypeShift_Num [FK]Sales_Clerk [FK]Guest_Number [FK]Agency_ID [FK]

RoomPrimary Key

number2 [PK1]ID [PK2] [FK]ID2 [PK3] [FK]Non-Key Attributes

statustype

CustomerPrimary Key

ID [PK1]Non-Key Attributes

nameaddresstelNofaxNocreditCard

Travel AgentPrimary Key

Agency_ID [PK1]Non-Key Attributes

Agency_NameAddressYTD_Commission

US CustomerPrimary Key

ID [PK1] [FK]Non-Key Attributes

NationalityPassportNumber

International CustomerPrimary Key

ID [PK1] [FK]Non-Key Attributes

Postcode

ReceptionistPrimary Key

ID2 [PK1]ID [PK2] [FK]

0..10..*

makes

1

0..*

makes

1 0..*

books

1 0..*

contacts

is_ais_a

Multiple Method View of InformationMultiple Method View of Information

Reception

Customer

Accounts

Reservation NumberGranted for

Provisionally BookedReservation

Customer Notified ofCredit Problem,

Reservation Denied

CancelReservation

Room BookedProvisionally

Notify Customer ofCredit Problem

Provide Client withReservation Number

Customer Agrees toTerms

Customer RejectsTerms

Check Customer Credit

Advise Customer

Calculate Room Price

Reject Terms

Deny Reservation

Provide Reservation #

Credit Not OK

Credit OK

Room Price

Room Price

Customer Credit Details

Room Price

Room Information

HTML/Web

SA SA RepositoryRepository

System ArchitectSystem Architect

Industry StandardsBPMN

XML SchemaEntity / Relation

UMLFrameworks

C4ISRDoDAF

www.popkin.com13

Evolution of System ArchitectEvolution of System Architect

‘88 ‘96 ‘97 ‘98 ‘99 ‘00 ‘01 ’02 ‘03

FirstWindows-

Based Modeling

ToolIntroduced

Structured Analysis

and Design

Object-oriented

Business Process Modeling

(IDEF)

UMLSupport

SQL Server As Underlying

Repository

TOGAF Support

Simulation, XML Support

C4ISR Framework

Support

ZachmanFramework

Support

Data Modeling

Multiple Methods

www.popkin.com14

Product Development Life CycleProduct Development Life Cycle

Quality Control Team

Development Team

Management Team

Business Team

Quality Review

System

Deployment

Test

Performance

& Multi-user

Test

Regression

Test

Execute Test

Cases

Review Test

Cases

Business

Acceptance

Test

Fix BugsIntegration

Test

Unit TestCode

Build Test

Cases

Program

Specification

Technical

Design & Audit

Requirements

Definition

Commercial ReleaseProject Proposal

Test Cycle

www.popkin.com15

Quality Assurance vs. Quality ControlQuality Assurance vs. Quality Control

• Quality Assurance• Quality Control

What’s the difference?

www.popkin.com16

Quality Assurance:Quality Assurance:

- System Development Methodologies- Estimation Processes- System Maintenance Processes- Requirements Definition Processes- Testing Processes and Standards- Measurement Programs to evaluate processes- Processes gap analysis- Concerned with all of the products that will ever

will be produced by a process

www.popkin.com17

Quality ControlQuality Control

- Relates and concern with a specific product or service;

- Verifies whether specific attributes are in, or are not in a specific product or service;

- Identifies defects for the primary purpose of correcting defects;

www.popkin.com18

Test Life CycleTest Life Cycle

QATestPlan

Quality Control Team

Development Team

Management Team

Business Team

Quality Review

SystemDeployment

Test

Performance& Multi-user

Test

RegressionTest

Execute TestCases

Review TestCases

BusinessAcceptance

Test

Fix BugsIntegrationTest

Unit TestCode

Build TestCases

ProgramSpecification

TechnicalDesign & Audit

RequirementsDefinition

Commercial ReleaseProject Proposal

Test Cycle

www.popkin.com19

Quality FactorsQuality Factors

Effort required to couple one system with anotherInteroperability

Extend to which program can be used in other applications – related to the packaging and scope of the functions that programs perform

Reusability

Effort required to modify an operation programFlexibility

Effort required to test a program to ensure that it performs its intended functionTestability

Effort required to locate and fix an error in an operational programMaintainability

Effort required to learn, operate, prepare input, and interpret output of a programUsability

Extend to which access to software or data by unauthorized persons can be controlled

Integrity

The amount of computing resources and code required by a program to perform a function

Efficiency

Extend to which a program can be expected to perform its intended function with required precision

Reliability

Extend to which a program satisfies its specifications and fulfills the user’s mission objectives

Correctness

www.popkin.com20

The System Architect MetamodelThe System Architect Metamodel

www.popkin.com21

IDEF0 – Integration Definition for Function ModelingIDEF0 – Integration Definition for Function Modeling

A22

Function/Activity_10

Output

MechanismCall

Input

Control

www.popkin.com22

Black Box Test ApproachBlack Box Test Approach

A22

Function X0

Control (Direct Dependencies)

Controlled Input

Call (to subFunction X1) Mechanism (Dependencies)

Actual Output

www.popkin.com23

White Box Test ApproachWhite Box Test Approach

A22

Method0

Control

Input

Call Mechanism

Output

www.popkin.com24

Test ApproachTest Approach• Black Box Approach

– Boundary Value Analysis (internal code)– Regression Testing (Test Partner and Loadgen)– Data Integrity Testing (Internal Tools)– Stress (Volume) Testing (Loadgen)– Backup and Recoverability (Internal Tools)– Functional Testing (Test Partner and Manual Testing)– Security Testing – Configuration/Compatibility Testing (VMWare images)– User Acceptance Testing (End Users)– Benchmark Testing (Loadgen)– Operational Readiness Testing (Loadgen and Test Partner)– Usability Testing (End Users/508 Compliance Guidelines)– Alpha/Beta Testing (Loadgen)

• White Box Approach– Memory Usage (BounceChecker)– General Resource Usage – Error Handling

• Gray Box Approach– Combination of Black Box and White Box Approach

www.popkin.com25

Boundary Value Analysis Example:Boundary Value Analysis Example:

Problem: Need to understand what will take place at the boundaries of the

behavior for each component.

Example: Field Name for the Diagram is required to accept unique name for the

diagram consistent of alpha-numeric characters up to 105 characters (with exception for special characters)

Boundary Value Analysis:

Absence of the name = rejectedSpecial Characters = rejected (this is on the boundary)null = rejected2alpha5 numeric characters = acceptedAll alpha characters = acceptedAll numeric characters= accepted106 characters = rejected

www.popkin.com26

Best PracticesBest Practices

• Peer Reviews• Defined Requirements Standards• Defined Quality Standards• Constant improvements to Quality Processes• Implementation of Automation Testing into Regression

and Performance Test Phases

www.popkin.com27

Merging Industry Standards into our Test MethodologyMerging Industry Standards into our Test Methodology

• 508 Compliance– 508 Compliance Checklist

• Methodologies: IDEF0, IDEF1, IDEF3– IDEF0/IDEF1/IDEF3 Compliance Checklist

• IEEE Standards– Used as guideline in defining of Quality Assurance Processes

• ISO Standards– Used as guideline in defining of Quality Assurance Processes

www.popkin.com28

508 Compliance Check List508 Compliance Check List

• Menu items have a unique, functioning access key.• Shortcut or popup menu items have a unique, functioning access key.• Dialog controls have a unique, functioning access key.• Dialogs can be closed by pressing the ESC key.• Shortcut or popup menus can be opened via the keyboard. Most keyboards have what's called an

application key. Pressing this key displays the shortcut menu for the selected object. • Drag and drop functionality can also be executed via the keyboard.• Dialogs have a logical and consistent tab order (left to right and top to bottom).• Color alone should not be used to indicate different states. For example, in access control red and

green colored arrows were used to represent checked in and checked out items.• The tabs on a tabbed control can be navigated by pressing CTRL+TAB.• Tree controls can be navigated by pressing the arrow keys.• Interfaces containing multiple windows and/or frames allow for focus to be shifted via the

keyboard.• Focus is always placed on the most recently opened or accessed window.• The application’s active window should display a visual focus indicator at all times so that users

can anticipate the effects of their keystrokes.• The application can read, use, and preserve system-wide user interface (UI) settings when

displaying customized controls or window content. The system-wide settings adopted by a user enhance the product’s accessibility; the application should use them where possible, and not disable or disregard them. An example of a system-wide setting is the ToolTip size.

• The application does not convey information exclusively by sound. If sound is the default method for conveying information, the application should provide other options to express this information.

www.popkin.com29

IEEE Software Engineering StandardsIEEE Software Engineering Standards

IEEE Standard for Software Project Management Plans (ANSI)1058.1 – 1987 (R1993)

IEEE Standard for Software Project Management Plans1058 – 1998

IEEE Standard for Software Reviews1028 – 1997

IEEE Standard for Software Verification and Validation – Supplement to 1012-1998. Content Map to IEEE 12207.1

1012a - 1998

IEEE Standard for Software Verification and Validation1012 – 1998

IEEE Standard for Software Unit Testing (ANSI)1008 – 1987 (R1993)

IEEE Recommended Practice for Software Requirements Specifications830 – 1998IEEE Standard for Software Test Documentation829 – 1998IEEE Standard for Software Configuration Management Plans828 – 1998

IEEE Guide for Software Quality Assurance Plans (ANSI)730.1 – 1995

IEEE Standard for Software Quality Assurance Plans730.1998

IEEE Standard Glossary of Software Engineering Terminology610.12.1990

www.popkin.com30

Defect Life CycleDefect Life Cycle

Retest

FixedRespond

Closed

ConfirmResolution

ReOpen

Design

Open

Hold

Verified

Issue

Quality Control Team

Development Team

Issue Originator/Tech Support

No

Yes

Require Additional Information

Reproduce

www.popkin.com31

GlossaryGlossary

• Organizations:– ARTS – Retail Tech Standards with Industry Data Model

• www.nrf-arts.org– BPMI – Business Process Management Standards body

• www.bpmi.org– DAMA – Data Management Association

• www.dama.org– DSDM – Dynamic Systems Development Method

• www.dsdm.org– NASCIO – National Association of State Chief Information Officers

• www.nascio.org– OMG – Object and XMI Tech Standards Body

• www.omg.org– Supply Chain Management standards with SCM data model

• www.supply-chain.org– Telecoms Tech Standards with Industry Process Mode

• www.tmforum.org– The Open Group

• www.opengroup.org

www.popkin.com32

GlossaryGlossary

• FEA – Federal Enterprise Architecture

• BRM - Business Reference Model• SRM - System Component Reference Model• PRM - Performance Reference Model• DRM - Data Reference Model• TRM - Technical Reference Model

• BPMN– Business Process Modeling Notation

• IDEF– Integrated Definition Language, a systems modeling technique using a specific graphical structure

• ORM– Object Role Modeling

• E/R– Entity Relation

• BPEL4WS– Business Process Execution Language for Web Services

• UML– Unified Modeling Language, an object-oriented design language

• XMI– XML Metadata Interchange

• DoDAF– Department of Defense Architecture Framework

• C4ISR– Command, Control, Communications, Computers, Intel-ligence, Surveillance, and Reconnaissance Architecture

Framework

www.popkin.com33

POPKINSOFTWARE

Popkin Software2 Rector StreetNew York, NY 10006

Tel: (646) 346.8578Fax: (646) 346.8501