APPLICATION STRUCTURAL UALITY - CAST · any risks of future performance degradation. This assessment uses the CAST Application Intelligence Platform (AIP) to automatically scan the

APPLICATION STRUCTURAL QUALITY

SYSTEM RESILIENCE REVIEW

Prepared for:

Constellation Corporation

Application Structural Quality | Design for Performance Review

2

Table of Contents 1. Introduction ....................................................................................................... 3

2. Application Summary ........................................................................................ 3

A. Changes in Application Health ................................................................................ 4

B. Benchmarking Analysis ........................................................................................... 5

3. Resilience Risk Profile ....................................................................................... 6

A. Aging Analysis of Critical Robustness Violations ..................................................... 7

B. Potential Points of System Failure ........................................................................... 7

C. Quick Wins for Better Resilience ............................................................................. 9

4. Appendix ......................................................................................................... 11

A. CAST AIP Health Factor Definitions ...................................................................... 11

B. CAST Application Intelligence Platform ................................................................. 12


3

1. Introduction

This assessment is an effort to determine the overall quality of the Cassiopeia and Orion applications and identify some of the root causes of current resilience issues, as well as any risks of future performance degradation. This assessment uses the CAST Application Intelligence Platform (AIP) to automatically scan the implementation of these applications to review the architecture, design, and code against current industry best practices and known design flaws that may impact resilience. CAST AIP applies over 1000 engineering checks based on standards and measurements developed by the Software Engineering Institute (SEI), International Standards Organization (ISO), Consortium for IT Software Quality (CISQ), the Institute of Electrical and Electronics Engineers (IEEE) and the technology provider industry. The resulting analysis identifies specific flaws in the software and aggregates this information into metrics to objectively quantify the structural quality of the application.

2. Application Summary

This assessment is focused solely on the technical implementation of the Cassiopeia and Orion applications (user interface to database), with no investigation of the functionality.

Table 1: Summary of applications analyzed


4

CAST AIP provides architects and engineers the ability to examine and drill down on critical application characteristics and attributes. The primary application health factor that is addressed in this assessment is Robustness, but the assessment also provides some insight into: Performance Efficiency, Security, Transferability, and Changeability. (For detailed definitions of these attributes, see page 11 in the Appendix.)

Table 2: Summary of application characteristics

A. Changes in Application Health

The Total Quality Index (TQI) combines all health factors into a single application quality score as an indication of the overall health of the application. The Robustness health factor score focuses on the engineering flaws and best practices that can have an impact on the runtime stability of the application, as well as the ability to effectively test and fix problems in the application. Typically, resilience and stability are tested during the load and performance test stages of the project lifecycle. Specific issues are traced via dynamic analysis and instrumentation. This is a heavy and risky approach to ensuring the application remains resilient in the face of the unpredictability of live use.


5

The scoring of the health factors and TQI is on a four-point scale, where 1.0 is a very risky application and a 4.0 is a very clean application. Most applications have a TQI in the 2.5-3.5 range. The TQI and the Robustness index analyzed by CAST AIP for these applications are:

Figure 1: Cassiopeia Health Factor Scores Figure 2: Orion Health Factor Scores

B. Benchmarking Analysis

CAST collects the anonymous analyses of many applications in an industry benchmarking database called Appmarq. This repository currently contains approximately 1250 applications, with an average size of 500,000 lines of code. These data are categorized by industry, business process, geography, technology, and other descriptive characteristics.


6

Figure 3 shows the average scores of Cassiopeia and Orion plotted on a chart with the distribution of scores for similar applications by technology and industry. The two applications appear to rank below the industry average for both Performance Efficiency and Robustness characteristics, with Robustness showing to be almost at the bottom quartile of the industry sample.

Figure 3: Industry benchmark shows a lag in Robustness and Performance

3. Resilience Risk Profile

Cassiopeia and Orion were analyzed against the standard CAST rule set of over 1000 software engineering industry norms. Out of the box, the CAST model identifies some of these norms as being “critical”–which means they have a high likelihood of turning into defects. Critical violations, thus, deserve specific attention and it is best to avoid introducing them in ongoing enhancement work.


7

A. Aging Analysis of Critical Robustness Violations

The aging analysis in Table 3 shows the number of critical violations in Cassiopeia and Orion, with a breakdown that shows how long these violations have been in each of the applications. Since CAST analyzed four releases, we can identify the violations that were introduced in the last release and those that were introduced in prior releases. This provides insight into opportunities for training and architectural guidelines.

Table 3: Aging analysis of critical violations

B. Potential Points of System Failure

The risk indices are a systemic analysis of the transaction chains running through the application. Since CAST reverse-engineers the architecture of the entire application, it is possible to prioritize the objects that are implicated by the highest number of paths through the application, and the transactions that have the highest number of critical violations in their paths.

Propagated Risk Index for Resilience

Propagated Risk Index (PRI) is a measurement of the riskiest artifacts or objects of the application based on their contribution to application robustness and risk and their impact on the rest of the application.


8

The Top 10 objects with the highest PRI are:

Table 4: Top 10 objects with highest PRI

Transaction Risk Index for Resilience

Transaction Risk Index (TRI) is an indicator of the riskiest transactions in the application and serves as a powerful metric to identify, prioritize, and ultimately remediate them. When experiencing stability issues in a specific area of the application, the TRI identifies the most structurally-risky entry points into the application. This is often a starting point for testing and remediation of problems with resilience. The TRI number reflects the cumulative risk of the transaction based on the risk in the individual objects contributing to the transaction. TRI is calculated as a function of the rules violated, their weight/criticality, and the frequency of rule violations across all objects in the path of the transaction.


9

The Top 10 transaction entrypoints with the highest TRI are:

Table 5: Top 10 objects with the highest TRI

In summary, the assessment of these two applications reveals several specific user entry points where fundamental stability and resilience risks exist in the structure of the code.

C. Quick Wins for Better Resilience

During the assessment, CAST AIP discovered some very dangerous architectural practices that typically undermine the resilience of applications such as Casseiopia and Orion. These critical violations may not be easily tied to specific defects experienced by the application today, but enough violation chains in the application will cause erratic behavior and difficulty to troubleshoot.


10

Table 6: Quick wins for improving resilience


11

4. Appendix

A. CAST AIP Health Factor Definitions


12

B. CAST Application Intelligence Platform

CAST connects into all major SCM systems, or can take source code in whatever format it is maintained in the organization. Source code is then processed and stored in the CAST Knowledge Base as metadata, which forms the basis for the analysis and information provided by CAST AIP. CAST looks at the entire application—including legacy components, packaged app customizations, and all modern distributed technology environments. Data from third party code analyzers can be integrated into the CAST Knowledge Base and displayed in AIP dashboards.

Figure 4: Working with CAST AIP

Documents

APPLICATION STRUCTURAL UALITY - CAST · any risks of future performance degradation. This assessment uses the CAST Application Intelligence Platform (AIP) to automatically scan the