Commercial off the Shelf (COTS) policy at DMO ...academy.amccentre.nl/thesis/I_A_Khrystova.pdfMaster of Science in Asset Management Control at the Hogeschool Zeeland written by Iryna

Commercial off the Shelf (COTS) policy at DMO: consequences

of implementing COTS products and components in naval

military assets

written by

Iryna A. Khrystova

2

Commercial off the Shelf (COTS) policy at DMO: consequences

of implementing COTS products and components in naval

military assets

Thesis submitted to the International Masters School

to fulfill the Requirements for the Degree of

Master of Science in Asset Management Control

at the Hogeschool Zeeland

written by

Iryna A. Khrystova

Defence Materiel Organization (DMO), The Hague, The Netherlands

November 2011

Scientific Supervisors: Ir.P.J.T. Bakker, MSc

Ir. Peter Van Gestel, MSc

Expert Supervisor: ing. P.J. Kense, MSc

3

ACKNOWLEDGEMENTS

First and foremost, I would like to express my sincere gratitude to my supervisors

Paul Kense, Peter Bakker and Peter Van Gestel, for their help, guidance, patience,

and constant encouragement during this research.

This study has been conducted at the Naval Maintenance & Service Agency (NMSA)

in Den Helder. Therefore, I would also like to express my gratitude to those people at

the NMSA who positively influenced my work. Special thanks go herewith to Dave

Sinay, Theo Druijven, Jan Hofstra and KLTZT Hans Mulder, LTZE 1 Marjon Gram-

Blauw, LTZE 1 Arie de Groot for spending their valuable time to review the work and

provide constructive feedback.

For their friendship, inspiration, motivation, and support, I would like to give my

thanks to the co-students of my cohort, and more in particular to my team partners

KVK(ret’d) Yves Delbrassine, KLTZT Arie Schaap and LTZ2OC Tom Stroop.

I would like as well to express my acknowledgements to the management of the

Guided Weapons Department for facilitating and their financial contribution to the

study “Asset Management Control” and to my colleagues from the Guided Weapons

Department who showed lots of interest during the progress of the study and in my

achievements while conducting this research.

Last but not least, I would like to thank my mother for giving me much strength and

support and my husband Jaap Kwakernaat for his love, care and understanding.

4

ABSTRACT

The use of COTS products and components has been suggested as implicitly

providing considerable benefits in terms of applying new technological developments

and saving cost and time. However, the success of the use of COTS items largely

depends upon a coherent policy and systematical engineering processes for

mitigating risks connected with their short life cycle, replacement strategy,

obsolescence character and as a consequence the financial impact on maintenance

support and availability of military assets.

Due to the lack of a systematic approach, early decisions that have been made to

use COTS products and components without consideration of consequences, turned

out to hamper an effective maintenance support of military assets. The assumed

”Faster, better, cheaper” features of COTS however, might invoke problems such as

rapid obsolescence, no clear definition of COTS as product group in the contracts

with suppliers, absence of Form-Fit-and Function (FFF) replacements, overpricing of

COTS components and non-supportability of particular COTS components.

This thesis defines elements to be taken into consideration in order to establish an

internal COTS policy, which aims at structurally taking into account different aspects

of COTS. These are to be preliminary contemplated in order to prepare and conduct

an effective support of naval military assets during the exploitation phase.

Within this approach, requirements for the policy foundation have been derived from

the outcome of a Gap analysis questions list and a set of assessment criteria

affecting the maintenance process effectiveness. The Gap analysis contents and the

assessment criteria for this thesis have been determined and refined through a

number of distinct study cases.

Conclusions and recommendations for the internal COTS policy have been made in

the scope of the total quality approach with interrelated domains and processes. It

has also been determined that in order to make COTS policy effective and to achieve

an effective maintenance process for naval military assets considering the use of

5

COTS products and components, minimum 75% of the assessment criteria which

have been proven through four study cases, are to be met.

6

1. CONTENTS

APPENDIXES B, D , E and F are removed from the public version of the thesis

because of company sensitive data.

7

2. LIST OF FIGURES

Figures 25 till 54 are removed from the public version of the thesis because of

company sensitive data.

8

3. LIST OF ABBREVATIONS

ADCF Air Defence and Command FrigateAMC Asset Management ControlCLAS Commando Landstrijdkrachten; Army CommandCLSK Commando Luchtstrijdkrachten; Air Force CommandCMS Combat Management SystemCOTS Commercial Off The ShelfCZSK Commando Zeestrijdkrachten; Naval ForcesDLM Depot Level MaintenanceDMO Defence Materiel OrganizationDMP Defence Materiel Process ECP Engineering Change ProposalFFF Form, Fit, FunctionFIBAA Foundation for International Business Administrative

AccreditationFMECA Failure Mode, Effects and Criticality AnalysisHSV Hydrographical Survey VesselILM Intermediate Level MaintenanceILS Integrated Logistic SupportINK Instituut Nederlandse Kwaliteit; Dutch Quality InstituteIMCS Integrated Monitoring Control System ISS In Service SupportKPI Key Performance IndicatorsLCC Life Cycle Cost(s)LORA Level of Repair AnalysisLRU Line Replaceable Unit MATEX Material Exploitation BudgetMATLOG Material LogisticsMTBF Mean Time Between FailureMTTR Mean Time To RepairMINDEF Ministry of DefenceMoD Ministry of DefenceMOTS Militarized off the ShelfMTTF Mean Time to FailureNDI Non-development itemsNMSA Naval Maintenance & Service AgencyNSN Nato Stock NumberOC Operational CommandOEM Original Equipment ManufacturerOLM Organic Level MaintenancePAM Project Aanpassing Mijnenbestrijdingscapaciteit; Mine-

Countermeasures Capability Modification ProjectPS Patrol ShipRAM Reliability, Availability and MaintainabilityRCM Reliability Cantered Maintenance RFQ Request For Quotation

9

RNLN Royal Netherlands NavyROTS Ruggedized off the ShelfSEWAC

O

Sensors, Weapons & Command Systems

SRU Shop Replaceable UnitTQM Total Quality Management WSM Weapon System Management / Manager

10

1. Introduction

4. INTRODUCTION

This thesis is a final scientific work in the scope of the Asset Management

Control (AMC) Master course. The thesis aims on analyzing whether or not the

current maintenance policy at the Defence Materiel Organisation (DMO) is effective

in relation to Commercial Off The Shelf( COTS) items. It also describes the risks

related to the obsolescence character of COTS and how to deal with consequences

of the use of COTS products in order to mitigate those risks.

The Naval Maintenance & Service Agency(NMSA) has requested to conduct a

research, which should contribute to the development of an internal COTS policy and

decision process in order to mitigate COTS related risks and identify, whether there is

any inconsistency in existing processes. Consequently, the outcome of this research

should bring feasible improvements on processes and recommendations on eventual

organisational changes.

Since COTS products have a short life cycle, a systematic approach with

consideration of their replacement strategy is an essential prerequisite of the effective

maintenance support. The absence of reconciliation between early decisions to use

COTS and the actual practice where no contemplation of replacement strategy and

risks related to COTS are taken into account, results in hampering an effective

maintenance support of military assets.

With this research, the nature of the problem will be identified by analysis of

current Actual (IST) processes and getting insights into backgrounds, causes, and

relationships of the problem. Consequently, recommendations for a Target (SOLL)

situation will be defined.

The results of this thesis can be valuable for ILS advisors, technical engineering

staff of the NMSA and the policy making department of the Defence Materiel

Organisation (DMO). It could be used as a reference for improving internal NMSA

processes and for updating DMO internal regulations. Therefore, the conclusions and

recommendations will be separately presented to the company supervisor

Mr.P.Kense and a copy of this dissertation will be handed over to the policy

department within DMO that deals with this topic.

Before working out the scientific part of the thesis, this introduction chapter will

first position “COTS” and what is generally understood under the COTS concept,

delimitate the field of research and give an overview of the organisation where this

11

1. Introduction

research has been conducted (section 1.2). In the section 1.3 a statement of the

problem which has been encountrered in the organisation will be formulated.

12

1. Introduction

4.1. COTS and their background

Allowing the use of COTS items in Defence systems aims to offer to Defence

organisations key opportunities and benefits such as, ,

- Elimination of the need to invest in the development and support of unique

items;

- Lower risks since the product is already established;

- Integration of new or latest technology (which is probably the most important

benefit for Defence organisations);

- Reduced cycle time;

- Lower life cycle costs;

- Improve reliability and availability;

- Access to commercial support services;

There are multiple attempts in the scientific literature to define what a COTS item is.

Though COTS definition matter will be further addressed in the chapter 3.1, it is

worthwhile to mention at this stage at least some of the existing definitions given by

Baron , and in the Commercial Item Handbook

“COTS items are commercial items that have been sold, leased, or licensed

in substantial quantities in the commercial marketplace and that are offered

to the Government without any modifications.”

“COTS are any item of supply other then real property that are of a type

customarily used by the general public or by nongovernmental entities for

purposes other than governmental purposes, and that has been sold, leased,

or licensed to the general public”.

In order to summarize COTS basis and determine COTS background, COTS

items may therefore include the following products, which are:

- Sold, leased, or licensed to general public;

- Offered by a vendor trying to profit from it;

- Supported and evolved by the vendor that retains the intellectual property

rights;

- Available in multiple, identical copies;

- Used without modification.

13

1. Introduction

Although COTS could bring significant benefits for Defence Organisations, using

COTS can invoke challenges. These benefits or challenges are also valid for military

goods and amongst them are:

- Performance in military environment;

- Costs for frequent upgrades;

- Integration of various commercial items;

- Cost of testing to ensure performance;

- Configuration Management (design changes are not in control by the buyer);

- Dependencies among system components;

- Limited control of frequency or content of COTS releases;

- Limited visibility of COTS items characteristics and behaviour;

- Market driven prices which being unpredictable (availability versus demand).

The extent to which COTS items are used in individual Defence projects depends

on the project and varies from project to project. For some projects, COTS could be

bought from one single supplier that can replace custom military equipment. For

other projects, military systems will be built from COTS being purchased from

different suppliers. This means that using or buying COTS equipment presents

unique challenges and requires some departure from acquisition “business-as-usual”.

Two of the most important areas for these items that are developed for the non-

Defence marketplace are ‘performance of trade-offs to meet Defence organisations

requirements and logistics support’..

Many commercial items were not designed to operate in the range of

environments where many military systems are exposed. It would be risky to mount a

commercial item being developed for low-level contaminant environments in military

equipment which is designated to operate in e.g. a desert area with high level of dust

or in environments where significantly high humidity, vibration or shock exist.

Reliability in these cases would evidently degrade substantially and the overall

performance of the military equipment would suffer. Thus, it is vitally important to

assess not only the potential cost savings using COTS items, but also any

degradation in the performance. In other words, it is essential to conduct trade-offs

analysis “Cost savings – performance”.

When COTS components are purchased, the Defence organisation rarely

receives any design data or assumes responsibility for configuration control.

Inherently, Form-Fit-Function procurement will be considered. The supplier in this

case then provides a certain way of logistics support.

14

1. Introduction

Logistics support becomes a less straightforward issue until the moment that a

COTS item fails. Should the customer limit his maintenance to remove and replace,

and ship faulty COTS items back to the supplier? If a COTS item is integrated into a

lager subsystem or system, does this failure possibly occur because of any interface

issues? Is there any method by which the customer can verify that this COTS item is

indeed defect before incurring the expenses and time delays associated with

shipping the unit back to the supplier?

In order to mitigate consequences related with the impact of logistics support on

costs and system readiness and availability, it must be determined during the

procurement phase which concept for logistics support is the most feasible and

practical for that COTS equipment. The logistics support spectrum methods could

vary from complete disposal of the item when it fails up to no organic support at all.

No organic support then implies that the contractor is to be held responsible for

supporting the COTS items throughout their life cycle. Logistics support could also be

combined with organic support and contractor involvement. Alternatively, it might be

only all organic support, which means that the Defence organisation is responsible

for supporting the item throughout its life cycle.

4.2. Field of research

The Defence Materiel Organisation (DMO) is a part of the Defence organization

which acts as a service centre that is responsible for materiel used by the Defence

organisation throughout its life cycle: from procurement up to maintenance and

disposal. The DMO is also responsible for creating and issuing internal materiel

policies within the Defence organisation. The DMO is at the service to all of the

operational users of the armed forces. Parts of the organization are clustered around

areas of expertise and types of materiel. This bundles strength, quality, experience,

and technologies in the area of materiel logistics services from available resources

within the Defence organisation. The organisation has three major categories of

materiel: for sea, for land, and for air1. The DMO is also responsible for inflow,

storage, and release of materiel. Being a military organisation, the DMO sustains

1 The Military Police (Marechaussee) is depending on the DMO for some of its assets and will not be taken into account in this research.

15

1. Introduction

knowledge about new and current weapon systems, including associated information

management systems.

Figure . DMO organisation chart

This bundling of logistics and materiel expertise ensures that the DMO clients receive

state of the art, well- maintained, safe and high-quality equipment. Figure 1 depicts

the organisation chart of the DMO. The headquarters of the DMO is located in The

Hague. DMO, logistics units and maintenance establishments are dispersed across

the country.

This research will be conducted at the Directorate of Logistics Agencies of

DMO and will put the focus on the Naval Maintenance & Service Agency (NMSA)

located in Den Helder. NMSA is the sustainment Agency having as a primarily

responsibility the maintenance of the maritime assets from the Navy Command

(CZSK). NMSA has to assure Materiel readiness of the maritime capital assets such

as ships and submarines in such way that they are ready to perform their missions.

In addition, it carries out work for the Army Command (CLAS) and the Air Force

Command (CLSK).

Figure 2 shows a more detailed organization chart of the NMSA.

Figure 2. Naval Maintenance & Service Agency organization

chart

The NMSA consists of the following divisions:

- Platform: hulls and hull related systems

- SEWACO: sensors, weapon and command systems

- Special products

- Logistics Services

The NMSA is involved in the design, construction and commissioning of new

ships using valuable maintenance experience of the own employees. It also provides

technical advice and support to its assets of responsibility anywhere in the world.

16

1. Introduction

Besides, the NMSA possesses an expertise in integration of weapon systems, based

on the principles of the Integrated Logistics Support (ILS).

One of the important missions of the NMSA is to maintain maritime assets during

the whole life cycle at optimal conditions and costs. Maintenance of propulsion,

electrical and electronic systems including weapon systems on board are, therefore,

the essential tasks of the NMSA. Other important missions of the NMSA are:

- elimination of Cost Drivers and Availability Killers;

- mission related alterations and modifications;

- alterations due to obsolescence, i.e. adaptations to the Fit-Form- and

Function.

The NMSA provides as well logistics support during the exploitation phase

including providing supplies to the operational units. The complete process is

supported by general and technical support services, which also take care about

docking.

In addition to the total package for maritime materiel, NMSA has at its disposal

expertise and tools for non-naval Defence assets such as:

- tools for calibration of some F16 systems ;

- maintenance of night goggles and night equipment for Commando Troops;

- maintenance of small calibre weapons;

- modification of anti-tank weapons;

- maintenance of guided missiles.

17

1. Introduction

4.3. Statement of the problem

4.3.1. General background information

The DMO has introduced in the beginning of 2005 Weapon System

Management (WSM) with the main objective to introduce a policy for managing the

maintenance of capital military assets at the lowest possible utilization cost. (Policy

Department, 10 December 2007). This policy aims to manage all material logistics

and financial processes, whereby the performance of a weapon system shall be

optimized during the whole life cycle. The basic assumption of the WSM policy is to

apply the principles of Integrated Logistics Support (ILS) and Life Cycle Costs (LCC)

which should lead to the most efficient logistic support during the whole life cycle of

the military asset (Policy Department, 10 December 2007).

Stavenuiter (Stavenuiter, 2002) specifies that ILS is an integral part of the system

life cycle: planning, design and development, testing and evaluation, production and

construction, utilization and maintenance, and phasing out. It covers “management

and technical activities to integrate support consideration into system and equipment

design, develop support requirements, acquire the required support, and provide the

required support during operational phase.”

4.3.2. Practical experiences with COTS

The NMSA (which organisation is shown in Figure 2) as a maintenance company

within the DMO has to assure that naval military assets are available for their

missions based on the principles of the ILS and WSM policies (Policy Department,

10 December 2007). The capability to maintain and sustain military assets relies

heavily upon a continuous availability of system components.

The NMSA has been confronted with the fact that COTS products and components

are often incorporated in products typically built for military purpose.

Some particular examples are:

- The Active Phased Array Radar (APAR) system integrated in the Air Defence

Command Frigates (ADCF) consists of four faces incorporating a total of 1792

transmit/receivers (TR) modules. These TR modules are COTS products.

18

1. Introduction

- COTS workstations and software products are integrated in the Combat

Management System (CMS) and Integrated Monitoring Control System

(IMCS) of the naval ships.

- A COTS based architecture forms the core of the Integrated Mine Counter

Measures System (IMCMS) on board of the Mine hunting vessels

4.3.3. Possible consequences on using COTS

“COTS characterizes standard products that defense and aerospace suppliers

offer off the shelf-from military-standard designs to commercial-grade components-as

opposed to custom products that are designed from scratch based on the customer's

specifications’’

Since COTS products have a shorter life cycle (Livingston, 2000) somewhere

from 6 months to two years, they are often subject of “…new version or being

dropped from supplier’s production line completely. This leads to the rapid

occurrence of obsolescence of commercial products”. (STANAG 4598, 19 April

2005).

Consequences of the frequent replacement of COTS products and their usage

during the exploitation phase became a crucial issue for the NMSA. During Depot

Level Maintenance (DLM), Line Replaceable Units (LRU’s) are repaired. If the LRU is

a COTS component, then the maintenance usually results in a complicated task

because of absence of information about the COTS item.

In the scope of the problem analysis, some interviews have been conducted with

ILS experts, employed at the NMSA. From these interviews, the following issues for

existing in-service support contracts came forward:

- There is no clear definition of COTS products/components in the agreements

with suppliers;

- There is no specification of COTS products regarding their design or internal

characteristics;

- Form- Fit- and Function (FFF) replacements are almost never available.

Consequences of the above-mentioned situation could be:

- Giving a chance to suppliers to declare any component of the asset as a

COTS one, resulting in an excuse of “non-supportability” of the particular

19

1. Introduction

component and proposal for adaptation of the system which then creates a

conflict of interests and leads to a quite complex negotiation process on how

to settle such cases;

- Overpricing of COTS components at later stages when components are

subject to obsolescence issue;

- Developing a monopolistic position or behaviour of the suppliers;

- Providing a minimal technical and logistics support of COTS components;

- Mitigating a warranty support regarding COTS products; unwillingness from

suppliers side to bear any responsibility for the system by placing all risks to

the NMSA, if COTS products are bought from any third party company;

- Starting modification projects caused by COTS components replacement

which leads to unforeseen life cycle costs and influences on the availability of

naval military assets;

- Changing maintenance strategy of naval assets because of the necessity of

periodic change of components in systems;

- Getting regular training for continuous development of personnel. Personnel

have to make more efforts every COTS Life Cycle period to choose new spare

parts or FFF COTS components.

The above list is surely not complete and might be extended with other arguments.

4.3.4. Internal policy and process

In general, policies and processes are essential. In case of COTS products with

their short life cycle, internal processes and organisational policy play significant role

because of their frequent replacement and impact on maintenance. Many

considerations must come together to make sure that the application of COTS

products and components will be successful, including important changes in the way

how naval assets are acquired and supported (STANAG 4598, 19 April 2005).

The initial acquisition process during the design phase of naval assets is

performed by the DMO in The Hague. During this process contractors and suppliers

are chosen to support the asset during its life cycle. Since the utilization phase and

maintenance process for these assets are performed at the NMSA in Den Helder, it

implies that any contractor or supplier that is chosen during the initial procurement

20

1. Introduction

phase, may not be changed or avoided during the utilization phase. From one side, it

seems an obvious and easy solution, if the NMSA deals only with one party. On the

other side, this leads to developing a monopolistic position of the suppliers. It is

particularly a case, if risks related to COTS product replacements and integration in

the existing naval assets are not initially stipulated in the main contracts. During

utilization and maintenance phase it is not always possible to cover these risks in the

in-service support contracts caused by lack of stipulations and not defined

responsibilities in the main agreements. Moreover, it is not considered that COTS

products subject to be quicker obsolete, which means that clear replacement strategy

must be defined.

From the current experience, it turned out that main suppliers purchase COTS

spare parts needed to support the asset from their own circle of different sub-

contractors or original equipment manufacturers (OEM) and then re-sale those ones

at the higher prices to the NMSA. There is no transparency at the moment how to

trace those OEM in order to purchase equivalent COTS products and components

directly.

Within the DMO there are presently two internal documents available referring to

COTS policies. The first document was issued in 2003 as guidance for the RNLN

(Notification within RNLN, December 2003). It was a first attempt to explain the

definition, the usage, and maintenance of COTS products and to list possible issues

that were raised during the preparation of maintenance and during the exploitation

phase of the asset. This document was supposed to be re-evaluated in 2004.

However, no updated version was published. The second document was issued in

September 2009 by the Policy Department (Instruction "'Obsolescence

Management'", 29 September 2009). This document has a temporary status, is

based on the main provisions stipulated in the STANAG 4597 ( "Obsolescence

Management"' STANAG 4597 , 21 April 2005) and two analyses, conducted by TNO

(R.C.T.Hans, June 2006) and NLR (Bardet, January 2003). This instruction is in fact

the first internal policy document dealing with with obsolescence problems. Yet, this

document is strongly based on theoretical stipulations of the STANAG and less on

actual practices encountered within the organization.

Further details in relation to the current COTS policy are addressed in the chapter

4.

21

1. Introduction

4.3.5. Acknowledgement of the problem

There are initiatives how to cope with obsolescence of COTS products, but these

initiatives should be updated with practical feedback on how to manage

obsolescence of COTS products that have already been integrated in assets

developed in accordance to military standards and specifications. This matter is

acknowledged by the experts of the NMSA , and by policy makers of the DMO .

Hence, there is an urgent need for a policy with respect to Obsolescence

Management, where COTS issues are addressed since internal COTS policy is one

of the inherent provisions to carry out an efficient Weapon System Management.

Based on the interviews with the NMSA experts ,, , , it can be concluded that

knowledge about aspects of COTS products is available within the organization,

though it is fragmented and not centralized. There seems to be a lack of coordination

on how to deal with potential problems arising from COTS usage. During the

interviews, it has been clearly stated that there is a need in knowledge sharing and in

a common database of the possible suppliers of spare parts and sub-systems.

4.3.6. Sub-conclusions and way ahead

The above sub-chapters have indicated sub-issues interconnected with each

other and derived from one general problem: absence of an unified and streamlined

COTS policy. From the problem study, it has been clearly observed, that a lack of

policy defining COTS as a group, describing how to deal with this group, indicating

hidden risks and giving suggestions how to mitigate these risks, has caused quite a

number of problems during the unitization and maintenance phase of naval military

assets.

The NMSA has encountered with obstacles, which could have been

surmounted, if new course to use COTS in military equipment and associated risks,

should have been considered beforehand during the design and acquisition phase,

and sufficiently defined in the contracts. Participation of NMSA experts in a new

design and acquisition phase has been very limited in the past, which resulted in lack

22

1. Introduction

of opportunities for design optimization and insufficient influence on performing

logistics support during exploitation phase.

The obsolescence character of COTS products means that an obsolescence

management of COTS products is a crucial issue for the NMSA. From the problem

analysis study, it became evident that contracts covering the obsolescence issues

require considerable investments. However, the outcome from the services is not

quite satisfactory for the NMSA due to a monopolistic position of suppliers with

respect to the prices for COTS components and their reluctance to allow a direct

involvement of the OEM.

Since suppliers are commercial companies with goals to get profit from their

business, it is of vital importance for the NMSA to not only re-consider, update, and

improve the internal procedures, but also to act professionally in negotiations with

suppliers in order to benefit from COTS employment.

It would be an improvement if the experience is shared and lessons learned

are implemented. However, existing knowledge about COTS products is still

incoherent and goals of using COTS are not quite evident. Cultural transition for

COTS deployment including new roles, responsibilities, and processes are not yet re-

considered.

An intention of this research is to analyze what is the effect of overcoming

these unforeseen obstacles in relation to COTS usage and their impact on the

effectiveness of the maintenance process. The reason why the study is only

concentrated on the effectiveness of the maintenance of the naval military assets is

that the effectiveness of the maintenance depends on many factors and in

particularly on well-defined maintenance support elements.

The other reason why a focus is put on the COTS maintenance and

maintenance support is due to the fact that maintenance is an integral part of the

utilization phase. In the literature, it can be found that 80% of the total life cycle costs

constitute costs committed during the utilization phase. Therefore, in the scope of this

research it will be examined, how the support elements especially for COTS products

are organized, what kinds of the issues exist, and how risks are covered in the

contracts. Upon availability of information, it will be investigated how prices for the

COTS components are developed in the scope of obsolescence contracts and why

FFF replacements are not available.

It is important to mention that this research has no purpose for cost

optimizing, yet to address a quality of the organized processes and procedures.

23

1. Introduction

The main research question, which is formulated in the paragraph 2.1, is

supposed to reveal the above-described problems or part of them. Disclosing the

positive or negative consequences of using COTS products and their impact on the

effectiveness of the maintenance process shall help to look at all benefits and

sources of problems and determine why these problems occur. Consequently, this

analysis will contribute to defining a COTS policy concept not only on the theoretical

approach, but on what is the most essential, based on the actual practices in the

organization.

Because of a tendency to use COTS is quite new for the DMO and being about

10 years, it is extremely important that the DMO shall act consistently from lessons

learned in order to proceed professionally with future projects.

24

2. Research Approach

5. RESEARCH APPROACH

This chapter specifies the scientific approach for this research. Here will be

addressed and explained how the research question is formulated and what the

objectives of this research are. A research model with its phases will outline the

whole approach and set boundaries for the research. Based on the research model

structure, the logical sub-research questions are formulated in order to achieve a

goal of this research project expressed in the main research question. Further, the

research strategy will be developed, whereby an explanation is provided for the

holistic case study. Holistic cases study strategy implies incorporating multiple cases.

The sub-chapter 2.6 defines these multiple study cases and the way in which those

cases are grouped for this research.

5.1. Research question

In section 1.3 “Statement of the problem”, organisational consequences related to

COTS items usage were addressed. All these issues are associated with logistics

support elements where maintenance strategy is one of the important bases. If

COTS are integrated in naval asset, a maintenance strategy for this asset has to be

elaborated based on the strategic policy developed within the organisation. The

maintenance strategy is to be assessed and adapted depending of the extent of

usage of COTS products and components.

Having examined the current policy related to COTS usage and support, and actual

organisational experience, recommendations and suggestions will be proposed on

what kind of improvements are to be implemented for the current maintenance

process for naval military assets.

Therefore, the research question for this research is expressed as follows:

What are the consequences of implementation COTS products and

components in naval military assets and what is the impact on the

effectiveness of the maintenance process?

25


5.2. Research objectives

The research objective is to provide suggestions and recommendations for

developing an internal COTS policy through:

- surveying the expectations of the stakeholders;

- understanding the issues which occur during the exploitation and maintenance

phase of military assets in relation to COTS;

- investigating backgrounds and causes of those issues;

- examining if the current maintenance policy in relation to COTS items is

effective (IST);

- determining principles and selection criteria to structure and systematize the

COTS process in order to mitigate risks related to the obsolescence character

of COTS products;

- analyzing the current process and make recommendations for Target (SOLL)

situation.

5.3. Research model

Upon developing the research, it is important to have insights into existing

theoretical understandings and to form a basis for elaborating proper research and

sub-research questions. Hence, an essential tool for getting such insights is to evolve

a conceptual model for the research, states Verschuren (P.Verschuren, 2007).

The theory of Verschuren describing how to conduct any policy research and to

build a model for research, has laid down a foundation for the conceptual

confrontation model, which has been solely elaborated for this specific research. This

model fits for a qualitative research, indicates, and compares an interaction effect

between variables (P.Verschuren, 2008).

A widespread and generally accepted scientific method is that of comparing

research objects. This is especially true approach to be used for qualitative forms of

research. In the comparison, it is recorded which similarities and differences are

26


established between objects of research. Based on the result of the comparison,

conclusions are drawn. In this particular case, a comparison is based on activity,

which is referred in this instance as a confrontation. The confrontation is a type of

relationship between two or more confronted entities, which further has found a basis

for the conceptual confrontation model.

The conceptual confrontation model for the COTS policy concept is depicted in

figure 3.

The Conceptual confrontation model focuses on the relation between independent

and dependent variables in such way that the following important aspects could be

verified:

- What is the outcome of the confrontation between X and Y?

- How is this confrontation result of X and Y to be seen in confrontation with Z?

- How can variable C be expressed as a function of X, Y and Z?

- How do variables X, Y and Z influence on the outcome C?

Figure 3. Conceptual confrontation model for COTS policy

concept

The confrontation model has been further worked out in the research model,

which combines all relevant elements, sources, study cases, and scientific analysis in

order to define a framework for this research. The conceptual confrontation model

shown in the figure 3 is embedded into the research model depicted in the figure 4.

The COTS policy concept has been elaborated based on the evaluation of the

maintenance effectiveness of each study case. Thus, the maintenance electiveness

(C) is chosen as a dependent variable. Criteria (Y), criteria (X) and experience from

the study cases (Zn) are independent variables, which explain variation in the

dependent variable (C). In other words, these are the causes.

27


Criteria (X) are derived from the scientific literature and determine here

prerequisites of effective maintenance in relation to usage of COTS products and

components. Through describing the actual situation with current policy, criteria (Y)

are defined.

Having compared criteria (X) with the criteria (Y), assessment criteria for effective

maintenance in relation to COTS have been developed. These assessment criteria,

defining a desirable situation and being a confrontation result of X-Y, have been

applied for the evaluation of each study case (Zn). In this way, it has been found

which problems indeed exist and what is the background of them. Results of these

evaluations have been used for the suggestions and recommendation for the COTS

policy.

The research model for COTS policy is divided in the following four phases:

Phase 1

• Review of different theories and research literature in relation to COTS. Those

theories are supposed to provide marginal values and criteria which could be used

as a basis for the policy assessment.

• Conduct preliminary investigation with experts of the NMSA.

• Review all internal instructions and policy related documents regarding COTS

management. Deduce how data from the internal documents is in conformity to

scientific literature. Develop dimensions for conceptual confrontation model

variables.

Phase 2

With the help of the research model, get insights in actual practices based on the

experience derived from four study cases (the list of these study cases is specified in

the section 2.6 Research data.

Phase 3

A relative comparison of the actual practices with the defined assessment criteria

shall provide an understanding of the background of the existing issues and

problems.

Phase 4

Assessment data will be processed from phase 3 in the form of suggestions and

recommendations for COTS policy concept.

28


Figure 4. Research model for the COTS policy

Theory and scientific literature review is described the chapter 3. The content of the

chapter 3 provides insights into existed knowledge about COTS, their risks and

impact on the maintenance. Results of the chapter 3 help to define criteria (X).

Internal policy documents are described in the chapter 4. Based on the results of the

chapter 4, criteria (Y) are subsequently determined. Consequently, both criteria (X)

and (Y) are compared in the chapter 5. Assessment criteria for the evaluation of the

study cases are derived from the outcome of the comparison between criteria (X) and

(Y) and are justified in the chapter 5.

29


5.4. Sub-research questions

From the main research question and research model shown in figure 4 the following

four sub-research questions could be formulated:

1. Which criteria for the effectiveness of the maintenance process as for

performance preparation will be considered in view of implementing COTS

products and components in naval military assets?

2. To what extent does the current policy comply with those criteria?

These first two questions are related to the part of the model shown in the figure 5.

The X-Y confrontation result “Assessment Criteria” represents in the figure 5 an

outcome of the accumulation of criteria X derived from the scientific literature and

criteria Y derived from the internal policy documents.

Figure 5. Relation of the research model with the sub-

questions 1and 2

30


The third sub question is formulated as follows:

3. To which extent does the maintenance process in each of the study cases

comply with the set of criteria derived from the research question 1?

Hereby a

confrontation between study cases (Z1, Z2, Z3, and Z4) and defined Assessment

Criteria are to be investigated. The part of the research model that is related to the

sub-question 3 is shown in figure 6.

31



question 3

32


Finally a fourth sub-question, leading to suggestions and recommendations is:

4. What have we learned from the analysis results of study cases (Z1, Z2, Z3,

and Z4) in a view of the maintenance process effectiveness in relation to

COTS? What are suggestions and recommendations for improving of the

maintenance process effectiveness and of the COTS policy?


question 4

33


5.5. Research strategy

Holistic case versus embedded case strategy has been adopted for this

research. In this case, the NMSA has been used as a single organisation as a whole,

within which four different projects were involved as embedded study cases. The

rationale of using four different study cases for this research was to focus upon the

fact to determine whether the findings of the first case occur in the other study cases.

Consequently, it is required to generalise from these findings.

The study cases have been examined regarding their practices with COTS

products and components. Pros and cons in every study case approach have been

analyzed and compared with the stipulated assessment criteria, which have been

derived from the scientific literature, theories, and internal documents.

For the result evaluation, the Gap analysis technique is used. Under the Gap

analysis technique is understood a method of the steps to be taken in moving from a

current IST state to a desired future SOLL state.

Those steps imply:

- Identify strategic objectives;

Assessment criteria have helped to define the strategic objectives and to obtain

important information about key areas and what is missing in these areas;

- Indentify current standings;

Information and data have been collected with regard to each of the strategic

objectives for each study case. The current standings have been defined, based on

the Gap analysis template. This template is described in the chapter 5 and extended

with the Gap analysis check list questions. In order to visualize an evaluation scale,

answers Yes, Partly, No, Not Applicable (N/A) are chosen to tick off the items from

the questionnaire.

- Create a plan of action;

After the information has been gathered, a plan for closing the gap between the

current and desired state of the organization could be made. Each “No” or “Partly”

reply means that a Gap has been indentified and action should be taken. In

conformity with the research model (fig.4) for this study, recommendations and

suggestions for the COTS policy have been outlined based on the Gap analysis

template.

- Back up the plan of action with data and analysis;

34


Findings from the Gap analysis template have been supported with appropriate data

and an analysis has provided a course of actions for improvement. The analysis of

the study cases Zn is to be found in the appendixes B, C, D and E.

The data collection techniques were various and they were likely used in

combination of each other. They have included data analysis, interviews,

observations at locations, and documentary analysis.

5.6. Research data

Research data has been collected from the sources as shown in the figure. 8.

Figure 8. Research subject and sources of the information

As already specified in the research strategy, four different projects were considered

as embedded study cases. These study cases are:

- Mine hunters (PAM);

- Hydrographical Survey Vessel (HSV)

- Air Defence and Combat Frigate (ADCF)

- Patrol Ship(PS)

The above-listed study cases have been divided in three sub-categories. The place

where the maintenance process is conducted, is taken as a distinguishing feature for

each category.

35


To category 1 where the manufacturer conducts the maintenance, belong the

following study cases:

- Mine hunters(PAM);

- Hydrographical Survey Vessel (HSV)

To category 2 where the maintenance is conducted by the NMSA, belong the next

study case(s):

- Air Defence and Combat Frigate (ADCF)

The study case Patrol Ship (PS) belongs to the third category. This study case is

considered as a new project being in the development. An intention to include the PS

study case in the list was to verify how the lessons learned from the past, have been

taken into newly developed and constructed naval military asset.

For every study case, a system has been chosen in which a considerable quantity of

COTS products and components are integrated.

Data has been obtained from portals on the intranet and projects documents. The

research has been limited to SEWACO equipment for each study case. Face-to-face

interviews have been planned and conducted with the representatives of different

departments of the DMO:

With the Naval Maintenance Service Agency (see fig. 2)

- Procurement

- Division SEWACO

- Division Platform

- Logistics Services (MATLOG)

With the Project Management

- ILS Management department

- Contract Management department

- Weapon System Management department

With the Policy Management

- Policy Department

The above-mentioned departments have been chosen for conducting interviews

because of their direct involvement into the maintenance process preparation and

implementation, collected knowledge and experience as far as COTS advantages,

disadvantages, and issues are concerned.

Project documents for each study case have included the following sources from

which information, evidence, etc., have been obtained:

- ILS study reports;

36


- Original contracts with manufacturer;

- In-Service Support contracts with manufacturer;

- Obsolescence reports;

- Price lists for COTS components;

- Quality plans;

- LCC reports.

37

3. Scientific literature review

6. SCIENTIFIC LITERATURE REVIEW

This chapter represents an overview of the examined scientific literature in

relation to COTS definition, existing COTS categories, type of risks being identified in

different areas, and how to deal with those risks for COTS based systems. A

comparison will be given what is the difference in a traditional and a COTS-based

system approach. Special emphasis will be put on the engineering activity area

where COTS deployment and sustainment are an integral part of it. Consequently, a

maintenance support approach will be addressed with a description of the essential

factors to be taken into consideration during maintenance and long-term support of

COTS. The cost of COTS and their obsolescence features will be separately

described and addressed in this chapter as being recognized in having the strongest

impact on life-cycle support.

6.1. The definition of COTS

The use of Commercial off the Shelf (COTS) products and components in large

systems has grown rapidly over the last years (Moriso M., 2002). Since in 1994 the

Secretary of US Defense William Perry has mandated to use COTS products to

support a flexibility and scalability of military systems (Perry, June 1994), hundreds of

studies have addressed this topic according to Claap (Claap J., March 2001).

McHale (McHale, 2003) has stated that from the time of Perry’s mandate “… most

military programs are saving millions of dollars by using COTS technology equipment

wherever possible, instead of designing every part of a program from the ground

up ...”.

Claap (Claap J., March 2001) has paid attention to the fact that at one side, by

using COTS there seems to be a cheaper, faster, better solution without any system

engineering and testing efforts. On the other side, a number of risks connected with

authentication, authorization, legacy systems risks, lack of control over upgrades,

capabilities, reliability, interface, configuration tracing, and many others should be

identified.

In spite of a large number of research papers about COTS, it is yet not clearly

determined what is understood under the concept COTS. It became clear from

literature that the definition of COTS is not unambiguous and allows quite a lot of the

38


different interpretations (Moriso M., 2002). The confusion with respect to the

definition has also been confirmed by Aartman (Aartman, 1999). In his report “What

is COTS technology?” issued by the Dutch National Aerospace Laboratory NLR, he

stated that there is a perplexity in the definition of COTS. Some sources imply under

COTS the individual electronic components while others suggest COTS to be

complete products and even processes.

In particular, Aartman (Aartman, 1999) defines COTS as products (systems,

equipment, software, components and spare parts), including processes, methods

and norms which are arisen from the commercial and industrial market sector. Moriso

(Moriso M., 2002) has tried to categorize COTS products characteristics and

admitted that the term COTS implies a broad coverage.

6.2. COTS categories hierarchy

Aartman (Aartman, 1999) and Baron (Baron, September 2006) have divided

COTS products depending on their development time and costs for the buyer. Figure

9 depicts the following COTS groups, their combination and hierarchy versus

development costs:

- Buy-off-the-shelf Commercial off the Shelf (COTS) which are unmodified, with

zero development for the buyer, no customization needed, and short

procurement lead products;

- Ruggedized off the Shelf (ROTS) are the COTS items developed by designer

for the extreme conditions;

- Militarized off the Shelf (MOTS) are COTS being modified to specific military

need;

- Commercial and Non-development items(NDI) integration: both of them are

not subject to any development process; NDI are developed according to

military specifications;

- Development with Commercial and NDI components and Full Scale

Development imply the high degree of development which is expensive and

time consuming.

39


Figure 9. Commercial to Custom Groups Hierarchy

As one moves up along the green line, typically development and procurement

cost, development time and specialization of the product will increase.

Based on the definition of each group, aforesaid groups could be merged into three

main categories:

Category 1 consists of commercial equipment which can be bought and mounted

into systems without any adaptations or modifications.

Category 2 consists of commercial equipment which can be bought, but it is subject

to a modification and should comply with a form- fit and function (FFF) requirement in

relation to the existing system.

Under Form is understood an unique and relevant physical characteristics (shape,

size, mass) that characterize a part for a particular use;

Under Fit is understood an ability of a part to physically mate with, interconnect to, or

become integrated with another part;

Under Function is understood an action that the part is expected to perform in

fulfilling its purpose.

FFF requirement implies that parts are interchangeable. A part is interchangeable

with another part when:

- The relevant functional and physical properties are at least equivalent in

performance, reliability, and maintainability;

40


- It can be used without requiring special procedures (such as selecting for fit or

performance) and without altering the part itself or any other part.

A part is often considered interchangeable with another when the form, fit and

function are identical.

Category 3 includes items that are to be developed by the industry with use of COTS

components.

A distinctive feature of COTS is their short life-cycle, which is usually estimated to

be around 6 till maximum 10 years (Livingston, 2000). That means that COTS

products will need to be frequently replaced in systems with a longer life cycle. Taking

into account that many military assets are expected to have a service life longer then

35-40 years (Livingston, 2000), or even be used for an extended time, their

vulnerability for obsolete subsystems and outdated technologies is clearly a risk

driver. Thus, the use of COTS products is at one side an attractive way to embed

successive generations of new technological products, but on the other side it is

complicated in connection with how to incorporate them in already existing military

assets.

41


6.3. Risks related to COTS

Using COTS products in complex military systems introduces risks that can be

related to the obsolescence character of COTS items. The literature recognizes the

following major domains where risks can occur, ,:

Operational requirements where risks are associated with using a COTS-

based system to meet the functionality and performance requirements of the

users;

Technical approach where risks are associated with the technical

characteristics of COTS products and their impact on the system into which

they must be integrated as components of a system;

Business strategy where risks are associated with the vendor of COTS

products, the need of continued availability of support for the products over

time, and funding profile over the life of the system;

Difficulty to evaluate the products where risks are connected with a lack of

detailed documentation before purchase. Evaluation is particularly difficult in

terms of Security, Interoperability, Robustness, Reliability, Availability,

Maintainability and Supportability;

Difficulty to keep the product under control where risks exist due to rapid

changes in technological developments, which lead to abrupt discontinuation

of products. It might have negative consequences in terms of logistics and

maintenance costs;

The “Secret or hidden Costs” where risks are to be considered in relation to

”Price versus Cost” to avoid any unnecessary COTS product functions. The

price must include guarantees, assistance, maintenance, run-time licenses,

upgrading agreements and property rights.

Livingston (Livingston, 2000) states that in order to mitigate the above-

mentioned risks, “..Resources have to be committed and combined engineering-

business strategies need to be implemented to sustain supply of obsolete parts long

enough to permit redesign and requalification”.

Rampino and Fiorilli have confirmed that in order to maximize the benefits and

minimize the problems associated with COTS based systems a number of

engineering analysis and trade-offs along the whole system life-cycle is required.

42


These authors have also stressed that the use of COTS products requires a specific

and systematic approach to avoid technical and project management problems and

high maintenance cost.

It is important to illustrate COTS benefits and risks in comparison to military

system developed in accordance with the customer specifications.

Development of the bespoke systems in accordance to customer specifications

cost initially more and it takes a longer time to deploy them. COTS systems

development should cost less because COTS components are already available at

the market from different sources. Therefore, COTS systems deployment can be

quicker and the buyer will be able to use a system that is built with state of art

technology sooner than in case of new development.

Parts selection to satisfy the customer requirements in case of the bespoke

system will be done based on the requirements to assure an application for robust

design. A screening will be identified for deployment at the military environment.

Support concept will be subjected to by the deployment in the extreme conditions.

For COTS based system parts, the selection process is done based on assertions of

a COTS manufacturer that claims a certain level of reliability in the military

environment. The latter is difficult to verify.

A system being under development or specially developed for military

deployment will have the Defence organisation involved in the design process, the

testing and design reviews in order to retain inherent RAM.. The lack of detailed

engineering and manufacturing data for COTS products could be partially blamed to

limited access to types of processes and process control procedures used by the

COTS manufacturer.

The availability of the bespoke system stays at the certain level and degrades

rapidly until the next refit or modification. For modification or refit, the parts, that are

purchased, have to ensure the desired RAM requirement. However, it often becomes

unprofitable for a part supplier to sustain the production of a particular product line in

small quantities. A renewal of the production line for often failed spares for military

applications drops rapidly because manufacturers do not see an adequate market .

An increased obsolescence rate became an issue for military systems as well.

Finally, those parts might be produced at a higher price which at the end increases

43


exploitation and maintenance costs of the military system. The cost of COTS is

further addressed in the sub-section 3.7.

The fundamental problem with COTS is rooted in the rapid product

obsolescence because new generation of COTS products are quickly brought to the

market,. Therefore, the level of maintenance support and availability of the spare

parts for the system, in the case of diminishing spares, becomes more costly.

Consequently, understanding COTS products obsolescence and related risks

requires development of mitigation strategies on how to limit their effects on system

maintenance. COTS obsolescence strategies subject is addressed in the sub-section

3.8.

6.4. Traditional versus COTS-based approach

Many literature sources emphasize that a systematic approach for COTS

products is essential. On the other hand, there is a limited number of researches

where such approach is completely elaborated. From that point of view, Carnegie

Mellon University’s Software Engineering Institute(SEI) in Pittsburgh (USA) is unique

because of their initiative to work out in detail how the use of COTS products affects

existing systems and which new processes are needed for successful use of COTS

products. This paragraph summarizes the main principles of this approach and

indicates some activities, which will be streamlined further in relation to the

maintenance process of COTS.

Notwithstanding the fact that COTS components are build to satisfy market

segment, still many organisation apply traditional linear processes as shown on the

left in fig.10. This traditional approach is referred as waterfall model where

development method is linear and sequential. Once a phase of development is

completed, the development proceeds to the next phase and there is no turning

back. Based on the experiences of the Software Engineering Institute (SEI) in

examining different projects, a fundamental change is required how the COTS –

based systems are engineered. This new engineering approach requires a

simultaneous definition and trade-offs in four spheres of influence as shown on the

right in fig.10.

44


Figure 10. Traditional and commercial to system

development

In a traditional way system, development starts from requirements, proceeds with

a thorough choosing system design and architecture, followed by implementing the

system, deploying the system, ending with testing and maintenance. “ ’In

commercially based system this paradigm is changed”, reports Place , Oberndorf and

. This change introduces significant challenges, which are to be supported during the

whole life cycle of the military system. Moreover, effective ways of using COTS

components requires a new way of doing business: new skills, knowledge, changed

processes, new roles and responsibilities.

New approach of the SEI implies looking at the continuous balance between four

spheres:

- Stakeholder needs and business processes;

- Marketplace;

- Architecture and design;

- Programmatic and risks.

Any of these four spheres may have impact on the three others. Thus, these four

spheres are interrelated and it cannot be proceeded without reconciliation of the

relationship or knowledge of three others. Furthermore, the activities that are carried

out for the system buit on from COTS products and components are revolving in

45


cycles means that tradeoffs will be repeated constantly through the whole of life cycle

of the system.

Stakeholder needs and business processes denote requirements (including quality

attributes such as performance, reliability), end-user business processes and

operational environment.

Marketplace denotes an available COTS technology and components and related

standards.

Architecture and design denote here essential elements of the systems and

relationships between them. These elements include structure, behaviour, use,

functionality, performance, different constrains.

Programmatic and risks denote management aspect of the process. These elements

include and consider cost, schedule, deployment and supporting, risk of changing

business processes.

A compilation between the four spheres of influence is depicted in figure 10

and a simplified representation of the synchronized-X-model is shown on figure 11.

The X –model has been developed by the Boeing Company and can be regarded as

an expanded version of the V-model. With the reference to INCOSE Systems

Engineering Handbook v.3.1 (2007) and other scientific literature related to the

subject of principles of system engineering, the V-model has been specified as the

one which provides a useful illustration of the system engineering activities during the

life cycle stages.

Dickerson and Mavris believe that the V-model is interdisciplinary. It depicts the

evolving baseline from the user requirements agreement to identification of a system

concept to definition of system components that will comprise the final product.

Blanchard underlines that a user requirement, which constitutes the baseline and

needs to be established from the very beginning, must be traceable from the top on

and down to the component level as necessary. This top-down approach with an

appropriate feed-back being incorporated, reflected on the both sides on the X-

model, is critical for the successful implementation of the system engineering

program where COTS components are used. It is established that these early

requirements have a great impact on the ultimate life cycle cost of the given system.

Time and project maturity moves from left to right. As shown in the X-model,

communication paths are now defined in two dimensions: feed forward/feedback

(horizontal dashed lines) and concurrent paths (vertical dashed lines).

46


Figure 11. X-model and four spheres of influence

Oberndorf has identified which activities fall into those four spheres of

influence. These activities are compiled with ones specified in the X-model and form

the activity area groups, which are depicted in fig. 12. Those activity areas are:

- Engineering;

- Business;

- Contract;

- Program-wide

Each activity area includes a number of activity sets depicted by blocks in fig. 12.

For instance, engineering area is straightforward and includes marketplace,

architecture and design, construction, system context, configuration management,

deployment and sustainment, evaluation. The contract activity area deals with issues

involved in contracting with main suppliers and integrators. Information from

engineering area is required for the business activity and vice versa. Business

activity area determines business process implications, developing cost estimates

and managing supplier. Program-wide activity area integrates the engineering,

business, and contract activity areas in the development and maintenance of COTS-

47


based system. COTS-based system strategy implies how a program governs all

other activities and their interrelationships. Due to changes in the COTS marketplace,

a program needs to evaluate its COTS-based strategy periodically and adjust it

accordingly.

Figure 12. COTS-based activity areas

Usage of COTS products and components represent a change of everyone in the

organisation, not only for the technical personnel. As already mentioned in this

chapter, new roles and skills are required. The more practices being used already in

the organisation, the easier to overcome cultural transition. Information sharing can

help to avoid mistakes of others.

Even though, the activity sets depicted in fig. 12 are combined in the blocks, they

form no process, but rather a model that can be applied for any project for further

detailed planning depending on the needs of the project.

Some activities are not different from custom-developed systems. For

instance, contract tracking and oversight activity are identical. The main idea is to

emphasize what is entirely new for COTS systems and has no counterpart in custom-

developed system.

Since this research is aimed at examining the consequences of COTS impact

during the sustainment (maintenance) phase of the naval military assets, further

48


attention will be paid only to the engineering activities area, namely on deployment

and sustainment. However, to look at the sustainment phase only without considering

any relation to cost factors and relationships with suppliers would not be seen as a

correct approach as far as COTS are concerned. Deployment and sustainment

activities encompass the delivery of the system to the end-users and support of the

system through routine maintenance.

The difference between a traditional and a COTS-based approach for the

sustainment phase is that for COTS-based systems there are continuous market

changes and new technology implement “construction” activities in the maintenance

process such as integration and test of COTS components. Traditional maintenance

and maintenance support approach is described in the appendix A.

COTS-based approach requires a constant process monitoring where it should

be thought over that, for instance, the availability of spare parts and support of the

replaced COTS could be limited to a few years. Besides, without constant trade-offs,

it will be unlikely that a supplier is going to exploit COTS only in the benefit of the

customer. The relationship between customer and supplier has also to be changed

considerably and both of them have to be able to agree on it. This causes a merge of

the traditional construction and maintenance caused by a higher frequency of new

component releases than expected or experienced in the past. All these aspects

result in a need to adapt the life cycle support phase of COTS in comparison with

traditionally developed systems.

49


6.5. COTS reliability , maintainability and availability considerations

Primarily owing to the advent of new technologies in design, complexities of many

systems have been increasing . COTS components may be embedded in the larger

customer systems. These components must be reliable, maintainable, and available

and must incorporate with larger system in order to give for the customer a benefit

from the advertised advantages as lower development and maintenance cost.

Blanchard emphasized on this that, if key design parameters such as reliability,

maintainability, and quality are sacrificed, the overall effectiveness of the system has

been decreased and the costs have been going up.

Despite the advantages of using COTS equipment, it should not be used without

fully understanding the implications of using it especially in a military environment.

The use of COTS in military systems is no longer dependant on an answer to a “’yes

or no” question, but rather to a question “to what degree’’. The Reliability &

Maintainability (R&M) activities for COTS are different than for new development

items. In accordance to ANEP-54 “’COTS hardware is less reliable then military

equipment”.

Commonly, COTS will not stand up to the full spectrum of thermal, vibration or

other environmental demands, often needed in military applications. In these

situations, protection is required to make certain that COTS systems are adequately

isolated and protected so they can properly perform. This conclusion has been made

in the Report of the Defense Science Board Task Force on Developmental Test &

Evaluation.

Since COTS component reliability is dependent on the environment, the

reliability of COTS items may differ significantly in military applications. Schneidewind

has indicated that COTS components are different from custom components with

respect to one or more of the following attributes: source, development paradigm,

safety, reliability, maintainability, availability, security, and other attributes. He has

also stated that COTS components are to be tested under operational profile

conditions in order to generate an empirical reliability assessment of COTS

components in the environment of the lager system and provide data for predicting

future reliability. However, if no documentation is available from the vendor, it is

50


difficult to find any resolution on how the other components of the system can tolerate

failed COTS component.

In case of maintainability, the situation is more complicated in accordance to

Schneidewind. If a failure occurs in a COTS embedded system, without proper

documentation the customer is not able to solve it. The vendor is not aware about the

problem as he is not on site. It leads to full dependence on vendor support to achieve

reliability and maintainability objectives.

In order to give an answer on the question “What will be the system availability

using COTS?”, Schneidewind suggests to store interchangeable and replacement

components. Maintenance costs will be kept low and availability is high by replacing

failed components with identical ones.

System availability is function of maintainability and reliability:

Availability = MTTF/(MTTF+MTTR) = 1/( 1+(MTTR/MTTF) (1)

In relation to COTS components application, Schneidewind has pointed out

that in order to achieve high system availability, COTS components are to be either

very reliable or there must be a strong supplier maintenance program. Schneidewind

has also concluded that, in order to employ COTS on mission critical systems,

decision should not be based only on their low development cost alone. Costs should

be evaluated on a total life cycle basis and RAM should be evaluated in a system

context. COTS suppliers should also consider making available more detailed

information regarding behavior of their systems, and certifying that their components

satisfy a specified set of behavioral properties.

Shaffer and McPherson concluded that “the maintenance concept for systems

using COTS products must also change accordingly. For COTS hardware this

typically involves adopting a lowest replaceable unit (LRU) e.g., circuit card, power

supply, display, disk drive etc. , swapping procedure and returning the failed item to a

manufacturer for repair or replacement’’.

51


6.6. COTS logistics support elements and COTS maintenance

COTS equipment must be maintained as well as custom-designed equipment, but

because of relatively recent adoption of COTS parts in dependable systems, issues

related to maintenance and reliability have not been well thought out. Drury points

the following important issues in relation to COTS maintenance:

- There is no difference in the importance of proper maintenance procedures

and proper reliability and dependability information between COTS and

military customs parts;

- It is vital that designers and users have reliability predictions for the COTS

equipment they will be using related to the functional environment. Often this

data is not available from manufacturer. Sometimes it may have to be

accumulated through field observation;

- Product quality needs to be monitored continuously to see if environmental

stress screening (ESS) is warranted;

- There is a significant difference between evaluating failure modes and

characteristics of COTS and custom designed equipment. In custom designed

equipment, a FMECA is used to evaluate the design of the system. With

COTS devices, this is generally not an issue. The level of detail about design

of a COTS assembly is usually not enough to perform FMECA, and therefore

evaluation must be limited to the interface of the assembly.

- It is important to keep track of failure and fault information for COTS

equipment. This because it provides better predictions about the use or failure.

Supportability of COTS must be assessed within the context of the key logistics

support elements. Figure 13 depicts such logistics support elements.

52


Figure 13. COTS logistics support elements

Possible maintenance strategy might include the following:

- Return faulty COTS to factory for repairs, possible with a pool of replacement

items to minimize turnaround time;

- On-site repair by contractor personnel;

- Provision of test equipment, procedures and parts for ILM and DLM.

Military maintenance facilities may be able to be used to replace faulty COTS. The

challenge is how to use in the best way existing facilities and support systems. The

next considerations are of importance:

- Degree to which the vendor or other military services already provide

maintenance support to customer;

- Responsiveness of such support activity (mean logistics downtime, need for

priority service, etc.)

- Degree to which military services are able to provide in-house maintenance

support, and the need in the technical training of the personnel;

- Need to minimize downtime

53


The DoD Guide for achieving Reliability, Availability and Maintainability (RAM)

recommends to consider RAM criterion for any COTS components selection. Further

the following criteria are to be taken into account if COTS components are used and

are to be sustained:

- Verify implications for the warranty and support if COTS components are to be

modified:

- Understand effects of a new application on the system environment;

- Indentify changes in maintenance and support concepts;

- Determine whether costs of required changes to the support system are

reasonable and affordable.

Pursuant to ANEP-54 the following implications of COTS inclusion in the generic

levels of maintenance and on the repair system are to be considered for keeping a

system maintainable:

Level 1: organic level maintenance

- Reliability data is a must for a feasible spares policy;

- Calculation of the needed inventory of spares is a difficult task;

- Documentation and training are to be available to perform LRU replacement

on the platform;

- COTS products must go via environmental stressing screening due to their

high level of infant mortality

Level 2: intermediate level of maintenance

The same requirements concerning spare parts inventory, training and

documentation apply as for the OLM.

Level 3: depot level maintenance (DLM)

- Repairs are to be performed by the vendor since the necessary maintenance

documentation is often not available

- Configuration management has to be the responsibility of the vendor.

If COTS components and products are used for a military application or are

already embedded in large systems, the program should carefully weigh important

factors including the environment, integration, maintenance, long-term support,

warranty, and integrated diagnostics of those items. These factors are summarized in

the chart 1.

54


Chart 1

Factor DiscussionEnvironment If the environment of the military application, is more severe than the commercial

application, reliability may be significantly less in the military environment.Integration COTS items may require new and different support requirements, i.e. support equipment,

item interface adaptors, the use of materials or fluids that are currently banned in that service. The acquisition activity and user should conduct a thorough analysis and risk assessment of integrating COTS items into user’s environment.

Maintenance For a true COTS item, the only military repair is to remove the failed item from the system and replace it with new. The manufacturer must do all maintenance of the COTS items for two reasons:

- Usually commercial suppliers will not sell the data needed to repair the item. To obtain such data, the government usually has to do reverse engineering and generate the data at considerable cost;

- Government attempts to do maintenance will normally void the warranty and the supplier will may refuse to incorporate whatever technology updates and being made new production items in the modified items.

Long-term support

Suppliers feel not obliged to support an item for a specific length of time. They may not provide much notice of plans to discontinue supporting an item. The government may:

- Choose to make a life-of-type buy;

- Use reverse engineering to develop a “make-to-print” specification and develop repair procedures;

- Identify another COTS item that is a ‘suitable substitute”Warranty Warranties of commercial items are usually null and void if the user attempts to modify or

repair the item. The user should determine if existing policy and procedures are adequate for the return of warranted items or if new policy and procedures are needed, especially for items that fail while system is deployed to an overseas location.

Integrated Diagnostics

Proposed COTS systems and units need to provide system status and functional information in compatible format to on-board and off system maintenance environments

Keil has specified the following set of seven criteria to be taken into account:

- Functionality;

- Reliability;

- Cost;

- Ease of use;

- Vendor reputation;

- Ease of customization;

- Ease of implementation.

Further the author has stressed that functionality, reliability and cost are significant

factors of importance in predicting COTS package value (see figure 14). However,

such attribute as functionality and reliability are the key drivers in the acquisition

process, whereas the cost becomes a key driver during the maintenance phase. A

cost increase during the maintenance phase might occur because of , for instance, a

55


system upgrade being not compatible with already embedded COTS components, or

a system upgrade with the latest COTS components just to receive a support from

supplier, and etc. Such situations could not be resolved if no documentation about

COTS components is available. Thus, it results in a follow-on event that the system

could not be maintained which in its turn leads to decreasing system availability.

Figure 14. Significant factors of relative importance

STANAG 4598 specifies the following aspects to be considered during

sustainment while applying COTS products and components:

- Need for the market knowledge and market analysis;

- Product integration testing and interfaces;

- Management of system modifications and upgrades;

- Logistics Support Contracts;

- Spares and configuration management process;

- Product technical support;

- Budgeting;

56


- Integrated Logistics Support(ILS);

- Technology refresh;

- Reliability, maintainability and quality assurance.

Information from different publications on which aspects of COTS should be taken

into consideration during sustainment phase of a military system is summarized in

Chart 2.

57


Chart 2

DoD Guide for achieving Reliability, Availability and Maintainability

Market knowledge Implication on warranty and support

Source Functionality Cost Maintenance

Product integration, testing and interfaces

Effects of new COTS application on the system

Development paradigm

Reliability Availability Manpower and personnel knowledge and skills

System modifications and upgrades

Changes in maintenance and support concepts

Safety Cost Suitability to the system Supply support and obsolescence

Spares and configuration management

Costs of required changes of the system

Reliability Ease of use Design Change Visibility Support and test equipment

Logistics Support Contract

Maintainability Vendor reputation Performance Training and training support

Product technical support Availability Ease of customization Obsolescence FacilitiesBudgeting Security Ease of implementation Lifecycle Storage and transportationReliability, maintainability and quality assurance

Reliability data Product upgrades

Technology refresh Vendor longevityDesign controlBackwards compatibility Availability of MaintenanceCompletionCompliance Certification Robustness to user InterfacesMaintenance OptionsTraining

58


6.7. COTS costs

Some hidden costs associated with COTS products during the whole life of the

system in comparison with custom developed system are very difficult to predict due

to the dynamic nature of COTS, unless a risk management program includes

proactive mitigation strategies specially oriented towards the unique risks of COTS.

Faster-better-cheaper solution of COTS in relation to development and acquisition

costs can be offset by the often more costly repairs of ineffectively managed risks.

Risks associated with COTS products and components cannot always be quantified,

but rather realistically estimated. There are costs associated with analyzing

performance, selecting product, and integrating, qualifying, supporting it. “The trend

towards COTS should not be driven by the product’s procurement cost alone”..

“One of the major lifecycle cost-driver associated with the use of COTS is a

lack of effective COTS-specific planning and budgeting. When program fails to apply

COTS risk mitigation strategies, the program then loses the advantage of proactive

planning and becomes increasingly reactive to emerging technology changes and

COTS product obsolescence situations. These situations limit management options

and force program to adopt sub-optimized and consequently more costly solutions”.

As it is already mentioned in the paragraph 3.5, the lack of reliable data and a

short COTS products life cycle, makes it difficult to predict accurately projected costs

for a life cycle support of a system, which might be threatened by the obsolescence

nature of COTS products. The cycle time between insertions of new COTS products

can be different based on the product class . For a TV monitor it might be about 10

years, whereas for a processing board it might be only 2 years. Thus, continuous

engineering tracking is to be done to avoid incurring the full costs of re-baseline in

order to make estimations of a complete update or a new release.

The graph shown in the figure 15 depicts systems containing Low or High

Level of COTS components during the total life-cycle of the asset. If the system is

developed with Low level of COTS components (<10%) then, maintenance strategy

and maintenance costs during the life-cycle of the system could be more or less

predictable in advance. If the system is developed with more than 50% of COTS

components, there is a considerable risk that associated costs will not be foreseen,

which creates in later stages substantial budget allocation issues. Inserting new

59


technology may cause the costs higher than predicted. At each potential insertion

point, the customer needs to make a decision as to whether he wants to incur the

immediate cost. The uncertainty area in the graph implies those unforeseen or

unpredictable costs. Therefore, in this case it seems appropriate to re-consider a

sustainment strategy on every 5-6 years depending on the situations with product

replacement. After this term, “..costs of supporting the original product configuration

escalated exponentially compared with upgrading to a new one”.

Figure 15. Life-Cycle strategy

60


6.8. COTS obsolescence

Obsolescence problems occur more often when using COTS products, and the

options offered to settle with them are limited. Obsolescence may occur for a

component or for the higher level assembly. In-service-support (ISS) contracts have

to contain mitigation strategies and stipulations on how to deal with obsolescence.

The most important prerequisite is that the customer has to be notified in advance

about obsolesce occurrence.

Strategies on how to react on this obsolesce notification may be classified as follows:

A.1. Fit-Form-Function replacement

In order to realize FFF replacement, the design and documentation data for

COTS products are to be available from the supplier. The compatibility of FFF

replacement is to be verified since the compliance is not always guaranteed by

the COTS vendor. Problems may occur if the same quality level is

unprocurable, then a substitution shall be activated.

A.2. Purchase of Life Time Buy spare parts

This strategy is applied when all components including spares and repair stock

will be purchased at the start of production.

In order to make an estimation on how many spare parts are needed during

the whole life-cycle of the asset, the reliability data for that part is required.

Since the reliability data for COTS is still an issue, there might be a risk that

spare parts will be bought either in insufficient quantity or an excess of them

will remain in stock when the system has to be phased out.

There is another risk which comes from the possibility that another part used

on the same assembly might become obsolete. If this occurs, the utility

associated with the Life Time Buy would be neglected. This solution is to be

avoided if possible. It can be applicable when a specific batch of items is in

production and their life cycle is well known and agreed.

A.3. Last Time Buy

Upon obsolescence notification by the supplier, a purchase of components is

initiated to cover all future demands for the program including spares and

repairs. This solution is applicable to a mature equipment and in isolated

events only, but not for new design/re-design. It is also applicable only if all

obsolete components on a module can be removed by the last time buys,

otherwise re-design shall be considered.

61


A.4. Substitution

Substitution is a replacement by a part with acceptable non-compliance. This

strategy is applied when no alternative components are available and a

deviation can be accepted by the Customer.

A.5. After-market supplier

This strategy is applicable for mature equipment and implies a purchase from

a supplier who has obtained the rights and facilities to carry on manufacturing

the part from the original manufacturer.

A.6. Upgrade sub-systems with newer technology

This solution requires the whole process of specification, selection,

procurement, design and integration.

A.7. Inventory survey

This solution is applicable for LRU and SRU which are not in production any

more, but still in service. By using internet tools, it shall be possible to allocate

obsolete items being for sale as excess stock.

62


6.9. Effectiveness of the maintenance process

‘t Veld has paid attention to the following criteria in relation to process

analysis:

- effectiveness

- efficiency

He has also suggested that in order to define if a process manifests any

effectiveness, it is necessary to determine a relation between end results of the

process in comparison with an initially specified norm. It means that the end result

has to comply with the result being aimed at.

Process effectiveness definition, in accordance to ‘t Veld is:

Effectiveness is a relationship between achieved result and intended result.

This statement is illustrated in the following formula:

, (2)

where

R actual – achieved result

R norm – intended result

If the achieved result R actual is better than intended R norm result, consequently the

effectiveness of the process is higher than 1 (or higher than 100%). If the achieved

result is lower than intended one, then the effectiveness of the process is smaller

than 1 (or smaller than 100%).

Blanchard and Stavenuiter have recommended to apply a Total Quality

Management (TQM) philosophy to evaluate the effectiveness of the process quality.

Stavenuiter defines TQM as “.. a total integrated management method that

addresses system/product quality during all phases of the life cycle and at each level

in the overall system hierarchy. It provides a before-the-fact orientation to quality, and

it focuses on system design and development activities, as well as production,

manufacturing, assembly, construction, logistics support, and related function”. The

TQM method can be used during the whole life cycle of the asset and links human

capabilities to engineering, production, and support processes. Process

management, process improvement and process measurements are fundamental

management approaches, which are to be used as appropriate by all managing

actors.

63


Therefore, maintenance process efficiency can be evaluated reflecting the

degree to which the logistics processes have been balanced to provide a deployable

and operationally effective system. Achieving process efficiency requires early and

continuous emphasis on the various logistics support processes along with the

design consideration.

Supportability analysis (SA) is an iterative analytical process by which logistics

support for new or modified system can be identified and evaluated.. In figure 16 all

inputs for the supportability analysis are depicted.

Regardless of the life-cycle phase, effective supportability begins with

development sustainment requirements to drive the maintainable and affordable

system to operational effectiveness. A key product of supportability analysis is the

maintenance plan which evolves all sustainment requirements throughout the life-

cycle. SA during the sustainment phase can help in adjusting the program. This

includes using SA to identify areas in the supply chain where performance is affecting

material availability, increasing ownership costs or missing areas of potential

improvements/savings.

64


Figure 16. Supportability analysis

Figure 17 shows how the maintenance process efficiency is related to the system

operational effectiveness. The relationships illustrated in figure 19 are complex. Thus,

all elements from the figure 17 are interfaced in more relative connections shown in

figure 18. In figure 18, it is clearly seen that all supportability elements influence the

process aspect, which in its turn can have an impact on supportability. If reliability

drives the maintenance requirements, the implemented maintenance process and

the quality of the spare and repair parts can affect the consequent reliability. In

addition, how the system is operated will influence the reliability and the logistics

processes can influence both of them. Finally yet importantly, each of the design and

process aspects drives the life-cycle costs. Achieving an optimal balance across

these complex relationships requires proactive, coordinated involvement of

organizations and individuals from the requirements, acquisition, logistics, users

along with industry.

Figure 17. System Operational Effectiveness

65


Figure 18. System Operational Effectiveness

Interrelationships

66


6.10. To recap

After reviewing of the scientific sources and literature in relation to COTS

products, it became clear that COTS products bring quite a number of advantages

and at the same time a variety of risks. The definition of COTS stays still ambiguous

and implies many interpretations with a broad coverage. However, in this research

the definition of Aartman and his COTS hierarchy structure have been taken and

accepted as the most appropriate ones, which covers distinctive features of COTS.

Since the COTS products are associated with unique risks, it is essential to

elaborate internal mitigation strategies to cope with these risks and conduct

engineering analysis during the whole system life cycle, .

The mitigation strategies have to be considered in the scope of the COTS-based

activity areas: engineering, business, contract, and program-wide,,,,. These areas

have been chosen based on experiences collected from the different programs and

projects. Thus, they reflect lessons learned which are of importance for the DMO for

elaborating of own COTS policy.

In relation to COTS maintenance, it has been outlined which issues and

important factors are to be taken into account and how to organise the logistics

support elements to maximize the benefits of COTS usage and mitigate related risks.

Having reviewed a large number of scientific sources and articles, based on practical

experience with COTS products, an overview of the essential elements have been

summarized, which play an inherent role for effective COTS life cycle support (chart

2). For example, COTS obsolescence and inadequate planning of the costs could be

probably seen as the major life cycle cost driver, which in its turn affect on the

effectiveness of the maintenance process.

It has been suggested to evaluate the effectiveness of the maintenance in the

scope of the Total Quality Management (TQM) philosophy, and focus on the

supportability analysis elements in combination with the aspects, which are stipulated

and described for COTS-based activity areas: engineering, business, contract, and

program-wide.

The findings from the literature have formed a basis for defining criteria (X) which

have been described in the sub-section 5.1.

67


68

4. Current policy at DMO

7. CURRENT COTS POLICY AT DMO

This chapter gives an overview of the existing guidelines being applicable at

DMO. The aim of this chapter is to summarise the main stipulations of the internal

policy documents and define which criteria are to be chosen for the assessment of

the study cases.

The sub-chapter 4.1 describes in details the material logistics guideline

“Obsolescence Management”. Sub-chapter 4.2 summarizes the main stipulations of

the Guidance MEDDMKM 23 which was issued in 2003. Sub-chapter 4.3 makes a

summary of conclusions, which are needed for possible revisions of these internal

documents.

7.1. Material logistic guideline “Obsolescence Management”

The current COTS policy by the DMO is defined in the material logistics guideline

“’Obsolescence management”’. This guideline was issued in September 2009 by the

policy making department of the DMO. It is based on the main provisions as

stipulated in the NATO STANAG4597 “Obsolescence Management” and two

researches conducted by TNO (R.C.T.Hans, June 2006) and NLR (Bardet, January

2003).

The guideline has a temporary status until ILS plans for the military systems are

to be amended. However, there is no clear schedule specified on which term those

plans are to be revised. It is also not clear when the current ILS guideline is

supposed to be adjusted in relation on how to deal with obsolescence issues.

The guideline states the future impacts of the obsolescence on:

- Performing of the maintenance and modifications;

- Availability of assets;

- System and its sub-systems.

The guideline mentions that it is known that by commercially purchased systems,

the risk of obsolescence is higher than for the military specific systems. This

69


statement is supposed to be underpinned with more arguments and facts in relation

with the DMO experience.

Obsolescence is caused by a production stop or changes in the product

components.

The guideline stipulates the following criteria and conditions to be considered to

prevent issues connected with obsolescence during:

1. Procurement phase

1.1. Make sure that the contract with a supplier has a clause that stipulates

all consequences of the obsolescence;

1.2. Consider an assessment of risks where costs and profits aspects shall

be evaluated;

1.3. Keep supplier responsible for obsolescence management;

1.4. Indicate in integral ILS plans how to deal with obsolescence issues

during the life-cycle of the system.

2. Exploitation phase

2.1. Conduct registration and analysis as soon as obsolescence issue

appears;

2.2. Make sure that sufficient stock of spare parts is available for items

which have been subject to obsolescence;

2.3. Conclude In-service contracts with third parties and guarantee that

obsolescence issues are the responsibility of the suppliers;

2.4. Elaborate an obsolescence management plan.

In order to develop an adequate obsolescence management strategy being either

reactive or proactive the guideline recommends applying a so-called risk matrix. Risk

matrix shall take into account possible costs, impact on the availability of an asset

and a probability of obsolescence occurrence, probability of obsolescence occurring

in terms of technology development or of the introduction of new legislation. These

risks are to be assessed during the pre-acquisition phase in order to minimize

operational risk in relation to obsolescence. The outcome from this matrix shall lead

70


to defining an adequacy of the applied policy. The guidance does not indicate exactly

how and with which generic tools and analytical methods such analysis is to be

carried out. If this task is supposed to be assigned to the supplier, the literature

indicates quite a number of analytical approaches used for obsolescence control.

The guideline does not suggest which department shall be responsible for the

registration and analysis of the obsolescence issues.

7.1.1. Implementation of the Guideline

In conformity with the conditions of the STANAG 4597, the obsolescence

management plan is essential to set up in order to prevent issues related to the

obsolescence. The guideline suggests two approaches on how to implement the

obsolescence management:

- External approach whereby a supplier will be kept responsible for solving

obsolescence problems;

- Internal approach whereby a system manager being responsible for the

system will be a controlling and managing party of obsolescence problems in

case if a proactive strategy is chosen.

Subsequently, the guideline states five options for internal obsolescence

management:

1. Use a reactive approach which means waiting until an issue or a problem

occurs.

2. Define the system breakdown structure with getting a quick overview of

sub-systems and parts relations. Determine consequences on un-

availability of those sub-systems/parts in case of occurring of

obsolescence.

3. Monitor all components and spare parts in the system in order to spot an

obsolete component in advance. Define all pre-conditions and

responsibilities in the contracts with suppliers.

4. Define the modifications schedule whereby the obsolescence problems will

be solved.

5. Apply Life-Time –Buy strategy

71


Thereafter, each of the above-mentioned options of the obsolescence management

is worked out in the guideline.

Under option 1 the reactive strategy is suggested. The reactive approach is

recommended in case if from the risk analysis is concluded to conduct a non-active

obsolescence management. It is mentioned in the document, that support for

instance is to be searched at the market whereby spare parts should comply with

FORM-FIT FUNCTION (FFF). However, it seems that this approach is rather

proactive than reactive. COTS FFF compatibility implies paradoxically an active

approach, where all possible risks connected with compliance, compatibility and

interfaces are to be taken into account. To oppose risks related with implementing

COTS components, a broader method has to be evolved and implemented.

The third option recommends defining all pre-conditions and responsibilities

regarding obsolescence in contracts with suppliers. Nevertheless, the guideline does

not specify which aspects are to be taken into consideration. Absence of such details

might be explained by the temporary status of this document.

7.1.2. Sub-conclusion to the implementation of the Guideline

The guideline is a first attempt to pay attention to obsolesce problems within

the DMO. Yet, this document is primarily based on theoretical stipulations of the

STANAG and less on actual practices encountered within the organization. This

document defines a general strategic management policy (see fig.19) which has to

configure long-term objectives of the organization and which gives an answer on

WHY this policy is so essential.

Figure 19. Strategic, tactical and operational levels of

management

Management on the tactical level is missing. The tactical plans which give an answer how to

plan and control individual organizational functions for improving obsolescence issues in

short or medium term are still to be elaborated. Operational level of management is available

at the NMSA, which focuses on ensuring that personnel at the work floor is instructed

72


correctly on the maintenance jobs to be performed at any particular time and that they are

provided with required material, tools and other facilities to get on with the work..

7.2. Guidance MEDDMKM 23

7.2.1. Implementation of the Guidance

This guidance MEDDMKM 23 version 1.0 was issued in December 2003 and

gives a broader overview of different aspects in relation to the use of COTS products.

It states that COTS are not only used by the platform systems, but also by the C4I

systems. It indicates that COTS products are cheap in acquisition due to applying of

new technologies. However, because of their short life-cycle period being about 5 till

6 years, a frequent replacement is essential. This guidance outlines different issues

raised during preparation of maintenance and in course of exploitation phase of the

asset.

The guidance refers to the following documents:

- STANAG 4597 “Obsolescence management”;

- STANAG 4598 “Guidance on the use of Commercial off the Shelf (COTS)

Technology”;

- Regulation “Supply management” 3VVKM16;

- Publication “ Integrated Logistics Support(ILS)” MEDDMKM 11;

- Regulation “Modification” MEDDMKM 22;

- Publication “Configuration management”;

- Regulation “’ Exploitation plan”;

- Regulation “ Maintenance concepts “

The responsibility for the contents of this guidance lies with the MATLOG Department

of the DMO. It was aimed to evaluate this guidance at the end of the 2004.

MATLOG issued an update of MEDDMKM 23 in order to alter of the modification

process since implementation of weapons systems management (WSM) in

2010/2011. Between 2003 and 2011 there was no evaluation on paper. Change of

MEDDMKM 23 is also needed to be prepared for ERP/SAP implementation in the

organization.

73


7.2.2. Summary of the Guidance stipulations

Using COTS means use new “state-of-the-art” technologies. It means that big

production quantities of components could result in a low price or cost of

procurement.

The development of a sustainment strategy depends on the following classification of

COTS:

- COTS which can be immediately utilized, whereby minimum adaptations are

possible;

- COTS products which are purchased from different suppliers for a developed

system. In this case the situation is complex and unique configuration is

required such as in the case of MILSPEC-configuration.

In general,, COTS products become quickly out of date because of rapid

technological progress. Therefore, it is important to consider at the earliest stage how

COTS products are to be replaced in the future and which effect will be created in

maintenance processes caused by the obsolescence character of COTS products.

During the procurement phase, replacement strategies are to be developed in

conformity with ILS procedures. In order to avoid complexity when COTS

replacement is required, it is wise to strive to apply open standards and have control

over interfaces. The replacement process can take place during a modification of the

system containing COTS.

The guidance stresses an importance to conduct a market research and collect

knowledge over different technological domains. It plays an essential role while

estimating the obsolescence problems in future maintenance phases. Good business

relationships with suppliers and exchange of the information could support and

stimulate developing insights for the coming maintenance strategies. The market

research is to be a part of the performed ILS analysis during a procurement phase.

Hereunder, the main essential elements for the COTS strategy which are to be

considered during all phases of the life-cycle of naval assets will be described shortly,

as derived from the field research of experts from NMSA.

1. Procurement phase

COTS products are purchased without any design details. Suppliers hold those

details and are reluctant to release or share that information. For instance, the

74


source code of software is a good example of such case. Therefore, during the

procurement it is essential that the supplier will be involved in the discussion about

an implementation, support and replacement of COTS products as early as possible.

The procurement phase is to be divided into sub-phases. A keen attention is to be

paid for the following aspects during:

Preliminary study phase

From the market research it is supposed to be verified which products comply with an

operational need:

- Only COTS products available;

- Products developed in conformity with MILSPEC;

- Both COTS and MILSPEC products are available;

- Products not available at all; thus they have still to be developed either in

accordance to a COTS or a MILSPEC strategy.

Study phase

If a choice is made for COTS products, then the subject of an applicability of COTS

products has to be continuously monitored during all following phases. The

consequences of their implementation are to be examined and worked out. A

replacement strategy is to be set out in global lines which will be deeper elaborated

during following phases. Participation of the supplier is hereby absolutely necessary.

Procurement preparation phase

In accordance to the ILS procedure all essential and non-essential components are

to be listed. Since a choice to implement COTS has been made, the decision shall be

taken if the maintenance of the final system is to be outsourced. Results of such

decision(s) are to be in compliance with configuration management of the definite

system. The supplier has to state how he is assumed to communicate with the DMO

regarding obsolete items and modifications.

75


Production phase

During the production, the required knowledge shall be collected by different actors of

the DMO with respect to different COTS technology fields. Consequences for the

own organization are to be worked out and implemented.

Requirements

During the description of the requirements, it is important to take into account that

COTS systems or sub-systems being built from COTS products have as an average

a lifecycle period of maximum 5 or 6 years. This means that during the whole

lifecycle of a ship some systems are to be substituted at least 3 or 4 times. However,

during the period to establish requirements, considering the realization time of those

requirements, technological developments are further progressed. Thus, it could be

that systems might be already obsolete before being put in service.

Initial design

Development of any system with COTS components is not as complex as the

development of a comparable system in accordance to MILSPEC. Therefore, it is

essential to keep in mind a possible interchangeable modular design. Modules which

could be expected to become quickly obsolete are to be regarded as a “black box”. In

this way it can be avoided that those systems become obsolete during long-term

realization projects.

Funding for systems containing COTS products shall be allocated under the

implementation costs and shall also take into account the costs related to the COTS

product interfaces and other adaptations.

Preparation of the exploitation

Usage of COTS shall have an influence on the utilization costs. Those costs could be

higher because of frequent replacement of COTS. The outsourcing of the

maintenance seems often to be more efficient. It is recommended to collect and

76


register knowledge from user’s experience, failure behavior and availability of the

asset to analyze how the maintenance could be more functional organized.

In- service support (ISS) or maintenance contract

The advantage of concluding such a contract with the supplier is to avoid any full-

sized changes of the own maintenance organization. The ISS contract must be

concluded for a definitive period with a clear agreement about a fixed price for

services and the expected performance level from the supplier. In this way availability

and reliability of the system will contractually be stipulated.

Disadvantages of an ISS contract are:

- Dependability from one supplier or service provider;

- Risk of increasing the life-cycle costs as soon as contract period is completed;

By conclusion of an ISS contract, special attention is to be put to:

- Obsolescence issues;

- Software issues( i.e. redesign of application software by non compliancy with

operating system software, software drivers or with the new COTS hardware);

- Elaborating of replacement strategies for COTS components and products;

- Test equipment or any other diagnostic devices;

- Configuration management( software and hardware);

- Contract type with supplier or service provider.

COTS Replacement strategies

The guidance recommends applying two types of strategy:

- Reactive;

- Proactive.

In order to apply any of the above strategies the following factors are of importance

to consider:

77


- Impact;

- Costs;

- Risk.

A choice of strategy will be determined depending on a combination of the estimated

above-mentioned risk factors. For the reactive strategy all three risk factors of impact,

cost and possibility of occurring are low. For the active strategy all risk factors are

high. Therefore, it requires constant alteration of the exploitation strategy based on

the experience data and facts. The supplier has to be involved in the change process

since the he has to have a leading role in solving of obsolete issues.

2. Exploitation phase

During the exploitation phase, COTS components are sensitive to be replaced

frequently. Thus, a replacement strategy is to be discussed and agreed upon during

technical meetings. It should be verified if systems containing COTS products comply

with the functional requirements. It is of importance to allocate funding in-time for

COTS procurement and replacement. Costs shall be allocated both in the

investment- and in the exploitation budgets. Budget allocation depends on the fact

from where requirements for COTS replacement are initiated.

Replacements could be classified depending the level of alterations as follows:

- Small-sized replacements are to be performed in accordance to the FFF principle.

- Big-sized replacements have a considerable impact on the system and therefore

are to be conducted in accordance to the strategy that has been determined in

advance. Involvement of the supplier and a market research are important.

- Modifications shall be performed in cases where FFF replacements are not

possible any more. Costs for the modifications for COTS replacements are to be

allocated in the exploitation budgets.

Spare parts and configuration management are inherent domains to be contemplated

in advance. In order to achieve certitude that COTS spare parts are compatible within

the current configuration of a system, required testing is to be executed by the

78


supplier. By means of configuration management the required configuration of

hardware and software in the system is to be supported and to be compiled with the

allowed combinations of operating system- hardware-software.

7.2.3. Sub-conclusion to the implementation of the Guidance

The guidance starts with a written statement to introduce COTS, explaining their

origin and usage. It gives a comparatively broad overview of COTS issues and

contains short descriptions of areas of attention, possible replacement strategies

during the whole life cycle of the system starting from procurement until the system

phase out. In addition, the guidance contains appendixes divided per life cycle

phases of the system where more detailed explanation is given on factors, which are

to be taken into account in case of COTS usage.

Information in the guidance is often repeated, therefore making it less conveniently

arranged for a reader. It would be much better readable and understandable if the

guidance structure would be revised, and if knowledge acquired from the appendixes

would be re-clustered with data in the main part.

In the guidance, strong attention is paid to initiate collecting market knowledge,

important aspects of concluding contract with service providers and strategies to be

applied during the exploitation phase of naval assets. However, there is no indication

which department has to be held responsible for collecting and monitoring new

developments on the market and how it is supposed to be documented. In addition,

the guidance attributes tasks and responsibilities to different departments in the

organisation. Since the DMO was subject of a reorganisation in 2006 and is re-

organised again in November 2011, it is of vital importance to bring this guidance

back in line with the organisation.

7.3. Conclusions

The material logistics guideline “Obsolescence management” may be considered

as a good initial attempt. However, it needs to be improved with practical experiences

and analysis from the operational level. This guidance indicates priority domains of

the DMO such as electronics, avionics and optronics. However, software

79


obsolescence also plays a significant role during deployment of a military system that

is also to be considered as one of the DMO domains and should not be left aside.

On the other hand, from this guideline, important criteria can be derived which are

to be used as attributive characteristics for developing of a COTS policy. These

criteria are described in the chapter 5 “Defining assessment criteria”.

Both internal policy documents material logistics guideline “Obsolesce

management” and the guidance MEDDKM 23 give sufficient background for

establishing an internal policy on how to deal with COTS issues during the

sustainment phase of naval military assets. Experience of the NMSA experts could

provide a vital input for updating of both documents based on the facts and

evidences collected from actual knowledge, direct personal participation or

observation from different projects being in an exploitation phase for the moment.

80

5. Defining assessment criteria

8. DEFINING ASSESSMENT CRITERIA

This chapter addresses the main criteria to be used for assessing the study cases.

Since the conceptual confrontation model (fig.3) has been chosen as a basis for the

research model (fig.4), this chapter shall indicate and justify why these particular

criteria have been selected. Consequently, a comparison between criteria (X) and (Y)

will be worked out and outcome from this comparison will be justified.

In accordance to the research model depicted in figure 4, criteria (X) are defined

based on the scientific literature review described in the chapter 3 and criteria (Y) are

determined from the internal policy documents described in the chapter 4.

8.1. Criteria (X)

With reference to the chapter 3.4, where COTS-based activities model has been

described (fig.12) in terms of four major activity areas, criteria X for evaluation of the

study cases have been consequently divided in four sections:

- Business activity area;

- Engineering activity area;

- Contract activity area;

- Program-wide activity area

Since this model has a generic application and has to be evolved depending on the

needs of the specific project, special importance in this particular case has been

given to cover all possible risks derived from COTS products nature.

Seeing that the emphasis in this thesis is put on the maintenance activities, a

selection work is built upon good representation of the engineering activities involving

processes or variations of them, where COTS products create concerns or risks,

which shall be taken into consideration.

Criteria selected for the business activity area cover aspects associated with

managing supplier including intergovernmental supplier relationships, COTS costs

and method of control. Such criterion as “COTS definition and COTS categories”

require an information from the engineering area.

Criteria derived from the engineering and business activity area are very essential for

the contracting activity area in terms of contract aspects, requirements and terms.

81


Contract activity area criteria address here COTS issues in light of supplier support

responsibilities and planned contract reassessments in order to accommodate a

possible process mismatch.

Criteria for the program-wide activity area are selected in view of their financial

consequences. One of them is a “COTS technology insertion and its planning during

the life-cycle”. It covers a strategy which is interrelated with technology refresh,

source of components, budget allocation and reflecting it in contracts with suppliers.

The purpose of including such criterion as “Risks identification approach and impact

on the maintenance” was to identify COTS-related risks as early as possible, adjust

maintenance strategy and plans in order to manage those risks. Criterion “COTS

price assessment” can be similarly included either in the business activity area or in

the program-wide part. Due to definite financial consequences of the COTS prices, it

is decided to consider this criterion under the program-wide part activity area.

Criterion “Knowledge management, information sharing about COTS practices,

performance, and operation” has no direct financial aspect. However, it has been

included here in view of the statement “information costs money”. Information sharing

concerning new products on the market, failure trend reports, obsolescence

approaches, COTS risks mitigations techniques during sustainment phase,

verification of the reliability data or lessons learned from other projects could be

useful learning tool within organization.

Criteria X , which have been selected because of their direct relation and influence

on to the COTS maintenance and their cause-effect on how the effectiveness of the

maintenance is to be achieved, are listed in the chart 3. These criteria may be also

considered as prerequisites for optimizing of the maintenance process where COTS

products and components are involved.

Chart 3

Activity Area Assessment criteriaBusiness COTS definition and COTS categories

Relationships with suppliersCOTS costs and method of control

Engineering Supportability elements analysis for effective maintenance conceptCOTS ILS supportTest equipment support COTS reliability and its verification method during sustainment phaseCOTS maintenance and replacement strategyCOTS obsolescence strategy( hardware and software, i.e. different strategies)

82


COTS failure and fault registrationTechnical developments on the market

Contract Main delivery contract with supplierType of support contract with supplierContract surveillance and evaluation approach

Program-wide COTS price assessment Risks identification approach and impact on the maintenance COTS technology insertion and its planning during the life-cycleKnowledge management, information sharing about COTS practices, performance , operation

8.2. Criteria (Y)

In order to sustain COTS-based systems and ensure efficient maintenance

processes, all relevant prerequisites for system maintainability and supportability are

to be defined during design phase of the system. Supportability analysis is a pre-

condition to ensure effective maintenance and minimize life cycle cost.

The guidance “’Obsolescence management” and the guidance MEDDKM23

specify important factors to be considered only during procurement and sustainment

phase. Therefore, evaluation criteria, which are derived from these two documents,

reflect only some factors related to procurement and sustainment (see chart 4).

Based on the current obsolescence policy of DMO and the guidance MEDDKM

23, the following criteria could be derived for the internal policy documents.

Chart 4

Phase Assessment criteriaAcquisition COTS Replacement strategy

Supplier responsibilitiesILS support for COTS replacement strategyObsolescence risk analysis

Sustainment

In-service contract with supplier Price indexation method for spare parts prices foreseen in the contractPerformance registration and analysis of obsolescence occurrenceCOTS obsolescence problems and technical meetingsLink of COTS replacements with system modificationsCOTS components obsolescence method

83


8.3. Assessment criteria

In order to achieve an effectiveness of the maintenance process for COTS

products and components, it is essential to weigh in advance all the possible

performance factors. Supportability analysis must be performed to verify

preconditions of maintenance flow and required infrastructure. Maintenance,

obsolescence and long-term support plans are the next steps in arranging the

relevant stipulations in the contract with a supplier. Supplier evaluation and

negotiating In-Service Support requirements are inherent conditions having on the

long-term contract an impact on the COTS costs and availability.

Having analyzed the criteria (X) and (Y), the conclusion has been made that internal

policy documents do not reflect all inherent factors that have an impact on

sustainment phase and which will later have an effect on the maintenance process,

performance and costs. However, it is noted that stipulated criteria in chart 4 are in

conformity with the ones being specified in the scientific literature. Yet, they are not

still sufficient in terms of achieving the expected efficiency of the maintenance in

relation to COTS products and components. The criteria being indicated in chart 3

have been combined together with the ones from chart 4. Therefore, chart 5 contains

the most significant criteria in order to assure that the COTS maintenance process

will be effective. It was chosen to keep the same activity areas being applied in chart

4 because of their generic character. Corresponding explanation for the criteria has

been indicated in the column “’Remark”.

Chart 5

Activity area Assessment criteria RemarkBusiness COTS definition and COTS

categoriesIndicate in which document any COTS definition is specified.Define what is understood under COTS/MOTS/ROTS or Development items.

Relationships with suppliers Indicate how relationships with the supplier and their sub-contractors are specified

COTS costs and method of control

Is there any method applied on how to control costs of the COTS products?

Engineering Supportability elements analysis for effective

Indicate which of the following analysis have been performed:

84


Activity area Assessment criteria Remarkmaintenance concept - Reliability predictions for

COTS/MOTS/ROTS;- LCC analysis;- Maintenance Task Analysis;- FMECA;- Spare parts analysis;- LORA;- Test equipment requirements

COTS ILS support Specify support elements of ILS support which have to be purchased to assure a future COTS replacement strategy:- Technical documentation;- Training program;- Spare parts supply;- Logistic support for repaired

items;Test equipment support Which requirements have been

stipulated for the COTS test equipment?How is the test equipment to be utilized?

COTS reliability and its verification method during sustainment phase

How is the COTS reliability verified? How do we know that products we buy from supplier are reliable? What kind of evidences is presently available?

COTS maintenance and replacement strategy

Indicate the clauses of the maintenance plan/system plan/ILS where the COTS replacement strategy has been elaborated What is the practical experience in the actual replacement strategy

COTS obsolescence strategy Which party is responsible for the obsolescence management of COTS?At which level is the obsolescence performed: LRU/SRU/component level?Which strategy is chosen to predict occurrence of COTS obsolescence?

COTS failure and fault registration

How is the COTS failure and fault registration preformed? Is there any periodic analysis conducted to verify a performance of specific COTS components?

Technical developments on the market

Which department is involved in market research and technical developments on the market? Is this knowledge for the project available in

85


Activity area Assessment criteria Remarkorder to negotiate more favorable conditions with suppliers?

Contract Main delivery contract with supplier

Point out which clauses of the main delivery contract with the supplier have any relation to the suppliers’ responsibilities in respect to COTS products and components

Type of support contract with supplier

Is the support contract based on performance based conditions?What are the responsibilities of the supplier in relation to COTS?Is this contract a long-term agreement or could conditions be re-evaluated and re-negotiated?

Contract surveillance and evaluation approach

Is the ISS contract to be evaluated? Is there a method applied to evaluate the supplier in relation to COTS matters?

Program-wide

COTS price assessment What is the price evolution for COTS components from previous periods? Indicate a price growth/decrease for the last 3 -5 years? Are there any reasons for a price growth/decrease?

Risks identification approach and impact on the maintenance

Are there any risks identified in relation to COTS costs, availability, obsolescence?What has been done to minimize these risks?

COTS technology insertion and its planning during the life-cycle

Is any planning made to assure technology insertion? On which term this planning is made?

Knowledge management and information sharing about COTS practices, performance , operation

In which way is the information and knowledge regarding COTS accumulated and shared in the organization?

For the evaluation of the above-mentioned criteria the Gap analysis techniques will

be applied. Under the Gap analysis technique is understood a method of the steps to

be taken in moving from a current IST state to a desired future SOLL state.

Those steps imply:

- Identify strategic objectives;

Activity areas and assessment criteria for each area in this case shall help to

define the strategic objectives and to obtain important information about key

areas and what is missing in the four areas (business, engineering, contract, and

86


program- wide) in order to focus on the process and quality improvement of the

process.

- Indentify current standings;

Information and data shall be collected with regard to each of the strategic

objectives for four defined activities area for each study case. The current

standings will be defined based on the Gap analysis template. This template

shall be made based on the structure of the chart 5, but extended with the Gap

analysis check list questions and visualisation of the evaluation scale in giving

an assessment in the form of the qualitative answers Yes, Partly, No, Not

Applicable (N/A).

- Create plan of action;

After the information has been gathered, a plan for closing the gap between the

current and desired state of the organization can be made. Each “ No” or “Partly”

reply means that Gap has been indentified and action should be taken. In

conformity with the research model for this study, recommendations and

suggestions for the COTS policy will be outlined based the Gap analysis

template.

- Back up the plan of action with data and analysis;

Findings from the Gap analysis template are to be supported with appropriate

data and analysis giving a course of action for improvement.

The Gap analysis template for assessing of each study case is presented in the chart

6.

87


Chart 6

N Activity area

Assessment criteria

Check list questions

Yes Partly

No

N/A Remarks and comments

1. Business

1.1 COTS definition and COTS categories

Is there any definition given regarding COTS products?

If Yes, please specify in which documents this definition is stipulated

1.2 Relationships with suppliers

Is only one contractor involved in the process? Are there any other sub-contractors?

Indicate how relationships with the supplier and their sub-contractors

88


N Activity area

Assessment criteria


Yes Partly

No


are established

1.3 COTS costs and method of control


If Yes, specify which one

2. Engineering

2.1 Supportability elements analysis for effective maintenance concept

Have the below listed analysis been performed for COTS products?- Reliability

predictions;- LCC analysis;- Maintenance Task

Analysis;- FMECA;- Spare parts analysis;- LORA;

89


N Activity area

Assessment criteria


Yes Partly

No


- Test equipment requirements

2.2.

COTS ILS support

Has the below-mentioned ILS support been purchased for COTS products in order to ensure a future COTS replacement strategy?- Technical

documentation;

90


N Activity area

Assessment criteria


Yes Partly

No


- Training program;- Spare parts supply;- Logistic support for

repaired items;

2.3 Test equipment support

Are there any requirements stipulated for the COTS test equipment?Is the test equipment utilized by the facility in Den Helder?

2.4 COTS reliability and its verification method during sustainment

Is the COTS reliability verified? Does the supplier indicate the reliability of the COTS components?

Indicate the method how it is verified.State some evidences

91


N Activity area

Assessment criteria


Yes Partly

No


phase being presently available

2.5 COTS maintenance and replacement strategy

Has the COTS maintenance and replacement strategy been elaborated?

Indicate the clauses of the maintenance plan/system plan/ILS where the COTS replacement strategy has been described.Give practical examples in

92


N Activity area

Assessment criteria


Yes Partly

No


the current replacement strategy

2.6 COTS obsolescence strategy

Is there any party responsible for the obsolescence management of COTS?Has been any strategy chosen to predict occurrence of COTS obsolescence?If yes, Is the obsolescence performed at:- LRU- SRU- Component level?

If yes indicate responsible party and its main

Describe the strategy

2.7 COTS failure Is the COTS failures In which

93


N Activity area

Assessment criteria


Yes Partly

No


and fault registration

registration preformed? Is there any periodic analysis conducted to verify a performance of specific COTS components?

system? Who is responsible? Who analyses those data?

2.8 Technical developments on the market

Is any market research performed? Is this knowledge available for the project in order to negotiate more favorable conditions with suppliers?

Which department is involved in market research and technical developments on the market? How often is this research

94


N Activity area

Assessment criteria


Yes Partly

No


performed?How is this knowledge shared with other staff?

3. Contract

3.1 Main delivery contract with supplier

Is the responsibility of supplier in respect of COTS products and components properly stipulated in the contract?

Point out which clauses of the main delivery contract with supplier have any relation to the supplier responsibilities in respect to COTS

95


N Activity area

Assessment criteria


Yes Partly

No


products and componentsWhat are the responsibilities of the supplier in relation to COTS?

3.2 Type of support contract with supplier

Is the support contract based on performance based conditions?Is this contract a long-term agreement or conditions could be re-evaluated and negotiated?

Is the support contract based on performance based conditions? Which

96


N Activity area

Assessment criteria


Yes Partly

No


performance aspects are taken into account?What are the responsibilities of the supplier in relation to COTS?If yes, indicate the term of the contract revision

3.3 Contract surveillance

Is the ISS contract ever evaluated?

If yes, describe the

97


N Activity area

Assessment criteria


Yes Partly

No


and evaluation approach

Is there a method applied to evaluate the supplier in relation to COTS matters?

evaluation process

4. Program-wide

4.1 COTS price assessment

Have the prices for COTS components been changed rapidly?Are they increased?Are they decreased?Is the price evolution for COTS components from the previous periods available?

Indicate a price growth/decrease for the last 3 -5 years

Indicate reasons for a price growth/decrease

4.2 Risks identification

Are there any risks identified in relation to

Where those risks

98


N Activity area

Assessment criteria


Yes Partly

No


approach and impact on the maintenance

COTS costs, availability, and obsolescence?Is any policy developed to minimize these risks?

have been described

Describe the approach

4.3 COTS technology insertion and its planning during the life-cycle

Is any planning made to assure technology insertion?

If Yes, indicate the term 5, 10 , 15 years

4.4 Knowledge management and information sharing about

Is the information and knowledge regarding COTS accumulated and shared in the organization?

99


N Activity area

Assessment criteria


Yes Partly

No


COTS practices, performance , operation

- Share point- System portal- Papers- Other

100

6. Assessment of the study cases

9. ASSESSMENT OF THE STUDY CASES

This chapter summarises the analyzed data for each study case. The assessment

criteria, which were determined in the chapter 5, have been applied for evaluating the

study cases in four pre-defined activities area: business, engineering, contract, and

program-wide. Every study case has been assessed in accordance with the data that

has been made available from the NMSA. An outcome of the analysis will partially

present an answer on the sub-research question 4.

9.1. Summary of findings

Appendixes B, C, D and E contain detailed analysis of each study case Zn in

accordance to the Gap analysis template. From these analyses, it can be seen to

which extent the maintenance process of each study case is coherent to the set of

assessment criteria. Summarized assessment results of the evaluated study cases is

reflected in chart 7.

Hereby the following facts are indentified for:

1. Business area:

- There is no clear definition what is understood under the term COTS. As a result ,

there is no specification of COTS items in the parts lists and thus no further

defined policy in relation to these items;

- In the case that maintenance is outsourced, multiple system integrators are often

involved in the maintenance process. Due to disagreements between system

integrators concerning support services, multiple ISS contracts for the same asset

have been concluded which created a dissonance in the maintenance strategy;

101


- In the case that maintenance is performed at the own facilities of the NMSA, a

sub-contractor management will be performed by the NMSA, which requires extra

knowledge, extra personnel capacity and evaluation approaches of the

subcontractors performance;

- COTS costs method of control is not established which led to exceeding of

budgets.

2. Engineering area:

- Supportability analysis has been performed partially for two of the four cases.

- COTS ILS support has been partially purchased during the acquisition phase

which resulted in impracticable situations to perform the maintenance tasks;

- COTS Test equipment support has not been included in the LCC calculation

which has increased the exploitation cost because of unexpected investment;

- COTS replacement strategy in three of the four cases is not defined;

- COTS obsolescence management is in one case not included in the ISS contract;

- Obsolescence management is performed at the micro component level while all

spare parts lists are composed at the SRU/LRU level;

- Obsolescence management has been set up based on results of a product

discontinuation notice, and procurement of items in sufficient quantity to support

the life cycle of the asset, or until the next upgrade. The Last Time Buy is a

reactive resolution.

- FFF replacements are available since they are included in the variable services of

the ISS contract, which means that this service is to be paid separately;

- Failure registration for COTS has been sufficiently followed only for two cases;

- Technical developments on the market activities are not followed in two cases

because of a lack of clear roles and responsibilities in the organisation:

3. Contract area:

102


- Main delivery contract with supplier has not specified any responsibility

concerning COTS items and products;

- ISS contracts do not contain any performance based conditions and do not

accommodate any reassessment possibility;

- There is not any particular method or instruction with attention areas how to

evaluate an ISS contract. Evaluations are performed per every single case

depending on the needs and situation in the project;

4. Program-wide:

- Price evaluation for COTS components is not structurally performed and is not

set-up as a process;

- COTS risks and their influence on the maintenance are not yet recognized in two

cases. In one case study risks are indentified in respect of COTS obsolescence;

- There is no planning made on how technology insertion will be carried out during

the life time of the assets in three cases;

- Need in the knowledge management regarding COTS practices, COTS risks, etc.

is essential. However, this knowledge is still not good structured.

103


9.2. Sub- conclusion

The summary of findings and chart 7 constitute a basis for recommendations for the

COTS policy concept. The following is concluded:

- For 10 criteria, there is a gap whereby a process is to be developed and set-up.

- For other criteria, a process improvement and streamlining is advised, and use of

the current practices for the policy development is to be recommended as

valuable input for this policy.

104


N Activity

area

Assessment criteria C

h

e

c

k

l

i

s

t

q

u

e

s

t

i

o

n

s

Study

cases

Yes Partl

y

1. Business 1.1 COTS definition and COTS categories

Is there any definition given regarding COTS products?

Z1 XZ2Z3Z4

1.2 Relationships with suppliers

Is only one contractor involved in the process?

Z1Z2 XZ3Z4 X

105


N Activity

area


h

e

c

k

l

i

s

t

q

u

e

s

t

i

o

n

s

Study

cases

Yes Partl

y

1.3 COTS costs and method of control


Z1 XZ2Z3Z4 X

2. Engineerin

g

2.1 Supportability elements analysis for effective maintenance concept

Have the below listed analysis

Z1 X

Z2 X

106


N Activity

area


h

e

c

k

l

i

s

t

q

u

e

s

t

i

o

n

s

Study

cases

Yes Partl

y

been performed for COTS products?- Reliability predictions;- LCC analysis;- Maintenance Task Analysis;

Z3 XZ4 X

107


N Activity

area


h

e

c

k

l

i

s

t

q

u

e

s

t

i

o

n

s

Study

cases

Yes Partl

y

- FMECA;- Spare parts analysis;- LORA;- Test equipment requirements

2.2. COTS ILS support Has the below-

Z1 XZ2 XZ3 X

108


N Activity

area


h

e

c

k

l

i

s

t

q

u

e

s

t

i

o

n

s

Study

cases

Yes Partl

y

mentioned ILS support been purchased for COTS products in order to ensure a future COTS replacement

Z4 X

109


N Activity

area


h

e

c

k

l

i

s

t

q

u

e

s

t

i

o

n

s

Study

cases

Yes Partl

y

strategy?- Technical documentation;- Training program;- Spare parts supply;- Logistic support for repaired items;

110


N Activity

area


h

e

c

k

l

i

s

t

q

u

e

s

t

i

o

n

s

Study

cases

Yes Partl

y

2.3 Test equipment support

Are there any requirements stipulated for the COTS test equipment?Is the test equipment utilized by the facility

Z1Z2Z3Z4 X

111


N Activity

area


h

e

c

k

l

i

s

t

q

u

e

s

t

i

o

n

s

Study

cases

Yes Partl

y

in Den Helder?

2.4 COTS reliability and its verification method during sustainment phase

Is the

COTS

reliability

verified?

Does the

supplier

indicate

the

reliability

Z1 XZ2Z3 XZ4

112


N Activity

area


h

e

c

k

l

i

s

t

q

u

e

s

t

i

o

n

s

Study

cases

Yes Partl

y

of the

COTS

componen

ts? 2.5 COTS maintenance

and replacement strategy

Has the COTS maintenance and replacement strategy been elaborated

Z1Z2Z3Z4 X

113


N Activity

area


h

e

c

k

l

i

s

t

q

u

e

s

t

i

o

n

s

Study

cases

Yes Partl

y

? 2.6 COTS obsolescence

strategyIs there any party responsible for the obsolescence management of COTS?Has been any strategy

Z1 XZ2Z3 XZ4

114


N Activity

area


h

e

c

k

l

i

s

t

q

u

e

s

t

i

o

n

s

Study

cases

Yes Partl

y

chosen to predict occurrence of COTS obsolescence?If yes, Is the obsolescence performed at:- LRU

115


N Activity

area


h

e

c

k

l

i

s

t

q

u

e

s

t

i

o

n

s

Study

cases

Yes Partl

y

- SRU- Component level?

2.7 COTS failure and fault registration

Is the COTS failures registration preformed? Is there any

Z1 XZ2Z3 XZ4

116


N Activity

area


h

e

c

k

l

i

s

t

q

u

e

s

t

i

o

n

s

Study

cases

Yes Partl

y

periodic analysis conducted to verify a performance of specific COTS components?

2.8 Technical developments on the

Is any market

Z1Z2Z3 X

117


N Activity

area


h

e

c

k

l

i

s

t

q

u

e

s

t

i

o

n

s

Study

cases

Yes Partl

y

market research performed? Is this knowledge available for the project in order to negotiate more favourable conditions

Z4

118


N Activity

area


h

e

c

k

l

i

s

t

q

u

e

s

t

i

o

n

s

Study

cases

Yes Partl

y

with suppliers?

3. Contract 3.1 Main delivery contract with supplier

Is the responsibility of supplier in respect of COTS products and components properly stipulated

Z1Z2Z3Z4 X

119


N Activity

area


h

e

c

k

l

i

s

t

q

u

e

s

t

i

o

n

s

Study

cases

Yes Partl

y

in the contract?

3.2 Type of support contract with supplier

Is the support contract based on performance based conditions (PBC)?Is this contract a long-term

Z1Z2Z3Z4

120


N Activity

area


h

e

c

k

l

i

s

t

q

u

e

s

t

i

o

n

s

Study

cases

Yes Partl

y

agreement or conditions could be re-evaluated and negotiated?

3.3 Contract surveillance and evaluation

Is the ISS contract

Z1 XZ2 XZ3 X

121


N Activity

area


h

e

c

k

l

i

s

t

q

u

e

s

t

i

o

n

s

Study

cases

Yes Partl

y

approach ever evaluated? Is there a method applied to evaluate the supplier in relation to COTS matters?

Z4

Program- 4.1 COTS price Have the Z1 X

122


N Activity

area


h

e

c

k

l

i

s

t

q

u

e

s

t

i

o

n

s

Study

cases

Yes Partl

y

wide assessment prices for COTS components been changed rapidly?Are they increased?Are they decreased?Is the

Z2 XZ3Z4

123


N Activity

area


h

e

c

k

l

i

s

t

q

u

e

s

t

i

o

n

s

Study

cases

Yes Partl

y

price evolution for COTS components from the previous periods available?

4.2 Risks identification approach and impact on the maintenance

Are there any risks identified

Z1

Z2

124


N Activity

area


h

e

c

k

l

i

s

t

q

u

e

s

t

i

o

n

s

Study

cases

Yes Partl

y

in relation to COTS costs, availability, and obsolescence?Is any policy developed to minimize these

Z3 X

Z4 X

125


N Activity

area


h

e

c

k

l

i

s

t

q

u

e

s

t

i

o

n

s

Study

cases

Yes Partl

y

risks?4.3 COTS technology

insertion and its planning during the life-cycle

Is any planning made to assure technology insertion?

Z1Z2Z3Z4

4.4 Knowledge management and information sharing about COTS practices,

Is the information and knowledge regarding

Z1 XZ2 XZ3 XZ4 X

126


N Activity

area


h

e

c

k

l

i

s

t

q

u

e

s

t

i

o

n

s

Study

cases

Yes Partl

y

performance , operation

COTS accumulated and shared in the organization?- Share point;- System portal;- Papers;- Other

127


N Activity

area


h

e

c

k

l

i

s

t

q

u

e

s

t

i

o

n

s

Study

cases

Yes Partl

y

Total score 22 13

128

7. Conclusions and recommendations

10.CONCLUSIONS AND RECOMMENDATIONS

In this chapter, the outcome of the research will be summarized. Consequently, all

conclusions made after conducting interviews, content analysis of the project

documentation, and which are based on the summary of findings described in

chapter 6, are listed in the below sub-chapters. These conclusions are the basis for

respective recommendations.

10.1. COTS internal policy concept as a qualitative approach

Internal COTS policy means not only a setting up the internal policy documents, but

also to consider a COTS management policy in the scope of the total quality

approach where other interrelated domains and required processes are to be taken

into account.

Conclusions:

For the short term

- Establish an internal “COTS management” policy as soon as possible taking into

consideration existing policy documents and practices in the organisation;

- Revise “Obsolescence management” instruction regarding streamlining of a tactical

management and reflect the practical experience within the organisation;

- Revise the Guideline MEDDMKM 23 in terms of the new structure of the

organisation. Consider new roles and responsibilities.

- Make it obligatory to specify COTS requirements in requests for quotation (RFQ);

For the long term

- Set up a COTS management/obsolescence group, which will be specialized in new

developments in this domain. The group will be able to advise project teams

regarding a program-wide approach, including technology insertion and eventual

risks related to any specific project and how to mitigate those risks;

- Look for an international cooperation with experience from other nations, for

instance the UK MoD.

Recommendations;

129


In order to situate the policy “COTS management”’ at the DMO, consider domains

depicted in figure 20 as being interrelated for elaboration of a COTS management

policy. Figure 20 depicts so-called INK model, which is appropriate in the scope of

the TQM for validation of organization on nine important TQM aspects. The INK

model is adapted especially for the COTS policy.

Figure 20. COTS management policy and interrelated

domains

The interrelated domains for COTS management policy at the DMO could be:

- Proactive leadership accepting the issue and willing to cope with it;

- Management of employees where employees are to be to encouraged to follow

training programs such as ILS, maintenance management, reliability centered

maintenance, and asset management control in order to consider the COTS

issues as a system approach method;

- Management of assets (read: Weapon System Management) means that the

weapon system management policy has to reflect COTS management aspects as

well;

- Management of processes such as COTS maintenance, COTS obsolescence and

contract management for in-service support have to be re-developed in the view

on how to mitigate risks related to COTS;

130


- Requirements of OPCOs are to reflect their wishes regarding systems and sub-

systems updates with new functionalities. In this way new technology insertion

terms could be defined and budgeted;

- Supplier management and supplier evaluation and follow-up is essential;

- Market research is an important aspect for a COTS policy where technological

developments are to be traced;

- Control and Finance measures are unavoidable for an effective COTS policy in

order to ensure that the costs of COTS replacements and obsolescence risks will

be governed;

- Improve and Update implies an iterative process during the life cycle which is

needed where COTS are applied.

- COTS management is to be seen as a quality approach where a Plan-Do-Check-

Act systematic method is to be applied, regularly to be evaluated and repeated. A

COTS policy management versus this so-called Deming Cycle is shown on fig 21.

“PLAN” – set up a plan for conducting of processes, set up targets and objectives

of the policy, consider specific COTS aspects, hazards and quality issues.

“DO” – structure roles and responsibilities in the organization, consider any

training requirements for the personnel, how it will be communicated, make sure

that procedures and work instructions for COTS related systems are clear,

perform the planned processes.

“CHECK” - control the maintenance processes performance, register deviations

from the objectives, measure financial results of these deviations;

“ACT” – Evaluate deviations and results, take corrective and preventive actions,

reconsider the current process if needed before the process start again, track

improvements, establish a reporting system and involve personnel in discussions

about possible improvements.

131


Figure 21. COTS policy Plan-Do-Check-Act approach

10.2. COTS definition and COTS categories

In order to define any policy activities for a definite group of products, the definition of

the product is essential. The definition and considerations for COTS products and

components, used for military systems could be formulated as follows:

Definition:

COTS in military systems are products and components which are developed in

accordance to the latest technological achievements, integrated in military assets

without or with minor modifications, having limited visibility into reliability and

frequency of their new updates, and also creating interdependencies.

Considerations:

132


COTS require a proactive approach being already required from the development

phase on, continued during the acquisition phase and tending to be very crucial

during the exploitation phase of military assets. Under proactive shall be

understood an approach whereby strong policies from the side of suppliers and

organizations are to be developed whereat planning of COTS replacements,

mitigating of COTS obsolescence issues and indicating financial consequences,

specifying COST support elements in contracts become of a great importance.

It is also recommended to classify COTS into three categories in accordance to fig. 9:

Category I COTS

Category 2 MOTS/ROTS

Category 3 Full scale development

It is essential to establish boundaries between these groups in order to apply an

adequate replacement strategy, obsolescence solutions and support approaches.

Supplier participation and cooperation is inherent.

It is recommended to indicate in the recommended spare part list the above-

mentioned categories which implicate the following strategy:

Category I COTS

The supplier shall provide an obsolescence alert when the specific items become

obsolete with a recommendation for a replacement or a last time buy.

Category 2 MOTS/ROTS

Items which fall under this category shall comply with the military specifications and

some extra handling might be needed from the side of the supplier to make them

suitable for a stress environment. This extra handling, which might be either extra

testing or extra protection layers in order to make these items ruggedized, must be

clearly described.

Possible recommendation for replacement could be last time buy, modification in

accordance to the FFF which also could involve the rest of the system.

Category 3 Full scale development

Items, which fall under this category for instance printed circuit boards, are

developed by the supplier with COTS components. These items are to be under the

complete responsibility of the supplier. If any item on the PCB becomes obsolete, the

NMSA should not have any concern and should not bear any financial consequences

133


to substitute any of them as long as the choice on which components are to used for

the PCB remains the responsibility of the supplier.

10.3. Recommendations

To mitigate risks with COTS products and components being implemented in naval

military assets and assure an effectiveness of the maintenance process, the following

precondition is crucial to guarantee exploitation costs as lower as possible:

In order to achieve effective maintenance process for the naval military assets

considering use of COTS products and components, minimum 75% of the criteria

specified for the assignment of the study cases are to be met:

Effective maintenance process = (Business, Engineering, Contract, Program-

wide Activities)

Considering results of the study cases assessment exhibited in the chart 7,

subsequent final recommendations could be given for each of the activity areas:

BUSINESS

- COTS definition and categories are to be mentioned in any requirements for asset

development or in any modification request;

- Requirements to any development of any naval military asset have to contain the

principles which are reflected in the STANAG 4597 "Obsolescence Management"

and 4598 "Guidance on the use of commercial of the shelf(COTS) technology";

- Requirements to any development of any naval military asset have to contain a

stipulation pursuant to NATO guide for codification AC/135;

- In case if the maintenance of the system is outsourced, one system integrator has

to be responsible for the development of the maintenance policy for naval military

asset. Understand and monitor the system integrator plans for maintenance

support, engage in meetings and information exchanges with the system

134


integrator. Encourage the system integrator to facilitate working relationships with

COTS vendors;

- In case if maintenance is carried out at the NMSA facilities, and the warranty

period already ran out, consider the following factors in relation to COTS sub-

contractor management:

• Make sure that in accordance to the main contract, the NMSA has a right to

deal with vendors and suppliers directly;

• Formulate requirements;

• Choose a list of key vendors or vendors of importance to the program;

• Visit the facility of those vendors to verify the quality norms;

• Participate in user’s group meetings to sustain a relationship with important

vendors

• Assign internal resources with skills and experience who are supposed to

sustain a sub-contractor management and relationships with different

vendors;

- In order to get control on financial aspects related with COTS products and

ensure synchronizing of the budget with the maintenance activities and

procurement of COTS products, subsequent measures are to be taken:

• Procurement of COTS products for different weapon systems is to be

planned, compared and reported during the life cycle of the systems;

• Spread the procurement of COTS products for different weapon systems in

time during the life cycle of the systems;

• Desynchronize COTS procurement in order to remain within the exploitation

budget and achieve flat budget spreading;

• Combine a financial procurement planning with the planned maintenance;

• Revise the ILS plan with cost estimations;

• Specify COTS procurement in the Road Map for all weapon systems;

• Estimate COTS procurement in the DIP budget instead of in the MATEX.

ENGINEERING

- Supplier has to be requested to conduct a Supportability Analysis for all system

components in including special test tools and equipment ;

- Supportability analysis is to be conducted during exploitation phase based on

performance drivers and taking into account a failure occurrence;

135


- Supplier has to be requested to specify which products and components are

considered as COTS and provide a “’COTS handling Road Map”’ which could be

a part of the LCC analysis;

- The LORA Worksheets have to contain parts which are specified on the LRU/SRU

level and which of the parts are to be considered as COTS;

- Integrated Logistics Support packages have to be bought during the development

phase of naval military assets and not to be postponed until the exploitation

phase;

- Plan costs for operational testing to ensure performance. Consider the next

practices:

• Include an analysis for test tools and equipment in the LCC and SA;

• Analyze what is the return of investment to purchase the test tools and

equipment. This analysis is to be requested in the scope of the maintenance

tasks preparation activities;

• Create a working group within a project team which can coordinate the

correctness of the analysis together with the system

integrator/contractor/vendor;

• Analyze whether it is economically favorable to test COST products in

cooperation with other customers;

- Supplier has to be requested to prove the reliability of system components in

accordance to the STANAG MIL-HDBK-217F and in accordance to “Non-

Electronic Parts Reliability Dataset(NPRD)” NRPD-95;

- For assemblies where no reliability data is available, the complexity on LRU level

has to be studied and reliability figures have to be evaluated using calculated

figures for COTS equipments with comparable complexity and function;

- The COTS maintenance and replacement strategy is directly influenced from the

short lifetime of COTS products and their quick obsolescence. Therefore, the

maintenance and replacement strategy might include the next options depending

on the system:

• Life Time Buy: which in most cases is applied at the NMSA and which

requires investments in the spare parts stocks till the end of the system life;

• FFF replacements: which has to be very good defined contractually and

which requires more efforts from the supplier/system integrator;

136


• Complete new replacement: with upward compatible equipment and being

planned in combination with the planned maintenance of the asset to avoid

a decrease in the asset availability. The following is hereby to be taken into

account:

o Planned maintenance is conducted based on a 4 years cycle;

o COTS products update shall be performed also based on a 4 years

cycle;

o COTS software source code update shall be conducted based on a

8 years cycle, since the software update costs are much higher than

the update of the hardware, and in some instances are

unpredictable.

The options 1 and 2 are more suitable for SEWACO.

Option 3 is more suitable for C4I.

• Indicate the strategy in the:

o System Plan (SP);

o Integrated Logistics Support Plan (ILS);

o Maintenance Plan (MP)

- Define a COTS obsolesce strategy taking into account the next steps:

• Request supplier to conduct analysis about obsolescence resolutions and

connected costs;

• Determine a level of obsolescence LRU/SRU/component;

• Set-up receiving a pre-advanced notification about terminating of

components and products production;

• Purchase of the spare parts after getting pre-advanced notification;

• Participate in the service bulletins;

• Make proper agreements with suppliers;

• Plan modifications with a new technology insertion;

• Arrange post contract support guaranties in respect of spare parts;

• Agree with the supplier about specific maintenance activities in case if

supplier is not going to continue to sustain the system;

137


• Set up in advance a sufficient stock of spare parts;

• Evaluate the situation and discuss with the supplier

- Any operational COTS failure has to be registered, processes and handled in

the MATRAX system. The procedure of the failure registration is not something new

in the NMSA. However, it is recommended to trace failures closely and follow the

next steps in the failure procedure:

• Assign responsible officers for filling in the MATRAX and tracing failure

occurrence;

• Involve users to make them understand why a failure registration is

important for an adequate COTS management and replacement strategy;

• MTBF (if known) is to be checked and compared with the one defined

during the ILS study;

• The failure outcome has to be finalized in the End report with supporting

engineering conclusions and results;

• The End report has also to be registered in the MATRAX system for

possible trend analyzes of the similar failures;

• Failures could be divided into the following categories: Incidental and

Structural. Structural failures could be discussed and assessed at the level

of international cooperation;

• A Common Problem Report is to be prepared, based on the assessment of

common structural failures;

- Define roles and responsibilities for evaluating technical developments on the

market. Theses task could be assigned to the Technical Groups, which are set up

in the scope of the in service support activities. The areas which will be covered

by the Technical Groups activities could be as follows:

• Track on available and emerging COTS technologies;

• Availability of COTS products on the market;

• Standards;

• Re-explore the market in view of the technical evaluation results for a

specific system;

• Sustain relationships with contractors/system integrators/vendors for future

plans;

• Cooperation with other nations which might use the same COTS products

138


CONTRACT

Acquisition phase

- The main delivery contract for purchase of naval military asset has to stipulate the

contractor responsibility in respect of COTS items, their codification, integrated

logistics support, availability of the technical documentation and drawings;

- Contractor has to guarantee FFF replacement;

- Concerning the COTS items codification, it is recommended to use the following

contract wording:

“’The contractor shall dispatch the data or arrange for dispatch of the data

from sub-contractors or suppliers on request from the Codification

Authority within the timescale specified in the contract.

The contractor shall include the terms of this clause or an equivalent

contractual instrument in any sub-contract(s) to ensure the availability of

technical data to the Codification Authority and for future support of the

system. If dispatch of the data takes place from the sub-contractor or

supplier, the contractor shall provide details of sub-contract numbers or

similar to enable the Codification Authority to approach the sub-contractor

or supplier direct for the data.”

Exploitation phase

- An ISS contract where COTS definitions and categories are to be part of, is to be

developed and drawn up during a new development and construction phase and

has to start at the same time with the exploitation;

- The ISS contract has to be concluded for a definite time, contain performance-

based conditions and accommodate any reassessments possibility;

- Performance-based conditions are to contain:

• A certain incentive clause stimulating innovation and related to outcome;

• Key Performance Indicators (KPI) must be developed and defined in order

to verify that the effectiveness of the maintenance process will be

achieved. Examples of the KPI’s:

System availability %;

Mean Time Between Failures;

Mean Time To Repair;

Estimated replacement value of components;

Schedule compliance;

139


Total maintenance costs(TMC);

Maintenance labor costs as a % of TMC;

Contractor maintenance labor costs as a % of the TMC;

Compliance with the allocated budget.

• Specified units of measurement (the metrics) of the performance and clear

procedure of measurement( when, where, by whom and how);

• Verification of performance based on the KPI ‘s;

• Risks are born by the party that is the best equipped to mitigate risks;

• A possibility of abandoning of unnecessary services;

- Constitute ISS contract with a clear COTS replacement and COTS technology

insertion strategy;

- Include obsolescence management aspects in the ISS contract and define

responsibilities;

- Describe in the ISS contract how financial risks are to be shared between NMSA

and the contractor. It could include the following options:

• Financial threshold setting: if below -- contractor pays, if above --- NMSA

pays;

• All resolution costs split by a % factor, e.g. 70/30, 50/50, 60/40

- Contract evaluation has to be carried out depending on the needs of each

particular program. The next steps could be recommended based on the current

experience in the NMSA:

Reconsider the scope of work and its completeness taking into account an

experience, analysis and reports for the previous period;

• Evaluate whether the current contract is in compliance with the actual need

and whether all services described in the contract are still applicable;

• Evaluate the lead time of the maintenance activities;

• Assess contract cost effectiveness on a yearly basis: product and services

versus costs, and product and services versus time;

• Calculate cumulative contract costs for the previous period and compare

with the historical forecast;

• Look for optimizing, if needed.

PROGRAM WIDE

140


- Conduct a price assessment for COTS components and products to determine

that the price is reasonable. Price analysis is a preferred approach for

determining price reasonableness. The information needed to make this

determination can be collected through market research. Examples of the types of

information being useful:

• Prices of commercial items sold to the public;

• Purchase histories, including prices paid for similar items;

• Prices based on catalogs or price lists.

- Include in the contract a clause to limit the profit margin of the suppliers for the

acquisition of COTS products and components. It can be assumed that it is

reasonable that this margin should not be higher than 10% form the original price

of the OEM;

- Request a contractor/system integrator/vendor to describe how the proposed

system will allow technology insertion and refresh through upgrades of individual

hardware, components, or software modules with new COTS components.

Consider the next aspects:

• The contractor’s description shall minimally address in his approach the

technical structure of the system, and how components from third- parties

will be allocated in the asset.

• The contractor shall be requested to estimate possible costs for the

technology insertion and refresh;

• Indicate the strategy in the:

o System Plan (SP);

o Integrated Logistics Support Plan (ILS);

o Maintenance Plan (MP)

- Risk identification approach implies that any COTS-related risk is to be identified

as soon as possible and whereby plans to manage those risks are to be made. It

is a wrong attitude to keep only the supplier responsible for risks identification.

Most common COTS risks for maintenance include:

• Failure to estimate the costs in relation to frequency of replacements due to

failure, obsolescence, and technology insertion;

141


• Failure to make a long-term commitment in contracts with the suppliers and

system integrators;

• Failure to re-evaluate internal processes, approaches and strategies;

• Failure to acknowledge and take into account a wide-ranging impact of

COTS issues in the organization and their financial consequences.

• Failure to collect COTS lessons learned from another projects and share

them with others.

- Knowledge information about COTS that can be shared shall include (however it

is not limited to) information about all assessment criteria applied in this research.

All programs for naval military assets have to keep and secure this information for

own use, and for others, who search for the experience from other programs. An

objective for information share is to avoid the same mistakes and to help others to

learn from the techniques, approaches, and facts, which have been proven to be

successful. It is recommended to implement the following activities:

• Assign roles and responsibilities for information collection in each particular

program;

• Decide which information of importance exists in relation to COTS cost

estimation, maintenance, obsolescence resolutions, etc.;

• Monitor how this information is used;

• Make sure that this information could be accessible to others and available

for sharing;

• Manage and optimize the information;

• Build information sharing in the work process and program reviews

meetings

Figure 22 shows a COTS based lifecycle model which could be useful to capture in

one complex view how all activity areas business, engineering, contract and program

wide are interrelated and result from each other.

10.4. Way ahead

If the principles of this thesis are accepted at the NMSA, it is advisable to set up a

working group in order to elaborate a decision tree analysis, which will be used for

the future projects where COTS products and components will be embedded in naval

military assets.

142


143


Figure 22. COTS life cycle model

144


145

Reflection on Research

11. REFLECTION ON RESEARCH

When I decided to conduct this research, I realized at that time that this research

could be a challenge. Facing this challenge for the duration of this study made me

more and more enthusiastic about addressing an issue which till now has not been

thoroughly explored in the DMO. During meetings with different people, concerned

with this subject and having interviews with them, it became clear for me that the

“subject COTS” is a matter of vital importance in our organization, meaning that this

study could be of great worth.

The challenge was hidden not only in the absolutely unexplored subject, but also in

fact that this research shall take place in new working environment. That means more

efforts are to be spent to understand and learn how the Navy assets readiness are to

be created, supported and improved. On one side, it took me quite a time to get

acquainted with different Navy projects and to obtain an access to the project

documentation and data for the evaluation. On the other side, it gave me wider and

more clear perception about operations of the NMSA and extended my network.

Thus, I look upon the whole process as a learning loop whereby the abstract theory

has been tested in practice by the active experimentation.

The scope of the research at first was very ambitious. From the beginning the idea

was to examine five projects. Because of lack of time to conduct still one more

assessment, it was decided to leave out of the scope of the research the project

“’Walrus Submarine”. Anyhow, the interview, which has been conducted with

technical personnel from the Technical Logistics Engineering section Mr.B.Scholte

and Mr.J.Roersma gave to me useful information about COTS usage in the platform

related departments.

Furthermore, my intention was to investigate how the obsolescence management

has been set up in defense organizations of other NATO countries. Therefore,

contacts were established with the English MoD, which has developed a detailed

process how to deal with the obsolescence issues. Unfortunately, those contacts

stayed not further developed which is regrettable, since we could surely learn more

from the experience of our partners. Hence, it is really recommendable to cooperate

at the international level with other Defense organizations for knowledge share and

information exchange. It is especially advisable for those projects where

146

Reflection on Research

obsolescence management is carried by the NMSA as for instance in case with

APAR system.

This research has been chosen as a qualitative assessment of the COTS usage for

developing an internal policy within the DMO. However, it would be undoubtedly a

major point to conduct a quantitative pilot study investigating financial consequences

of COTS application.

147

Appendix A

APPENDIX A

TRADITIONAL MAINTENANCE AND MAINTENANCE SUPPORT

APPROACH

Dependability management standard defines maintenance as..”a combination

of all technical and administrative actions including supervisory actions, intended to

retain an item, or restore it to, a state in which it can perform a required function”.

Basic terms relating to maintenance and their relationship in accordance to the

dependability management are illustrated in Fig.23.

Figure 23. Interrelationship of maintenance terms

A maintenance policy defines the general approach for the provision of

maintenance and maintenance support, based on the objectives and policies of

owners, users and customers. It influences the decisions made on maintenance

148

Appendix A

activities and resources during the complete life cycle of an item. This includes

monitoring of the performance of the enabling system and services, identification,

classification, reporting of deficiencies and failures of the enabling systems and

services, and the resolution of those deficiencies and failures.

A maintenance concept is a specific maintenance approach developed for

items using deferent levels of maintenance based on the indenture levels. It makes

use of maintenance support resources within the framework of the maintenance

policy and is performed by a maintenance echelon.

Items can be subdivided into a hierarchy (for example, facility, system, assembly,

equipment and component) or indenture levels against which maintenance tasks are

prescribed. The set of maintenance tasks to be carried out at the specific indenture

level are referred to as the level of maintenance.

Maintenance echelons are the organizational units where maintenance is carried out.

Preventive maintenance may be carried out at regular intervals or according to

prescribed criteria to reduce the probability of failure or degradation in order to retain

the functioning of an item or to detect a hidden fault.

Corrective maintenance restores the functions of the item after failure has occurred

or performance fails to meet stated limits.

Management commitment includes:

- Establishing maintenance policy;

- Planning for maintenance and maintenance support during the design and

development phase;

- Making decisions on trade-offs between functional needs, capability, life-cycle

cost and reliability, maintainability and maintenance support;

- Providing and improving maintenance and maintenance support resources

during operation and maintenance phase.

For some items this responsibility begins and remains with the manufacturer. For

others, the manufacturer has an initial responsibility for identifying maintenance

support needs but the end user or operator takes the final responsibility for

completing the planning process and implementing the results.

The following definitions of the reliability, maintainability and availability are

defined in the literature:,.

Reliability (R) is concerned with a probability and frequency of failures (or lack

of failures). A commonly used measure for reliability for repairable systems is the

mean time between failures (MTBF). Reliability is also a measure of the system

149

Appendix A

performance that affects system availability, mission accomplishment, operating

support cost. Reliability determines how often repairs are needed. The less often that

a system has a failure, the less it will cost to operate and support the system. The

reliability of any repairable system is a significant factor in determining long-term

costs to operate and support the system.

Maintainability (M) is defined as the measure of the ability of an item to be

restored or retained in a specific condition within a specific time. Maintenance should

be performed by personnel having specified skill levels, using prescribed procedures

and resources, at each prescribed level of maintenance and repair. Maintainability is

a measure of how effectively and economically failures can be prevented through

preventive maintenance and how quickly system operations can be restored

following a failure a failure through corrective maintenance. A commonly used

measure for maintainability in terms of corrective maintenance is the mean time to

repair (MTR). Maintainability is a design parameter, while maintenance consists of

actions to correct or prevent a failure effect.

Availability (A) is defined as the percentage of time that a system is available

to perform its required functions. It is measured in a variety of ways, but it is

principally a function of downtime. Availability can be used to describe a component

or system but it is most useful when describing the nature of a system of components

working together.

R, M and A are interrelated. However, even when a “good” level of reliability is

achieved, some failures are expected. The effects of failures on availability and

support cost of repairable systems can be minimized with a “good” level of

maintainability. A system that is highly maintainable can be restored to full operations

in a minimum of time with a minimum expenditure of resources.

Objectives of the traditional maintenance support are to:

- Process and record items’ performance and capacity;

- Process and control items’ availability and reliability;

- Use external resources and contracts for maintenance services;

- Use organizational structure and define responsibilities;

- Use condition monitoring techniques and tools;

- Consider cost and other constrains.

In traditional maintenance support approach maintenance tasks are indentified by

one or a combination of the following approaches (see fig.24):

- Adopting manufacturer’s recommendations:

150

Appendix A

- Analyzing items using structured approach such as reliability centered

maintenance (RCM) based on a Failure Mode, Effects, and Criticality Analysis

( FMECA ).

Figure 24. Maintenance support

Level of repair analysis (LORA) may be performed to determine the optimum

maintenance echelons. The output from the detailed LORA makes it possible to give

the assignment of a level of maintenance for each item and provide input into

maintenance task analysis and the identification of the maintenance support

resources. Based on the LORA, it will be possible to finalize the maintenance

concept for each item. The next decisions are to be made:

- Whether maintenance personnel are provided by the organization or whether

they are obtained from external resources;

- Who provides spare parts and materials;

- Where special tools and transportation, test and support equipment is

sourced;

- Infrastructure that needs to be provided to implement maintenance policies.

151

Appendix A

In order to keep a system operational and available for performing its mission,

replacing of faulty parts is involved.

There are three generic levels of repairs:

Level 1: organic level maintenance (OLM)

These repairs are performed on the operational platform and consist of replacement

of Line Replaceable Units (LRU). LRU are the highest level of assembly and are to

be easily replaced with identical parts. The purpose of this type of repair is to

minimize the down time of the equipment.

Level 2: intermediate level maintenance (ILM)

Faulty LRU are returned to ILM repair facilities. They are examined using required

test and repair equipment. This could be done to component level, if relevant repair

documentation, training and spare parts components are available for to Shop

Replaceable Unit (SRU) level.

Level 3: depot level maintenance (DLM)

This maintenance is performed on the SRU level. The faulty SRU’s are either

repaired or discarded and replaced by new unit. All repaired and replaced units are to

be under configuration control management.

152

BIBLIOGRAPHY

"Obsolescence Management"' STANAG 4597 . 21 April 2005. STANAG 4597

"Obsolescence Management"'. Brussel, Belgium : NATO Standardization Agency,

21 April 2005. STANAG 4597.

5-698-1, TM. 2007. Reliability/Availabity of electrical and mechanical systems for

C4CI. 2007.

60300-3-14, IEC. 2004. Dependability management. Part 3-14:Application Guide-

Maintenance and maitenance support. 2004.

A.King. 2001. Commercial Off- the-Shelf Sotware Benfits and Burdens. The

Edge Perspective. MP-00W0000142, 2001.

AAP-20. 2010. AAP-20 Phased Armaments Programming System(PAPS). 2010.

Aartman. 1999. What is COTS technology? sl : National Aerospace Laboratory

NLR, 1999.

ANEP-54, NATO Standardization Agency. 1998. "Guidlines for COTS

insertion"'. 1998. ANEP-54.

Atlas Electronik. Information on COTS items. 2004. Bremen : sn. issue 2.0.

B.Scholte, J.Roersma. 2010. Notes of the meeting. Den Helder, 2 June 2010.

Bardet, S.M. January 2003. Obsolescence management within the Royal

Nederlands Navy/MLD. sl : National Aerospace Laboratory NLR, January 2003.

NLR-CR-2003-051.

Baron, Sally J.F. September 2006. COTS foundation:essential background and

terminology. sl : International Public Procurement Conference Proceedings,

September 2006.

Benjamin S. Blanchard. 2004. Logistics Engineering and Management (6Th ed).

Upper Saddle River : Pearson Education, Inc, 2004.

Berg, M.van den. 2010. Greep op veroudering. The Hague, August 2010.

Beyond Cost: Drivers of COTS Application Value. Mark Keil, Amrit Tiwana.

May/June 2005. sl : IEEE SOFTWARE, May/June 2005.

Blanchard, B. 2004. Logistics Engineering and Management. sl : Pearson

Education, Inc., 2004.

153

Blanchard, Benjamin S. 1995. Maintainability: a key to effective serviceability

and maitenance management. New York : John Wiley &Sons, Inc., 1995.

Businessdictionary. 2010. http://www.businessdictionary.com/definition/Form-

Fit-and-Function-F3). Businessdictionary.com. [Online] 2010.

C.Albert, L.Brouwnsword. 2002. Evolutionary process for integrating COTS-

based systems. Pittsburg : Software Enegeneering Institute, 2002. Technical

Report CMU/SEI-2002-TR-005.

Claap J., King A.,Taub A. March 2001. COTS Solutions: Opportunities and

Obstacles. March 2001. pp. Volume 2, number1.

COTS ‘Solutions’: Opportunities and Obstacles’. J. Clapp, A. King, A. Taub.

Criscimagna, Ned H. 1998. COTS equipment and Non-development items. sl :

Realability Analysis Center, 1998.

D.Sinay. 2010. Notes of the meeting. Den Helder, 12 July 2010.

Defence, Office of the Secretary of. November 2001. Commercial Item

Handbook. USA : sn, November 2001.

Defense Acquisition University. 18 March 2011. Defence Acquisition Guidbook.

sl : https://dag.dau.mil, 18 March 2011. p. p.61.

Defense Science Board Task Force. 2008. Develomental Test & Evaluation.

Washington, D.C. 20301-3140 : Office of the Under Secretary of Defense For

Acquisition, Technology, and Logistics, 2008.

Department of Defense: USA. 2005. Guide for achieving Reliability, Availability,

and Maintainability. sl : http://www.everyspec.com, 3 August 2005.

Dickerson C.E., Marvis D.N. 2004. Logistics Engineering and Management (6Th

ed). Saddle River : Pearson Education, Inc., 2004.

DMO. 2008. Leaflet "Defence Materiel Organisation". 2008.

Drury, Adrian. 1999. Maintenance and Dependability. sl : Carnegie Mellon

University, 1999.

Duurland, Erwin. 2009. System Analysis: Module 07 – Element 01. 2009.

E.Hendriks. 2010. Notes of the meeting. Den Helder, 28 May 2010.

Emblemsvag, Jan. 2003. Life-cycle costing. Hoboken, New Jersey : John Wiley

& Sons, Inc., 2003.

Enotes.com. 2009 . http://www.enotes.com/business/q-and-a/discuss-difference-

between-strategic-tactical-97313. [Online] 23 August 2009 .

154

F.Schneidewind, Norman. 1998. Issues and Methods for Assessing COTS

Reliability, Maintainability, and Availability. sl : Software Maintenanace

Conference, 1998.

Federal Aviation Administration. 1 Jan 2010. Risk Mitigation Guide: Practical

Methods for effectice COTS acquisition and lifecycle support. Risk Mitigation

Guide. sl : Federal Aviation Administration, 1 Jan 2010. Revision 3.2.

H.Mulder, KLTZT. 2010. Notes of the meeting. June 2010.

Haines, Linda. March-April 2001. “Technology Refreshment within DoD”. sl :

Information Technology, March-April 2001.

http://www.businessdictionary.com/definition/Form-Fit-and-Function-F3). [Online]

http://www.damennaval.com/nl/company_product-range_holland-class-

patrol-vessels.htm.

http://www.thalesgroup.com.

http://www.thalesgroup.com/News_and_events/Group_focus_jan08_Integrated_M

odular_Mast/.

https://dag.dau.mil. Defense Aquisition Guidebook, chapter 5.

Instruction "'Obsolescence Management'". 29 September 2009. Instruction

"'Obsolescence Management'". Den Haag : Directorate "'Maintenance Policy"', 29

September 2009. version 2.

International Hydrographic Organization. 1998. IHO Standards for Hydrpfrahic

Surveys. Monaco : Intenational Hydrographic Organization, 1998. 4th edition,

special publication N44.

Jones, James V. 2006. Integrated Logistics Support Handboek. New York : The

McGraw-Hill Companies, 2006.

Livingston, Henry. 2000. Diminishing Manufacturing Sourcesand Material

Shortages(DMSMS), Mangement Practices. sl : Solid State Devices Comittee,

2000.

Marco Buratti, Daniele Del Brusco. 2000. The Obsolescence Management

Based on a "'Pro-Active"' Approach in Conjunction with a "Pre-Planned"

Technology Insertion Route. Budapest, Hungary : RTO SCI Symposium

"'Strategies to Mitigate Obsolescence in Defense Systems Using Commercial

Components"', 2000. RTO MP-072.

Mark Saunders, Philip Lewis, Adrian Thornhill. 2007. Research Methods for

Business Students. Forth edition. sl : Prearson Education Limited, 2007. p. 140.

McHale, J. 2003. Space the final frontier for COTS. 2003.

155

McHale, John. August 2003. Space: the final frontier for COTS. (special report).

August 2003.

MIL-HDBK-338B. 1988. Electronic Reliability Design Handbook. 1988.

Moriso M., Torchiano M. 2002. Definition and classification of COTS:proposal.

Orlando : sn, 2002.

NATO, AC/135. Guide to the NATO Condification System AC/135.

Notification within RNLN. December 2003. Handleiding "'Commercial off the

shelf". sl : RNLN, December 2003. version 1.

Oberndorf, Tricia. 2000. An activity framework for COTS-based systems.

Pittsburgh : Carnegie Mellon Software Engineering Institute, 2000. CMU/SEI-

2000-TR-010 ESC-TR-2000-010.

P.Engert, J.Clapp. Common Risks and Risk Mitigation Actions for Management

of COTS-Based Systems. sl : Electronic Systems Center of Air Force.

P.Kense. 2010. Notes about COTS philosophy. Den Helder : sn, 2010.

P.Kense, W.Polle. 2010. Notes of the meeting. Den Helder, 13 April 2010.

P.Verschuren. 2008. Probleemanalyse in organisatiee en beleidsonderzoek.

Amsterdam : Boom academic, 2008. Vol. ISBN 978 90 473 00878.

P.Verschuren, H.Doorewaard. 2007. Het ontwerpen van een onderzoek. Den

Haag : LEMMA, 2007. ISBN 978-90-5931-496-2.

Perry, William J. June 1994. “Specifications & Standards - A New Way of Doing

Business”. Washington DC : Secretary of Defense, June 1994. Memorandum.

Place, Patrik. 2000. Guidance on Commercial-Based and Open systems for

Coast Guard Program Managers. Pittsburg : Software Engeneering Institute,

2000. Special Report CMU/SEI-2000-SR-013.

Policy Department. 2007. Handboek Integrated Logistics Support. Den Haag :

Ministry of Defence, 2007. version 2.0.

—. 10 December 2007. Weapon System Management, version 1.9. The Hague,

NL : Policy Department, Maintenance Policy, 10 December 2007.

R.C.T.Hans, M.P.W.Gillis,. June 2006. Aard en omvag obsolescence binnen de

Landmacht en de Marine. Den Haag : TNO, June 2006. TNO-DV- 2006 A213.

Reliability Analysis Center. COTS/NDI Technology. sl : ITT Research Institute.

S.Rampino, M. Fiorilli. 2000. C3I System Acquisition and maintenance in

relation to the use of COTS products. Brussel, Belgium : RTO IST

Symposium"'COTS products in Defence Application"', 2000.

156

SD-2, Defense Standartization Program Office. 2010. DoD Acquisitions Buying

Commercial Items and Nondevelopment Items. US : Defense Standartization

Program Office, 2010. SD-2.

STANAG 4598, NATO Standardization Agency. 19 April 2005. "Guidance on

the use of commercial of the shelf(COTS) technology". Brussel : NATO, 19 April

2005. STANAG 4598.

Stavenuiter, John. 2002. Cost Effective Management Control of Capital Assets.

Lelystad, The Netherlands : Anker B.V, 2002.

T.Oberndorf, L.Brouwnsword, C.A.Sledge, PhD. 2000. An Activity Framework

for COTS-based systems. Pittsburg : Software Engineering Institute, 2000. p. 5,

Technical report CMU/SEI-2000-TR-010.

Thales Nederland. 26 April 2005. APAR shines again in live firing tests. sl :

http://www.thales-nederland.nl/nl/news/archive/2005/april26-2005.shtml, 26 April

2005.

Veld, Jan in 't. 2007. Analyse van bedrijfsprocessen. Groningen : Noordhoff

Uitgevers bv, 2007.

Wikepedia. Technolojy push. http://en.wikipedia.org/wiki/Technology_push.

[Online]

Y.Delbrassine, KVK(ret’d). 2010. Notes of the meeting. 10 June 2010.

157

Documents

Commercial off the Shelf (COTS) policy at DMO ...academy.amccentre.nl/thesis/I_A_Khrystova.pdfMaster of Science in Asset Management Control at the Hogeschool Zeeland written by Iryna