Embed Size (px)
Citation preview
SPIKE Seminar1
Process Improvement and Risk Management in OTS (Off-The-Shelf)
Component-Based Development
Jingyue Li
Norwegian University of Science and Technology
7th September 2004
SPIKE Seminar2
Agenda
• Section 1: An Empirical Study of Variations in COTS-based Software Development Process
• Section 2: Risk management in OTS (COTS and Open Source) based development
SPIKE Seminar3
Section 1: An Empirical Study of Variations in COTS-based Software Development Process• State-of-the-art
• Research motivation and research questions
• Research methods
• Results
• Discussions
SPIKE Seminar4
State-of-the-art
• Process of the whole software development lifecycle– Boehm et al.: Both the waterfall model and evolutionary
development process were regarded as unsuitable for COTS-based development.
– NASA COTS-based development process
– SEI EPIC (Evolutionary Process for Integrating COTS-based system)
– Seven theses about COTS-based development
• COTS component selection and evaluation processes– MAUT, MCDA etc. (decision making algorithm drvien )
– RCPEP (requirement driven)
SPIKE Seminar5
Research Motivation
• Limitations of previous studies– Focused on propose new revised process without empirical
evidence
– Small project sample with similar contexts
– Difficult for project managers to customize their processes based on their project context (developer skill, applicaiton domain, requirement ”toughness” etc.).
• Our motivation– To find out how to customize a COTS-based development process
based on their project context.
SPIKE Seminar6
Research Questions
• RQ1: What was the actual process used in the projects using COTS Components.
• RQ2:What are the commonalities and possible variations in COTS-based development processes.
• RQ3: What are process scenarios of the projects using COTS components successfully and not successfully? What are the relationship between possible risks and process scenarios
SPIKE Seminar7
Research Methods
• Semi-structured interviews– Interview guide include both open and close questions
– Definitions: COTS• Is either provided by some other organizations in the same company, or provided by
external companies as a commercial product.
• Is integrated into the final delivered system.
• Is not a commodity, i.e. not shipped with an operating system, not provided with the development environment, not generally included in any pre-existing platforms.
• Is not controllable by the user, in terms of provided features and their evolution. Our addition: This normally means “black box”, i.e. no source code available.
• Pre-test of interview guide
• Data collection method
• Data analysis method
SPIKE Seminar8
Results - Background information
• Companies (13 Norwegian IT companies)– 1 small sized companies (3<staff size <15)
– 7 medium sized companies (15 =<staff size <=100 )
– 5 large sized companies (staff size > 100)
• Projects (16 finished project using one or more COTS components)
• Respondents– 3 IT manager, 4 project manager, 5 software architects, 4 developers
– 9 of them have more than 10 years SW development experience
• COTS Component samples– Following COM/DCOM, CORBA, EJB, etc.
– C++, or Java library
SPIKE Seminar9
Results - Answers to research question RQ1
• RQ1: What was the actual process used in the projects using COTS Components.
• Data: – Process can be classified into three categories:
• Pure waterfall• Waterfall with prototyping• Incremental mixted with prototyping
– The main process was decided before they started to think about using COTS component or not.
• Answer: the so-called COTS-based development process is to customize the traditional development process due to the use of COTS components.
SPIKE Seminar10
Results - Answers to research question RQ2
• RQ2:What are the commonalities and possible variations in COTS-based development processes.
• Data: Commonalities– Some new activites added
• Build vs. buy decision
• COTS component selection
• Learn and understand COTS component
• Build glueware and addware
– New roles added• COTS component knowledge keeper (12 projects)
SPIKE Seminar11
Results - Answers to research question RQ2 (cont’)
• Data: Variations– The phase to make the build vs. buy decision and the COTS
component selection– The COTS component selection and evaluation process
• SP_fam: Familiarity-based selection process• SP_unfam: Internet search, trial-based selection process
– Step 1: First, developers used a search engine to browse the Internet, using keywords to express decided functionality. This usually gave them a handful of candidates.
– Step 2: If there were more than 2 to 3 possible candidates, they selected 2 to 3 of them very quickly based on some key issues, such as license issues, cost and vendor reputation etc.
– Step 3: After a small number of candidate COTS components had been decided, they downloaded a demo version of these from the web and test the COTS components.
SPIKE Seminar12
Results - Answers to research question RQ2 (cont’)
• Answer: There are some common new activities and one new role added in the COTS-based development process. The possible variations are when and how to perform these new activities
SPIKE Seminar13
Results - Answers to research question RQ3
• RQ3: What are process scenarios of the projects using COTS components successfully and not successfully? What are the relationship between possible risks and process scenarios
• Data:
– 12 successful projects and 4 unsuccessful projects
– 6 Scenarios
– 6 Lessons learned
SPIKE Seminar14
Results - Answers to research question RQ3 (cont’)
ID Characterizations Lesson learned
Sc1 Two projects used waterfall processes.
Made the build vs. buy decision and the COTS component selection in the implementation phase.
Selected unfamiliar COTS components.
Lesson 1: The COTS component selection should be implemented no later than the design phase in the waterfall processes.
Lesson 2: They should inform the customer after they decided to use COTS components.
Sc2 One project used waterfall process.
Selected COTS components in design phase but mainly on their functionalities.
Selected unfamiliar COTS components.
Lesson 3: They should consider more on how to integrate the COTS components. If the COTS components were hard to be integrated, they should build alternative components themselves.
Sc3 One project used incremental & prototyping process. Selected the COTS components in design phase but mainly on their functionalities. Selected unfamiliar COTS components.
Lesson 3: See above
Scenarios of 4 unsuccessful projects
SPIKE Seminar15
Results - Answers to research question RQ3 (cont’)
Scenarios of 12 successful projects
ID Characterizations Lesson learned
Sc4 Two projects used waterfall process.
Made the build vs. buy decision and selected COTS components in the requirement phase.
Selected familiar COTS components.
Lesson 4: Selected the COTS components mainly based on the easiness of integration.
Sc5 One project used waterfall with some prototyping process.
Made the build vs. buy decision and selected COTS components in design phase.
Selected familiar COTS components.
Lesson 4: See above
Sc6 Nine projects used incremental with prototyping processes.
Seven projects selected COTS components in requirement with familiar COTS components.
Two projects selected COTS component in design phase with unfamiliar COTS components.
Lesson 5: Using COTS components in the incremental & prototyping process helped the success of the project.
Lesson 6: Rank the integration of unfamiliar COTS components as high risk task and integrate such components before other components.
SPIKE Seminar16
Results - Answers to research question RQ3 (cont’)
• Answer: COTS-based development could be performed successfully using waterfall and evolutionary processes. However, different process scenairos may face different risks.
SPIKE Seminar17
Variations in COTS-based Software development Processes
Design
Waterfall
Incremental& prototyping
SP_unfam
Waterfall with unfamiliar COTS component
Waterfall with familiar COTS component
Incremental & prototyping with familiar COTS component
Incremental & prototyping with unfamiliar COTS component
Requirement Implementation
RequirementSP_fam
DesignSP_fam
Implementation
RequirementSP_unfam
DesignSP_unfam
Implementation
RequirementSP_fam
DesignSP_fam
ImplementationSP_fam
SP_fam: Familiarity-based selection processSP_unfam: Internet search, trial-based selection processPossible COTS-based development process customizations
SPIKE Seminar18
Discussions
• Comparing with related work– Boehm et al.
– EPIC & NASA process
– Proposed COTS selection processes
• Possible threats to validity– Internal validity
– External validity
– Construct validity
– Conclusion validity
SPIKE Seminar19
Conclusions
• Use of COTS components can be done as part of traditional development processes (e.g. waterfall and evolutionary) – there is no special “COTS-based development process”.
• Successful use of COTS components in such processes, however, requires that some new activities and roles are introduced in order to reduce risks. Typical new activities are the build vs. buy decision, COTS component selection, and COTS component integration. A new role is that of a knowledge keeper.
• Two COTS component selection processes have been verified in practice, i.e. familiarity-based selection process and process combining Internet searches with hands-on trials.
• Two of the new activities, the build vs. buy decision and the COTS component selection, can be placed in different development phases (requirements, design, or implementation). This depends on project context, especially on the familiarity with possible COTS components and the flexibility of requirements.
• A set of explanatory scenarios and guidelines have been synthesized to assist in the customization of the traditional development processes with the new activities and roles.
SPIKE Seminar20
Section 2: Risk management in OTS (COTS and Open Source) based development
• State-of-the-art
• Research motivation and research questions
• Research methods
• Results
• Discussions
SPIKE Seminar21
State-of-the-art
• OTS component based development processes• Risks in component-based development
– Component functionalities may not sufficiently match system requirements. – Whenever an assembler employs a component from a less-reputable source, it is
difficult to gauge the quality of the corresponding component-based system.– Component-based assembly requires personnel with a different set of skills than the
one employed in traditional life-cycle development etc.
• Risks in COTS component based development– Insufficient effort planned for integration, i.e. glue-code development and testing.– COTS components could not be sufficiently adapted to changing system
requirements.– Ineffective management of the vendor relationship etc.
• Risks in Open Source Component based development– Security– Quality etc.
SPIKE Seminar22
State-of-the-art (cont’)
• Proposed risk management strategies– Involving customers into the “acquire” vs. “build” decision.
– Significant quality features must be tested before finally selecting OTS components.
– A most likely OTS component learning curve must be accounted for in cost estimation.
– Involving OTS-knowledgeable individuals in all analytical processes.
– Using a stringent configuration control etc.
SPIKE Seminar23
Research frameworkDecide the whole
development process(When and how? RQ2)
Make vs. Buy decision(Why? RQ1)
OTS selection(When and how? RQ2)
OTS integration
Have found the proper OTS ?
OTS maintenance and evolution
Y
Start project level risk mitigation (RQ3)Y
N
Non-OTS process
Decide to use OTS ?
N
Start OTS level risk management (RQ4)
Start project level risk identification and evaluation (RQ3)
P1
P2
P3
P4
SPIKE Seminar24
Research questions
• RQ1: What are the main motivations of using OTS components instead of building in-house?
• RQ2: What are the real common OTS-based process models, and what are the possible variances?
• RQ3: What project-level risks have been identified and their relationships with the project contexts? Which risk reduction methods have been performed from the project level and the results?
• RQ4: What OTS component-level risks have been identified and their relationships with OTS component characteristics? Which risk reduction methods have been done for specific OTS components and the results?
SPIKE Seminar25
Research methods
• Survey (questionnaire) • Sample selection strategies
– Norway We gathered a company list from the Norwegian “Census Bureau ”. We included
mostly companies which were registered as IT companies. Based on the number of employees, we selected the 115 largest IT companies (100 IT companies and 15 IT departments in the largest 3 companies in 5 other sectors), 150 medium-sized software companies (20-99 employees), and 100 small-sized companies (5-19 employees) as the original contacting list.
– Italy We first got 43580 software companies from yellow pages. We then randomly
selected companies from them. For these randomly selected companies, we read their web-site to ensure they are software companies or not. 196 companies were finally clarified as software companies, and were used as the original contacting list.
– Germany We selected name list from a company list from an organization similar to the
Norwegian “Census Bureau”. We then used the existing IESE customer database to get contact information of relevant IT/Software companies, in line with the Norwegian selection.
SPIKE Seminar26
Research methods (cont’)
• Data collection procedure– The final questionnaire was first designed and pre-tested in English
(internal and external preview). It was then translated into the native languages and published on the SESE web survey tool at Simula Research Lab.
– Possible respondents were contacted by telephone first. If they have suitable OTS-based projects and would like to join our study, a username and password will be sent to them, so that they can log into the SESE web tool to fill in the questionnaire (they can also use paper version).
– The respondents who didn’t want to answer the questionnaire was also registered so that we could find the underlying reasons, e.g., no OTS-based projects, “busy”, not interested etc.
SPIKE Seminar27
Preliminary results
• Data collection is still on-going
• 15 filled-in questionnaire (9 from Norway, 6 from Italy) till the middle of August.
• Summarized the answers to RQ3 based on current data.
SPIKE Seminar28
Answers to RQ3 - Project level risksID Risks Relative
frequency (1..5)
Rg Requirements were changed a lot 3.33
Rb Effort to select OTS components was not satisfactorily estimated 2.79
Ro Provider did not provide enough technical support/ training 2.33
Rc Effort to integrate OTS components was not satisfactorily estimated 2.29
Rl It was difficult to update the system with the last OTS component version 2.29
Ra The project was delivered long after schedule 2.20
Rh OTS components could not be sufficiently adapted to changing requirements 2.20
Ri Project could not (re)negotiate requirements with the customer, if OTS components could not satisfy all requirements
2.08
Rj It was difficult to identify whether defects were inside or outside the OTS components 1.93
Rk It was difficult to plan system maintenance, e.g. because different OTS components had asynchronous release cycles
1.93
Rf OTS components negatively affected system performance 1.87
Rn Information on the reputation and technical support ability of provider were inadequate 1.85
Rm OTS components were not satisfactorily compatible with the production environment when the system was deployed
1.80
Rd OTS components negatively affected system reliability 1.60
Re OTS components negatively affected system security 1.50
SPIKE Seminar29
Answers to RQ3 (cont’) – Project level risk management
ID Risk management action Relative frequency
Mh Did integration testing incrementally (after each OTS component was integrated 3.64
Md OTS components qualities (reliability, security etc.) were seriously considered in the selection process 3.33
Me Effort in learning OTS component was seriously considered in effort estimation 2.87
Ma Customer had been actively involved in the “acquire” vs. “build” decision of OTS components 2.73
Mf Effort in black-box testing of OTS components was seriously considered in effort estimation 2.46
Mb Customer had been actively involved in OTS component selection 2.33
Mg Unfamiliar OTS components were integrated first 2.29
Mj Maintained a continual watch on the market and looked for possible substitute components 2.20
Mi Local OTS-experts actively followed updates of OTS components and possible consequences 2.14
Mc OTS components were selected mainly based on architecture and standards compliance, instead of expected functionality
2.13
Mk Maintained a continual watch on provider support ability and reputation. 2.07
SPIKE Seminar30
Answers to RQ3 (cont’) –Relationships between risks and risk management
Risks Risk management actions Correlation P-value
Ra Ma, -.414 .013
Ra Mb -.386 .008
Risks Risk management actions Correlation P-value
Rd Md -.365 .035
Rd Mh -.366 .030
Risks Risk management actions Correlation P-value
Rh Ma -.414 .000
Rh Mb -.325 .028
Cost-estimation risk
Quality risk
Scope and requirement risk
Somers’d in SPSS Version 11.0
SPIKE Seminar31
Preliminary conclusions• Comparing with related work
– The scope and requirement risks are the relative most frequent risks. Our results also show that customer involvement can help to reduce the scope and requirement risk. Customers can not only be involved into the “acquire” vs. “build” decision, they can also be involved in the selection of OTS component.
– Estimation of selection and integration costs in OTS-based projects is perceived challenging. Most proposed risk mitigation methods focus on solving this problem by experienced project manager or a formal cost-estimation model. Our results show that customer involvement may help make it easy to estimate.
– Most previous studies regard quality (reliability, security, or performance) risks are very challenging in OTS component-based development. Our results show that they are not as frequent as assumed. Project managers used careful selection and incremental testing to help to mitigate OTS components’ negative effect on the quality of the system.
– For maintenance risks, these are perceived important but with less level of control.
SPIKE Seminar32
Discussions
• Methodology lessons learned– Definition of IT companies?
– Sample selection (Representative IT companies may not be the representative sample of OTS component-development)
– Compare the representativeness of different selection strategies in different countries
– Motivation of survey?
– Compare with structure interview
SPIKE Seminar33
Questions & Comments ?
