cds.cern.ch · INFSO-RI-261611 2010-2013 © Members of EMI collaboration PUBLIC 1 / 41 EUROPEAN MIDDLEWARE INITIATIVE DSA2.3.4 - PERIODIC QA AND QC REPORTS. EU DELIVERABLE: D4.3.4

INFSO-RI-261611 2010-2013 © Members of EMI collaboration PUBLIC 1 / 41

EUROPEAN MIDDLEWARE INITIATIVE

DSA2.3 .4 - PE R I O D I C QA A N D QC RE P O R T S

EU DELIVERABLE: D4.3.4

Document identifier: EMI-DSA2.3.4-1277603-Periodic_QA_and_QC_Reports_M36.docx

Date: 30/04/2013

Activity: SA2

Lead Partner: CERN, CINECA

Document status: Final

Document link: https://cds.cern.ch/record/1277603

Abstract: This document is the report on the compliance with and results of the quality assurance and quality control process during the third year of the EMI project.

https://cds.cern.ch/record/1277603

I. DELIVERY SLIP Name Partner/Activity Date

From

Alberto Aimar, Giuseppe Fiameni, Michele Carpenè,

Maria Alandes Pradillo,

CINECA, CERN/

SA2 30/04/2013

Reviewed by Andrea Ceccanti

Jon K. Nilsen Florida Estrella

INFN/SA1 UIO/JRA1

CERN/NA1 23/05/2013

Approved by PEB - 23/05/2013 II. DOCUMENT LOG

Issue Date Comment Author/Partner

1 30/04/2013 Draft v1.0 for review Maria Alandes Pradillo, Alberto Aimar/CERN Giuseppe Fiameni/CINECA

2 20/05/2013 v1.2 for review Maria Alandes Pradillo, Alberto Aimar/CERN Giuseppe Fiameni/CINECA

3 22/05/2013 v1.3 for review Maria Alandes Pradillo, Alberto Aimar/CERN Giuseppe Fiameni/CINECA

4 23/05/2013 Final v1.0 approved by PEB PEB III. DOCUMENT CHANGE RECORD

Issue Item Reason for Change 1 2

IV. DOCUMENT AMENDMENT PROCEDURE

This document can be amended by the authors further to any feedback from other teams or people. Minor changes, such as spelling corrections, content formatting or minor text re-organization not affecting the content and meaning of the document can be applied by the authors without peer review. Other changes must be submitted for peer review and to the EMI PEB for approval.

When the document is modified for any reason, its version number shall be incremented accordingly. The document version number shall follow the standard EMI conventions for document versioning. The document shall be maintained in the CERN CDS repository and be made accessible through the OpenAIRE portal. V. GLOSSARY

ABI Application Binary Interface

ACR Approved Change Request

API Application Programming Interface

CDS CERN Document Server

DSA2.3.4 - PERIODIC QA AND QC REPORTS Doc. Identifier:EMI-DSA2.3.4-1277603-Periodic_QA_and_QC_Reports_M36.docx Date: 30/04/2013


CG Change Request

CSIC Consejo Superior de InvestigacionesCientificas

DCI Distributed Computing Infrastructure

DMSU Deployed Middleware Support Unit

EGI European Grid Infrastructure

EMT Engineering Management Team

ETICS eInfrastructure for Testing, Integration and Configuration of Software

FPVA First Principles Vulnerability Assessment

GGUS Global Grid User Support

ITIL IT Infrastructure Library

KPI Key Performance Indicator

kSLOC Kilo Source Lines Of Code

MCB Middleware Coordination Board

NGI National Grid Initiative

PEB Project Executive Board

PTB Project Technical Board

QA Quality Assurance

QC Quality Control

RfC Request for Change

SLA Service Level Agreement

SQAP Software Quality Assurance Plan

SQC Software Quality Control

SU Support Unit

VC Validated Change The complete EMI glossary is available at https://twiki.cern.ch/twiki/bin/view/EMI/EmiGlossary.

https://twiki.cern.ch/twiki/bin/view/EMI/EmiGlossary



VII. COPYRIGHT NOTICE

Copyright (c) Members of the EMI Collaboration. 2010-2013. See http://www.eu-emi.eu/about/Partners/ for details on the copyright holders. EMI (“European Middleware Initiative”) is a project partially funded by the European Commission. For more information on the project, its partners and contributors please see http://www.eu-emi.eu. This document is released under the Open Access license. You are permitted to copy and distribute verbatim copies of this document containing this copyright notice, but modifying this document is not allowed. You are permitted to copy this document in whole or in part into other documents if you attach the following reference to the copied elements: "Copyright (C) 2010-2013. Members of the EMI Collaboration. http://www.eu-emi.eu". The information contained in this document represents the views of EMI as of the date they are published. EMI does not guarantee that any information contained herein is error-free, or up to date. EMI makes no warrantees, express, implied, or statutory, by publishing this document.



TABLE OF CONTENTS 1. INTRODUCTION ......................................................................................................................................... 6

1.1. EXECUTIVE SUMMARY ............................................................................................................................. 6 1.2. PURPOSE AND SCOPE ............................................................................................................................... 7 1.3. DOCUMENT ORGANIZATION .................................................................................................................... 7

2. QUALITY ASSURANCE ............................................................................................................................. 8 2.1. STATUS OF SOFTWARE QUALITY ASSURANCE PLAN ................................................................................ 8 2.2. STATUS OF THE POLICIES ......................................................................................................................... 8 2.3. STATUS OF THE QA DOCUMENTATION AND TOOLS .............................................................................. 8 2.4. METRICS AND DASHBOARDS ................................................................................................................... 8

2.4.1 Verification Dashboard ............................................................................................................... 8 2.4.2 Request for Change Dashboard ................................................................................................. 10 2.4.3 Automated Metrics Generation .................................................................................................. 11

3. QUALITY CONTROL ............................................................................................................................... 12 3.1. VERIFICATION OF RELEASED PRODUCTS ................................................................................................ 12

3.1.1 The verification methodology..................................................................................................... 13 3.2. THE EMI 1 RELEASE .............................................................................................................................. 13

3.2.1 Verification activity .................................................................................................................... 13 3.2.2 EMI 1 Coding and Testing Metrics report ................................................................................. 18

3.3. THE EMI 2 RELEASE .............................................................................................................................. 19 3.3.1 Verification activity .................................................................................................................... 19 3.3.2 EMI 2 Coding and Testing Metrics report ................................................................................. 24 3.3.3 EMI 2 General Metrics report ................................................................................................... 25

3.4. THE EMI 3 RELEASE .............................................................................................................................. 26 3.4.1 Verification activity .................................................................................................................... 26 3.4.2 EMI 3 Coding and Testing Metrics report ................................................................................. 32 3.4.3 EMI 3 General Metrics report ................................................................................................... 33

3.5. STATIC ANALYSIS OF PRODUCT PACKAGES (EPEL COMPLIANCE) .......................................................... 34 4. CONCLUSIONS AND OUTLOOK ........................................................................................................... 38

4.1. EMI 3 IMPROVEMENTS .......................................................................................................................... 38 4.2. OUTSTANDING QUALITY ACHIEVEMENTS OF EMI 3 ............................................................................... 39 4.3. OUTLOOK AND POST-EMI PLANS .......................................................................................................... 39

5. REFERENCES ............................................................................................................................................ 40



1. INTRODUCTION

1.1. EXECUTIVE SUMMARY This report, as in the previous years, presents the changes to the quality assurance policies and metrics and the actual results of how EMI products adhere to the policies and the values of each metric measured on the released EMI product.

This report focuses on the status of the EMI QA activity during the third year of the project on the EMI 1 and EMI 2 Updates and on the EMI 3 Release that were performed in that period. Policies and metrics that have not changed in the third year are summarized and references to the complete information are included.

The policies and metrics have basically not been changed in the third year of the project, as expected, and this is very good indication the policies have been adequate and useful for the goals of the project.

As with last year, the metrics focus on measuring, monitoring and steering the testing process and fostering the adoption of EMI packages to the EPEL/Fedora and Debian standard. The reason for this is to give to EMI products a certain level of tested quality to ease the inclusion in the main Fedora and Debian package repositories.

The two dashboards [R36] [R37] developed in the QA activity, used by Quality Control and product teams, have shown to be a fundamental tool to display the status of products and processes. The Verification Dashboard gathers and process information from the release tracker, and the certification and test reports to present a single view of the status a product in a release. The RfC (Request for Change) Dashboard is a tool that offers a unique entry point to track software request for changes for all EMI products, aggregating data coming from all the RfC trackers in use by EMI PTs.

The compliance level of QA policies and procedures has been measured and reported weekly by the QC team which was mainly focused on verifying that software products included in EMI 1, EMI 2 and EMI 3 releases and updates were compliant with the Production Release Criteria, and verifying that the released products were compliant with the EPEL and Debian packaging criteria.

The QA status and the main achievements of the EMI project in the third year are:

• 100% of EMI products now provide build from sources in addition to binary distributions;

• 100% of the high and emergency RfCs are covered by associated tests in all recent EMI 1 and EMI 2 updates and for the EMI 3 Major release;

• 100% of EMI products have Deployment Test, Basic Functionality Test, 83% Automated Regression tests, all cases in which test could be automated;

• 99.3% of EMI products were immediately accepted by the EGI Quality Criteria (149/150);

• 90% of the packages are EPEL compliant with less than two errors per package;

• The largest EMI packages (ARC, dCache, DPM, etc) are now already in EPEL; in 2010 no major EMI product was in EPEL;

• 90% of the products aiming at Debian compliance have less than 10 errors.

All QA metrics and statistical data summarized in this document are extracted automatically and continuously from the release trackers and the integration tools used to release EMI products.



1.2. PURPOSE AND SCOPE

The purpose of the Periodic QA and QC Report is to inform and critically analyse the progress of the software quality assurance process within the EMI project and to report on the quality verifications performed on the EMI products and their results. In the past the QA and QC reports were two separate documents. After the first EC review, it has been suggested to merge the reports as they had very interlinked purposes and scopes.

The initial section on Quality Assurance describes the changes to the QA process implemented in the current year of the project and reports on the changes to the QA policies and on the implementation of a tool aimed at facilitating and automating QC verifications.

The main purpose of the Quality Control section, as mandated by the Software Quality Assurance Plan (SQAP), is to summarize the status and performance of quality control of the EMI software development activities. It summarizes the results of the EMI software components’ reviews and provides details about the availability and execution of unit, functional and compliance tests for the EMI components.

This report focuses on the status of the QC activity during the third year of the project on the EMI 1 and EMI 2 Updates and particularly on the EMI 3 Release.

1.3. DOCUMENT ORGANIZATION The document, as in the previous years, is organized as follows:

• Chapter 1 is the Executive Summary of the reported work;

• Chapter 2 presents the status of the Quality Assurance plan, including updates on polices and quality metrics;

• Chapter 3 reports the results of the Quality Control activities concerning EMI 1 and EMI 2 updates and EMI 3 release;

• Chapter 4 concludes this deliverable with a summary of the third year improvements and achievements.



2. QUALITY ASSURANCE

2.1. STATUS OF SOFTWARE QUALITY ASSURANCE PLAN The Software Quality Assurance Plan (SQAP) [R1, R3, R5, R6] did not change during the last year. The SQAP was consolidated at the end of October 2011 and since then it has been a reference document describing the SQA strategy of the project.

2.2. STATUS OF THE POLICIES The policies describing the EMI release process [R8] did not change during the last year. They were reviewed for EMI 2, which introduced the SL6 and Debian 6 platforms, and since then they have guided EMI members throughout the different EMI major releases and updates.

The stability of the policies is a confirmation of the maturity reached by the QA activities within the project and their full adoption by the PTs.

For information here are the references to all the QA policies of the EMI project:

• Release Management [R8]

• Change Management [R11]

• Configuration and Integration [R9]

• Packaging [R28]

• Testing [R13, R34]

• Documentation [R35, R28]

• Certification [R10]

• Requirements [R28]

2.3. STATUS OF THE QA DOCUMENTATION AND TOOLS Due to the well-known and established process steered by the Release Manager, the EMI QA documentation [R12, R13, R14] was stable with no changes necessary during the third year of the project. For the same reason, very little support was needed.

2.4. METRICS AND DASHBOARDS There were no new metrics [R2] defined in the last year. EMI 3 metrics reports have been produced regularly following the same approach used in EMI 2 and the data is reported and commented in the QC sections of this document.

Also in the third year of the project the QA dashboards were an essential tool to monitor the products and the processes of the project.

2.4.1 Verification Dashboard The Verification Dashboard [R37] introduced in the previous year was used daily by the QC team and by the PTs before submitting their products for release. It gathers and process information from the savannah tracker, the certification and test reports and presents a consolidated view of the status of quality checks for a product in a release.



During the verification process, QC uses the verification dashboard to review the automatic checks, and to provide answers to the manual checks needed to generate the verification report. As shown in Figures 1 and 2, via a single web portal the whole content of the release can be viewed: one can browse through all products and all the reports of the certification, documentation and testing performed by the PT.

Figure 1: Quality Control verification dashboard with all the EMI updates and releases



Figure 2: Quality Control verification dashboard with details of a specific product checks

2.4.2 Request for Change Dashboard The RfC (Request for Change) Dashboard [R36] is the other important verification tool that offers a single entry point to track software changes, like defects and new features, for EMI products. This dashboard is aggregating data coming from all the RfC trackers in use by the PTs. The EMI QA policies define a common release process that is followed by the PTs, including the minimum set of states present in the tracking tools. With this dashboard one can see the status of each RfC across the whole EMI project.



Figure 3: EMI RfC Dashboard

2.4.3 Automated Metrics Generation The automation of metrics generation is a major achievement for SA2. Automated metrics generation facilitiates the production of metrics reports and fulfil requests for new metrics. In this way, the metrics activity gives support to other activities of the project that can benefit from metrics results. All QA metrics and statistical data used in the project reported in the weekly reports and summarized in Section 4 of this document are extracted automatically from EMI product and processes.



3. QUALITY CONTROL

The aim of the Quality Control activity is to verify the application of Quality Assurance (QA) processes and policies, identifying unacceptable or non-conform results, helping the Project Executive Board (PEB) to undertake corrective actions to mitigate negative impact on future project results.

This document presents the results of the Quality Control activities being measured during the course of the third year of the project that has been mainly focused on releasing and updating the EMI product for a total of 132 product releases:

• 9 updates for EMI 1, for a total of 25 products

• 12 updates for EMI 2, for a total of 57 products

• Third (and final) major EMI 3 release, for a total of 50 EMI products

The measurements collected are provided for EMI 1 and EMI 2 updates released from June 2012 to April 2013, grouped by major release, and for EMI 3 that was released at the end of February 2013.

These results are presented in aggregate views and compared with past results to emphasize the continuous improvement of released products.

The role of Quality Control task has already been described in detail in the previous documents and the third year of the project was completed, a summary list of the main QC tasks is provided below:

• Verify that EMI products satisfy the Production Release Criteria;

• Perform a security assessment of released products, at least the most critical ones;

• Verify that released products satisfy distributions packaging criteria for EPEL and Debian;

• Prepare reports where results of periodic control activities are reported pointing out any existing nonconformity or deviation.

3.1. VERIFICATION OF RELEASED PRODUCTS The release of each software product during EMI year 3 has been done according to the same exact sequence of steps followed during the verification of the past EMI updates.

According to the procedure, Release Tasks containing basic information about the product (name, version, release notes, the list of RfCs and packages) are created on the Savannah tracker each time a new product needs to be released, then the Release Manager approves the release of each product according to the release policy [R25, R5].

Once the release task transits from “Open” to “Certified” the QC takes care of verifying that all the mandatory checks are passed so that the product can be deployed on the EMI testbed. The task transits to “Ready for Testbed” and, when the product deployment has been successfully completed, the task transits to “Deployed on the Testbed”.

Finally the QC performs the last checks and changes the task status to “Verified”.

The list of mandatory checks a product needs to comply has not changed after EMI 2 release:

• All RfCs are listed in the release task;

• All RfCs are Tested and Relevant RfCs have an associated Regression Test, listed in the release task;

• VCS Tag is available in the certification report;



• The Certification/Test Reports are compliant with SA2 policies and complete in all sections;

• The ETICS subsystem configuration for the product builds without errors;

• Binaries and source packages are available in the repository and listed in the release task;

• Unit test results, functionality test results, regression tests results have been provided;

• Release notes are specified in the tracker and are compliant with SA2 policies;

• Test plan [R13, R22] is referenced from the release tracker;

• Documentation is referenced from the release tracker and up-to-date.

All these checks need to be satisfied to put the task on “Ready for Testbed”. If the product deployment is successful, the task is moved first to “Deploy on the Testbed” and then to “Verified”.

During the verification of the last EMI 1, EMI 2 updates and the EMI 3 major release no particular issues were found and the verification of all the components proceeded easily.

3.1.1 The verification methodology During the verification activity, from June 2012 to April 2013, 25 products were tested for the EMI 1 updates (16-24), 57 products were tested for the EMI 2 updates (1-11) and 50 products were tested for the EMI 3 major release. For each product verified the “Verification Report” was generated and included, as well as product information, into the Savannah Task created by the Release Manager.

The most part of the checks were performed using the “Verification Dashboard”, the web application developed by SA2 to automate the verification process. This tool reduces the number of manual errors significantly also providing QC people with products information in the form of CSV summaries. These summaries are then used to generate graphs to be included into the reports. Complete tasks verification requires today only a couple of minutes and a very few tasks submitted were found incomplete; as a consequence the verification of new updates and EMI 3 products was completed easily.

In the following sections, measurements about the EMI 1, EMI 2 updates and the EMI 3 release are reported, including some conclusions on the achieved performance and final considerations.

3.2. THE EMI 1 RELEASE

3.2.1 Verification activity During the verification activity from June 2012 to April 2013, 25 products for the EMI 1 updates (from 16 to 24) were fully verified.

Status of the verification of the products During the verification of EMI 1 updates almost all the tasks were found compliant, the table below shows results for EMI 1 updates from 16 to 24, where only 1 task had to be accepted after a minor modification.

State transition Product passed the QA verification in EMI1, updates 16-24/Total number of products

From Certified to Open 25/25 From Ready for Testbed to Open 25/25 From Ready for Testbed to Certified 24/25 From Verified to Deployed on Testbed 25/25



Availability of software tests The following charts present the measurements collected for the EMI 1 release and updates concerning the availability of software tests (unit and regression) and their coverage for high/immediate priority defects.

Figure 4: Availability of unit tests for EMI 1 updates (Y1, Y2 and Y3)

Figure 5: Availability of regression tests for EMI 1 updates (Y1, Y2 and Y3)

The number of available regression tests has gradually improved over the course of the project ranging from a 47% of availability for the EMI 1 Major release in Year 1 to 82% for the most recent updates in Year 3. Results here presented have been collected starting from the test reports provided by PTs during the certification of their respective products. One should notice that not all products can be tested with useful unit tests and that regression tests on distributed scenarios are often very complex.



Comparing the Figures 4 and 5 with the charts below (Figures 6, 7 and 8) the improvement of products quality using two different sources of information is noticeable. The first source is the test reports and the second sources are the EMI feature/bug trackers.

Figure 6: Number of Associated Tests for High/Immediate priority defects

The overall availability of tests has progressively increased over the time and the coverage of tests for high/immediate priority defects follows a positive trend.

Concerning High/Immediate priority defects, for the EMI 1 updates (starting from 16 to 24) a good level of coverage can be observed while in the past (i.e. update 8) results were unsatisfactory. In the case of associated tests for features, results obtained seem good (no new features without an associated test). In addition the fact that in Figure 7 fewer High/Immediate Priority bugs needed to be fixed is a indicator of the good quality of the software in particular after the few major updates of the first year.

According to the collected measurements, the following conclusions can be made:

• The number of developed tests (i.e. unit, regression) has progressively improved and all the features/bugs marked with high/immediate priority are covered;

• The number of unit tests has slightly improved while regression tests percentage is stationary at 80%.

Availability of binary/source packages

During the verification of products for the period from June 2012 to April 2013 a significant share of binary and source packages was provided by PTs. This must be considered a very good result for the project. During the verification process, it may still happen that packages are missing, for example because the PT didn’t provide the correct link. As in the past, the QC team stops the release process



and report one non-conformity. This kind of issue never occurred during the verification of EMI products in the last year as the teams are now familiar with and comply with the established release and documentation process.

The charts below report how many packages, both binary and source, were present during the verification of EMI 1 updates.

Figure 7: Availability of binary packages (Y1, Y2 and Y3)

Figure 8: Availability of source packages (Y1, Y2 and Y3)

As we can see in Year 3 a 100% score was achieved: for all EMI 1 updates 16-24 the binary and source packages were always provided.



Certification report conformance

During the verification of EMI products, the QC checks if the Certification Reports are compliant with the SA2 Certification Report Template. In case of inconsistency, the QC can view the corresponding error from the Verification Dashboard. In the EMI 1 updates from 16 to 24 no task was found Non-Compliant; all the tasks were associated with a correct Certification Report.

The chart below shows the percentage of Certification Reports compliant with the SA2 policy in the EMI 1 release (71%) and for the updates 1-16 (86%) and 16-24 (100%) that confirm an admirable progression in the QA practices of the development teams.

Figure 9: % of compliant certification reports in EMI 1 major release and updates (Y1, Y2 and Y3)

Percentage of passed checks

Figure 10: % of passed checks for EMI 1 release and relative updates (Y1, Y2 and Y3)



The number of passed checks is collected directly from the Verification Dashboard; this number represents the number of mandatory checks that passed the verification for each single task. QC people are able to download a “Passed Mandatory Checks summary file” generated automatically from the dashboard.

The graph above shows the percentage of passed checks in the EMI 1 updates from 1 to 15 and from 16 to 24. The number of passed checks has grown continuously starting from EMI 1 major release: for the EMI 1 release the percentage of passed checks is the 63%, the EMI 1 updates from 1 to 15 have the 92% of passed checks, the EMI 1 updates from 16 to 24 have the 99% of passed checks.

3.2.2 EMI 1 Coding and Testing Metrics report Coding and testing metrics reports are prepared per update and contain measurements to evaluate testability and maintainability quality characteristics. The following table reports average results of collected measurements.

Quality characteristic

Metrics Required Level Assessment Actual Result

Testability

Number of Test Plans per released software product.

One per software product.

100% Completely fulfilled

Number of Test Reports per released EMI software product.



Number of mandatory tests per EMI software product.

Ideally 100%, however an improvement per product is also a good indicator.

Major improvements can be observed reaching approximately the 90% of compliance for the most recent updates.

Number of RfCs tracking a defect (High/Immediate priority) with an associated regression test.

Ideally 100%, although an increased value per product per release is also a good indicator.

100% Completely fulfilled Available results report a positive trend for last EMI 1 updates al have regression tests

Number of RfCs tracking a new feature with an associated functionality test.


100% Completely fulfilled From June 2012 to April 2013 (Figure 8) this was fulfilled at 100%.

Number of development tasks tracking a new feature with an associated functionality test.


100% Completely fulfilled (Figure 6).

Number of passed Certification 100% for the checks 100%



Checks considered in the Production Release Criteria.

Completely fulfilled (Figure 9)


3.3.1 Verification activity During the verification activity from June 2012 to April 2013, 57 products for the EMI 2 updates (from 1 to 11) were fully verified.

Status with the verification of the products

The number of compliant tasks in EMI 2 updates has greatly increased during the last year. Sometimes occurs that PTs still do not provide regression tests or unit tests, but despite existing non-conformities only one task was found non-compliant and thus rejected.

State transition Product passed the QA verification in EMI 2, updates 1-11/Total number of products


Availability of software tests

The following charts present the measurements collected for the EMI 2 Major release and updates from 1 to 11 on the availability of software tests (Unit and Regression) and their coverage for high/immediate priority defects.

Figure 11: Availability of unit tests for EMI 2 major release and updates (Y2 and Y3)



Figure 12: Availability of regression tests for EMI 2 major release and updates (Y2 and Y3)

Although the percentages were already high both for Unit and Regression tests a small improvement can be seen. The Unit tests availability has improved from 69% to 75% while the Regression tests availability is stationary to 83/84%.

One should notice that for some products it is not easy to provide useful unit tests because the products are part of a distributed service or not easily embeddable software for unit testing.

The availability of Unit Tests have been the most deficient control for the EMI 1 and EMI 2 releases, but at the end a good result has been achieved for this measurement. Results on test availability have been collected directly from test reports provided by respective PTs, now these can be compared with results from EMI feature/bug trackers.

Figure 13: Number of associated tests for high/immediate priority defects in EMI 2 and updates



The chart above presents measurements gathered directly from release trackers of middleware consortiums on the test coverage for High/Immediate priority defects for EMI 2 major release and updates from 1 to 11. Actually the coverage level for High/Immediate priority defects is approximately of the 90%.

According to the collected measurements, it is possible to conclude that:

• The availability of unit tests and regression tests has further improved, even if unit tests are always slightly less than regression tests;

• The coverage for high/immediate priority features/bugs for EMI 2 updates appears satisfactory.

The charts above show how the information collected from the test reports of released products differs from that of software release trackers, but in the end one gets good and consistent results.

For completeness Figures 15 and 16 show the Number of Associated Tests for Defects/Features only for the EMI 2 updates 1-11.

Figure 14: Number of Associated Tests for Defects in EMI 2 updates 1-11



Figure 15: Number of Associated Tests for Features in EMI 2 updates 1-11

As one can see, for EMI 2 updates 1-11 the 70% of features (14 over 20) report a functionality test. A good improvement respect to the EMI 2 major release (27%).


Figure 16: Binary availability in EMI 2 release and updates 1-11 (Y2 and Y3)



Figure 17: Binary availability in EMI 2 release and updates 1-11 (Y2 and Y3)

All PTs provided right binary and source packages for their products and correctly listed them in the release task for EMI 2 updates from 1 to 11. This result confirms a great achievement for the Quality Control as the maximum result possible has been measured for the EMI 2 updates as well as for the EMI 2 major release.

Certification report conformance

All PTs provided compliant certification reports for the EMI 2 release and the updates for a total of 108 products releases.

Figure 18: % of EMI 2 release and updates providing updates (Y2 and Y3)




The graph below shows that the percentage of passed checks in EMI 2 updates, in the period from June 2012 to April 2013, has grown with respect to EMI 2 major release. The percentage has reached the 100% of passed checks as opposite to EMI 2 where a lower percentage (97.26%) was achieved.

Figure 19: % of passed checks for EMI 2 release and updates (Y2 and Y3)

3.3.2 EMI 2 Coding and Testing Metrics report Quality

characteristic Metrics Required Level Assessment Actual

Result

Testability



Completely fulfilled


One per released software product.




100% Deployment Test. 100% Functionality Test 84% Automated Regression tests 75% Unit Tests



For EMI 2 Updates 1-11 the 90% of RfCs have an associated regression test.



Figure 18 shows that today the 70% of features (14 over 20) report a functionality test for EMI 2 updates 1-11. A good improvement respect to the only EMI



2 major release (27%)

Number of passed Certification Checks

100% for the checks considered in the Production Release Criteria.


3.3.3 EMI 2 General Metrics report Quality


Result

Suitability Number of implemented development tasks.

100% Measurements for this metric are not available.

Functionality compliance

KPI KJRA1.1 Number of EMI service interfaces and libraries passing standard compliance tests

JRA1 aims to increase the number of successful tests at least by two per reporting quarter [R4]

All standard interfaces were tested and the test passed from 0 to 8 during EMI 2.

Security

Number of EMI Security Assessments.

At least one per EMI software product targeted for this.

Two products were test during the project.

Number of fixed security vulnerabilities reported in the EMI security assessments.

100% security vulnerabilities according to the proposed deadlines.


Number of fixed EGI SVG.

No EGI SVG in this year.

Recoverability

Number of services providing high-availability setups.

After identifying which services are suitable for this setup, the number should increase over time until the end of the project.

These measurements are part of the EGI Quality Criteria and all EMI products passed them.

Understandability

Number of missing mandatory documents.

Ideally 0, although a decreasing number per product per release is also a good indicator.

These measurements are part of the EGI Quality Criteria and all EMI products passed them.

Number of EMT [R22, R23, R24] tasks tracking documentation issues.

< 5 per week. The documentation issues were a handful during the whole year.

Operability

Number of services providing service control and status mechanisms.

All services should provide this feature. The number should increase over time until the end of the project.

All services provide this features also verified by the EGI Acceptance Criteria

Number of services providing configuration tools.

Measurements for this metric are not available.

Testability Number of Test Plans. One per EMI software Product.




Adaptability

Number of supported platforms.

1 for EMI 1 (SL5/64bits), 4 for EMI 2 (SL6/32, SL6/64, Debian 6/32 and Debian 6/64, to be decided for EMI 3.

Debian packages are available for several major products.

Installability

Number of standard installation tools per supported platform.

At least one per supported platform.

Yum, Apt

Number of standard package formats per supported platforms per released product.

At least 4: binary packages and tarballs, source packages and tarballs.


Number of non-compatible software licenses.

All licences in the EMI product are compatible.

Co-existence

KPI KNA2.4 Number of EMI products included in standard repositories, Linux distributions, etc.

80% of the client components, selected services based on requirements [R4]

This is the major achievement planned for EMI 3. Currently several major products are already in the EPEL release. Many EMI developers are now certified EPEL packager.


3.4.1 Verification activity During the verification activity from June 2012 to March 2013, 50 products for the EMI 3 release were fully verified and EMI 3 (Monte Bianco) has been released on March 11th, 2013.

Status with the verification of the products

The number of compliant tasks in EMI 2 updates has greatly increased during the last year. There are PTs with no regression tests or unit tests, but despite existing non-conformities only one task was found non-compliant and thus rejected.

State transition Product passed the QA verification in EMI 3/Total number of products


Availability of software tests

The following charts present the measurements collected for the EMI 3 Major release on the availability of software tests (unit and regression) and their coverage for high/immediate priority



defects. Results about software tests have been compared with EMI 1 and EMI 2 updates for the last year, in order to show the positive trend of such measurements.

Figure 20: Availability of Unit Tests for EMI 3 major release vs. EMI 1 and EMI 2 updates

Figure 21: Availability of Regression Tests for EMI 3 major release vs. EMI 1 and EMI 2 updates

Although the percentages were already high both for unit and regression tests a small improvement can be seen. The unit tests availability has improved from 57% to 72% comparing EMI 1 updates to EMI 3, while the regression tests availability is stationary to 83%.

The availability of unit tests has improved continuously from EMI 1 to EMI 3; again this is a good result.



Figure 22: Number of associated tests for high/immediate priority defects in EMI 3

Figure 23: Number of associated tests for features in EMI 3 major release

The charts above present measurements gathered directly from release trackers of middleware consortiums on the test coverage for High/Immediate priority defects and associated tests for features for the EMI 3 major release.



The first chart (Figure 22) shows that all high priority issues were tested, in Figure 23 instead one can see how many new features have been introduced in EMI 3 and that approximately for the 43% of these provide a functionality test.

Standard Compliance and Interproduct Tests

The charts below present measurements collected from the Verification Dashboard on the Compliance/Interproduct Tests for EMI 1, EMI 2 and EMI 3 major releases. One can see a substantial improvement from EMI 1 to EMI 3 major release.

Figure 24: Availability of Standard Compliance Tests for EMI 1, 2 and 3

Figure 25: Availability of Standard Interproduct Tests for EMI 1, 2 and 3




All PTs provided right binary packages for their products in EMI 3 major release. A small drop occurred for source packages availability from EMI 2 updates (100% of packages provided) to EMI 3. As one can see also the 100% of products in EMI 3 provided both source and binary packages.

Figure 26: Availability of binary packages for EMI 3 vs. EMI 1 and EMI 2 updates

Figure 27: Availability of source packages for EMI 3 vs. EMI 1 and EMI 2 updates

Debian packages

As a major novelty about packaging, for EMI 3 it has been defined a list of products which must provide also mandatory Debian packages besides standard binary and source files. Debian package names must be included into the certification reports, and then the QC takes in charge to verify that the PTs provide all the right .deb files.



The list includes: AMGA, CREAM, LCG-Util, dCache, ARC suite, lcg-info-clients, gridsite, UNICORE-HILA, UNICORE-Client6, and VOMS.

For all these products correct Debian files have been committed to the EMI official repository.

Certification report

All PTs provided compliant certification reports for the EMI 3 release and the updates for a total of 50 products releases.

Figure 28: % of certification reports for EMI 3 major release vs. EMI 1 and EMI 2 updates


The graph below shows that the percentage of passed checks in EMI 3 major release dropped a little bit respect to EMI 1, EMI 2 updates. The percentage has reached the 89% of passed checks as opposite to EMI 2 where the higher percentage (100%) was achieved.

Figure 29: % of passed checks for EMI 3 major release vs. EMI 1 and EMI 2 updates



Delay on the release schedule

The final chart of this section presents the delay on the release schedule that occurred for the EMI 3 release. The ratio between the number of months of delay and those planned for preparing the release has still decreased as its value stayed below 1% of the planned time needed to deliver.

In practice the result is that the release management of the products, by SA1, was managed and achieved exceptionally better than the average software projects were a 10% delay is considered an excellent result.

Figure 30: Delay of the EMI 3 release

3.4.2 EMI 3 Coding and Testing Metrics report Quality


Result

Testability





One per released software product.




100% Deployment Test. 100% Basic Functionality Test 83% Automated Regression tests 72% Unit Tests



100% fulfilled





Figure 23 show that the 43.24% of features (16 over 37) report a functionality test. This is a reasonable result given that the several minor features were verified in other large tests.

Number of passed Certification Checks

100% for the checks considered in the Production Release Criteria.

100% fully passed the certification tests.

3.4.3 EMI 3 General Metrics report Quality


Result

Suitability Number of implemented development tasks.

100% All development tasks planned were implemented.

Functionality compliance

KPI KJRA1.1 Number of EMI service interfaces and libraries passing standard compliance tests

JRA1 aims to increase the number of successful tests at least by two per reporting quarter [R4]

The compliance tests improved significantly for the EMI 3 release of 4 per each release.

Security

Number of EMI Security Assessments.

At least one per EMI software product targeted for this.

Two security assessments were done during the project.

Number of fixed security vulnerabilities reported in the EMI security assessments.

100% security vulnerabilities according to the proposed deadlines.

No security vulnerabilities discovered in EMI 3

Number of fixed EGI SVG. No request received.

Recoverability

Number of services providing high-availability setups.

After identifying which services are suitable for this setup, the number should increase over time until the end of the project.


Understandability

Number of missing mandatory documents.

Ideally 0, although a decreasing number per product per release is also a good indicator.

Results of the review of EMI 3 products documentation could not calculated in time to be included in this deliverable.

Number of EMT [R22, R23, R24] tasks tracking documentation issues.

< 5 per week. Results of the review of EMI 3 products documentation could not calculated in time



to be included in this deliverable.

Operability

Number of services providing service control and status mechanisms.

All services should provide this feature. The number should increase over time until the end of the project.


Number of services providing configuration tools.


Testability Number of Test Plans. One per EMI software Product.


Adaptability

Number of supported platforms.

1 for EMI 1 (SL5/64bits), 4 for EMI 2 (SL6/32, SL6/64, Debian 6/32 and Debian 6/64, to be decided for EMI 3.

Fulfilled. All platforms defined are available. Debian packages are available for all products that were needed.

Installability

Number of standard installation tools per supported platform.

At least one per supported platform.

Yum, Apt

Number of standard package formats per supported platforms per released product.

At least 4: binary packages and tarballs, source packages and tarballs.


Number of non-compatible software licenses.

Co-existence

KPI KNA2.4 Number of EMI products included in standard repositories, Linux distributions, etc.

80% of the client components, selected services based on requirements [R4]

This is the major achievement Currently several products are already in the EPEL release. Many EMI developers are now certified EPEL packager.

3.5. STATIC ANALYSIS OF PRODUCT PACKAGES (EPEL COMPLIANCE) According to the EMI policy defined to guarantee the quality of released products [R4] the packages have been validated for specific platforms using the RPMLint plugin to verify the generated RPMs comply with Fedora Guidelines. The RPMLint plugin implements the metric defined in the EMI Metric Specification [R2]. Packaging guidelines and policies adopted are defined by Fedora, EPEL and Debian in the following documents:

• Fedora Guidelines and Policies [R29]

• EPEL Guidelines and Policies [R30]

• Debian Policy Manual [R31]

As an important novelty, starting from EMI 3 release the same implementation for Debian packages has been introduced equipping ETICS with a Lintian plugin to validate Debian packages.



EPEL compliance

Since the main characteristics of the RPMLint implementation have been already discussed in details in the previous document (EMI-DSA2.3.3-D4.3-Periodic_QA_and_QC_reports) here one can say that such implementation supports various RPM checks that verify different aspects of an RPM package, the number of RPMLint errors of each product (the parameter that defines the metric) should reduce to zero over a given time period according to an S-curve. During packages analysis two categories of errors are generated: high or low.

The latest results of the RPMLint plugin are collected and updated two times per week on the twiki page [R32]. Based on these results new statistics are calculated and the number of errors is also calculated per EMI product.

EMI 2 Status

The following table shows the RPMLint measurements over all EMI products at the time of the EMI 3 release. As one can see the EMI 2 status is still that no products were passing the EPEL verifications without errors.

Errors % Packages 0 0.0% 1 4.3% 2 39.7% 3 17.6% 4 14.1% 5 9.6% 6 7.8%

> 7 6.9%

Figure 31: Number of RPMLint errors per package EMI 2

EMI 3 Status

The following table shows the same measurements per product for the EMI 3 major release, as one can notice the effort put in the EPEL compliance brought more than 50% of the packages to be compliance and 90% of the packages to have less than 2 errors.

0%

10%

20%

30%

40%

50%

1 2 3 4 5 6 7 80 1 2 3 4 5 6 > 7



Errors % Packages 0 48.5% 1 29.6% 2 10.9% 3 5.1% 4 3.7% 5 1.1% 6 0.7%

> 7 0.3%

Figure 32: Number of RPMLint errors per package in EMI 3

Registration in EPEL [R30]

A very important achievement is that the QA activity and measurements have steered and induced the teams to become compliant with the EPEL rules and to submit their product to the EPEL repository.

In 2010 there were only two products in the EPEL repository: VOMS and gridsite.

The following products are in EPEL: ARC, BDII, CANL, dCache, DPM, GFAL/lgc_util, Gridsite, LFC, lcg-info-clients, and VOMS

All these are the major EMI products in terms of packages and SLOC covering at least the 50% of the whole artefacts of the project. And another important progress is that now there are a dozen of developers familiar with EPEL and several developers are even official EPEL reviewers.

In addition, one has to consider that Java-based products cannot always reach a fully compliant level because of dependency management, and that sometimes the RPMLint plugin causes false positives for some products. In any case for the EMI 2 and EMI 3 releases the quality process for the packaging has reached a good level and the results obtained confirm a positive trend about EPEL compliance of the EMI software

0%

10%

20%

30%

40%

50%

1 2 3 4 5 6 7 80 1 2 3 4 5 6 7 0 1 2 3 4 5 6 > 7



Debian / Lintian

The verification for Debian started during the third year and is still in a less established process. Nonetheless very positive results are already been noticed.

The following table shows the Lintian measurements per product for the weeks from the 20th of March to the 16th of April 2013, collected for the mandatory products and associated to the ETICS source code branch B2_DEV and B3_DEV.

EMI 2 EMI 3 Number of

Errors Product

13.04.01 (B2_DE

V)

13.04.09 (B2_DE

V)

13.04.16 (B2_DE

V)

13.03.27 (B3_DE

V)

13.04.01 (B3_DE

V)

13.04.09 (B3_DE

V)

13.04.16 (B3_DE

V) AMGA 2 2 2 N.A. 2 15 15 CREAM 14 14 14 3 14 1 N.A. LCG-Util N.A. N.A. N.A. N.A. N.A. N.A. 50 dCache, 2 2 2 1 2 1 1

ARC suite 1 1 1 7 1 4 7 lcg-info-clients N.A.

N.A. N.A. 0 N.A. 0 0

gridsite 10 12 10 5 12 6 6 UNICORE_

HILA 9

N.A. 9 9 9 9 9 UNICORE-

Client6 2 1 2 1 1 1 1 Table 3: Number of Lintian errors EMI 2 vs. EMI 3

From the table above it is possible to see that the number of Lintian errors is very low for the most part of the products. Some improvements have been achieved, for example this is the case of CREAM or gridsite, for the other products values are more or less stationary.

The work done for EPEL is an excellent example of how products can now reach Debian compliance following established and agreed foundation during the third year of the EMI project.



4. CONCLUSIONS AND OUTLOOK

During the third year of EMI a lot of work has been done for the verification of new products. This was the result of a high number of released products for EMI 1 and EMI 2 updates and the new EMI 3 major release (Monte Bianco).

Verification and RfC Dashboards have been refined to improve the feedback to QC team, allowing a better understanding of products quality as well as a good support for taking decisions. These tools had already reached a good level of performance during the second year, and not so much development effort was needed in the third year. Human resources, in this case, was invested in analysing available data and understanding how to assist PTs provide “high quality” software packages.

The joint effort between PTs and the QC team has contributed to continuously improving measures, reaching in most cases with very high percentage values.

From June 2012 to April 2013, Regression Tests and Unit Tests’ availability has improved continuously together with EPEL/Debian compliance metrics and the overall number of errors/inconsistencies has dropped. This consideration, together with the awareness that EMI has produced a high-quality stable software suite, is a positive indicator for the sustainability of EMI middleware after the end of the project.

Equally notable is the EMI releases and update passed at 99.3% (149/150 products) at the first verification the EGI UMD Quality Criteria [R25]. This achievement was acknowledged and praised by the EGI Management at the EGI Community Forum 2013.

4.1. EMI 3 IMPROVEMENTS The major achievements of the project form the QA and QC point of view are:

• EMI 3 (Monte Bianco) successfully released together with EMI 1 and EMI 2 new updates for more than 130 release and updates.

• All products follow a standardized release process involving testing and certification. Products documentation is homogeneous and well structured.

• Products are packaged in a homogeneous way and part of them fully complies with EPEL standard packaging policy.

• 100% of the products now provide build from sources in addition to binary distributions.

• Unit and regression tests and the regression coverage of RfCs have clearly improved, reaching a level of at least 80% of all products.

• The release of EMI 3 according to the agreed timetable is the major achievement considering it was executed in parallel with the maintenance of EMI 1 and EMI 2 releases, work on the new updates and the shorter time available for its preparation.

• A major improvement on all metrics as the project progressed towards EMI 3. There are notable improvements in compliant certification reports, passed mandatory checks and testing of high priority issues.

• Compliance to standards (GLUE 2.0 SRM, etc) and interproduct testing were introduced and performed.

• The overall number of defects affecting released products is following a negative trend decreasing both per product and per priority.



• EPEL compliance has been reached for all products for which it is possible; some Java packages depending on libraries not available in EPEL cannot become complaint.

• Debian compliance has well advanced and is now measured. There were clear signs of improvements in the packaging of products that wish to become compliant.

4.2. OUTSTANDING QUALITY ACHIEVEMENTS OF EMI 3 On testing:

• 100% of the High and Emergency RfCs are covered by associated tests in all recent EMI 1 and EMI 2 updates and for the EMI 3 Major release;

• 100% of the products have Deployment Test, Basic Functionality Test, 83% Automated Regression tests, all cases in which test could be automated;

• 99.3% of the EMI products were immediately accepted by the EGI Quality Criteria (149/150).

On EPEL compliance:

• 90% of the packages are EPEL compliant with less than two errors per package;

• The largest EMI packages (ARC, dCache, DPM, etc) are now already in EPEL, in 2010 no major EMI product was in EPEL.

On Debian compliance:

• 90% of the products aiming at Debian compliance have less than 10 errors.

4.3. OUTLOOK AND POST-EMI PLANS As defined in the initial three-year strategy, the goal of the QA and QC activity was to bring together the processes and practices of more than 30 EMI PTs to move towards common standards in releasing, packaging, documenting and distributing the grid software.

This has been achieved and the teams now can benefit from an established and common process of distributing source packages and continue to comply with the EPEL and Debian guidelines.

The artefact developed for the EMI QA activity can continue to be useful in the future:

• The policies are well endorsed and will continue to be followed by the PTs. They are available in an open repository [R39] for future QA activities such as those fostered by the MeDIA initiative.

• The metrics and the dashboards developed and the metrics established are going to be available for follow-up activities. Discussions with EGI are on going to use these tools and verification for the future UMD releases.

• The build tools, ETICS in particular, is in SourceForge [R38]. It is going to be supported by the partners of the iMarine project.

• The PTs that have a working on-site infrastructure will use their own testbed and the processes to continue to execute organised and structured testing of their products.

After three years of intense collaboration, from four completely different ways of preparing and delivering software, the development teams have now achieved a common way of testing, distributing and documenting all the major European grid middleware products with important advantages for the quality and the ease of installation of these products and their future sustainability.



5. REFERENCES

R1 Quality Assurance Plan, https://cdsweb.cern.ch/record/1277599/files/EMI-DSA2.1-1277599-QA_Plan-v1.2.pdf

R2 Quality Assurance Metrics https://twiki.cern.ch/twiki/bin/view/EMI/TSA23

R3 Quality Assurance Wiki Page https://twiki.cern.ch/twiki/bin/view/EMI/SQAP

R4 EMI Quality Model twiki page https://twiki.cern.ch/twiki/bin/view/EMI/QualityModel

R5 Software Release Plan, https://cdsweb.cern.ch/record/1277545/files/EMI-DSA1.2-1277545-Software_Release_Plan-v1.0.pdf

R6 Software Maintenance and Support Plan, https://twiki.cern.ch/twiki/bin/view/EMI/DeliverableDSA11

R7 Technical Development Plan https://cdsweb.cern.ch/record/1277544?ln=en

R8 Release Management Wiki Page https://twiki.cern.ch/twiki/bin/view/EMI/TSA13

R9 Configuration and Integration Policy https://twiki.cern.ch/twiki/bin/view/EMI/EmiSa2ConfigurationIntegrationPolicy

R10 Certification and testing policy https://twiki.cern.ch/twiki/bin/view/EMI/EmiSa2CertPolicy

R11 Change management policy, https://twiki.cern.ch/twiki/bin/view/EMI/EmiSa2ChangeManagementPolicy

R12 DSA2.2.1 - QA Tools Documentation, https://twiki.cern.ch/twiki/bin/view/EMI/DeliverableDSA221

R13 Test Plan and Template https://twiki.cern.ch/twiki/bin/view/EMI/EmiSA2TestPlan

R14 Software Quality Assurance Plan Documentation, https://twiki.cern.ch/twiki/bin/view/EMI/SQAP#SQAP_Documentation

R15 First Principles Vulnerability Assessment http://www.cs.wisc.edu/mist/VA.pdf

R16 Consejo Superior de InvestigacionesCientificas http://www.csic.es

R17 First principles vulnerability assessment Proceedings of the 2010 ACM workshop on Cloud computing security workshop, James A. Kupsch, Barton P. Miller, Elisa Heymann, Eduardo César

R18 Vulnerability reports, http://www.cs.wisc.edu/mist/includes/vuln.html

R19 Security Assessment Plan, https://twiki.cern.ch/twiki/bin/view/EMI/SA1QCSAP

https://twiki.cern.ch/twiki/bin/view/EMI/TSA23

https://twiki.cern.ch/twiki/bin/view/EMI/SQAP



R20 Production Release Criteria, https://twiki.cern.ch/twiki/bin/view/EMI/ProductionReleaseCriteria

R21 Software Test plans list https://twiki.cern.ch/twiki/bin/view/EMI/QCTestPlan

R22 EMI development tracker https://savannah.cern.ch/task/?group=emi-dev

R23 EMI EMT Meetings http://indico.cern.ch/categoryDisplay.py?categId=3077

R24 EMI Release Management Process, https://twiki.cern.ch/twiki/bin/view/EMI/EmiSa2ReleaseManagementPolicy

R25 UMD Quality Criteria https://documents.egi.eu/public/RetrieveFile?docid=364&version=5&filename=EGI-GENERIC-QC-V2.pdf

R26 EMI Metrics Specification https://twiki.cern.ch/twiki/pub/EMI/QualityModel/EMI_SA2_MetricsSpecification_v1.4.doc

R27 EMI Quality Model https://twiki.cern.ch/twiki/bin/view/EMI/QualityModel

R28 EMI packaging policy https://cern.ch/twiki/pub/EMI/EmiSa2PackagingPolicy/EMI_SA2_Packaging_v_2_1.pdf

R29 Fedora Guidelines and Policies http://fedoraproject.org/wiki/PackagingGuidelines

R30 EPEL Guidelines and Policies http://fedoraproject.org/wiki/EPEL/GuidelinesAndPolicies

R31 Debian Policy Manual http://www.debian.org/doc/debian-policy/

R32 EMI EPEL Compliance https://twiki.cern.ch/twiki/bin/view/EMI/EMIEPELcompliance

R33 EMI Quality Assurance reports http://emiqa.web.cern.ch/emiqa/reports/latest

R34 EMI Testing Policy https://twiki.cern.ch/twiki/bin/view/EMI/EmiSa2TestPolicy

R35 EMI Documentation Policy https://twiki.cern.ch/twiki/bin/view/EMI/EMISa2DocumentationPolicy

R36 EMI Request for Changes Dashboard http://emi-rfc.cern.ch

R37 EMI Verification Dashboard http://emi-verification.cern.ch

R38 ETICS SourceForge project repository http://sourceforge.net/projects/etics/

R39 Public Repository of the EMI QA Policies https://cds.cern.ch/record/1550332

http://indico.cern.ch/categoryDisplay.py?categId=3077

Documents

cds.cern.ch · INFSO-RI-261611 2010-2013 © Members of EMI collaboration PUBLIC 1 / 41 EUROPEAN MIDDLEWARE INITIATIVE DSA2.3.4 - PERIODIC QA AND QC REPORTS. EU DELIVERABLE: D4.3.4