Interim Evaluation of the Clean Sky 2 Joint Undertaking … Sky 2 Interim Evaluation Report 30 June 2017 6 7.2.2 Operational effectiveness 61 7.2.2.1 CS Programme Management

Interim Evaluation of the Clean Sky 2 Joint Undertaking

(2014-2016) operating under Horizon 2020

Experts Group Report

Written by: Cheryl Atkinson Helge Pfeiffer Piotr Doerffer

Expert group: Cheryl Atkinson, Helge Pfeiffer, Piotr Doerffer, Heather Allen, Michael Dooms June 2017

Clean Sky 2 Interim Evaluation Report 30 June 2017

2

Interim Evaluation of the Clean Sky 2 Joint Undertaking

(2014-2016) operating under Horizon 2020

Experts Group Report

Written by Cheryl Atkinson

Helge Pfeiffer

Piotr Doerffer

Expert group Cheryl Atkinson, Helge Pfeiffer, Piotr

Doerffer, Heather Allen, Michael Dooms

LEGAL NOTICE

This document has been prepared for the European Commission however it reflects the views only of the authors, and the Commission cannot be held responsible for any use which may be made of the information contained therein.

More information on the European Union is available on the Internet (http://www.europa.eu).

Luxembourg: Publications Office of the European Union, 2017

ISBN: 978-92-79-69188-1 doi: 10.2777/160825 KI-01-17-535-EN-N European Union, 2017 Reproduction is authorised provided the source is acknowledged.

Europe Direct is a service to help you find answers

to your questions about the European Union.

Freephone number (*):

00 800 6 7 8 9 10 11

* The information given is free, as are most calls (though some operators, phone boxes or hotels may charge you).


4

EUROPEAN COMMISSION Directorate-General for Research and Innovation Directorate H Transport Unit H.3 Aviation

Contact: Francky Callewaert E-mail: [email protected] European Commission B-1049 Brussels

Table of Contents

1 EXECUTIVE SUMMARY ............................................................................................................................. 7

2 INTRODUCTION ..................................................................................................................................... 12

2.1 PURPOSE OF THE EVALUATION .................................................................................................................... 12 2.2 SCOPE OF THE EVALUATION ........................................................................................................................ 12

3 BACKGROUND TO THE INITIATIVE ......................................................................................................... 13

3.1 DESCRIPTION OF THE INITIATIVE .................................................................................................................. 13 3.1.1 The Clean Sky Initiative ................................................................................................................... 13 3.1.2 Intervention logic ............................................................................................................................ 16 3.1.3 Consistency of the JU with EUs general transport objectives ........................................................ 18

3.1.3.1 White Papers on transport 2001 and 2011 ............................................................................................ 18 3.1.3.2 Vision 2020 and Flightpath 2050............................................................................................................ 19 3.1.3.3 ACARE .................................................................................................................................................... 20

3.2 BASELINE ................................................................................................................................................ 21

4 EVALUATION QUESTIONS ...................................................................................................................... 25

5 METHOD/PROCESS FOLLOWED ............................................................................................................. 26

5.1 PROCESS/METHODOLOGY ......................................................................................................................... 26 5.2 LIMITATIONS ........................................................................................................................................... 27

6 IMPLEMENTATION OF THE CLEAN SKY JOINT TECHNOLOGY INITIATIVE ................................................ 28

6.1 IMPLEMENTATION IN GENERAL .................................................................................................................... 28 6.2 STRUCTURE OF CLEAN SKY 2 JU .................................................................................................................. 29 6.3 BUDGET ALLOCATION ................................................................................................................................ 29 6.4 LEADERS AND AFFILIATES ........................................................................................................................... 31 6.5 CORE PARTNERS ....................................................................................................................................... 32 6.6 CALLS FOR PARTNERS ................................................................................................................................ 33

7 ANSWERS TO THE EVALUATION QUESTIONS ......................................................................................... 40

7.1 MAIN ACHIEVEMENTS AND EFFECTIVENESS OF IMPLEMENTATION ....................................................................... 40 7.1.1 Main Achievements Direct achievements .................................................................................... 40

7.1.1.1 IADP LPA .............................................................................................................................................. 40 7.1.1.2 IADP Regional aircraft (Reg) ................................................................................................................... 41 7.1.1.3 IADP Rotocraft ....................................................................................................................................... 42 7.1.1.4 ITD Airframe ........................................................................................................................................... 44 7.1.1.5 ITD Engines ............................................................................................................................................. 45 7.1.1.6 ITD SYS ................................................................................................................................................... 47 7.1.1.7 TA ECO Design ........................................................................................................................................ 48 7.1.1.8 TA Small Air transport ............................................................................................................................ 48 7.1.1.9 TA Technical evaluator ........................................................................................................................... 48 7.1.1.10 ACARE Objectives ................................................................................................................................... 49 7.1.1.11 Impact .................................................................................................................................................... 50

7.1.2 Effectiveness of implementation ..................................................................................................... 52 7.2 CLEAN SKY 2 JOINT UNDERTAKING'S PERFORMANCE IN 2014 - 2016 ................................................................ 53

7.2.1 Clean Sky 2 JU mission and governance .......................................................................................... 53 7.2.1.1 Governing board (GB) ............................................................................................................................ 54 7.2.1.2 The JU Programme Office ...................................................................................................................... 55 7.2.1.3 IADP and ITD Steering Committees ........................................................................................................ 57 7.2.1.4 States Representatives group (SRG) ...................................................................................................... 58 7.2.1.5 Scientific Committee (SciCom) ............................................................................................................... 59 7.2.1.6 Other Stakeholders ................................................................................................................................ 59 7.2.1.7 Conclusions on Mission and Governance............................................................................................... 60


6

7.2.2 Operational effectiveness ............................................................................................................... 61 7.2.2.1 CS Programme Management ................................................................................................................. 61 7.2.2.2 Clean Sky Quality Partners and Research ............................................................................................. 63 7.2.2.3 CS Communication ................................................................................................................................. 65

7.2.3 Operational efficiency ..................................................................................................................... 67 7.3 EU ADDED VALUE .................................................................................................................................... 68 7.4 COHERENCE ............................................................................................................................................ 71 7.5 RELEVANCE ............................................................................................................................................. 74

8 CONCLUSIONS ....................................................................................................................................... 75

9 RECOMMENDATIONS: ........................................................................................................................... 78

9.1 THE DELEGATION AGREEMENT ................................................................................................................... 78 9.2 ADMINISTRATIVE SIMPLIFICATION ............................................................................................................... 78 9.3 THE H2020 AERONAUTICS INNOVATION PIPELINE ....................................................................................... 78 9.4 STIMULATE SUBCONTRACTING .................................................................................................................... 78 9.5 AN HOLISTIC APPROACH FOR AERONAUTICS RESEARCH ................................................................................... 78 9.6 INCREASED TRANSPARENCY ........................................................................................................................ 78 9.7 INCREASE INSIGHT .................................................................................................................................... 79 9.8 SYNERGY WITH NATIONAL RESEARCH ........................................................................................................... 79 9.9 PROMOTE ECONOMIC IMPACT .................................................................................................................... 79 9.10 ENERGIZE AND ENABLE ACADEMIC PARTICIPATION ......................................................................................... 79

10 ANNEXES ............................................................................................................................................... 81

10.1 CONSIDERATIONS ON STATE AID FOR PRIVATE COMPANIES AND THE 5% DILEMMA ................................................ 81 10.2 BIBLIOGRAPHY ......................................................................................................................................... 83 10.3 INTERVIEWS CONDUCTED ........................................................................................................................... 85 10.4 ABBREVIATIONS ....................................................................................................................................... 85 10.5 TABLE OF FIGURES .................................................................................................................................... 86 10.6 TABLE OF TABLES ...................................................................................................................................... 87 10.7 DISCUSSION PAPER CROSS TJUS LEARNINGS AND RECOMMENDATIONS FROM THE INTERIM AND FINAL EVALUATIONS OF CLEAN SKY 1&2, SESAR(2020) AND SHIFT2RAIL ...................................................................................................... 87 10.8 ANNEX COORDINATOR SURVEY ................................................................................................................. 115 10.9 ANNEX - SURVEY PUBLIC CONSULTATION .................................................................................................. 128

7

1. EXECUTIVE SUMMARY Scope This document presents the results of the first Interim Evaluation of the Clean Sky Joint Undertaking (CSJU) established during the Horizon 2020 Programme (2014-2021), carried out in parallel with the Final Evaluation of the predecessor Seventh Framework Clean Sky 1 programme. These evaluations were mandated by the regulation establishing the CSJU and was conducted by a team of independent experts from January 2017 to June 2017. The evaluation was carried out following the Terms of Reference of the Evaluators retention contract, which addressed the requirements of the revised guidelines of the Better Regulation Package as well as the main evaluation criteria: relevance, efficiency, effectiveness, coherence and EU added value. In addition, the criteria: openness, transparency and research quality are considered. The evaluation is intended to inform the European Commissions views on the effectiveness of the CSJU and shape the implementation of future Public-Private Partnerships (PPP) for the purpose of promoting R&D in the aeronautics domain. The Clean Sky Research Programme The CSJU is responsible for the execution and management of a multidisciplinary research program focusing on opportunities to accelerate the industrial implementation of green technologies in air transport vehicles. Its approach is to use demonstrators (TRL 6) to validate technological concepts that can meet the societal objective of mitigating the environmental impact of air transportation but are beyond the research investment capacity of industry to develop. The intention was to pave the way for evolutionary concepts to replace the normal incremental product development strategy. The objectives for Clean Sky 2 were based on the Societal Challenges of Decision 2013/742/EU, specifically the Challenge under Part III to contribute to improving the environmental impact of aeronautical technologies, including those relating to small aviation, as well as to developing a strong and globally competitive aeronautical industry and supply chain in Europe. The Joint Technical Programme [2] of Clean Sky 2, the most comprehensive public document describing the whole scope of the CS2 programme, explicitly states The renewed ACARE SRIA was completed in 2012, with ambitious goals for a sustainable and competitive aviation sector. These include a 75% reduction in CO2 emissions, a 90% reduction in NOx and 65% in perceived noise by 2050 compared to 2000 levels, and 4 hour door-to-door journey for 90% of European travellers These substantial emissions reductions and mobility goals require radically new aircraft technology inserted into new aircraft configurations. Building on the substantial gains made in Clean Sky, Clean Sky 2 aims at meeting the overall high-level goals with respect to energy efficiency and environmental performances Demonstrators were defined for the most used vehicles in the Air Transport market the large passenger aircraft, the regional aircraft and the rotorcraft and were configured for laboratory, ground or flight test validation depending on the technologies incorporated. The technologies were developed by discipline based units for air vehicle (structure, aerodynamics etc.), engines and systems. An Eco Design focus group developed life cycle analysis tools and influenced all of the development work. A Technology Evaluator tracked the extent to which the environmental objectives could be considered achievable. What are the main achievements of Clean Sky 2?


8

The CSJU was established as an EU body subject to EU Financial Regulations, with funding from the Horizon 2020 budget appropriation of 1,755.5 M allocated for up to 40% to the co-opted members (the IADP/ITD Leaders and their Affiliates) and for up to 30% to the Core Partners that were joined as members following open calls for Core Partners early in the CS2 programme. All of these participating entities match the EC contribution to their scope of work with in kind contributions. The remaining 30% (526.6 M) were allocated to supporting partners on the basis of open calls for topics contributing to each of the core research areas and more than 600 participants are expected to be joined to the CSJU. As in CS1, a high quality of research capability was realised in a geographical distribution that approximated the economic contribution in aeronautics. A good balance of industrial, research and aeronautics institutes of higher education is being achieved and the high engagement of SMEs realised in CS1, as a result of the characteristics of demonstrator projects, appears to be continuing. Universities that have enriched their curriculum with the capacity to model, build and test innovative parts that were integrated in the demonstrators continue their involvement and new industrial research partners will augment their product development capability in executing their part of the Clean Sky programme. The Clean Sky 2 programme will host the final demonstration phase of flight test for the highly ambitious Counter Rotating Open Rotor engine and the Laminar Flow wing developed in Clean Sky 1. Clean Sky 1 also contributed a broad range of opportunities for future development which are being investigated and progressed into the demonstrator configurations that will be validated in Clean Sky 2. At the time of writing most of the opportunities have been assessed and the down selection for development form the basis of the detailed planning. What are the main findings of the evaluation? Clean Sky has achieved widespread recognition around the world for the unprecedented level of collaboration of its research participants in a focused and coherent research programme that significantly reduces the fragmentation of other funding instruments. The PPP approach is effective in the governance and the execution of the programme through the CSJU Programme Office. Its dedicated support to the members in administrative matters and the monitoring and interventions of its technically competent project officers provide opportunities for improved research management that are highlighted in the recommendations. A well-crafted Communication Strategy mobilises participation and broadcasts Clean Sky activities and accomplishments leading to a focusing of national and industrial research priorities in its wake. In terms of the main evaluation criteria: Effectiveness The relatively smooth transition from Clean Sky 1 promises to continue the remarkable achievements that have been made there, both in technological advances and in the realisation of a well functioning partnership in the programme execution. This is largely attributable to the recognition by all stakeholders that Clean Sky is the right approach to the coordination of demonstrator oriented aeronautics research and their resulting dedication to making it work. The continuation a JU to realise the ACARE Strategic Research agenda environmental objectives sustains the momentum for a focused coordinated response to the need for the greening of air transport. The alignment of the Clean Sky multipoint research agenda with the product development strategies of its members, both long and short term, continue to motivate their investment in, and wholehearted support for, the Clean Sky programme. The Commission policy of integrating the diversity of Horizon 2020 funding instruments in a single

process infrastructure has greatly complicated the transition from Clean Sky 1 and is eroding the

effectiveness of the tailored administrative approaches that supported the predecessor programme.

9

Efficiency

The Clean Sky programme office has a broad portfolio of operational tasks, related to the

administration of the programme participation and the monitoring of research progress, combined

with ad hoc responsibilities such as establishing the ESIF relationship. The rigorous review schedule,

the high frequency of meetings inherent in the governance structure, and the geographical reach of

the programme participation places a high travel burden and demands almost 24/7 reachability for

the staff members, who fortunately all think they have the best job in the world. Overall, the Clean

Sky Programme Office performs remarkably well and efficiently.

Relevance

Political developments underwrite the continuing relevance of reducing the environmental impact of

air transport. The adoption in 2015 of the historic Paris Agreement and its ratification in November

2016 underscores the global intention to resist climate change. The International Civil Aviation

Organisation (ICAO) agreement in February 2016 on a CO2 standard for new aircraft, followed by

their accord on global market-based measure to control CO2 emissions from international aviation in

October, highlight an emerging regulatory framework. It is felt that regulation will be instrumental in

stimulating industrial uptake of the most ambitious Clean Sky accomplishments.

The radical, disruptive technology lines that the Clean Sky programme has pursued are the

foundations for aircraft industry product steps that skip a generation of evolutionary development

and make the Clean Sky concepts the new normal in the operating fleet and the biggest challenge

is to ready them for the next plus 1 aircraft fleet renewal cycle.

The policy and rationale that underlay the Clean Sky programme in 2007, and its continuation in 2014, are still in line with the current challenges in the Air Transport sector and the portfolio of tasks entrusted to the Clean Sky Joint Undertaking, and the effective execution of them in Clean Sky 1, continues to underwrite the PPP approach.

EU Added Value

The European Union is the only place on the planet that could have realised Clean Sky and the JTI

approach is the only way that the critical mass for the success of the Clean Sky research agenda could

have been obtained. The JTI approach has been able to focus the aeronautical research community

on the goal of mitigating the environment impact of aviation, a mission that would not have been

undertaken based only on market incentives, on a much larger scale than would otherwise be

realised.

The European Parliament has noted that "the concept of European added value must not be limited

to advanced cooperation between Members States but should also contain a visionary' aspect". [3]

There is no doubt that that Clean Sky research agenda, which targeted to double the rate of progress

in fuel consumption reduction and half the time to market for the resulting technologies, is

ambitious.

Coherence

Four aspects of the coherence of the Clean Sky programme have been evaluated.

the internal coherence in the Clean Sky programme itself is very high and, in spite of the

complexity of the programme interfaces, the participants have a common vision

H2020 collaborative research is not providing sufficient support for bottom up project proposals

that fill the innovation pipeline with ideas that, if shown to be feasible in L1 research, may be

developed to a higher TRL in Clean Sky.


10

There have in the past been difficulties in implementing a working relationship that achieves the

necessary complementarity and synergy with SESAR and this relationship has been improved in

the transition to Clean Sky 2.

Coordination with national research programs has not been explicitly accomplished although

anecdotally such alignment is reported. As Clean Sky has not been granted the monopoly that

SESAR has in ATM research a persuasive approach to coordination with national research

programs is the only recourse.

Openness and Transparency

The CSJU has been implemented in an open and transparent manner. The Governing Board records

are complete and publically available as are the Annual reports, which have developed over time to

give high quality insight into the programme. Clean Sky activities and accomplishments are well

promoted. The website and social media presence has made great strides in providing broad visibility

of the programme activity and will target improved accessibility technical documentation.

Research Quality

The Clean Sky PPP is clearly achieving its objective of grouping the best quality aeronautics research

entities in a programme of common European interest with a clear relationship to the societal

challenge of climate change and benefitting the competitiveness of European aeronautics

stakeholders.

Clean Sky has produced world class Aeronautical Research, Development Test and Evaluation

(RDT&E) that enables promising technologies for the European Aeronautical Industry to be

developed and tested in large scale demonstrators.

Recommendations

The reviewers chose to offer a top ten list of recommendations from the variety of critical observations made in the evaluation and defer to the greater expertise of the CSJU in developing specific solutions. Three of the recommendations are related to day-to-day operations, both present and future: the need to reconsider the forced implementation of H2020 one size fits all infrastructure which is detrimental to the operational effectivity of the CSJU, the opportunity in the JU approach for a trust based relationship that can reduce the administrative burden for beneficiaries and the suggestion that many of the open call topics are inefficient use of the complex call process and subcontracting should be stimulated. A further three recommendations address the participation of the broader aeronautical research community in Clean Sky: the need to replace the collaborate research shortfall of the H2020 programme with open calls from within the existing Clean Sky programme, the advantage to having the JU operate the full spectrum of aeronautics research in future programmes as a way of balancing participation across the European aeronautics research spectrum and the incentive to take measures that cultivate academic participation in Clean Sky and ensure the future engagement of our best and brightest in our industry. The remaining four recommendations address the quality of research management by the CSJU. The achievement of the Clean Sky objectives can be strengthened through better synergy with nationally funded research, we advocate the refinement of the CSJU in programme monitoring to include measures of the ability of the partners to both choose high quality research targets as well as to accomplish them (where accomplishment should be considered to include the economic impact realised through projected as well as actual exploitation.) These advances in research performance insight should be accompanied by greater transparency in the flow of funding for a comprehensive view of the gains being realised by public investment in European aeronautics research.

11

We will trust that our reflections above are of influence in the evolution of Clean Sky 2 and the

definition of a potential successor aeronautics research programme based on the Clean Sky

experience and expertise.


12

2. INTRODUCTION

2.1 Purpose of the evaluation

Article 11 of the COUNCIL REGULATION (EU) No 558/2014 [1] establishing the Clean Sky 2 Joint Undertaking provided for the external reporting for the Clean Sky programme. In addition to annual reports on the progress achieved to the European Parliament and to the Council, two interim evaluations and a final evaluation, carried out with the support of external experts, are prescribed. This first interim review, carried out in conjunction with the final review of the Clean Sky 1 (CS1) programme [4], will also contribute to the H2020 Interim Evaluation. Detailed and thoughtful guidance for the evaluation was provided by the European Commission (Commission) Directorate-General for Research & Innovation, Transport (H) in an appendix to the retention contract for the selected experts [5]. As the first interim evaluation, carried out just two years into the programme, the progress made in achieving the objectives set for the Clean Sky 2 (CS2) programme and the extent to which the CSJU was managed and operated efficiently are the central themes. There is less emphasis on the technical achievements, which are rather limited at this early stage. In addition, an assessment of the openness and transparency with which the CS2 programme is being conducted will reflect on the advances made since the CS1 programme was implemented in 2008. As well as influencing the ongoing operation of the CS2 programme, the results of this evaluation will also be used to improve the implementation of the Joint Undertakings in general, and contribute to the formation and the ex-ante impact assessment of the possible next generation JUs. The results of this evaluation will be used by the Commission to inform the European Parliament and Council, national authorities, the research community and other stakeholders of the achievements and outcomes realised by the Clean Sky 2 JU operating under Horizon 2020.

2.2 Scope of the evaluation

The scope of the evaluation covers the process of the transition from Clean Sky 1 to Clean Sky 2 and the early achievements in CS2. It will target a helicopter view of the life cycle of the Clean Sky 2 programme evolution thus far. It presents the incentives for the initiative and discusses the implementation of the Clean Sky 2 Council Regulation with attention to the adjustments that were needed to accommodate the new H2020 framework. A best effort has been made to ground the observations and conclusions in this report in the documented evidence from the Clean Sky 2 JTI but it also includes personal perceptions obtained through stakeholder interviews. It has been a great honour to conduct this evaluation and the reviewers are grateful for the time and care that has been taken by the Commission, the JU staff and other stakeholders to provide the information we requested and thoughtful, well considered responses to our many questions. We are in awe of the openness and honesty of all of the stakeholders we engaged with in the course of this exercise. While we cannot claim in any sense to have fully evaluated the Clean Sky 2 programme and presented in this report all of the nuances that influenced its execution, we sincerely hope that we have highlighted those aspects that have most significantly impacted the aeronautical research community and the public trustees that have shown their confidence in the cooperativeness and professionalism of this community by renewing their investment in the Clean Sky 2 JTI.

13

3. BACKGROUND TO THE INITIATIVE

3.1 Description of the initiative

Joint Technology Initiatives (JTIs) were a major new feature of the seventh framework programme (FP7), introduced to support key areas of research and technological development that can contribute to Europes competitiveness and quality of life by providing for Community contribution to the establishment of long term public private partnerships (PPP). In Horizon 2020 they were continued as Joint Undertakings (JUs) that are founded on the provision, under Article 187 of the Treaty on the Functioning of the EU (TFEU)[6], that the Union may set up Joint Undertakings or any other structure necessary for the efficient execution of Union research, technological development and demonstration programmes and are Union bodies under Article 209 of the EU Financial Regulation. The Clean Sky 2 JU was established in Regulation No 558/2014 [1] of 6 May 2014. The JTIs under FP7 originally emerged from the European Technology Platforms (ETPs), identified for Aeronautics in the Clean Sky regulation as the Advisory Council for Aeronautics Research in Europe (ACARE) and are described in the FP7 council decision [7] as These initiatives, mainly resulting from the work of European technology platforms and covering one or a small number of selected aspects of research in their field, will combine private sector investment and national and European public funding, including grant funding from the Research Framework Programme and loan finance from the European Investment Bank. Horizon 2020 continued the FP7 JTIs without specific new provisions on size, scope and purpose, but with options to adapt their structure. The public-private partnerships in the form of Joint Technology Initiatives (JTIs) launched under the Seventh Framework Programme may be continued using structures better suited to their purpose. [8] at (40) as reflected in article 25 of the H2020 Establishment act [8].

3.1.1 The Clean Sky Initiative

The original, principal objectives of the original Clean Sky JTI, as captured in the founding regulation [9], were:

To provide integration and demonstration at the level of the system as a whole to decrease the risk for private investment in developing new environmentally friendly aeronautics products.

To accelerate the development of Air Transport technologies to realise the earliest possible deployment of contributions to Europes strategic environmental and social priorities.

To ensure the coordinated use and efficient management of the funds assigned to the Clean Sky JTI.

These are reflected in the first objective of the Clean Sky 2 Council regulation [1] at (2), which addresses the finalisation of the research activities started in Clean Sky 1.

a) to contribute to the finalisation of research activities initiated under Regulation (EC) No 71/2008 and to the implementation of Regulation (EU) No 1291/2013, and in particular the Smart, Green and Integrated Transport

Greening of air transport remains a key objective, including in the small aviation market, and the targets for the key environmental parameters (CO2, NOx and noise) are revised and expressed as the gain to be realised relative to a 2014 technology baseline. In addition, the aspect of the global competitiveness of the industry has become a Clean Sky 2 objective [1] at (2).

Challenge under Part III Societal Challenges of Decision 2013/743/EU;


14

b) to contribute to improving the environmental impact of aeronautical technologies, including those relating to small aviation, as well as to developing a strong and globally competitive aeronautical industry and supply chain in Europe.

This can be realised through speeding up the development of cleaner air transport technologies for earliest possible deployment, and in particular the integration, demonstration and validation of technologies capable of: (i) increasing aircraft fuel efficiency, thus reducing CO 2 emissions by 20 to 30 % compared to state-of-the-art aircraft entering into service as from 2014; (ii) reducing aircraft NO x and noise emissions by 20 to 30 % compared to state-of-the-art aircraft entering into service as from 2014.

These objectives were based on the recognition of the economic benefit of the air transport sector to the European Union both directly, in a strong contribution to employment and GDP, and in aviations role in the overall competitiveness and growth of the European Union. Technological advances in aeronautics that mitigate the environmental impact of the future global fleet could prevent constraints on the growth of air traffic due to emissions and on airport capacity due to noise limitations. However private investment in these technologies is only partly justified by the economies of the fuel consumption reductions that would accompany reductions in emissions and the competitive advantage to be gained is highly dependent on the prevailing oil price level. With the environmental costs to society thus far external to the air transport operators and manufacturers, public funding of the related research is a necessary additional incentive for the industry to invest in clean technologies. Competitiveness in the aeronautics sector requires exceptional levels of research investment, a higher than normal level of risk, long return on investment and lower return on investment than in other sectors. However it is a leading edge industry with significant spill over benefits in other sectors and the European Unions long commitment to building and maintaining world class capability in aeronautics manufacturing has had a significant economic impact in the community. It is also widely recognised that the main global competitors, such as the United States, enjoy a significantly higher level of public support for aeronautics research and that other strong competitors in various niche markets (Brazil, China, Russia, South Korea and India) have emerged in recent years. In comparison to this global trend, the Framework programmes, while generating significant contributions in innovations and concepts, have not been able to sufficiently emphasise the validation of complex systems at a high level of integration. The Clean Sky programme focuses on technologies that offer the potential for step changes in performance but that are currently seen as too high a risk to be funded only privately. It emphasises the realisation of high technology readiness demonstrators that could position industry for accelerated exploitation at reduced risk and influence the timing of new product development. The scope of the Clean Sky JTI addressed the breadth of three main axis all segments of civil air transport (regional aircraft and rotorcraft in addition to the economically dominant large commercial aircraft); the full depth of the supply chain (including engines, systems and materials supporting technologies) and the product development life cycle (to the limit of what may be realised with public funding), through an integrated approach leading to multiple full scale ground and flight test demonstrators. In the course of the execution of Clean Sky 1, and one of its important lessons learned and achievements, is that this approach enabled many of the participants to retain (or re-train) their internal capability for product development. The Clean Sky JU was established in 2008 to run until 31 December 2016, with a budget of 1.6 B equally shared between the European Commission (EC) and the aeronautics research community members. Its members were 11 Industry representatives (and their affiliates) and a research organisation in the role of Leaders. Additional members were to be added through open calls for Core Partners and many short term Partners will be joined through Calls for Proposals.

15

The current, second generation Clean Sky JU was charged with the top level management of the research activities of the initial Clean Sky programme for its last two years beyond the end of the Seventh Framework (to encompass the exploitation phase of the product development life cycle) and to transition to the Clean Sky 2 research agenda. Its role is to monitor the technological progress and impact of the research activities during the overall programme and explicitly includes the management of the vast amount of knowledge that the programme will generate. The JU staffing consists of contract and temporary agents under (until September 2016) Executive Director Eric DAUTRIAT, a highly qualified industrial research manager, who shaped and inspired the Clean Sky programme from its inception. Clean Sky 1 demonstrated that this unprecedented level of coordinated participation in support of common objectives with coherent targets provided a focal point around which aeronautics industry leaders and their knowledge networks could coalesce. A greater degree of cooperation between national, EU and industry sponsored research across the supply chain was achieved and the involvement of new actors, including from other industrial sectors, was stimulated by the high visibility of such an ambitious programme. This was the momentum that Clean Sky 2 would strive to maintain. The broad lines of the organisation of the Clean Sky JU and the architecture of the research agenda are discussed here and elaborated in other sections of this report, as needed to address specific evaluation criteria. The Clean Sky 2 research is organised in 9 units consisting of, at aircraft level, the Integrated Aircraft Demonstrator Platforms (IADP). The tier 1 suppliers of airframes, engines and systems form the Integrated Technology Demonstrators (ITDs), and transverse ITDs define the units whose work interfaces with each of the other units. This is shown below, with the respective funding levels at the outset of the programme. The matrix character of the relationships between these ITDs is reflected in the accompanying illustration and reflects the experience from the Clean Sky 1 programme with the coordination of technology development in this supply chain plus organisation model. A Technology Evaluator (TE) led by DLR assesses the environmental performance of the technologies developed in CS at sub-system, system and system of systems level (Figure 1). It was realised in CS1 as the first available European complete integrated tool delivering direct relationship between advanced technologies, still under development, and high-level local or global environment impact.


16

Figure 1 Clean Sky Program Logic and Set-up [10]

The demonstrators of the Clean Sky programme are configured to achieve technology readiness level (TRL) 6 based on the OECD definitions as directed in [1]. The higher product development level of TLR 7 (prototype) is beyond the scope of the programme. It is, however, noted by experts that the TLR concept needs to consider its technological context, i.e. a TRL will usually differ when assessed e.g. at component or system level, thus, a technology at TLR 4 on one platform can drop back to TRL 2 on another. The limiting criteria, according to [11] seems to be "non-marketability" but it is clear that the Clean Sky demonstrators may not have the characteristics of a prototype fully functioning in an operational environment.

3.1.2 Intervention logic

It is current practice to present an intervention such as the Clean Sky PPP initiative in the overall context of the European research priorities and the global influences on the industrial sector [3]. While not exhaustive, this analysis of the Clean Sky context and incentives for the JTI approach does provide key checkpoints for this evaluation and the assessment of whether the intervention has been implemented as intended and realised its aspirations. The intervention logic of the Clean Sky initiative is rooted in two major NEEDS that follow from the objectives for the Framework 7 research priorities relevant for the aeronautics sector. These are:

Technological breakthroughs for environmental impact mitigation The ACARE SRA environmental targets are so ambitious that they will not be reached without technological breakthrough, i.e. radical changes in technology requiring a substantial amount of research and validation but represent global leadership in signalling clear environmental objectives. Industrial readiness of green technologies a demonstrator based programme will ensure supply chain preparedness and product competitiveness of green technologies and level the playing field relative to global competitors with more public funding than in Europe.

17

SPECIFIC OBJECTIVES to be realised by the implementation of the Clean Sky 2 JTI to meet these needs are:

Validated green technologies arising from concepts developed within collaborative research projects (Framework programmes) and brought to system level demonstrator (TRL 6) in the Clean Sky programme.

The operational flexibility to cooperatively optimise the ongoing research programme to evolving industry priorities and to fast track promising technologies and thereby obtain long term commitment and investment from the aeronautics research community.

The mobilisation of a critical mass of resources to accelerate the realisation of common objectives through demonstrators with a high degree of functional integration.

Continuity and consistency of research activities over the programme life cycle to reduce the fragmentation of the collaborative research lottery and increase the efficiency of access to public funding.

Centralised programme management with efficient systems to reduce the administrative burden/barrier -*for research entities.

An innovation pipeline response within CS2 to the very limited H2020 collaborative research funding for aeronautics

The RESOURCES to be applied in the PPP intervention are:

The research capacity of the highest quality eligible actors, deployed to well-defined and rational research objectives.

An adequate level of funding for the programme objectives in a balanced contribution and distribution among the participating entities.

Streamlined and efficient procedures for grant administration and programme monitoring and control.

The ACTIVITIES to be conducted are:

The implementation of the joint technology proposal via development and work plans.

The adaptation of the work plan to external factors and intermediate research results.

The management of the programme.

The monitoring and communication of the outcomes (knowledge management, dissemination, exploitation).

The influential EXTERNAL FACTORS are emerging competition, economic conjuncture (air traffic growth, fuel price evolution), trade agreements, global climate change mitigation measures, change of political support and offshore ownership of European actors and changes in political support due to influences such as Brexit and the US intention to withdraw from the Paris Accord. As well as a host of validated demonstrators and green aircraft concepts, the OUTPUTS of the intervention will be research infrastructure, such as new simulation tools, updated test capacity (wind tunnels, test benches, flight test vehicles, test instrumentation etc.) and generated knowledge. The RESULTS of the intervention are expected to be validated Clean Sky technologies with the potential to realise the ACARE environmental targets and at a high level of technology readiness to accelerate market introduction. Spill-over results will be leading edge knowledge that can be applied in other industrial sectors.

The IMPACTS will be:

Strengthened research and innovation capacity in the aeronautics research eco-system

Strengthened industrial competitiveness

Mitigation of the environmental impact of air traffic growth

Avoidance of noise limitations to airport utilisation increase

Increased mobility and economic growth for EU citizens


18

These aspects are summarised in the intervention logic diagram below (Figure 2).

Figure 2 Intervention logic diagram

3.1.3 Consistency of the JU with EUs general transport objectives

A policy designed to stimulate research and development in the aeronautics sector, leading to environmentally efficient aircraft, is one pillar of the global strategy presented in the communication COM(2005)459 of September 2005 to reduce the climate impact of aviation, which was endorsed by the Council (2 December 2005) and the Parliament (4 July 2006). Stimulating aeronautics R&D is complementary to measures such as the Commission proposal to include aviation in the EU Emission Trading Scheme (ETS). In addition it is widely recognised that greener aviation technologies will contribute towards mobility within an enlarged EU, which will be particularly important for accession states where traffic is growing rapidly from a low base. 3.1.3.1 White Papers on transport 2001 and 2011

The general transport objectives in the period when Clean Sky 1 was designed were mainly based on the White Paper on transport [12] developed in 2001 and its successor documents which were adopted by the EC as a roadmap for research policy development. This is tasked to the Clean Sky 2 JU in its Statutes [1] at Annex I (2(c)):

(c) focusing efforts within ITDs, IADPs and TAs on key deliverables that can help the Union meet its environmental and competitiveness goals, including as outlined in the Commissions White Paper from 2011 entitled Roadmap to a Single European Transport Area Towards a competitive and resource efficient transport system

The White Paper underlines that the RTD priorities in the aeronautics field will focus, on the one hand, on lessening the environmental impact of engine emissions and noise and improving aircraft safety . It further states that As regards the environment, the aim is to compensate for the increase in air traffic by reducing CO2 emissions by 50 % and NOx by 80 % and by reducing aircraft noise by 10 dB in order to cut the perceived noise level by 50 %. Research will focus on aircraft technology, low-drag aerodynamics and flight operating procedures. Clean Sky continues to be well aligned to these objectives.

19

In general the policies of the White Paper for transport are strongly passenger-centred and driven by economic aspects in a more competitive integrated, safe and intermodal transport system. The updated White Paper for transport published in 2011 continues to demand that Improving the efficiency of aircraft and traffic management operations has to be pursued in the air sector. It adds however Developing and deploying new and sustainable fuels and propulsion systems and targets Low-carbon sustainable fuels in aviation to reach 40 % by 2050 . [13]. This is not within the scope of Clean Sky 2.

3.1.3.2 Vision 2020 and Flightpath 2050

In 2001, the report of the Group of Personalities "European Aeronautics: A vision for 2020 " [14] pioneered an integrated vision of the European Air Transport System (ATS) for the next 20 years and became a popular reference for research policy development, although it is no official adopted policy document of the EC. It established, as its top-level objectives, the need to respond to society's needs and to secure European leadership in the aeronautics field." Society's needs embrace the whole range of benefits that all citizens of Europe expect of the air transport industry now and in the future. These benefits are direct, as in the quality and price of travel, and indirect, as in the preservation of security and safety in a more global world. They encompass the personal needs of travellers and the collective needs of non-travellers who want to live in quiet, pollution-free neighbourhoods." It is clear that the range of CS research also addresses these aspects. In 2010, an updated Flightpath 2050 vision formulated by a high level group was published by DG RTD and DG Move. It specifically addressed the extraordinarily long lead times required for sustainable innovation in aeronautics. The following goals directly apply to Clean Sky area of interest:

Europe will maintain leading edge design, manufacturing and system integration capabilities and jobs supported by high profile, strategic, flagship projects and programmes which cover the whole innovation process from basic research to full-scale demonstrators.

Streamlined systems engineering, design, manufacturing, certification and upgrade processes have addressed complexity and significantly decreased development costs (including a 50% reduction in the cost of certification). A leading new generation of standards is created.

In 2050 technologies and procedures are available allow a 75% reduction in CO2 emissions per passenger kilometre to support the ATAG target and a 90% reduction in NOx emissions. The perceived noise emission of flying aircraft is reduced by 65%. These are relative to the capabilities of typical new aircraft in 2000.

Aircraft movements are emission-free when taxiing.

Air vehicles are designed and manufactured to be recyclable.

Europe is established as a centre of excellence on sustainable alternative fuels, including those for aviation, based on a strong European energy policy.

European research and innovation strategies are jointly defined by all stakeholders, public and private, and implemented in a coordinated way covering the entire innovation chain.

A network of multi-disciplinary technology clusters are created based on collaboration between industry, universities and research institutes.


20

Strategic European aerospace test, simulation and development facilities are identified, maintained and continuously developed. The ground and airborne validation and certification processes are integrated where appropriate.

All of these objectives, with the exception of sustainable fuels, are within the scope of the Clean Sky 2 research programme. 3.1.3.3 ACARE

Following the objectives of the high level group laid down in the Flightpath 2050 objectives, the Advisory Council for Aeronautics Research in Europe (ACARE), produced again a set of more detailed recommendations, that are now called Strategic Research and Innovation Agenda (SRIA) in 2012 and an updated edition was published and presented at the Paris Airshow in Le Bourget in June 2017 [15]. The Joint Technical Programme [2] of Clean Sky 2, the most comprehensive public document describing the whole scope of the CS2 programme, explicitly states The renewed ACARE SRIA was completed in 2012, with ambitious goals for a sustainable and competitive aviation sector. These include a 75% reduction in CO 2 emissions, a 90% reduction in NO X and 65% in perceived noise by 2050 compared to 2000 levels, and 4 hour door-to-door journey for 90% of European travellers. These substantial emissions reductions and mobility goals require radically new aircraft technology inserted into new aircraft configurations. Building on the substantial gains made in Clean Sky, Clean Sky 2 aims at meeting the overall high-level goals with respect to energy efficiency and environmental performances Thus, the ACARE SRIAs have been influential on Clean Sky 2 objectives and the environmental impact mitigation targets in the programme are directly linked to this reference.

21

3.2 Baseline

The Clean Sky 2 programme is a follow on to the successful Clean Sky JU established in the Seventh Framework research programme. Aeronautics research has taken place under the European Framework Research programmes which have been operating since the mid 80s with progressive budget increases and continual changes to the priorities, allocations and instruments within each programme. While industrial competitiveness was generally part of the scope of the successive workplans, the Fourth Framework was the first implementation detailing a Transport agenda in which for air transport, research will focus on reducing congestion of airspace and of airports, particularly taking into account the results of transport telematics, as well as on further improving human safety and reducing the negative impact on the environment The evolution of the Framework program budgets and the approximate appropriation for aeronautics (exclusive of air transport) is shown in Table 1. It can be seen that the growth in aeronautics research has not kept pace with the general trend. In addition, collaborative research funding dropped dramatically in Horizon 2020 with major impact on the research community not engaged in Clean Sky 2. ID Framework Programme Period Budget

(B) appr. Budget for aeronautics (B )

Approx. Budget for aeronautics collaborative research (B)

FP1 First 19841987 3.8 na na

FP2 Second 19871991 5.4 na na

FP3 Third 19901994 6.6 na

19901991 0.35 [16] 0.35 [16]

1992-1994 0.71 [16] 0.71 [16]

FP4 Fourth 19941998 13.2 Na na

1995-1998 0.245 [16] 0.245 [16]

FP5 Fifth 19982002 15.0 0.700 [16] 0.700 [16]

FP6 Sixth 20022006 17.9 0.85 [16] 0.85 [16]

FP7 Seventh 20072013 50.5 2.1 [16] 0.950 [16]

FP8 Horizon 2020 (Eighth) 20142020 80 2.1 0.185

Table 1 Key financial data for the different FP's

The Fifth Framework programme specifically addressed aeronautics research which, being considered close to market had previously been largely conducted in the context of the EUREKA instrument and over 600 projects are identified as related to FP5 calls for aerospace. In general the Framework Programmes made extensive use of the Specific Targeted Project (STP) instrument which in FP6 involved an average of 9 participants and received around EUR 2 million of EC contribution for a period of three years. In comparison, FP6 Integrated Platform (IP) contracts (40% of the funding) engaged 25 participants and received around EUR 9.5 million of EC contribution for a period of four years. The topic aerospace was allocated 6.4% of the overall FP6 budget. The success rate of Aerospace proposals in FP6 was 30% averaged over all instruments and 57% for IPs. Private participation (BES, substantially Industry) in FP6 projects is around 55% compared to 30% for the overall programme. In FP7 the overall success rate dropped to 20%. The higher success rate for IPs is related to the participation of stakeholders in the preparation of the Work Program and in particular to the very targeted large project scope and objectives that did not induce submissions from unprepared consortia or allow for their success. Considering the effort in expertise, time and travel to establish a collaboration and align the capabilities of the prospective partners into a cohesive research program, an unsuccessful proposal has a significant negative economic contribution. However many institutions depend on research funding for their growth (or stability!) and proposals therefore must continue to be submitted. It is


22

worth noting here that these same institutions are besieged with fully funded PhD candidates from countries with an emerging aeronautical industry. The aeronautical work plans for the Framework programmes, in which the calls for proposals were defined, were largely discipline oriented thematic descriptions of the desired project result. Generally several successful proposals were selected for funding under each call topic. While on the one hand this format provided plenty of scope for bottom-up creativity in project definitions, it did not provide for any continuity of the knowledge acquired from one project to the next (which was often granted to a different consortium of research organisations). Nor could this approach filter out duplication and repetition of research activity or research on matters which were already state of the art. The Technology Platforms (or IPs in FP6) were an exception where the research objective, and quite probably the related proposals, were crafted with the support of the ETP organisation for aeronautics, ACARE and IMG4, The European Aeronautics industry network for R&T and targeted an industrially defined need. A typical IP would aspire to reach TRL 4 (Component and/or breadboard validation in laboratory environment basic (low fidelity) technological components are integrated to establish that they will work together) or possibly TRL 5 where higher fidelity breadboard technology is integrated in a representative, possibly simulated, environment. This is a significant advance from the conceptual and analytical/experimental work which precedes these levels. The next advance, to a representative model or prototype system, to be tested in a relevant environment, is a huge step forward in industrialisation and manufacturing of the system components and is the domain of the TRL 6 Demonstrators that are the essence of Clean Sky. However, in the steps towards realising a demonstrator there are many technology requirements that are unfulfilled at the start of the integration effort and a great many inventions, from concept to prototype are swept up in the race to demonstrate the original technological concept. It is then apparent that the undifferentiated bottom up calls and funding lottery of the traditional framework instruments could not create the unity of purpose and focus of capability that can be realised in a Joint Technology Initiative such as Clean Sky. In view of the key role of the Project Officer in the Clean Sky JU it is necessary to consider the historical approach to project management, which was carried out by similarly titled Project Officers in the Commission Directorate (RTD Transport) responsible for the work programme, the calls and the management of the evaluations of the proposals. The multitude of small projects were primarily monitored through the negotiation of a grant agreement consistent with the proposal and the timely appearance of deliverables and a web presence. However the large Integrated Projects were more closely monitored, with the ongoing support of external experts, and serious efforts were often needed to resolve scope and partner commitment issues. The strong aeronautics background of the Commission project officers and their broad familiarity with the sector enabled them to effectively meet this challenge. However, running research projects and maintaining contact with beneficiaries has transitioned to the Innovation & Networks Executive Agency (INEA), who has engaged limited sector expertise, and FP7 projects have been transferred as RTD project officers retire. Horizon 2020 calls do not clearly distinguish project size but it is believed that no very large projects have been initiated thus far.

23

Figure 3 Relative financing for framework programmes over the last frame programs relating the trend for global

funding to the share appr. allocated for aeronautics and in that context, for collaborative projects (CP).

Although the support for aeronautics research in the Framework programmes steadily increased, it

did not follow the same trend as the overall budget. Of particular concern is the very limited H2020

budget for aeronautics collaborative research, the breeding ground for innovations and their first

steps in demonstrating industrial feasibility. The original (September 2011) Commission proposal

intended to double the FP7 budget for H2020 to approximately 100 B and the CS2 applicants

correspondingly proposed 1800 M. The sharing between collaborative research and CS2 would have

been similar to that of FP7. When the H2020 budget was settled at approximately 70 B, the budget

for transport and aeronautics was almost unchanged with respect to FP7 (2,1 B to 2,4 B). Hence

the commitment made for CS2 and SESAR JU reduced the budget for collaborative research from 1.2

B in FP7 to less than 0.2 B under H2020. This is in contradiction with the intention that

collaborative research is the core of Community research funding with JTIs intended to address only

selected aspects of research in their field. [7, 8]

Figure 4 EU contribution to the different program under FP7 and H2020

This is expected to have a long term impact on the quality of European product development in aeronautics. Tentatively, alternative measures will be taken by the CS2 JU to ensure that the innovation pipeline continues to produce the quality of research that has sustained the industry for two decades and formed the basis for the demonstrator programmes running today. The FP7 Clean Sky programme demonstrated the value of the JTI approach in efficiently coordinating a large and integrated research agenda and providing long term certainty to its main participants. The continuation in Horizon 2020 is based on the contribution that greener aviation technologies will contribute towards mobility within an enlarged EU, which will be particularly important for accession states where traffic is growing rapidly from a low base. It is attenuated by the strong Horizon 2020 industrial competitiveness agenda. Many lessons were learned in CS1 on how great the challenge of


24

demonstrator projects is but industry continues to see the long term benefit of their investment in the JU.

25

4. EVALUATION QUESTIONS This interim evaluation of the Clean Sky Joint Undertaking addresses the need for an in depth assessment of whether this second generation of public-private partnership is implemented in an open, transparent and efficient way. A broad range of topics are considered in the factual inquiry (document review) and the solicitation of opinions in surveys and interviews. The evaluation criteria of the Better Regulation Package [3]; relevance; efficiency; effectiveness; coherence and EU added value are addressed as appropriate. The evaluation begins with an examination of the BACKGROUND AND DESIGN OF INITIATIVE AND INTERVENTION LOGIC in which the prevailing funding instruments for aeronautics research are discussed and the original incentives for the Clean Sky JU implementation and the background to the CS2 programme are identified. This insight is presented in section 3 of this report. Section 6 of this report IMPLEMENTATION OF CLEAN SKY JOINT TECHNOLOGY INITIATIVE reviews the formation of the Clean Sky JU for the H2020 framework, and the manner in which funding was allocated in the course of the Clean Sky programme, to assess the extent to which openness and transparency was achieved. The research work carried out since 2014 in the Clean Sky JU is reviewed in section 7.1 MAIN ACHIEVEMENTS AND EFFECTIVENESS OF IMPLEMENTATION. The technical accomplishments (and setbacks) are presented for each ITD as is the progress in achieving the ACARE targets set for the programme. Research related aspects such as dissemination, exploitation and the value added of the research results is also addressed. A number of aspects of the overall functioning of the JU are addressed in section 7.2 JU PERFORMANCE IN 2014-2016. The overall operation of the JU is evaluated in section 7.2.1 JOINT UNDERTAKING MISSION AND GOVERNANCE to assess whether the regulatory framework has been coherently implemented and whether the governance structure, and each of its components, are effective in supporting the JU. Section 7.2.2 OPERATIONAL EFFECTIVENESS looks more closely at the functioning of the JU Programme Office to assess whether their methods and procedures have been effective in realising a new paradigm for the aeronautical research community. The performance of the JU Programme Office in their management role is the subject of 7.2.3 OPERATIONAL EFFICIENCY. In section 7.3 EU ADDED VALUE the effectiveness of the large scale demonstrator programme, in comparison to alternative research deployment schemes, is discussed. The relationship between the Clean Sky programme and the related research activities such as H2020 collaborative research, SESAR and nationally funded aeronautics research is presented in Section 7.4 COHERENCE. The incentives for continuation of the Clean Sky programme into Clean Sky 2 due to the ongoing contribution to societal objectives is discussed in section 7.5 RELEVANCE. Section 8 presents the overall CONCLUSIONS of the evaluation and identifies the aspects that are working well along with those that need more attention. It emphasises that the main key to the success of the CS2 programme will be the commitment and dedication of both the public and private partners in meeting the expectations of the initial concept for the JTI. The evaluation closes in section 9 RECOMMENDATIONS with a set of ideas that might positively influence the functioning of the CSJU in the short term as well as some that may only be achievable in a successor Clean Sky programme.


26

5. METHOD/PROCESS FOLLOWED

5.1 Process/Methodology

The Clean Sky Final Evaluation and the Clean Sky 2 Interim Evaluation were conducted in parallel by two domain expert reviewers (one shared with SESAR) and a rapporteur, all appointed mid December 2016. A team including the S2R and S2020 (SESAR) reviewers was led by two horizontal experts that contributed cross-cutting input. A comprehensive and carefully articulated Terms of Reference (ToR) [5] contract annex outlined the daunting task ahead. An initial documentation set addressing the establishment of the JUs, the annual reports and the two CS Interim Assessments [17, 18] were provided in preparation for the Kick-off meeting in Brussels in mid-January. It was immediately noted that the previous Interim Assessments were conducted by a dedicated team of not less than 5 evaluators over a period of 7 months and set accordingly a much higher granularity for technical insight than we are able to achieve. The Kick-off meeting (see also a history of the evaluation in Table 2) provided an opportunity to meet the staff of the various Commission entities involved in the Evaluation, to learn of the interrelationship of our work with the H2020 Interim Review and the related schedule challenges, and to meet the Clean Sky management team and our appointed JU interlocutors (a very wise and necessary arrangement for which we are all grateful). A first Evaluation Team meeting followed and provided an opportunity to exchange views on the approach to the task. Following this Kick-off event the team moved into familiarisation and information gathering phase with interviews beginning just 5 weeks later. Two days of interviews with many of the JU staff provided background and key attention points which effectively guided the follow-up planning. These interviews coincided with a Rapporteurs meeting in Brussels which was attended for Clean Sky by teleconference. Unfortunately, despite the best efforts of the Commission meeting organiser, the poor tele-conferencing facilities did not support meaningful participation. However it was clear that the Clean Sky Evaluation team would be challenged to make a contribution to the H2020 Interim Evaluation due to its very short lead time for our input. Period Key Activity

December 2016 Evaluator contracts. Initial documents (establishment, interim assessments and annual reports)

January 19 2017 Commission Kick-off meeting and first evaluation team meeting in Brussels

Urgent input to coordinator survey

Statistical data (Corda) for calls available.

February 2017 Inventory of documentation and interview requirements

Preparatory discussions with JU management

Mid February First set of documents from JU (Dev Plan, CSMM, WP 2016-2017)

Feb 28, March 1 Interviews with JU staff (rapporteur connected via teleconference)

March 2 Rapporteurs meeting (rapporteur connected via teleconference)

March 7 Coordinators survey results posted

March 17 Public Survey Results posted

March 13 to 31 JU colocation of the rapporteur

March 21/22 Clean Sky Forum and CS Closing Event, GB,SC,SRG meetings, Interviews (GB, ACARE, DLR, Honeywell)

April 26-28 Annual Review meeting Rotorcraft IADP in Turin (Italy)

April 27 ITD Steering committee meetings held

May 3-5 Annual Review meeting Systems ITD in Linkoping (Sweden)

June 22 Final meeting Table 2 Key activities of the evaluation

The Clean Sky Closing Event on March 21 and 22 drew many of the Clean Sky stakeholder representatives to Brussels and the JU had planned coincident meetings of most of their governing entities. The Clean Sky JU generously provided office accommodation and supervised access to the shared drive through which all of the programme documentation is available in a very orderly and accessible directory. The last three weeks in March provided thus a very efficient closure of the Information Gathering and Interview phase of the evaluation effort.

27

The Annual Reviews of two of the key WORKPACKAGES in Clean Sky 2 (the Large Passenger Aircraft IATP and the Systems ITD) fell within the timeframe of the evaluation process and were attended in part by one of the members of the evaluation team. The JU arranged coincident related site visits where the Clean Sky demonstrator hardware could be seen and appreciated. Additional interviews were conducted with MEP Marian Marinescu on SESAR and Clean Sky content, as well as Thales Avionics and Airbus Helicopter representatives. Stakeholder input through the Coordinator and Public Surveys was analysed and absorbed into the evaluation report outline.

5.2 Limitations

A comprehensive review of the technical foundations of the Clean Sky programme has been beyond the scope of this evaluation. It is more important then that this evaluation presents and analyses the relationship that the Clean Sky JTI continues to have on the eco-system of aeronautics research and the impact that it has had on the productivity and effectiveness of this community. The evaluation findings in this report may be compromised by limitations of the utility of the management information available to the JU Programme Office, albeit generously shared with the reviewers. Insight in the composition of the membership and partners, and the contribution of each to the technical objectives of the programme, seem to be spread across a multitude of spreadsheets that each reflect the purpose for which they were created without elaboration of their data sources. While there is no doubt that the JUs Project Officers and the Executive Director know the Clean Sky programme so intimately that they can make a presentation at the drop of a hat, or find a presentation in response to a reviewers inquiry, we did not see management information at the level of maturity that would be expected. Many important relationships, such as the start and finish TRL of a project; its participants, budget and timeline and its relationship to a demonstrator (or the cause of its failure to contribute) should be more accessible. The deliverables, dissemination activities and eventual exploitation success (or not) should form an integral part of the JU Programme Offices management information system. The reviewers experienced many set backs to their original aspirations regarding the quality and content of this evaluation due to the severely limited time for its creation combined with the challenge of obtaining an overview of the aspects of the programme that were relevant to the evaluation.


28

6. IMPLEMENTATION OF THE CLEAN SKY JOINT TECHNOLOGY INITIATIVE

6.1 Implementation in general

The Joint Technology Initiatives (JTIs) are public-private partnerships (PPPs) in industrial research at a European level. They were a major new feature of the seventh framework programme (FP7), introduced to support key areas of research and technological development that can contribute to Europes competitiveness and quality of life. They were set up in 2007-2008 under the seventh framework programme (FP7) in five strategic areas aeronautics and air transport; public health; fuel cell and hydrogen technologies; embedded computing systems; and nano-electronics. Bringing together industry, the research community, in some cases regulators and the EU, to define common research agendas and invest in large-scale multinational research activities, the JTIs are concrete examples of the European Union's efforts towards strengthening its competitiveness through scientific excellence, openness and innovation. The transition to a second generation of the JUs under H2020 was promoted by the former President of the European Commission, Jos Manuel Barroso in his 2013 statements on an accelerated continuation of the JTI's under H2020. Period CS timeline

Early 2000s Recognition of a lack of large scale projects

2001 Vision for 2020 followed by the ACARE SRA

app. 2005 Initiative by informal industrial group.

June 2006 Clean Sky Workshop (ASD) with 200 participants.

23 June 2006 Study on the proposed Aeronautics JTI (structure and Rules for

Participation), Bertolini, Huguet

October 2006 Proposing members join MoU for Clean Sky

2006-2007 Definition of the ITD content

February 2007 Additional members join the MoU

March 2007 CLEAN SKY JTI Proposal submitted

May 2007-May 2008 FP7 Clean Sky CSA launched coordinated by Airbus SAS.

13 June 2007 Ex-Ante Evaluation of Clean Sky Impact Assessment

20 June 2007 Clean Sky JTI announced by Science and Research Commissioner, Janez Potocnik at Paris Airshow.

December 2007 CSJU established as PPP

2008 Technical evaluation of Clean Sky

4 February 2008 Clean Sky regulation published

September 2008 First ITD coordination meeting

15 June 2009 Clean Skys first Call for Proposals.

20 November 2009 Full autonomy of CS

12 September 2012 14 lead-members sign a Letter of Intent for CS2 under H2020

19 June 2013 The future Clean Sky 2 Leaders give a Joint Technical Proposal

(JTP) to the European Commission at Paris Le Bourget.

10 July 2013 The European Commission launches an innovation investment package that paves the way for the continuation of the Clean Sky Joint Technology Initiative (JTI) within the EU Horizon 2020 Framework Programme.

7 May 2014 The Council of the European Union agrees to extend the Clean Sky Joint Technology Initiative (JTI) within the EU Horizon 2020 Framework Programme with a budget of 4 billion.

24 October 2014 The members of Clean Sky 2 Joint Undertaking Scientific

Committee are selected.

23 July 2015 2nd Call for Proposals for Clean Sky 2 is launched.

22 October 2015 3rd Call for Core Partners is launched.

29

6.2 Structure of Clean Sky 2 JU

The Clean Sky JU was implemented precisely as required by its Establishing Regulation [9] including the governing Statutes in Annex 1. As in CS1, the central decision making body is the Governing Board (GB) comprised of the European Community, represented by the Commission, and 16 members in the Leader role. On a rotating basis there are further 6 representatives (1 per IADP/ITD) of the Core Partners, integrated following the selection of Core Partners through calls, and 6 of the CS1 associates (1 per ITD) until CS1 is closed at the end of 2017. Each member has one vote. Whereas in CS1 the EC has a veto in strategic, financial and procedural matters affecting the responsibilities of the Commission as a public trustee, in CS2 they hold 50% of the vote. Decisions not reached in consensus require an 80% majority of all member votes. The CS1 ITD Steering Committees remain constituted and there are 6 IADP/ITD Steering Committees for the CS2 scope of work to take responsibility for the technical and financial management of the ITD participants. There is further a Steering Committee, established by a GB decision, for the Technology Evaluator and Coordinating Committees for the Eco-Design and Small Air Transport transverse activities. The JU Programme Office is led by the Executive Director (ED) and consists of a support staff unit (i.e. administrative, legal, financial, human resources etc.), a project coordination role headed by the Chief Project Officer (to the good for the programme, a person who has been involved in Clean Sky since its inception) and a Clean Sky 2 Programme Manager. The day to day management of the research agenda is done by the team of Project Officers. The statutory General Forum that engaged Partners who were joined to the programme through selection in calls for project proposals is not present in CS2. The National States Representatives Group (NSRG) of CS1 is continued as the States Representatives Group (SRG) with a similar role and the Scientific and Technical Advisory Board (STAB), which was formed in CS1 by the ED, is in CS2 the statutory Scientific Committee. (SciCom) Whereas the CS1 Statutes explicitly mentions the Advisory Council for Aeronautics Research and Innovation in Europe (ACARE) in the JU tasks and the ED role this is not the case in the CS2 Statutes.

6.3 Budget allocation

The overall CS2 EC contribution of 1,755.5 M was, in view of the need to engage a very broad aeronautical research community, allocated for up to 40% to the co-opted members (the IADP/ITD Leaders and their Affiliates) and for up to 30% to the Core Partners that were joined as members following open calls for Core Partners early in the CS2 programme. All of these participating entities match the EC contribution to their scope of work with in kind contributions. In addition the Members also contributed to the running costs of the JU Programme Office, a total of 78 M. A minimum of 30% of the EC contribution was ring-fenced for disbursement in calls for proposals that drew shorter term Partners into the programme at a funding rate of 100% for research and innovation actions (RIA) for all applicants, and 75% and 100% for profit and non-profit-entities in the case of innovation actions (IA)." Both the Core Partner and Partner calls were executed by the CSJU. CS2

EU contribution 1.755 B [1]

Share for Leaders (members) 40% 702 M

Core partners 30% 527 M

Partners 30% 527 M


30

Table 1 EU contribution broken down to the different types of participants, taken from [9], the running costs are not considered.

The proportion of the EU funding allocated to each ITD and IADP and was, as in CS1, intended to roughly match approximately with the sectorial share, expressed by the GDP contribution of the sector (large passenger aircraft, engine, helicopters etc.). The preliminary distribution was given by the CS2 Establishment Regulation [9]. IADP/ITD/TA Abbr. Distribution

percentage [1] Distribution funding [2] in M

Large passenger aircraft - IADP LPA 24% 521.0

Regional aircraft IADP REG 6% 104.2

Fast Rotorcraft IADP FRC 12% 190.2

Airframes - ITD AIR 27% 347.1

Engines ITD ENG 19% 289.8

Systems ITD SYS 14% 246.5

Technology Evaluator TA TE 1% of IADP/ITD values 17.2 Eco-DESIGN TA ECO 2% of IADP/ITD values 39.0 (part of all IDPs/IADPs) Small Air Transport - TA SAT 4% of IADP/ITD values 68.0 (part of AIR, ENG, SYS)

Table 3 EU contribution broken down to the different types of ITD and IADP, the transverse activities (TA) and the projected costs for running the CSJU, taken from [1] and [2].

Figure 5 EU funding allocation to the different IADPs, ITD's, TA and JU running cost [2]. The numbers are given in M.

The CS1 Interim Evaluations and the CS2 Impact Assessment [19] proposed the introduction of a contingency budget to be allocated by the GB based on proposals by the ED. This was not implemented is CS2 but a provision to adjust budget allocations between the IADP/ITDs/TAs was made. Additional insight in the operational execution of the programme can be found in the level of funding to various types of work. For example wind tunnel testing was an important precursor of flight test demonstrations and was used to eliminate concepts and optimise the final configuration. Similarly, a significant effort would be expected in manufacturing both in hardware and tooling to realise a demonstrator and appropriate ground and air vehicle test facility were also a significant investment. Unfortunately the JU could not provide this breakdown (Table 4). LPA REG FRC AIR ENG SYS

Concept and configuration studies

Wind tunnel models and testing

Manufacturing -tooling and materials

31

Demonstrator Test Facility

Innovative Instrumentation Table 4: Categories of activity per ITD, M

The recommendation in the CS2 Impact Assessment [19] to provide a contingency budget to be distributed by the GB was not implemented but the ED was given a discretionary range of 10% in budget transfers within and between ITDs and IADPs. Furthermore, an affected IADP/ITD no longer has a veto on budget transfers. These measures may improve the agility of programme adaptation in CS2 as compared to CS1. The engagement of co-operations, such as through the ESIF programme, may provide leverage funding for executing the CS programme.

6.4 Leaders and Affiliates

The Industrial Leaders that co-founded the CS2 JTI together with the Commission, and are established as members in [20], are the same entities as in CS1 (Airbus Helicopter was formerly Eurocopter) plus DLR for the TE, Evektor and Piaggio for the SAT and MTU has joined Safran and Rolls Royce in the Engines ITD. The resulting composition of Leaders is shown in Table 5 It is noted that the ownership and trade name of several members might have changed in the course of the programme but the original naming is used here for continuity.

AgustaWestland SpA and AgustaWestland Limited

Airbus SAS

Alenia Aermacchi SpA

Dassault Aviation

Deutsches Zentrum fr Luft- und Raumfahrt (DLR) e.V.

EADS-CASA

Airbus Helicopters SAS

Evektor

Fraunhofer Gesellschaft zur Frderung der

angewandten Forschung e.V.

Liebherr-Aerospace Lindenberg GmbH

MTU Aero Engines AG

Piaggio Aero Industries


32

Rolls-Royce Plc

Saab

Safran SA

Thales Avionics SAS

Table 5 Leaders

6.5 Core partners

Core Partners (in Clean Sky 1 associates) are added to the membership of Clean Sky following a selection through a Call for Core Partners. About 30% of the programme budget is for Core Partners. The calls, and all of the relevant documents, are published on the H2020 Single Portal for participants and follow the H2020 process to ensure openness and transparency. The evaluation panels are comprised of independent experts only, the topic managers may participate in an advisory role but are not part of the panel as in Clean Sky 1. The selection of Core Partners was completed at the end of 2016 through 4 waves of calls with, on average, 20 topics per call (Figure 5). The total number of topics was 75. The distribution of topics a

Documents

Interim Evaluation of the Clean Sky 2 Joint Undertaking … Sky 2 Interim Evaluation Report 30 June 2017 6 7.2.2 Operational effectiveness 61 7.2.2.1 CS Programme Management