Testing and trialling

Good practice guide for testing and trialling new technology

for Britain’s railways

April 2014

2 RSSB

’The railway sector is seeing significant investment, but it comes with the need to deliver greater value for money – to deliver more for less – so that the sector can develop and prosper for the benefit of all stakeholders. Government and the wider rail industry see innovation as a key enabler for this to happen, and the ability to efficiently test and trial new technology and products is essential for this.

The processes for introducing new technology and products onto the operating railway are often seen as unclear as are the responsibilities at each stage of the process. In this guide, the Technical Strategy Leadership Group (TSLG) aims to improve understanding of the process of product introduction and associated testing and to help all those with an interest in introducing new technology to Britain’s railways – through sharing good practice on testing and trialling. The guide is a partner document to the TSLG website which identifies the testing and trialling facilities available to the industry.

The guide is intended as the key reference document for all testing and trialling activities. It provides a thorough description of the testing and trialling landscape for Britain’s railways, together with practical information and experience about how to engage with it and apply innovation. We trust it will prove a valuable resource for all those developing equipment and systems for use on the railway and for those considering commissioning tests or trials.’

Francis How, Technical Director, Railway Industry Association Jerry England, Director of Engineering, Network Rail

3RSSB

When compared with other modes of travel, rail provides substantial benefits in terms of environmental performance and urban regeneration. However, to secure these benefits for the longer term and enhance them, rail has to become more sustainable, particularly in light of the current economic climate. This means change, which needs effective innovation, which needs effective testing and trialling, which ensures risks are appropriately managed when new technology is introduced. The first edition of this guide shared good practice on testing and trialling to promote more efficient introduction of new technology.

Work has continued since then, instigated by TSLG, to address concerns that barriers in testing, trialling, standards, and consistency in the acceptance of innovation could have a negative impact on the introduction of innovative technology or products for GB railways. In particular TSLG felt that it would be worth investigating if better use could be made of test results from other sources and environments to speed up the process of introducing innovation.

A study for TSLG has concluded that barriers to innovation associated with inappropriate testing are not as significant as had been suggested, and that this was not the key issue in bringing innovation to GB rail.

About this guide

4 RSSB

In most cases, testing requirements for development or product acceptance are reasonable. Nevertheless, there are economic impacts associated with defining and completing testing requirements, and these cause delays in the introduction of products and technology. Delays can be significant for small- and medium-size companies. Given that a significant number of the issues identified arose out of a failure to identify stakeholder expectations, TSLG recommended that additional efforts be made in a number of areas, notably to improve up front formal negotiation and planning of acceptance criteria and tests. This updated edition incorporates the findings of that work and seeks to offer guidance to successfully implement these.

The guide is aimed at anyone who has an interest in testing and trialling, whether they have considerable experience in the rail sector or are new to it. It should be of particular interest to:

• Those thinking of commissioning tests or trials, particularly those working for a railway undertaking or infrastructure manager who haven’t been closely involved in this vital area of product introduction before.

• Test house and facility operators, by conveying to a new client the sort of issues that need to be thought about when approaching them to discuss potential tests.

• Those who are developing new technology for

5RSSB

Britain’s railway, particularly those new to the industry or seeking to gain a foot hold in it.

To avoid being a large and inaccessible tome that risks never being opened, this guidance is pitched at a relatively high level and applies across all asset groups. It gives advice that is as applicable to engineers seeking to introduce large complex systems such as new rolling stock, as it is to those bringing in components such as new rail fixings. It aims not just to provide advice on the testing required for product approval, but testing required throughout the whole product development lifecycle; many of the principles are as applicable to concept development as they are to product approval.

Some, perhaps most, of what is in this guide may seem rather obvious – it’s certainly not rocket science - and is relatively straightforward to take into account when conducting testing and trialling. Nevertheless, it can so easily be forgotten or overlooked with costly consequences. To those who see the content in this way, the guide will be a useful aide memoire and they may even wish to contribute to future revisions. To others, particularly those just starting out to earn their spurs in this most important part of the innovation process, it should provide an invaluable compendium of what should and shouldn’t be done. Ignore it at your peril!

While the guide is not intended to replace expert guidance, it should lead to more efficient dialogue with

6 RSSB

© RSSB Copyright 2014 Rail Safety and Standards Board.

This publication may be reproduced free of charge for research, private study, or for internal circulation within an organisation. This is subject to it being reproduced and referenced accurately and not being used in a misleading context. The material must be acknowledged as the copyright of Rail Safety and Standards Board and the title of the publication specified accordingly. For any other use of the material please apply to RSSB’s Head of Research and Development for permission. Any additional queries can be directed to enquirydesk@rssb.co.uk. This publication can be accessed via the RSSB website www.rssb.co.uk.

A PDF version of this document is available to download from www.futurerailway.org/Pages/testingandtriallingfacilities.aspx. The PDF has hyperlinks and bookmarks to aid navigation through the different sections of the document and to other supporting information.

experts. Considering its content may reduce uncertainty, and the risk of abortive tests and cost, as well as improve the chance of success.

The guide has been compiled by the Railway Industry Association (RIA) on behalf of the Testing Facilities Steering Group (TFSG), a subgroup of the Technical Strategy Leadership Group (TSLG). It would not have been possible without the assistance of the many organisations listed in Section 11 for which RIA and TFSG are grateful. Notwithstanding this support, there is probably much that could have been included in this issue of the guide that would give further value to those involved in testing and trialling. With this is mind, comments are welcomed and should be submitted to TFSG via enquirydesk@rssb.co.uk. TFSG will review this guide and consider the need for further iterations in response to feedback.

7RSSB

About this guide 3

Contents 7

1 How to use this guide 9

2 Scope of this guide 12

3 Definitions 14

4 Why do we need to test? 15

4.1 Testing as part of the technology development process 16

4.2 Testing as part of the product development lifecycle 18

4.2.1. Testing to demonstrate compliance with legislation 21

4.2.2. Testing to demonstrate compliance with the contract 22

5 Real or virtual? 26

6 Legislation and standards 32

6.1 Context 32

6.2 European standards and legislation 33

6.3 Domestic standards and legislation 35

7 Where can we test? 39

8 How to go about testing 41

Contents

8 RSSB

8.1 Preparation 43

8.1.1. Context and stakeholders 43

8.1.2. Planning 46

8.1.3. Objectives and success criteria 50

8.1.4. Roles and responsibilities 52

8.1.5. What is to be tested 55

8.1.6. Facilities and equipment required 56

8.1.7. Facilities and equipment required 60

8.2. Testing 62

8.3. Reporting 63

8.4. Reviewing how it went 64

9 Special requirements when trialling 65

10 For more information 70

11 Acknowledgements 73

Annex A: Planning checklist 74

Annex B: Processes to support on-track testing and authorisation for placing into service of rolling stock 78

References 80

9RSSB

This guide could be read from beginning to end and has been written to tell the ‘testing and trialling story’ in a logical sequence. However, the Test Facilities Steering Group (TFSG) is acutely aware of the demands on people’s time and that the value in a guide such as this is to make the content as accessible as possible. The guide has therefore been structured in a way that signposts content to facilitate quick and easy reference, in the hope that it will become a valued reference guide for those interested in testing and trialling new technology for Britain’s railways.

Although the document can be read as hardcopy, it has been developed to be read in its electronic form so readers can make use of the many hyperlinks, which not only take you to other parts of the document (green hyperlinks), but to other reference material that can be found on the internet (bold hyperlinks). In an attempt to illustrate the points being made and add further interest, anecdotes and case studies are given throughout the guide in grey boxes.

Briefly, the guide is structured as follows. The scope is outlined in Section 2 followed by key definitions in Section 3. Section 4 explores the role of testing and trialling during the development of new technology and the product development lifecycle, while Section 5 discusses the ever increasing role of computer based simulation in this. Section 6 sets out the legal

1 How to use this guide

10 RSSB

framework and relevant industry standards that we need to comply with when testing to demonstrate compliance against mandatory requirements. Section 7 signposts where to get more information on facilities that can be used for testing and trialling work, while Section 8 sets out good practice for testing, followed by Section 9 which gives special requirements for trialling. Section 10 signposts where to go for more information and expert advice, and the guide concludes with Section 11 which acknowledges the many organisations that have assisted with its compilation.

11RSSB

12 RSSB

What this guide covers What this guide does not coverTesting for Britain’s mainline railway systems: Network Rail managed infrastructure. Information for London Underground and other systems has been included where readily available.

Light rail, heritage railways and foreign rail administrations, although many of the principles will be transferable.

Testing and trialling using physical facilities such as laboratory test rigs, as well as test tracks and other sites dedicated to testing and trialling purposes such as sections of the operational railway.

Computer simulation, although a section is included on its ever increasing role in the testing arena.

Testing of new technology that exists in a physical form, including mechatronic systems.

Testing of software when not part of a physical system, such as when not part of a mechatronic system. Further information relating to software can be found on the management of change page of the RSSB website and EN50128:2011 ‘Railway applications. Communication, signalling and processing systems. Software for railway control and protection systems’.

2 Scope of this guide

13RSSB

What this guide covers What this guide does not coverMost of the guide concentrates on testing although there is a section on trialling, and many of the principles described for testing will apply to trialling as well.

Routine testing, such as following a vehicle exam (see Railway Group Standard GM/RT2004), or electrical appliance testing, although many of the principles may still apply. Equally, testing following incidents and accidents is excluded (see Railway Group Standard GM/RT2273).

All types of railway asset (such as rolling stock, track, and signalling) with asset specific standards referenced where readily available. However the principles tend to be defined at a higher level and therefore apply across all assets.

Guidance on seeking product acceptance, except where acceptance affects testing and trialling. Further guidance can be found on the Network Rail website and in its Company Standard NR/L2/EBM/029 (see Section 10).

Technologies, products, and systems.

Manufacturing processes and their management systems.

14 RSSB

While the following terms can mean many different things to different people, this guide uses them with the following meanings:

• Test – to demonstrate using a physical process (as opposed to computer-based) that the equipment being tested is capable of achieving a specified level of performance for the test environment and conditions used; the testing using facilities away from the operational railway. This may include testing on service lines during a possession.

It may also include commissioning where new technology is being made ready for operational service and which might involve testing such things as the man-machine interface, finalising operational processes and staff training.

• Trial – to test performance using facilities on the operational railway in revenue earning service; the word ‘test’ having the meaning given above.

3 Definitions

15RSSB

The dictionary1 definition of the verb ‘to test’ is ‘to try (something) out to ascertain its worth, safety or endurance’. In the context of this guide, and indeed the wider engineering field, this means testing to demonstrate performance against required functionality. Such functionality may be nothing more than conceptual aspirations of a nascent innovation, or it could be acceptance criteria formally defined by a regulatory body for a mature technology. Testing is not just about gaining acceptance and demonstrating that something is safe to operate, however important that may be; it is an integral part of the process of technology development long before it becomes mature enough to warrant acceptance.

It is perhaps useful to consider testing in the context of both technology development and product development; the latter being a subset of the former and taking place in the latter stages of the technology development lifecycle.

1. Paperback Dictionary, Collins, 2004

4 Why do we need to test?

16 RSSB

4.1 Testing as part of the technology development processTesting is a vital aspect of the development of new technology. The box on the following page defines the technology readiness levels (TRL)1 which describe the different stages that a new technology goes through during development. Testing enables a technology to progress from TRL2 where it is nothing more than a concept through to TRL9 where it is proven and accepted into the operational environment.

It would be unwise to wait until full-scale testing to determine the value of a new technology; less expensive small case testing (in a laboratory for instance) is normally done first. Nevertheless, testing enables the technology to be assessed at each stage, whether it be in a laboratory or full-scale testing or trialling in an operational railway environment. The design can then be corrected or improved if necessary and the next stage of development informed (or even shelved).

Testing serves a number of purposes during the exploration and assessment of a new technology and it is wise to explicitly agree its purpose. For example, to:

• Understand the behaviours, so that the effect on adjacent systems can be understood.

• Establish boundaries of performance, so that a specification can be created.

1. ‘Development of a technology roadmap and action plan for the GB railways – short-term roadmap’, R&D project T809, RSSB, October 2010.

17RSSB

• Demonstrate viability and value, by exceeding some required threshold of performance.

• Explore failure limits and behaviours, to consider risk and safety.

Only when the purpose is clear can the appropriate tests be designed and the requisite performace targets and thresholds be negotiated.

The next section considers the role of testing during the later TRLs, during product development.

18 RSSB

4.2 Testing as part of the product development lifecycle

Consider now a technology that has emerged from the ‘valley of death’ (TRLs 4-6) and is starting to be commercialised as a new product and integrated into a system.

The familiar V lifecycle diagram (next page) is derived from one published by the Institution of Railway Signal Engineers (IRSE), and depicts the different stages of development that a system will undergo throughout its entire lifecycle; from its birth as an outline definition (in response to a business need) through gradually increasingly more detailed design phases, through to testing phases, commissioning, operation and maintenance and finally decommissioning. Verification and validation (proving) activities include testing to ensure that the requirements defined earlier on in the lifecycle have been delivered.

Whether we’re talking about rolling stock or infrastructure components, as the stages of testing progress, the environment will move from the laboratory, to test tracks and then onto the operational railway. As this happens, precise control over input conditions will gradually deteriorate making way for an environment that is gradually more representative of the operational railway but less determinate.

19RSSB

‘Competence Guidance for Train-Borne Train Control Systems’ © IRSE 2009

20 RSSB

The impact test cannon, one of a series of factory test rigs for testing vehicle bodyside windows to Railway Group Standard GM/RT2100.

© Independent Glass, Glasgow

Each stage of the testing process has its place and there are pros and cons to each stage. While it can be difficult in a laboratory or factory to fully simulate service conditions, it does enable performance to be tested in a known and consistently repeatable (test) environment that is difficult to replicate in a demonstrable (measurable) way on the operational railway. Examples include:

• Tests of electronic systems in climatic chambers to ensure that they continue to perform as required over extreme temperature ranges.

• Accelerated fatigue testing of sleepers on laboratory test rigs to demonstrate acceptable life.

• Testing of vehicle bodyside windows.

• Sled testing of anatomical test dummies in seats to test interior crashworthiness.

In the later stages of the product development lifecycle, testing becomes an essential part of the acceptance process (although it is by no means the only part). Testing at this stage can typically be divided into two areas: testing to demonstrate compliance with legislation and testing to demonstrate contractual compliance.

21RSSB

4.2.1 Testing to demonstrate compliance with legislation

This is usually aimed at demonstrating that the new technology will be safe and compatible with the system in which it is due to operate. These ‘type tests’ are normally performed on the first of build example or prototype; subsequent builds can then simply be subject to tests that prove they are the same as those which were type tested.

• Testing to demonstrate safety is usually carried out in accordance with test protocols and performance criteria set out in standards. Section 6 gives more information.

• Testing to demonstrate compatibility with the system in which the technology is expected to operate may not be subject to such clear requirements. These tests need to assess the impact of the new technology on the system in which it is expected to operate and vice versa. This is about systems integration, ensuring that interfaces perform as required. Examples include:

• A new sleeper ‘engaging’ with the ballast and rail retaining mechanism.

• New trains interfacing with the permanent way, power supply, signalling system, the wider environment and, operationally, with traincrew and passengers. EMC testing of new trains to demonstrate compatibility with the signalling system has in the past been particularly problematic.

Even technology that has been proven in other environments may not be suitable for the rail

22 RSSB

environment, for instance, a road/rail crane will behave differently when on rail wheels to road wheels and could be more unstable. Another example is Network SouthEast running a trial on a commercially available hand-dryer in the toilet of a train as part of the ‘tractionisation’ process, ie to ensure that reliability of the hand-dryer would not be compromised by the accelerations seen in the rail environment.

Whether testing to demonstrate compatibility or testing to demonstrate safety, the criticality of these tests means that the results are usually required to be assessed by independent third parties to give the required level of assurance. Choice of the third parties is governed by the assurance regime that has been selected and can include Notified Bodies (NOBOs), Designated Bodies (DEBOs) and Vehicle Acceptance Bodies (VABs). Section 6 gives more information.

4.2.2. Testing to demonstrate compliance with the contract

This is dictated by the requirements set by the client in the contract and will usually be carried out as part of type testing. By way of example, ATOC1 suggests the following stages of testing to gain client acceptance for new trains (notwithstanding those associated with legislative compliance):

• Fault free test track running to enable preliminary acceptance at the ‘factory gate’.

• Commissioning both static and dynamic tests to

1. ‘The Twenty Point Plan, Fleet Reliability Focus Group’, CR/TP 1203 Issue 7, January 2011.

23RSSB

gain enough confidence for trains to be run on Network Rail managed infrastructure.

• Test runs, accumulating mileage and proving experience on Network Rail managed infrastructure to enable provisional acceptance.

• Operation in passenger service under the operator’s safety management system.

• Final acceptance when modifications arising from service experience are completed and sufficient fault free running has been achieved.

Examples of the sort of testing that takes place to demonstrate compliance with the contract are:

• Vehicle reliability – a period of mileage accumulation on a test track to demonstrate fault free running prior to operational service.

• Performance – proving, for instance, that a new tamper can work at a sufficient speed.

• Capability – to confirm how far beyond the required limits a machine can actually continue to operate safely. For example, the maximum installed cant a crane can operate safely when fully extended.

• Vehicle ride quality – to confirm that the accelerations that the passengers will feel as a result of vehicle movement are acceptable.

The opportunity can also be taken during testing programmes (or at least while the new equipment is visiting test facilities) to achieve other objectives besides testing alone. For example:

• Training those who will use the new equipment in an

24 RSSB

environment where risks can be better controlled. This could include emergency scenarios which are difficult if not impossible to simulate elsewhere.

• Proving the man-machine-interface between people and complex on-track machines to inform production processes – if people can’t work with it then there’s no point in commissioning it.

• Fine tuning the maintenance strategy to help maintenance staff and improve care of the new equipment.

A ballast regulator used for distributing and profiling track ballast. An example of one of the complex on-track machines used to maintain the infrastructure, which need ‘pre-production’ tests and trials to prove the man-machine-interface and the processes required to use the machines effectively.

25RSSB

Be very careful of other purposes gradually creeping into a test programme. For example, people might be interested to ‘see what it will do if …’ Unless the purpose, justification, protocols, and parameters are properly designed as part of the test programme planning, such tests can cause costs and, more importantly for those seeking to market new products, delays that are hard to control and that increase commercial risk.

26 RSSB

This guide concentrates on testing in the physical environment; however, advances in technology during the last few decades now make computational simulation a worthy alternative. Although the robustness of most numerical models means that physical testing will continue to be needed for some time to come, virtual testing can offer significant benefits. It is not uncommon in many industries these days, for simulation to be widely used as a way of avoiding the delays and costs of traditional approaches to testing and trialling. But when should the virtual test environment be chosen over the physical?

In practice the distinction is rarely so marked. Nowadays a ‘testing’ programme is unlikely to involve only physical tests or only computer simulation; rather it is increasingly common for one to inform the other, in an iterative sequence, throughout the test programme. For instance, in the automotive industry, testing typically comprises:

• Laboratory component testing supported by simulation.

• Full vehicle simulation.

• Only then, full-scale physical crash tests.

A lot of money can be saved by using computer simulation:

5 Real or virtual?

27RSSB

• At the start, to inform development of the new technology, establishing how performance varies with changes in design parameters, to optimise the design.

• Towards the end of the design process, to assist with problem solving.

However computer models will only ever provide estimates; they are only as good as the data that goes into them and that data is likely to have come from testing, whether it’s testing to establish material properties, or testing to establish boundary conditions that describe the operational environment. Furthermore, validation of some form is usually required and that ultimately means physical testing at some stage.

Striking the right balance between the physical and virtual test environment is important, based on the pros and cons of each technique, but ultimately driven by the need to make the development time as short as possible and thus time to market. Here are a few

When asked why scale models were being used for assessing the impact of wave action on costal defences, instead of computer modelling, Professor William Allsop, Technical Director of the Coastal Structures Group at HR Wallingford, said that numerical models aren’t yet rigorous and robust enough to commit to building real structures... to understand how the structures would work, models were required. Interview for BBC’s Countryfile programme broadcast on 20 March 2011.

28 RSSB

things that can influence that balance:

• Cost – virtual testing is not necessarily cheaper than physical testing. It can be expensive up front because computer models need to be developed.

• Time – virtual testing can be quicker because physical testing requires components to be manufactured, which takes time particularly for low production volume prototypes.

• The number of parameters – if the purpose of testing is to establish the impact of different design features on performance, then virtual testing is very good at quickly predicting how performance is affected by varying large numbers of parameters.

• Availability of the necessary and representative testing facilities/equipment and engineers with the necessary competence to manage, plan and conduct the tests as well as analyse the results.

In their work on whole train dynamics for RSSB project T118, Atkins and AEA Technology Rail (now DeltaRail) simulated collisions in a rake of vehicles studying the effect of four different parameters – analysis and presentation of the results can be quite challenging!

29RSSB

• Availability of suitable computers and, more importantly, competent engineers to construct the computer models and analyse the results.

• The availability of suitable numerical models for virtual testing and how well validated they are – how well understood are the physical phenomena? Are the models empirical (based on observations from previous tests) or deterministic (using mathematical models derived from theory)? Computer models have to be validated at some time or other using physical testing, and some validated models are available commercially, such as finite element analysis (FEA) and computational fluid dynamics (CFD) packages. Validation will only apply over a specific range of conditions, and by implication these models will remain unproven outside those ranges, when further physical testing is likely to be required. More recently, there has been a trend for parts of

DeltaRail’s VAMPIRE software is widely regarded as being a well validated computer model for simulating the wheel rail interface and comes with a number of validated vehicle models. New vehicle models are typically validated using test results from physical testing of the vehicle as a whole, or perhaps individual components. © DeltaRail Group Ltd

30 RSSB

computer models to be validated by physical testing. For example, the head of an anthropomorphic test device (ATD), colloquially ‘crash test dummy’, being subject to physical testing to validate that part of a computer model. Remember, the model is only as good as the data used to build it; garbage in – garbage out!

.... A computer model of the vehicle, which has been validated against the laboratory ΔQ/Q and X-factor tests, is used to predict the Y/Q ratio as a time history when negotiating prescribed track inputs...

Clause 1.2.4.1 - compliance method 2 in GM/RT2141 Issue Three, June 2009 ‘Resistance of Railway Vehicles to Derailment and Roll-Over’

31RSSB

• Client confidence – the most easily understood outputs of computer simulation involve watching animations on a computer screen, which have seemingly been determined by a grey box with little human involvement. Intrinsically, this is less convincing than watching physical tests, even though the latter will typically cover fewer scenarios and the test environment may vary. The client needs to buy in to the approach, particularly if it involves computer simulation in totality.

More information on computer based simulation can be obtained from the CFMS consortium, a non-profit organisation dedicated to driving forward the boundaries and reducing the cost of simulation-based design techniques.

32 RSSB

As explored in Section 4 and in particular Section 4.2, testing to demonstrate compliance with legislation and standards is just one of a whole range of reasons for testing; however it is one of the most important and high profile. This section has therefore been devoted to clarifying which legislation and standards set out relevant requirements; something that can be particularly confusing for those new to the rail industry.

6.1 Context

In the days of British Rail, requirements for the mainline railway network were set by the British Railways Board and much of the testing was done in-house by that organisation under its Director of Mechanical & Electrical Engineering or Director of Research. Not surprisingly things have changed. The railway has been privatised and now many different organisations are involved with setting standards and demonstrating compliance. This has introduced many complexities with which Section 8 endeavours to assist; however there has been a more fundamental change.

As part of the European Union, Britain is subject to European Law. Over the last couple of decades the European Commission has issued many directives, standards, and regulations that have a significant impact on the way we do business. The intent is to free up markets by reducing barriers to trade and, for the railway in particular, to work towards an integrated and

6 Legislation and standards

33RSSB

interoperable European railway network. This promises significant benefits in the longer term, if not sooner, however in the short term at least it has meant change and complexity while the railway network is in transition and gradually becomes more ‘interoperable’. Testing is no exception and anyone involved in testing on the railways needs to consider not just domestic standards and legislation but European also. The rest of this section elaborates on both of these areas.

6.2 European standards and legislationThe European Commission has a number of statutory instruments at its disposal to enact legislation and affect the way industry goes about its business. Further information on each of these instruments can be found on the European Commission’s website; however some of the most relevant to the railway industry are the following EC directives, the instruments used to align national legislation:

• The Railway Safety Directive 2004/49/EC, aims to harmonise the approaches to managing safety throughout the European Union. Its relevance to this guide is that it sets out the overall framework in which testing contributes to proving that something is safe to operate.

As an EC Directive, the UK government is required to transpose the requirements into domestic legislation and this has been achieved through the Railways and Other Guided Transport Systems (Safety) Regulations 2006 as amended (commonly known as ROGS, and last amended May 2013).

34 RSSB

• The Railway Interoperability Directive 2008/57/EC (RID), sets out the conditions that need to be met to achieve interoperability of the community’s railway network. Of particular relevance to this guide is the amendment of Annexes II, V and VI by Directive 2011/18/EU, which split the verification procedure for subsystems into two parts: an EC verification procedure (by a Notified Body (NOBO)) and a ‘verification procedure in the case of national rules’ (by a Designated Body (DEBO)). Further guidance is provided in Commission Recommendation 2011/217/EU on the authorisation for the placing into service of structural subsystems and vehicles. As a recommendation, it is not mandatory but it does give what is widely recognised to be more clarity and good practice; member states therefore need to have a good reason for not following it.

Although the two mentioned above have had the most significant impact, there are many more directives which are relevant to the railway industry. Two of relevance to testing are:

• The Non-Road Mobile Machinery Directive (NRMM) which sets emissions limits for diesel engines and of which there are four directives: the ‘mother’ Directive 97/68/EC and three amendments, the most relevant of which is Directive 2004/26/EC.

• The Machinery Directive 2006/42/EC which specifies requirements for On-Track Machines when working on the railway (as opposed to travelling on the railway between work sites).

35RSSB

Another significant implication of the RID has been the development of the Technical Specifications for Interoperability (TSI). A list of the TSIs and their current status is provided on the RSSB website along with more detail on the European legislative arena as it pertains to rail.

The final pieces of the European legislation and standards jigsaw are the EuroNorms published by the European Standards Organisations: CEN (mechanical), CENELEC (electrical) and ETSI (telecommunications) which specify standards for components, systems and processes (including product testing) that are recognised throughout the European Union. These standards are not mandatory in their own right, unless they have been called up by the TSIs, but they may become mandatory for specific contracts by virtue of being called up by a contracting entity.

As can be seen, the European legislative and standards framework is complex. Those needing greater awareness are encouraged to seek advice from experts by contacting the relevant organisations listed in Section 10.

6.3 Domestic standards and legislationWe have seen how the European standards system is gradually developing to improve harmonisation across member states of the European Community and ultimately improve efficiency and reduce costs. However, there are a number of areas where it has not yet been possible to harmonise practice across member

36 RSSB

states and this is where domestic standards come into play.

Domestic standards, in particular Railway Group Standards (RGS), will continue, for the foreseeable future, to have a number of roles in the context of the European Standards system. RSSB provides an explanation of whereRGSfitintheEuropeanstandards system on its website, and the diagram that depicts the relationships between the key ingredients provides a particularly useful summary.

There are many standards that are relevant to testing on the mainline railway; too numerous to mention in this guide. However, the high-level standards for each of the main asset types, are shown in the table on the following pages. Most of these reference further relevant standards. Section 10 provides advice on how to source these standards.

The Railway Industry Association also co-ordinated the development of a number of ‘RIA Specifications’ prior to the introduction of CEN and CENELEC standards. Most have now been superseded by EuroNorms, but more information can be obtained from RIA’s website.

Network Rail has recently introduced a streamlined innovation and suggestions scheme which identifies their challenges and priorities, encouraging those with relevant ideas to submit an outline of the innovation.

37RSSB

Asset types Mandatory AdvisoryGeneric (applies to multiple types of asset)

GE/RT8270, Assessment of Compatibility of Rolling Stock and Infrastructure

Not applicable

Rolling Stock GM/RT2000, Engineering Acceptance of Rail Vehicles (note that, in response to ROGS, RSSB has recently issued a non-compliance against this standard to allow alternatives to the VAB approach that it mandates)

GM/RC2510, Code of Practice for Acceptance Testing of Rail Vehicles

GM/RC2559, Railtrack Approved Code of Practice: Safe Testing of Rail Vehicles on Railtrack Controlled Infrastructure

GM/RC2515, Engineering Development of Rail Vehicles - Code of Practice

GM/GN2688, Guidance on the Structural Design of Rail Freight Wagons including Rail Tank Wagons

Network Rail Stakeholder Relations Code of Practice: Introducing new vehicles or changes to vehicles

Annex B gives an overview of the processes to support on-track testing and authorisation for placing into service

38 RSSB

Asset types Mandatory AdvisoryOn-Track Machines GM/RT2400,

Engineering Design of On-Track Machines

Not applicable

On-Track Plant See relevant NR company standards

RIS-1530-PLT, Rail Industry Standard for Engineering Acceptance of On-Track Plant and Associated Equipment

Signalling GK/RT0045, Lineside Signals, Indicators and Layout of Signals

GE/RT8026, Safety Requirements for Cab Signalling Systems

GK/GN0645, Guidance on Lineside Signals, Indicators and Layout of Signals

Permanent way GC/RT5021, Track System Requirements

GC/RT5033, Terminal Tracks - Requirements for Buffer Stops, Arresting Devices and End Impact Walls

Civils and structures

GC/R5112, Rail Traffic Loading Requirements for the Design of Railway Structures

RIS-7700-INS, Rail Industry Standard for Station Infrastructure

Tele- communications

GE/RT8082, GSM-R Cab Mobile, Great Britain Open Interface Requirements

Electrification GE/RT8016, Verification of Electrification Systems and Interactions with Other Systems

39RSSB

The 24/7 railway and the need to protect performance, increasingly preclude the possibility of testing in the normal operational environment. We therefore prefer to test at dedicated facilities away from the operational railway. There is an increasing number of cost-effective facilities available to Britain’s railway industry. These are provided by Network Rail and a variety of other industry stakeholders and facility providers, either in Britain or elsewhere, either specific to the rail industry or other sectors. These can be found at dedicated test centres, such as Network Rail’s Innovation & Development Centre at Melton, or may be single test rigs used by a supplier or university, but which could be made available to third parties. Regardless of what equipment is needed for testing, competent personnel are also needed; facilities are no good without the expertise necessary to design and run representative tests, preferably based locally to minimise logistics and costs.

7 Where can we test?

Not surprisingly, Network Rail also shares TfL’s view. In the March/April 2010 issue of Rail Magazine, Jim Morgan, Network Rail’s Principle Programme Sponsor for ERTMS is reported as saying ‘Anybody who wants to install ERTMS in the UK on our network will have to install it on Hertford and prove it will work’, referring of course to its new Hertford North Integration Facility.

In an article in the Evening Standard of 9 March 2011, Peter Hendy, Commissioner for Transport for London, is reported as saying ‘we will never give contractors free access to the railway at weekends’ adding that the engineering work will have to be done at night, and the signalling software tested off-site.

40 RSSB

Information on facilities available to Britain’s railway industry for testing and trialling has been collated by RSSB and made available on the future railway testing and trialling web page.

Feedback would be welcomed on:

• How to develop this facility page further

• Facilities that are needed but that appear unavailable

Please send any comments to TFSG via enquirydesk@rssb.co.uk.

41RSSB

Having explained why we want to test, the legal framework that applies, and where to find information on test facilities; this section covers how to go about testing.

The timeline of any testing programme can be separated out into four discrete phases as follows:

• Preparation – the most important part of the whole process.

• Testing – obtaining the results and analysis.

• Reporting – documenting the evidence.

• Reviewing – identifying how it can be done better next time.

Good practice for each of these phases is outlined on the following pages.

8 How to go about testing

42 RSSB

New and innovative system of lightweight signals and structures – delivering high performance and cost savings.

Demanding expectations and short time-scales can be achieved if:

• The supplier has the technology and ability to:

• Think ‘out of the box’

• Apply tested solutions

• The client has a need with driven benefits:

• Bigger rewards/pressures provide increased focus

• Input is captured from stakeholders:

• End product addresses stakeholder needs

• Gets buy-in from users

• Both sides (client and supplier) contribute to the objectives:

• ‘Push-pull’ effect

• Don’t do it in one go:

• Get concepts and mandatory requirements right and accepted

• Then attend to the detail

©Variable Message Signs Limited

43RSSB

8.1 Preparation

Testing is a crucial phase – however it is usually under time pressure, reputations are often at stake and there are frequently commercial consequences far in excess of the value of the item tested hinging on the results. So it is important that all involved understand the wider context in which the test is being run and through this it helps to see why various stakeholders behave as they do.

So, in the following sections there is continuing emphasis on the importance of understanding and addressing the perspectives of all those involved as tests are conceived, planned, and agreed. During preparation for the test make sure everybody is clear whether the test is to open up new understanding or to converge on a go/no-go decision. As roles and responsibilities are clarified think also of people’s incentives as well as their power, personal professional risk, and their professional capability. The better the preparation, the better the foundation for cost-effective and timely success – from the perspective of all parties.

8.1.1. Context and stakeholders

Section 4 discussed ‘why we need to test’, identifying that testing might take place at any stage of the life of a maturing technology or product and for differing purposes. Only at the later stages, when the application is well-understood and there is a clear specification does testing for acceptance come to the fore. Before that there may be broader and less precise questions

44 RSSB

– does this technology or product provide the sort of performance that would solve our problem at an acceptable risk? How far can we push the performance in this particular application? Does this have any unanticipated consequences? Do we fully understand the implications of developing or adopting this technology or product?

Different stakeholders will have different views of the most important questions to be answered. A supplier may feel that previous experience abroad should be evidence of product quality, whereas a customer may wish to test compatibility with legacy systems unique to GB rail. A supplier may be anxious to break into a new market with a reference site, whereas a customer may not feel the need for yet another supplier to add to their current portfolio.

Downsides may be different for different people. A supplier may be sensitive not just to the cost of a test, but also fear that delay might mean missing a window of opportunity for lucrative contracts or time sensitive markets. Hence ‘adding one more test’ might provoke a reaction disproportionate to the incremental cost. Similarly, a project manager may be more interested in limiting the risk to his or her project than in obtaining system-wide success through innovation or the last few percent of performance – performance visible only long after s/he may have been pilloried for an expensive overrun. The stakes get higher in safety-related matters. Individuals may be concerned about personal career risk as well as commercial risk as they decide

45RSSB

what testing is necessary to demonstrate satisfactory safety performance. What further testing might they feel to be necessary to demonstrate their personal diligence as well as delivering the objective benefits of a test?

Finally there might be the matter of understanding technology potential. In some instances, customers may not yet understand the implications of their requirements – they are still exploring what questions to ask. Many suppliers are frustrated when they have not grasped where their customer is on the road to technology acquisition, or when the customer has not thought through the system-wide potential of a solution.

Hence the customer and the supplier must share their perceptions of:

• The maturity of the new technology, product, subsystem or system.

• The extent to which the interfaces are well understood or characterised.

• The degree to which the customer understands how they will use the product or system, and the extent to which all the requirements have been captured and specified.

• The extent to which the required performance can be unambiguously defined in ways that are testable.

• The degree of step change which will need to be rolled out by the customer, against the commercial benefit and the risks to either party.

46 RSSB

Only then can the purpose of the test be clarified and the test be designed to demonstrate what is needed.

8.1.2. Planning

Without exception, all the experienced test engineers, customers, and suppliers that helped to compile this guide said that planning is the most important part of the process.

Testing uses valuable resources; whether stakeholder time or facility time which can cost thousands of pounds each day. Effective planning ensures that you use your resources efficiently in the most expensive phase of the test cycle – the testing phase itself. As the old adage goes ‘proper planning prevents particularly poor performance’!

For the purposes of communication and change control, the result of the planning process should be captured in a ‘test plan’. This is the ‘bible’ for the whole testing programme.

The test plan should be the outcome of discussions and agreement between supplier(s) and customer(s), recognising that for complex systems there may be teams of both supplier’s and customer’s engineers involved. These discussions should explore the context for the testing and consider the perspectives of those involved (Section 8.1). The objective of these discussions is to negotiate a shared set of expectations about the tests, their purpose, the criteria for success, boundaries, the methods and protocols and who is

47RSSB

going to do what. In other words an agreed way forward at the outset.

The starting point is to agree what risks the customer (and also the supplier) wishes to address by testing. What uncertainties and questions will the test resolve? What performance needs to be demonstrated – and why that particular level – and under what operating conditions or environment?

Knowing this enables the supplier and the customer to propose and agree constructive and effective tests that will demonstrate the outcome and deliver the certainty that both parties seek. What tests will best demonstrate the performance that addresses the needs (technical and commercial) of the customer and other stakeholders that will use the new equipment?

48 RSSB

Difficult though it is, a valuable step is to consider what the implications might be of a failed test in respect of a given product. What useful information can be gathered? What contingencies are available to enable the product to be taken forward to the benefit of both supplier and customer and would there be an impact on price/performance over that originally specified. Such thinking is particularly valuable during technology development.

A critical issue, especially for equipment that might have been used in other applications or on other railways, is the admissibility of prior test results or prior operational or maintenance experience. This is where a clear understanding of the objectives of the tests becomes so important. If the background concerns are those illuminated by past test experience – for example long-term reliability, then past tests or operational evidence may be useful. But if the concern is about aspects unique to the application then transfer of past experience may be harder.

There is increasing interest across the industry in the philosophy of ‘plug and play’, for example to minimise field-testing. This shifts the focus of testing from the performance of the product in the field to the robustness and consistency of controls over manufacturing – so testing will move increasingly to the demonstration that product conforms to type, every time with complete repeatability and reliability. This shift to testing process rather than product will require suppliers and customers to consider anew their approach and the tests they choose to perform and assess.

Finally, the parties should work together to identify what

49RSSB

might be the best choice and sequence of tests – what gives the most insight for the least risk and cost?

And if necessary, loop around the process, revisiting the questions of benefits and insights from the tests, understanding the extent to which they address key concerns and finding the best choice and sequence of tests, the best use of prior experience and therefore the most cost- and time-effective approach for both supplier and customer.

With this negotiated plan in place, all the parties involved have a view of the future tests, why they’re needed, their criticality to each party, how they’ll be assessed and what the implications and impact are on wider plans.

Without a shared set of expectations, somebody is likely to get a nasty and possibly expensive surprise during the test programme.

The agreed test plan should define:

• The objectives of the tests and the success criteria

• What are the concerns and risks to be addressed by the tests and/or what performance criteria are being applied and why?

• Roles and responsibilities

• Who is going to do what?

• What is to be tested?

• Consider the boundaries of the equipment to be tested and give careful consideration to configuration control.

• Methodology

• Facilities and equipment required.

50 RSSB

8.1.3 Objectives and success criteria1. Understand why the test is being done, and what the

results of the tests will and will not be used for.

• Resolve concerns and uncertainties

• Explore risks and show them to be under control

• Assess performance and show it meets needs

• Explore and understand boundaries of potential

• Understand insights gained and future potential and possibilities

• Treat and resolve emergent open points and unforeseen issues

2. Define the scope of the tests and the objectives; what does the testing need to demonstrate? Is it just to demonstrate compliance against mandatory standards, are there further contractual requirements to be met, or is it a good opportunity to determine the safe limits of operation? (See Section 4.) To what extent can test results and experience from other environments be used to inform the objectives?

3. Define the requirements that need to be met to pass the test, including relevant standards that have to be complied with. Consider the following:

a) The standards may not just pertain to the design of the new technology and cover approval

The following sections outline many components of the test plan so, for ease of reference, a checklist has been provided in Annex A.

51RSSB

requirements; they may also pertain to its operation and the delivery of the tests.

b) Ensure that the requirements are fit for purpose. If they have been defined by the client or a third party, work with them to ensure they are appropriate, particularly when integrating new technology into equipment that may have been around for decades.

c) Ensure that the requirements satisfactorily describe how the interfaces with the system/environment are required to interact with the equipment or new technology, and vice versa.

Note: The interfaces with the system and environment may not only be mechanical and electrical, they may also be human and climatic, etc. Some may not be controlled so easily when finally in service operation, such as weather and vandalism.

d) A further complication is that interface requirements may not yet be defined, because the equipment that the technology has to interface with has not yet been installed. For example designing electric traction for IEP on the Great Western Mainline that has not yet been electrified.

4. Define the inputs to the test. Conclusions cannot be reliably drawn about the behaviour of the test system if the inputs are not understood; for example: how can you reliably compare the ride quality measured over different pieces of track if you don’t know the track quality and the operating speed over it?

One experienced testing engineer referred to a case where innovative converter interlacing on new rolling stock led to higher frequency EMC than that specified in the requirements. This led to unexpected interference with lineside equipment.

52 RSSB

8.1.4 Roles and responsibilities

Is there an identifiable individual championing the need for and the progress of the product – someone who is convinced of the economic and technical merit of the technology, product, or system, and hence the need to test it? Experience shows that the best progress is made when there’s an enthusiastic and committed person driving the testing forward. And progress is better still when there are champions inside each of the main organisations involved so they can push forward within their own organisations. So identify your champions – and worry if you don’t have any.

In today’s railway, most testing programmes are going to involve at least two companies, if not many more. This isn’t necessarily a bad thing, but it does mean that there are more risks to manage because there are more interfaces.

As in any project, stakeholder management and communication between them is critical for success. Here are some things to consider when managing stakeholders, that is, anyone who is affected by the testing programme, including those who will be doing the legwork as well the client and approvals bodies:

1. Identify all stakeholders who need to be involved and think about their interests and concerns (both corporate and personal). There could be a large number of different organisations involved: operators, infrastructure managers, vehicle owners, suppliers of the equipment, test facility operators, specialist engineering support, regulatory bodies, neighbours,

53RSSB

independent assessors, and approvals bodies (NOBOs), and of course the end clients. They will all have differing requirements and interests.

The organisations and individuals involved will each have different perspectives and concerns. Explicitly think through the perspectives of these – as individuals. What’s in it for them? Perhaps they’re indifferent; for example, is this just another product test when there are adequate numbers of alternatives already on the market? Is there a downside for them? Are they about to take a difficult decision that might come back to haunt them, and hence they choose to be more cautious than you would otherwise expect?

2. Ensure that all stakeholders have a common understanding of the objectives and the test plan, especially across disciplines, where the interfaces typically lie and thus risks. Early engagement with those authorised to approve the new technology is invaluable.

3. Obtain agreement from all stakeholders – a word of caution – this can take more time than the testing itself. Getting stakeholders engaged in a testing programme that might not happen for two years may not be easy – they will have more pressing priorities! Agreement will be facilitated by obtaining clarity on how the testing will benefit each of the stakeholders involved. Getting agreement from any approvals bodies involved as well as the client is particularly important.

Soon after privatisation in the mid 90s, South West Trains fitted sanding devices to its Class 159 fleet. Resources from nine different organisations where involved in the development, testing and fleet wide fitting on 22 units, yet the whole programme took only 10 months. Large numbers of stakeholders does not necessarily mean extended timescales, but stakeholders do need to be managed.

54 RSSB

4. Allocate roles to all stakeholders – a RASIC matrix is good practice which identifies those who need to be: responsible, accountable, supported, informed, and consulted on each activity. Remember interfaces present risks and they’re not just confined to trains and infrastructure; they also occur between people both within the same company and in different companies as well. Effective communications are essential for successful delivery. Something to bear in mind in particular for testing is whether or not tests need to be witnessed by the end client, approvals body or other independent party.

5. Listen and talk to the experts – see Section 10. They can assist throughout this complex process.

Engage senior management to provide ‘air support’ for the project. Often the decision to introduce an innovation or new approach is made at a relatively senior level. It is important for the project managers to engage with the senior leadership to make sure that innovative projects are explicitly given additional time or resources to introduce innovation on the first project or demonstration – otherwise the innovative approach will be expected to deliver against the expectations for a standard approach.

55RSSB

8.1.5. What is to be tested?

It almost goes without saying, but clearly, any test programme requires clarity on what is to be tested:

1. Define in an unambiguous way what equipment or new technology is to be tested; use part numbers if you can. This may sound straightforward but when the technology being tested is part of a larger system with a number of complex and different interfaces (human, electrical, mechanical) it may not be straightforward at all – let alone getting all the stakeholders to agree on it! Large projects use systems such as DOORS (Dynamic Object-Oriented Requirements System) for managing requirements.

2. Define the interfaces that the equipment or new technology has with the system and the environment in which it is to operate.

3. Maintain configuration control throughout the testing phase before transferring to an equivalent control in operational practice

In testing new technology and equipment the issue of interfaces and environment becomes more critical because:

• Sometimes the adjacent systems are not fully understood or fully characterised because it is an experimental system.

• Sometimes legacy systems do not provide the full range of interface functions, signals or data.

• The equipment and the environment may interact in ways that are not fully predictable.

56 RSSB

8.1.6. Methodology1. Define the test methodology. Remember standards

are good at specifying such things as controls, accuracy and error limits, but the correct test must be done in the first place. There is no substitute for engaging with the experts (see Section 10).

2. Consider how much testing is required to achieve the objectives, for example the number of scenarios that need to be tested (variation in inputs) and the consistency required from the results (variation in outputs). Also consider the order and sample size in which different tests will be done – is this the order that gives the most important and useful information for the costs incurred. These are key factors in planning and project costs. Consider:

a) The risk associated with failure of the new technology when in its operating environment. The level of testing should be proportionate to the risk, for instance, a wheelset will be subject to much more testing than an interior light.

b) How novel the technology is, the nature of the novelty, and thus the extent and nature of the testing needed.

c) Where the technology has been developed and manufactured.

d) The extent to which the technology has or has not already been tested outside Britain’s rail industry.

e) The confidence clients have in tests carried out elsewhere, considering issues such as how representative such tests might be, the standards

57RSSB

regime applied, and the familiarity of all involved with the UK’s operating environment and practices.

f) The extent to which Britain’s rail operating environment can be cost-effectively simulated (whether physically – on a test track for instance, or virtually – in a computer).

g) The number of tests for each set of input conditions to get statistically significant results.

3. Consider phasing tests with supplier/customer reviews between them to allow all those involved to reflect upon the intermediate findings and to decide the need for and best direction for completion of the test programme and any outstanding testing. In particular, for exploratory testing when the capability of a new technology or product is being evaluated such reviews can lead to a much more cost-effective outcome because lessons and issues are explicitly managed in context (See Section 8.4).

4. Try to strike the right balance between what could be foreseeably needed during the testing phase and what must get done. Also consider:

a) What else could be measured at the same time? It might not seem essential now, but it could prove to be later during analysis. For example, if testing a train’s performance in low adhesion after applying a suitable lubricant to the track, could it be worthwhile measuring adhesion between each train pass and not just at the beginning and the end?

58 RSSB

b) Combine tests with other tests only when you’re confident that they won’t affect each other.

5. Identify the risks associated with the test programme and how they are to be managed. This should concentrate not only on safety but also the environment and other business risks; for example, What might delay the testing programme? What happens if it doesn’t yield the required results? Remember also to consider that risks might not just manifest themselves in the equipment being tested, but also in the environment. For example, electromagnetic interference from a new train may cause station systems, such as lifts, to fail; or even affect colliery traffic below the railway.

6. Define timescales. Testing programmes do not always run to plan so contingency and flexibility is essential:

a) Have all reasonable eventualities been planned for and variables managed? When testing on service lines, for instance, be sure that the intended lines will deliver the required test conditions such as power supply and track quality. When ride testing, for example, how will consistent speed and track quality be delivered?

b) What should be done if the results fall outside expectations – how will the testing programme be modified? Such planning might be informed by prior computer simulation (see Section 5).

c) Build review gates into the plan to prompt decisions on whether to: proceed, do more simulation, or revise testing protocols.

59RSSB

d) If on-track testing is required, consider the amount of time it will take to get test vehicles to site, take control of infrastructure and hand it back; these significantly erode the available testing time. Don’t underestimate the amount of time it can take to get approvals to run trains that aren’t already in the timetable; it can take two years when other operators need to be involved to give up paths. If existing train operators are not involved in the test programme, it can take many months to set up a track access agreement.

7. Consider dependencies. The tests are likely to be dependent on completion of several activities such as supply of the equipment to be tested and provision of appropriate facilities or equipment. Consider also what activities are dependent on successful completion of the tests.

8. Define what evidence is required and needs to be captured throughout the remaining phases of the testing lifecycle. Agree the deliverables to be provided by testing, the pass/fail criteria, and the reporting requirements.

9. Define how data is to be captured, analysed and uncertainty established. There are standards for the instrumentation used to take measurements during testing and for software used to subsequently conduct the analysis. Speak to the experts to find out more (see Section 10).

10. Above all ensure that the test plan is cost-effective, proportionate to risk and experience, realistic, and deliverable.

60 RSSB

8.1.7. Facilities and equipment needed1. Define the facilities and test equipment that are

sufficiently representative of the intended operating environment to enable the objectives to be met (see Section 7). Taking test tracks for rolling stock as an example, considerations will include:

a) Speed of operation.

b) Compatibility of all interfaces – a substantive issue in many cases.

c) How representative are track quality, and other factors?

d) Does they need to be representative?

e) Safety of operations, risk management, and others.

f) Proximity to manufacturing facility.

g) How easily can you demonstrate to the customer and other third parties that the test track is sufficiently representative of the intended operating environment?

Similar principles apply for asset groups other than rolling stock.

2. Ensure the facilities have appropriate accreditation. ISO/IEC 17025:2005 ‘General requirements for the competence of testing and calibration laboratories’ is commonly cited as an appropriate standard, however it is aimed at laboratories and may not be appropriate for all testing facilities. It is the highest level of accreditation for testing and, consequently,

61RSSB

can be expensive to attain and retain. It might be more appropriate for facilities to adopt the following essential principles and be verified by an appropriate independent party such as a NOBO (Gaskill, 2007):

a) Test facilities are appropriate for the testing planned.

b) The quality management system complies with ISO9001:2008 Quality management systems - Requirements, including procedures for managing instrumentation and data processing.

c) Staff competency is effectively managed using an appropriate competency management system.

Note: ‘Competence’ is defined by the Office of Rail Regulation (2007) as ‘the ability to perform activities to the standards expected in employment; it is a combination of practical and thinking skills, experience and knowledge’. This has been clarified by the Institution of Railway Signal Engineers (2009) by reference to work by Baker & Durrant (2008):

d) The staff who undertake the testing are independent; have no vested interest in the outcome of the tests.

The level of assurance associated with the tests, and thus the level of confidence in them, will dictate how you can use the results.

62 RSSB

8.2. Testing

As has previously been stated, the most important part of testing is planning. If the planning has been thorough, the rest of the testing programme stands the best chance of successful delivery, even when contingency plans have to be engaged because of unexpected (but planned for) eventualities.

The testing phase should be all about delivering the plan set in Section 8.1.2, concentrating on:

1. Checking nothing has changed since the plan was produced, if necessary, revising the plan accordingly.

2. Ensuring the equipment to be tested arrives on time.

3. Ensuring that the equipment being used to conduct the test complies with the requirements set in the plan, and has the necessary approvals.

4. Ensuring everyone is clear on the plan, their roles and responsibilities.

5. Ensuring that the test conditions and environment comply with the plan and are adequately described in the test records.

6. Ensuring that everyone who is taking part in the testing has the competencies defined in the plan, including any necessary certification.

Again, in the case of ‘plug and play’ equipment there will be a need to test manufacturing facilities and processes to provide assurance about product consistency. Tests ‘to type’ are a specific domain not covered further here.

63RSSB

7. Being prepared to be flexible, as even the best plans cannot cater for everything – be ready to conduct reviews in the wake of the unexpected.

8. Ensuring data is being collected as planned. As one experienced test engineer put it, the biggest single risk of tests failing, is measuring equipment that performed perfectly well in a lab, not being able to cope with the harsh railway environment. This can be due to electrical problems as well as mechanical, particularly 50Hz interference.

9. Review the emerging results to ensure they are as expected, and that further tests can benefit from the experience obtained.

8.3. Reporting

The reporting phase is particularly critical as without structured, documented evidence, the tests will have been for nothing. In many cases, it will form a significant element of the submission made to third party approvers. In others cases, it will be the feedback into the product development process. During technology development the results of tests will be used to determine the direction and scope of further development routes.

Reporting should be in accordance with the plan set in Section 8.1.2 and should set out the following in an objective and auditable way:

1. What was tested – see Section 8.1.5.

2. The objectives of the test, referencing legislation or approval requirements – see Section 8.1.3.

64 RSSB

3. How it was tested (the test method), and how it addresses the objectives – see Section 8.1.6, and the key parameters and thresholds sought.

4. The facilities used – see Section 8.1.7.

5. The test conditions and environment at the time of the test.

6. Who performed the tests, citing evidence of relevant competency.

7. Pass/fail criteria.

8. The test results.

9. How the results compared with the criteria - the conclusions.

8.4. Reviewing how it went

When the first 3 phases of the test lifecycle have been completed, the final phase should review how the whole programme went, to identify learning points, inform future test programmes, and aid continuous improvement. Good practice is given in ISO9001:2008 ‘Quality management systems – Requirements’ but this should already be an integral part of all effective quality management systems.

65RSSB

Most of the principles that apply to testing will also apply to trialling; however there are some requirements that will be specific to trialling.

As explained in Section 3, although the terms can mean different things to different people, the distinction we have drawn in this guide between testing and trialling is that trialling takes place on the operational railway in revenue earning service.

The level of risk associated with trialling therefore increases by orders of magnitude, particularly safety risk, because passengers are involved, not to mention

9 Special requirements when trialling

Not all trials lead to new technology being adopted, as these remains of Brunel’s atmospheric railway remind us. Carriages were ‘sucked’ along by a partial vacuum acting on a piston that ran within the pipe. Leather strips that helped to seal the pipes proved ineffective leading to overworking of the pumps that created the vacuum.

66 RSSB

greater numbers of staff. Business risk also increases, in areas such as operational disruption and loss of reputation.

Trialling will not therefore take place with new equipment or technology until the confidence in the level of benefits to be returned outweighs the risk.

If we consider the development timeline of a new technology as it progresses through the various technology readiness levels (see Section 4.1), the progression from testing through to trialling is achieved by increasing confidence allowing restrictions in the test environment to be reduced until it is possible to move into the (more unrestrained) operational environment. Trialling does not involve novel equipment unless that novel equipment has already been subject to testing that has yielded sufficient confidence.

67RSSB

‘Note on test tracks – they are invaluable for getting past first base in developing the technical safety case for a new train, and for validating later changes to the design. Pre-delivery endurance running on test tracks is useful as a sophisticated build quality check, but a true indication of reliability only emerges from experience on real infrastructure, which tends to draw out many more issues.’

The Twenty Point Plan, Fleet Reliability Focus Group, CR/TP 1203 Issue 7, January 2011, published by ATOC.

You should consider these issues when embarking on trialling:

• Follow the principles outlined earlier in this guide; most of them apply equally to trialling as they do testing. However, bear in mind that because the risks are different, the controls may also need to be different.

• The big difference with trialling is that it will be done on the operational railway in revenue earning service. That means operation under the safety management system of the railway undertaking,

68 RSSB

whether it be a train operator or the infrastructure manager, and approval under ROGS (see Section 6.3). Early engagement with the railway undertaking and its commitment is essential; without it there is no point in proceeding.

• The scope of the railway undertaking or the test facility provider’s insurance may also be an issue, as trialling new technology may invalidate existing cover; it has certainly been a problem in the past following privatisation. Make sure that the appropriate asset and 3rd party liability cover are in place.

• Finally, note that trials are new and usually interesting. They can be a welcome distraction from delivering the daily routine service, whether it be operating trains or delivering paths on the infrastructure. Trials get a lot of a management attention to ensure they run smoothly and ensure the maximum chance of success. So if a trial proves to be successful, bear in mind that when it is rolled out and integrated into the operational environment there might not be quite the same level of management attention and it might not be quite so successful!

69RSSB

Key messages are: 1 Commitment of the train operator is essential for timely progress. Their core business is to move passengers reliably from A to B not to conduct trials per se, therefore any trial that doesn’t contribute to such aims needs to be incentivised appropriately. 2 Approval processes are there for a reason. The trial which took longest to mobilise has proven to be reliable in operational service;

design tweaks were required on the other following an initial period in-service. Robust independent review contributed to improving the design and ensuring it was well documented. 3 Ensure those involved have sufficient experience of trialling on the operational railway, and ensure that all parties are incentivised to co-operate with each other and make timely progress. This needs to be recognised at the outset and built into the plans because there will be an impact on resources (financial and man-power) as well as programme. Listen to the experts!

Finally, consider experience from two trials managed by the same consultancy but on trains operated by two different companies. Both trials involved adding monitoring and data recording equipment to DC rolling stock and operating the vehicles in service. One trial involved developing a complex bespoke distributed data acquisition system, and the other used more conventional PC-based acquisition with fewer channels. The former took twice as long to mobilise but not just because of more complex architecture.

70 RSSB

The advice given in this guide is only a starting point for those seeking to test new technology. If more information is needed, most organisations will offer free initial advice within reason. Contact details are shown below, along with other information mentioned in this guide that is available on the internet:

Source Contact detailsAdvice – CFMS consortium

More information on computer based simulation can be obtained from the CFMS consortium: http://www.cfms.org.uk/

Advice – Consultancies

Refer to the RIA website for a list of consultancies that may be able to assist: http://www.riagb.org.uk

Advice – Rail Research UK Association

RRUK-A similarly also has a number of university based experts experienced in testing: http://www.rruka.org.uk

Advice – RSSB RSSB has a number of industry experts experienced in testing most parts of the railway system and are particularly well placed to advise on European issues. RSSB has also prepared advice on the management of engineering change. Previously covered in the Yellow Book, this now covered by EC Common Safety Methods legislation. http://www.rssb.co.uk

Advice – The Railway Industry Association

RIA has a small technical section who may also be able to assist: http://www.riagb.org.uk

10 For more information

71RSSB

Source Contact detailsFacilities – Organisations with facilities available for testing and trialling

http://www.futurerailway.org/Pages/testingandtriallingfacilities.aspx

See also Section 7.

Legislation – Domestic

Further advice can be found on the government’s website: http://www.legislation.gov.uk

Legislation – European

Further advice can be found on the European Commission’s website: http://ec.europa.eu

Network Rail – Innovation & Suggestions Scheme

Outlines of innovations that may provide solutions to one of Network Rail’s challenges, can be submitted at http://www.networkrail.co.uk/aspx/12000.aspx

Network Rail – Product Acceptance

Information on Network Rail’s product acceptance process can be found at: http://www.networkrail.co.uk/aspx/3262.aspx

Standards – British and European Standards

Available to purchase from BSI: http://shop.bsigroup.com

Standards – European Standards Organisations

CEN (mechanical standards): http://www.cen.eu/cen/pages/default.aspx CENELEC (electrical standards): http://www.cenelec.eu/ ETSI (telecommunications standards): http://www.etsi.org/

Standards – International Standards

http://www.iso.org/

72 RSSB

Source Contact detailsStandards – London Underground Standards

Available by subscription from IHS on line at: http://www.lulstandards.co.uk/

Standards – Network Rail Company Standards

Available by subscription from IHS on line at: http://www.ihs.com/en/uk/products/industry-standards/collections/uk-network-rail/index.aspx

Standards – Railway Group Standards

Available free of charge from:

http://www.rgsonline.co.uk/default.aspx

Technical Specifications for Interoperability (TSI)

As with many documents of this type, the situation is continuously changing. RSSB provide a regularly updated status report on the situation with TSIs which includes direct links to the TSIs themselves: http://www.rssb.co.uk/Library/standards-and-the-rail-industry/TSI_status_summary.pdf

73RSSB

The authors would like to thank many experienced individuals from the following companies and acknowledge others who preferred to remain anonymous who have assisted with compiling this guide and without whom it would not have been possible:

• Alstom Transport

• Angel Trains

• Arthur D. Little

• Association of Train Operating Companies

• Bombardier Transportation UK

• DeltaRail Group Ltd

• Department for Transport

• Frazer-Nash Consultancy

• Freightliner

• Hitachi Rail Europe

• HR Wallingford

• Independent Glass

• Interfleet Technology

• Kilfrost

• London Undergound

• MIRA

• Network Rail

• Porterbrook

• Railway Industry Association

• Rail Safety and Standards Board

• Serco

• Siemens

• Technical Programme Delivery Limited

• Thales

• University of Birmingham

• Variable Message Signs Limited

• Virgin Trains

11 Acknowledgements

74 RSSB

Ref Element Covered Comments8.1 Common view of scale of

economic benefits of the innovation, product, or technology

8.1.1 Context and stakeholders1. Stakeholders and their role2. Stance of stakeholders

(up-side or down-side)3. Senior staff support and

resources (budget, time) for the project

4. Committed project champion in client organisation?

5. Existing context and evidence (existing test data, knowledge) shared between shareholder (and agreed)

6. Stakeholders aligned on what is needed beyond existing context and evidence

8.1.2 Planning1. Performance requirements,

risks, and uncertainties to be resolved

2. Tests agreed to meet requirements and to resolve risks and uncertainties

Annex A: Planning checklist

75RSSB

Ref Element Covered Comments3. Agreed scope to use test or

operational experience from elsewhere

4. Sequence and timing reviewed to give best benefits, earliest, for lowest costs

8.1.3 Objectives and success criteria

1. Why the test is being done and what the results will be used for?

2. The scope of the tests and the objectives

3. The requirements that need to be met to pass the tests, including relevant standards

4. Clarity about timeline for tests to be done and expectations about time for client/supplier to respond, provide feedback and process results

5. The interfaces between the equipment under test and adjacent systems

6. The inputs to the test7. The evidence required from

the testing

76 RSSB

Ref Element Covered Comments8.1.4 Roles and responsibilities1. Identify all stakeholders, their

interests and concerns2. Ensure a common

understanding of the test plan3. Obtain agreement4. Allocate roles to all

stakeholders5. Listen and talk to the experts

8.1.5 What is to be tested?

1. The equipment/new technology

2. The interfaces it has with the system in which it is to operate

3. Test configuration control system

8.1.6 Methodology

1. The test methodology

2. How much testing is required to achieve the objectives

3. The balance between what could foreseeably be needed and what must get done

4. The risks – not just safety risks but environmental and business risks

5. Timescales taking into account contingency and flexibility

77RSSB

Ref Element Covered Comments6. Dependencies – what could

the test programme affect and what could affect the test programme

7. The evidence required from the testing

8. How the data is to be captured, analysed and uncertainty established

9. Ensure the test plan is realistic/deliverable

8.1.7 Facilities and equipment required

1. Facilities/test equipment that are sufficiently representative of the intended operational environment

2. Ensure the facilities have appropriate accreditation

3. Ensure staff competency

78 RSSB

Annex B: Processes to support on-track testing and authorisation for placing into service of rolling stock

79RSSB

Annex B: Processes to support on-track testing and authorisation for placing into service of rolling stock

© Interfleet Technology Limited 2009

80 RSSB

Baker, John and Durrant, Paul, 200 ‘Developing and Maintaining Staff Competence, Comparisons with Rail Industry Experience’.

Gaskill, Stephen, 2007, The role of the Notified Body in the European rail industry, available from http://www.europeanrailwayreview.com/1525/err-magazine/past-issues/the-role-of-the-notified-body-in-the-european-rail-industry/ Viewed 27/02/2014.

Institution of Railway Signal Engineers, 2009 Competence Guidance for Train-Borne Train Control Systems.

Office of Rail Regulation, 2007 Developing and Maintaining Staff Competence .

References

81RSSB

Notes

82 RSSB

Notes

83RSSB

Notes

RSSB Block 2 Angel Square, 1 Torrens Street, London EC1V 1NY.www.rssb.co.uk

Tel: 020 3142 5300Fax: 020 3142 5663 Email: enquirydesk@rssb.co.uk