MDI/WP17 Internal Report 1

The OTR Vessel Issues or

The Proof of Murphy’s Law1

Collected by

M. Holz, J. Kruse, D. Nölle, D. Renner, S. Vilcins, Ch. Wiebers and K. Wittenburg

May 9, 2016

1 If anything can go wrong, it will. O’Toole’s Comment on Murphy’s Law: Murphy was an Optimist

OTR CHAMBER ISSUE

[WP17/MDI] PAGE 2

OTR Vessel Issue

Abstract 49 vacuum vessels for screen monitors for the European XFEL (E-XFEL) were produced and delivered with insufficient quality and had to be replaced by newly produced vessels. This summary describes the situations, the observations, the ideas to counteract and the final solution of the issue. Some “Lessons Learned” are discussed to avoid similar problems for the future.

Introduction As an essential part of the XFEL accelerator diagnostic system about 60 screen stations will be installed at different locations along the machine to image beam transverse profiles. They are intended to reliably measure the beam profiles along the XFEL with sufficient accuracy and resolution for lattice matching and machine optimization. Some stations include also a complementary system of the wire scanner monitors for the long bunch trains with high intensity. To avoid coherent OTR radiation the stations are equipped with scintillation screens and their readout is performed by high resolution Scheimpflug optics [1]. The vacuum vessel2 of the screen stations are made of a massive monolithic stainless steel block, with integrated CF Flanges milled into the steel block. All connections with movable parts (screen mover and Wire Scanner) are flanged to this vessel. A vacuum approved fused silica optical window flange system is used to transport the screen light out of the vacuum onto a camera. The windows itself has to be of imaging optical quality, i.e. flatness better the λ/2 of the main wavelength λ. Two types of vacuum vessels were required, one long version which includes the flanges for the two orthogonal wire scanners (OTRB, see Fig. 1) and one short version without these flanges (OTRAC, Fig. 2). 29 of Type OTRB and 20 of type OTRAC were required and produced for the XFEL project. Five OTRAC vessels were previously manufactured for the FLASH accelerator by the company VACOM [2] by using their steel without any significant problems. These vessels are still installed in FLASH and are in regular use.

Technical situation The stainless steel needed for the vessel production was ordered by DESY after a public open tender process. The delivered stainless steel of type 1.4435 ESU (AISI/SAE 316L, UNS S31603) with a certified hardness of 157 HB (Hardness Brinell, see Fig. 3) was delivered by Fa. Hödtke [3]. The material was produced by Stahlwerk Breitenbach [4]. Although the DESY/MDI specifications [5, 6] required a hardness of 160-200 HB the material was accepted due to a lack of other suppliers. The material was cut into required pieces by the supplier.

The assignment to produce of the vacuum vessel of type OTRAC was given to the company FMB [7] after a public call for tender. The material for the production was provided by DESY (see above). The production was done by a subcontractor while FMB did the low particle cleaning and the assembly of the final vessel with optical windows, the adapter for the vacuum part of the screens (“movers”) and an extension on the opposite flange. The windows (CF63) were ordered from Vaquec-Scientific ([8], Fig. 4), a pure distributor. The manufacturer of the windows was not published by Vaquec-Scientific. A few windows of the same type are in use up to now in the FLASH2 beamline without any problems. These parts were provided by DESY to the companies. All vessels were delivered after assembly to DESY with vacuum leakage test (Figure 5). All extensions on the vessel were mounted with typical CF copper gaskets [9]. There was no information how the remaining two open flanges (for the beam pipe) were sealed during the leakage test. Two of the chambers were completely sealed the CF copper gaskets since they underwent successful particle concentration tests and Residual Gas Analysis (see Figures 6 and

2 For historical reasons the common name for the vacuum vessel is still “OTR Chamber”

OTR CHAMBER ISSUE

[WP17/MDI] PAGE 3

7). Due to these successful tests no extra check of the vessels and their CF flanges was done. One has to note that the steel was low hydrogen annealed during the production of the vessels.

The assignment to produce of the vacuum vessel of type OTRB was given to the company Hansa Press- und Maschinenbau GmbH [10] after a second public call for tender. The material for the production was provided by DESY (see above). Since MDI had no experience with Hansa Press in producing UHV components, we ordered first a prototype flange and a prototype of one vessel. Both prototypes were checked, UHV tested and accepted. Therefore the production of all vessels was accepted as well. One vessel from the beginning production line was UHV tested at DESY without any problems. The cleaning and mounting of all parts (windows, adapters, extensions, blind-flanges for wire scanner) was assigned to FMB. These parts were provided by DESY. The windows (CF63) were ordered from Vaqtec-Scientific, a pure distributor. The manufacturer of the windows was not published by Vaquec-Scientific. All vessels were delivered to DESY with vacuum leakage test, while 10% of the vessels received particle concentration test and certificated Residual Gas Analysis. All extensions on the vessel were mounted with typical CF copper gaskets. Note that all flanges were sealed with rubber O-rings during the leakage test. 10% of the chambers were completely sealed the CF copper gaskets since they underwent successful particle concentration tests and Residual Gas Analysis. Due to these successful tests no extra check of the vessels and their CF flanges was done. Note that the steel was NOT low hydrogen annealed during the production of the vessels.

Both types were finally produced in 2013 (OTRA Feb. 2013, OTRB July 2013). At that time no test equipment and limited manpower was available at DESY-MDI. Therefore all vessels remained stored, sealed and wrapped in plastic foil to avoid particle contamination of the vacuum part. Both companies agreed to start the guaranty period later, after the final acceptance tests at DESY.

The Problems In March 2014 MDI started to prepare five OTR stations for the XFEL injector. All vessels had to be delivered to the DESY vacuum group MVS with a residual gas analyses certificate (RGA). Meanwhile this technical equipment to provide such a test was available at MDI.

The first problem appeared after a final assembly of a complete OTRAC vessel (including the mover and the screen) since the RGA contained unacceptable gas contamination. The reason was found in a part of the mover in which some residual cleaning liquid was not properly removable. By heating this part (and later all these parts) in a vacuum oven the problem disappears at following RGAs. Note that the mover was mounted on the adapter while this adapter remained mounted on the vessel and was not removed from it at this time.

The second problem: In April 2014 we discovered during tests after final assemblies of all parts that some of the OTRAC vessels couldn’t become vacuum-tight. The location of the leakage was detected at the optical window. A second observation was, that there was some white crystals visible on the vacuum window (see Fig. 8a and 8b). These crystals were visible at almost all stored OTR vessels of both types on the CF63 windows. The same type of window (but size CF100) from the same supplier was installed on a third type of OTR vessel (OTRD, produced, assembled and stored at DESY), which did not show these problems. After discussions with FMB it turned out, that FMB cleaned the window in an ultrasonic bath with fluids. Note that the DESY cleaning specification [11] (delivered to FMB) does not specifically exclude “wet cleaning” of vacuum windows while it recommends “wet cleaning” in an ultrasonic bath with pure water and a suitable cleaning fluid of all vacuum parts to achieve particle free conditions. A chemical analysis of the white crystals showed the same lead components as the soldering material (see Figure 9a and 9b).Probably the crystals were created during a chemical reaction between the cleaning fluid and the sealing (soldering) of the window. This process required some time to become visible. This sealing (soldering) contains lead, which was not documented by the supplier. As a result, complete new sets of windows (CF63, CF100) had to be ordered from two other companies (Hositrat [12] and VACOM). The ordering was done in a few steps to get as fast as possible some

OTR CHAMBER ISSUE

[WP17/MDI] PAGE 4

urgent required windows. Also 15 new CF100 optical windows were ordered due to the unacceptable lead sealing of these windows. It was planned to exchange these windows on the OTRD vessels at a later time. None of the new windows got a “wet cleaning” but all were carefully cleaned with Isopropanol and by pressed nitrogen blow until the remaining particle detection was well below the allowed value.

The third problem: In April 2014 it was clear that all vessels have to be re-cleaned since the lead crystals contaminate their vacuum part. Therefore all the vessels were disassembled. Immediately after the disassembly of the first vessel we observed that the “used” knife edges of the CF flange of the vessels were deformed (see examples in Figures 10). Our first guess was that the process of low hydrogen annealing softened the material, later we found the same problem on the OTRB vessels which were not low hydrogen annealed. To overcome this problem, we succeeded to seal 3 OTRAC vessels with Taper Seal gaskets (see Fig. 11) and with existing CF63 windows from our stock. However, these tests were done with un-cleaned vessels so that these cannot be used for final installation. Due to that positive test, we ordered Taper Seal Gaskets; unfortunately there were not enough on the market to satisfy the DESY request and the delivery date was too long for our first needs of OTRAC vessels. Therefore a new order to manufacture five urgent required OTRAC vessels was accepted by the XFEL project team. After discussion with the DESY vacuum group it was accepted to seal all remaining vessels with Taper Seal Gaskets. At this time no Taper Seal Gaskets were available at DESY to perform more tests. To improve the situation of the deformed knife edge all OTRAC vessels were send to the manufacturer to repair the knife edges (Mai 2014). Meanwhile we checked the OTRB vessels as well and similar knife-edge deformations were detected after disassembly of the mounted parts. At that moment our theory about the softening of the material due to low hydrogen annealing became wrong and the general hardness of the raw-material became questioned. First measurements at DESY showed a hardness far below the certificated 157 HB. The qualified company Germanischer Lloyd [13] made 50 measurements on various vessels of both types and on pieces of residual raw material. Values between 129 HB to 141 HB were measured without any correlation between the two different types and production lines (see Fig. 12, 13a and 13b). After discussion with the DESY vacuum group it became clear that such a week material cannot be accepted for the vacuum requirements of the XFEL (July 2014). This was confirmed by various unsuccessful sealing tests with normal and tempered (annealed) Taper Seal Gaskets on an OTRAC vessel end of July 2014 and a little time later on a few other repaired vessels. Another test with meanwhile ordered CF type Aluminum gaskets were successful, but these gaskets were not accepted for the XFEL vacuum system. To satisfy the urgent request of the XFEL project also 8 OTRB vessels were ordered at the company VACOM which promised to have the requested material (1.4429 ESU) with sufficient hardness above 160 HB [14] on stock. This company had proven their reliability by manufacturing successfully the first 5 OTRAC vessels for FLASH (see Introduction). It was decided at that moment to keep the present design of the vessels since these 5 OTRAC are still successfully working at FLASH providing very good measurement results. In parallel two ideas were tested to harden the existing vessels: 1. One OTRB vessel was heated in an oven and cooled fast by nitrogen [15]. This should increase the nitrogen content in the steel und should provide more hardness. This test failed, the material became even softer. 2. Another idea was tested on two refurbished vessels; the tested process is named Kolsterising which is a diffusion of carbon into the surface which should result in a hardening of the surface. The Kolsterising was done by BODYCOTE [16]. Unfortunately the tests with these two vessels were also not successful. Probably the thickness of the hardened surface was too thin and not strong enough to survive the copper gasket pressure on the soft grounding. Since all attempts to seal the produced vessels were not successful or not accepted by the vacuum requirements, 15 OTRAC and 21 OTRB vessels have to be manufactured again. The order was placed in Sept. 2014 at the company VACOM who had enough of the prescript material available and had been shown to deliver sufficient quality vessels. A detailed inspection process was defined and performed between DESY-MDI and VACOM which helped receiving the vessels in useful quality. The low particle cleaning of the vessels was done by VACOM as well. None of the new vessels produced by VACOM failed at the required vacuum- and RGA-tests. These tests and the final assemblies were

OTR CHAMBER ISSUE

[WP17/MDI] PAGE 5

done almost immediately after delivery of each produced and cleaned vessel at DESY-MDI during 2015. The fourth problem: As pointed out in the “second problem”, there were optical windows from the supplier Vaqtec-Scientific with CF100 flanges mounted on a third type of vessel, on the OTRD chambers. All ten vessels were tested to be vacuum tight and clean. Although these windows did not show the “second problem” (no wet cleaning) we liked to exchange these ten windows due to their lead sealing. Therefore new windows of this size were ordered at the company VACOM. The OTRD vessels with the mounted old window were stored for about 1.5 years until beginning of 2016 when the planed exchange of the window should start. Now another problem rose: Without any further action all 10 windows showed cracks in the glass near the metallic flange, most close to the screws (Fig. 14). Up to now, it is not clear yet if the windows are still vacuum tight. An inspection of the mounted windows showed three results: 1) the flanges were tighten flange-to-flange with zero gap, 2) stainless steel screws and nuts with “Molykote” grease were used to mount the flanges and 3) normal copper sealing were used to seal the flanges. This kind of mounting might put too much force on the windows. However, the crack did not appear immediately during the mounting procedure but during storage in the hall where the vacuum group stores their vacuum equipment. We try to understand some existing Mounting Instructions of another Vacuum-window supplier (Fig. 15) in more detail and ask this supplier: 1) "Use a torque wrench to ensure equal sealing torque. Do not over tighten." What do you

exactly mean with over tighten? I could not find any hint for the specified torque. The answer: “The recommended use of a torque wrench is intended to allow the user to maintain an even application of torque to each bolt rather than any specific torque value. The uncontrolled variables affecting the required torque in this situation are bolt/nut material, thread grade, use of washers and whether any dry lubricant has been applied. With regard to the “Do not over tighten” perhaps this should be read as “only tighten enough to attain a leak tight, vacuum seal” “. 2) “Always use annealed copper gaskets and washers on both sides with each nut and bolt.” The answer: “The use of annealed gaskets has been accepted practice within the vacuum industry for many years. The main reason as I understand it is to reduce the bolting force required to attain a leak tight vacuum seal. The reduced bolting force required has the effect of reducing the risk of damage to the viewport due to distortion of the flange and consequently compromising the glass to metal seal. I am in agreement with the statement that it is much more important to equally tighten the screws / bolts, further to this they should be tightened in a pattern which distributes the bolting forces equally across the viewport. The pattern should be repeated with gradually increasing torque until a leak tight vacuum seal is achieved. The caveat to this statement is that if the flange faces are seen to be metal to metal and a vacuum seal have not been achieved the assembly should be disassembled and the knife edges inspected for damage.” All windows are now exchanged and care was taken not to over tighten the flanges.

Lessons Learned From the first problem: The best way of avoiding vacuum contamination by residual fluids is a design which does not allow fluids to remain in some hidden places like blind holes or so. If this cannot be avoided the parts have to be baked out in a vacuum oven before testing on an UHV pump. From the second problem: Not many companies wrote down the chemical compounds of solder used for their window sealing. Under normal condition the windows may behave well in vacuum environment. For particle low environment additional cleaning is required and with some sealing types a cleaning in an ultrasonic bath and/or special cleaning fluids might result in chemical reactions with the soldering. Therefore one should test properly the cleaning procedure applied to the window and should expect long term reactions as well. If possible one should avoid “wet cleaning” of vacuum windows. A careful wiping with pure alcohol or Isopropanol together with pressed air (oil-free!) or nitrogen blow can be sufficient to reach particle low conditions.

OTR CHAMBER ISSUE

[WP17/MDI] PAGE 6

From the third problem: A) The hardness of stainless steel used for UHV CF-Flanges must not be below 160 HB (to be on the save side better not below 170 HB). However, the hardness calculated from the tensile strength might be different from the hardness measured by the Brinell hardness test definition3 since it depends strongly on the stainless steel type and processing. Therefore a defined and unified test procedure to measure the hardness in HB should be applied to the steel before it can be accepted for UHV CF-Flange production, especially if the documented hardness is calculated by the tensile strength. A test procedure description will be prepared in the near future. B) Most tests with various gaskets to seal a CF-flange made out of not hard enough stainless steel failed. It might be possible to seal it properly the first time with typical copper gaskets but after that the knife edge is deformed and unusable for further sealing. Very soft gaskets made from Aluminum avoided the deformation, so that some successive sealing tests were done successfully. Even normal Taper Seal Gaskets could not fix the problem; the reason still has to be examined in more detail. Annealed Taper Seal Gaskets were not able to seal even a normal and un-deformed CF-flange and are therefore useless as UHV gasket. C) Low Hydrogen Annealing in our case did not have a significant influence on the hardness of (at least) this type of steel (1.4435 ESU) since there was no correlation in our hardness measurements between steel with and without this annealing process. D) Hardening of the existing vessels: Two ideas failed; one vessel was heated in an oven and cooled fast by nitrogen and the Kolsterising process. Probably the thickness of the hardened surface was too thin and not strong enough to survive the copper sealing pressure on the soft grounding. However, a third test was done together with the company Dreistegen [17] which hardened the flanges of an OTRAC vessel by “cold nitrifying”. After that process three successive leak tests with normal copper gaskets were done successfully on that vessel. More tests are foreseen in the near future. From the forth problem: It was observed that the windows of the CF100 flange cracks after some time during storage and not immediately during the mounting process. There is no clear definition available from the supplier for the tightening strength, so that the two lessons might be 1) only tighten enough to attain a leak tight, vacuum seal and 2) use a torque wrench to maintain an even application of torque to each bolt. Both is true for using normal and annealed copper gaskets although the weaker annealed gaskets will lower the risk of over tightening and annealed gaskets are recommended by the window supplier. The use of Molykote graphite grease on the screws of the flange is not recommended since it gains the risk to over tighten the flange and the resulting deformation might crack the window.

3 In former standards HB or HBS were used to refer to measurements made with steel indenters. The unit HBW has a more precise definition (H from hardness, B from Brinell and W from the material of the indenter, tungsten (wolfram) carbide). The Brinell test implies pressing a 1, 2.5, 5 or 10 mm tungsten carbide ball into a sample material and measure the diameter of the impression left. The loads used vary from 1 kgf to 3000 kgf. Applicable Standards: ASTM E10 / ISO 6506 / JIS Z 2243

OTR CHAMBER ISSUE

[WP17/MDI] PAGE 7

References [1] Scintillating Screen Monitors for Transverse Electron Beam Profile Diagnostics at the European XFEL Ch. Wiebers, M. Holz, G. Kube, D. Nölle, G. Priebe, H.-C. Schröder DESY, Hamburg, Germany Proc. 2nd International Beam Instrumentation Conference, IBIC2013 16-19 September 2013, Oxford, UK [2] VACOM Vakuum Komponenten & Messtechnik GmbH, Gabelstraße 9, D-07749 Jena [3] Hödtke GmbH & Co. KG, Industriestr. 2-6, D-25421 Pinneberg [4] Schmiedetechnik Breitenfeld GmbH, Breitenfeldstrasse 22, A-8662 St. Barbara-Mitterdorf [5] Technische Spezifikation Nr.: Vakuum 003/2008 Version 1.3 / 17.03.2009 Geschmiedete Ronden aus Material 1.4435 / ESU (316L / ESR) (Vakuumanwendungen) FS-BT und MVS EDMS Nr.: D00000001385231 Rev: B Ver: 1 Status: Released Dat.: 25.06.2009 [6] Internal MDI note, not published: Spezifikation zum Material, 21.02.2012 [7] FMB Feinwerk- und Messtechnik GmbH, Friedrich-Woehler-Str. 2, D-12489 Berlin [8] Vaqtec-Scientific, Thulestrasse 18b, D-13189 Berlin [9] Technische Spezifikation Nr.: Vakuum 004/2008 Version 1.2 / 07.11.2008 Flachdichtungen aus Kupfer für ConFlat®-Flanschverbindungen (Vakuumanwendungen) FS-BT und MVS Technische Material Spezifikation EDMS Nr.: D00000001385281 Rev: B Ver: 1 Status: Released - Dat.: 3. Jul 2009 [10] Hansa Press- und Maschinenbau GmbH, Jacobenweg 18, D-22525 Hamburg [11] Technische Spezifikation Nr.: Vakuum 005/2008 Version 1.6 / 22.09.2010 Richtlinien für UHV-Komponenten bei DESY U. Hahn (FS-BT), K. Zapfe (MVS) EDMS Nr.: D00000001425511 Rev: H Ver: 2 Status: Released Dat.: 22.09.2010 [12] Hositrat Lindnergasse 2, D-93047 Regensburg [13] Germanische Lloyd Prüflabor GnbH, Tempowerkring 11, D-21079 Hamburg [14] Technische Spezifikation Nr.: Vakuum 002/2008 Version 1.2 / 07.11.2008 Geschmiedete Ronden aus Material 1.4429 / ESU (316LN / ESR) (Vakuumanwendungen) FS-BT und MVS

OTR CHAMBER ISSUE

[WP17/MDI] PAGE 8

Technische Material Spezifikation EDMS Nr.: D00000001385161 Rev: B Ver: 1 Status: Released - Dat.: 25. Jun 2009 [15] HÄRTEREI REESE Chemnitz GmbH & Co. KG, Otto-Schmerbach-Straße 19, 09117 Chemnitz [16] Bodycote Hardiff BV, Paramariboweg 45, 7333 PA Apeldoorn, Niederlande [17] Dreistegen GmbH, Dreistegen 7-9, D-52156 Monschau

OTR CHAMBER ISSUE

[WP17/MDI] PAGE 9

Figures

Fig. 1: OTRB vessel during vacuum test. The vacuum pump adapter is connected to the beam pipe flange. One of the two flanges for the wire scanners is on the top, the other one on the opposite of the viewing port (CF50). The optical window (not visible) is mounted at the tilted surface below the pumping adapter. The screen mover is connected on the opposite while the extension is visible on the front.

Fig. 2: OTRAC vessel. The optical window on the CF63 flange and the mounted adapter for the mover are visible. The beam pipe connection and the adapter are closed temporarily by a simple flange.

OTR CHAMBER ISSUE

[WP17/MDI] PAGE 10

Fig. 3: The 3.1 material test certificate of the ordered stainless steel type 1.4435 ESU. This certificate was requested by DESY.

OTR CHAMBER ISSUE

[WP17/MDI] PAGE 11

Figure 4: Tender from Vaqtec for CF63 Windows

OTR CHAMBER ISSUE

[WP17/MDI] PAGE 12

Figure 5: Vacuum test chart of an OTRAC vessel after first delivery

OTR CHAMBER ISSUE

[WP17/MDI] PAGE 13

Figure 6: Particle Concentration chart of an OTRB vessel after first delivery

OTR CHAMBER ISSUE

[WP17/MDI] PAGE 14

Figure 7: Residual Gas Analysis chart of an OTRAC vessel after first delivery

OTR CHAMBER ISSUE

[WP17/MDI] PAGE 15

Figure 8a: White crystals on the window

Figure 8b: Zoom to the window edge

OTR CHAMBER ISSUE

[WP17/MDI] PAGE 16

Figure 9a: EDX analysis of the white crystals on the window

Figure 9b: EDX analysis of the soldering material

OTR CHAMBER ISSUE

[WP17/MDI] PAGE 17

Figure 10: Examples of the damaged knife edges of the flanges after disassembly of the window, adapter or extension.

OTR CHAMBER ISSUE

[WP17/MDI] PAGE 18

Figure 11: Taper Seal Gaskets; from http://www.sev-vacuum.com/sub02_page01_e.html

OTR CHAMBER ISSUE

[WP17/MDI] PAGE 19

Figure 12: Defined locations (red circle) on the OTR vessels for the hardness tests of Germanische Lloyd

OTR CHAMBER ISSUE

[WP17/MDI] PAGE 20

Fig. 13a: Three out of 50 different hardness measurements by Germanischer Lloyd showing hardness between 129 and 139 HB.

OTR CHAMBER ISSUE

[WP17/MDI] PAGE 21

Fig. 13 b,c: Distribution of the hardness measurements results by Germanischer Lloyd showing a mean value of far below 160 HB on both vessels (OTRA/C (upper) and OTRB (lower)).

OTR CHAMBER ISSUE

[WP17/MDI] PAGE 22

Fig. 14: Cracks on the window. Some cracks are visible near the flange on the vacuum side of the window. The gauge of 0.03 mm did not fit between the two flanges showing that the flanges were tightened without any gap between the flanges.

Fig. 15: Mounting procedure instruction from the window supplier Hositrad