R2E Mitigation Project Status & Outlook August 22 nd 2011 (R2E) Mitigation Project:CMAC Meeting, August 22 nd 2011 Radiation To Electronics Status & Outlook Radiation To Electronics Status & Outlook M. Brugger on behalf of the R2E Project !!! Many Thanks To Everybody !!! Snapshot R2E Mitigation Project Status & Outlook August 22 nd 2011 Overview Radiation Levels & Evolution Observed Equipment Failures 2011/2012 & Beyond TS, XmasBreak & Patch-Solutions Mitigation Plan & Strategy 2 R2E Mitigation Project Status & Outlook August 22 nd 2011 Reminder on The Challenge 3 LHC Tunnel Shielded Areas Critical Areas Safe Areas Tested & Designed COTS & Untested Electronics Electronics Electronics Electronics Surprises will happen (and happened) R2E Mitigation Project Status & Outlook August 22 nd 2011 Measured & Expected Rad-Levels 4 Weekly ReportDetailed Analysis Link1 Link1 & Link2Link2 Comparison & Extrapolation x50 R2E Mitigation Project Status & Outlook August 22 nd Operation up to Week 30 (2010 excluded): -> ~1,8 fb -1 (nominal: x30 for lumi scaling) Scaling might be non-linear for areas being dominated by direct losses (& distributions) and/or vacuum contributions! Luminosity Dominant Measured Radiation Levels R2E Mitigation Project Status & Outlook August 22 nd 2011 Radiation Levels & Luminosity 6 R2E Mitigation Project Status & Outlook August 22 nd 2011 Radiation Levels: Follow-Ups! UX45: Increase of radiation levels in UX45 12 & 22 RadMon counts so far REs: Increase of radiation levels 21 39 RadMon counts so far ARCs: Increase of radiation levels very low radiation so far ( more likely to have SEEs ending the fill since some other failure modes depend rather on time? In terms of behavior with time the failures reflect the cumulative luminosity (see slide before) R2E Mitigation Project Status & Outlook August 22 nd 2011 Performed R2E Mitigation Actions Shielding: P6 (RA63/UA63 and RA67/UA67) (gain ~factor 5-10) UJ22/23/76/88/87 (gain ~factor 10) RR77/73 (gain ~factor 10) US85 Safe-Room (gain ~factor 10) Relocations: Fire-Control Racks UJ56/76, US85 (safe) RTU relocated from safe room in UJ56/76 (safe) Cryo-relocations/valve replacement in UX85 (safe) UPS from UJ76 (safe) Fire-Detectors: US85, other points prepared (safe) PLCs from US85 (safe) Replacements & Upgrades: QPS Firmware Upgrade (ISO150 failures) (transparent) US85 24V Power Supply -> replaced by old model (more robust) Cryogenics communication problem: automatic reset (transparent) 17 R2E Mitigation Project Status & Outlook August 22 nd 2011 Mitigation Actions: in the pipeline Shielding: UJ14/15 (gain ~factor 5-10) -> xMasBreak Relocations: PIC/BIC/WIC relocations (UJ14/16/56/76) (safe) PLCs from US45 [if required] (safe) US85 relocation of cryogenics equipment [under study] and 24VDC distribution (safe) P1/5: Fire Detectors (safe) Replacements & Upgrades: QPS ISO150 Patch will be fully applied until xMas (transparent) Cryo current-lead temperature: hardware modification (safe) Cryogenics ET200 S->M replacement (transparent) Collimation control: software update possible (robuster) PLCs: redundant system studied (to be tested) FIP -> for some equipment: bricoFIP option (robust) 18 R2E Mitigation Project Status & Outlook August 22 nd 2011 Failures & Correlations 19 With previous mitigation actions (shielding, relocations), only SEE failures expected for the remaining 2011 run xMasbreak actions aim to reduce failure rate further for 2012 run Most-Critical Equipment & Patches/Relocations Cryogenics (9 events), patch applied and working nicely PIC/BIC/WIC (5 events), relocation foreseen for next TS + other patches R2E Mitigation Project Status & Outlook August 22 nd 2011 Critical Areas & Equipment 20 Equipment UJ76 Fire/ODH control & detectors Cryogenics Instrumentation (Profibus) AUG controlVacuum UPSBeam Television Monitor Electrical Distribution & ControlRamses Remote-Reset & TimingGSM Diagnostics Access System ControlBeam Loss Monitors Power ConverterBeam Position Monitors (temporary) EthernetOptical Fiber Cooling and VentilationWorldFip Less radiation critical equipment Fire/ODH detectos to be relocated Current Lead Heaters ok for the levels QPS known patch solution Equipment RR73/77 Fire/ODH detectorsBeam Position Monitors Power ConverterCryogenics Instrumentation (FIP) QPSWorldFip Cooling and VentilationCurrent Lead Heaters Power InterlockElectrical Distribution Power Converters Big Unknown Various critical equipments Several tested at CNGS and shown high radiation sensitivity Few cases allow for patch solutions Global Solution Required Equipment Inventory (EDMS # )EDMS # R2E Mitigation Project Status & Outlook August 22 nd 2011 Power-Converters & Tolerance Parts have been tested at CNRAD Expected sensitivity ~1x10 -8 (= one failure every ~1x10 8 cm 2 per converter) LHC Observations so far six failures confirmed as SEE (most sensitive module: AC/DC part) + few cases of 2 nd failure signature New Test-Area H4IRRAD: LHC-like conditions (spectra) About 1 year of nominal LHC within one week of test Final analysis ongoing! What will we learn: confirm LHC observations verify weak links and investigate patch-solutions (important for 2012 and 2014/15/16 operation) optimize global mitigation strategy 21 SPS Beam Jura SideNorth Area: Saleve Side Power-Converters are the Key Point for the final R2E Mitigation Strategy R2E Mitigation Project Status & Outlook August 22 nd 2011 What it Means in Reality 22 TEST THIS: HERE BEFORE AND AFTER (W)HOW ??? R2E Mitigation Project Status & Outlook August 22 nd 2011 Power-Converter R&D TE-EPC Planning for Rad Tolerant Converter Project Y. Thurel 23 R2E Mitigation Project Status & Outlook August 22 nd 2011 Superconducting Links Horizontal Links (RR73/77) 600A converters (120A not possible) TZ76 preparation foreseen consistently Vertical Links (P1/P5) Common interest with LHC upgrade, allowing the PCs to be brought on the surface 24 A. Ballarino et al. R2E Mitigation Project Status & Outlook August 22 nd 2011 Roadmap for 2011 Detailed radiation field analysis (Monitoring, TLDs, MD) Additional radiation tests (H4IRRAD, CNRAD) Provision/preparation of mitigation actions xMasBreak (shielding P1, some relocations, preparations) LS1 (shielding P1/5/8, relocations: P1/5/7/8) RadTol Power Converters and/or Super-Conducting Links TS & xMasBreak improvements Additional patch-solutions Evaluation of additional weak points (P4, REs) R2E Review in November 25 R2E Mitigation Project Status & Outlook August 22 nd 2011 Current Situation 2011: failures will continue to occur but not limit luminosity reach (patches -> 'shadow' of other problems) Continued testing/monitoring/analysis will allow to optimize R2E mitigation measures R2E Review in November 2012: xMasBreak actions will lead to further improvement impact of possible 25ns operation? (residual gas densities in the ARC & P4!) 26 R2E Mitigation Project Status & Outlook August 22 nd 2011 Mitigation Options 27 SHIELDINGRELOCATION CIVIL ENGINEERING RAD-TOL DESIGN No Major CE Improve & Gain TimeSolve & Gain Time Solve & Remain Flexible What Can Be Done R2E Mitigation Project Status & Outlook August 22 nd 2011 CNGS Incident -> Its Radiation (SEEs) !!! Task-Force to look at LHC Issues (ups) Solution for CNGS (Study, Implementation) Radiation To Electronics (R2E) Study Group Short-Term Actions (Safety 1 st Priority) Full Analysis of LHC Areas (Calculations,) R2E School and Getting Mobilized Medium-Term Actions (Focus on what can be done quickly) Power-Converter Review R2E-Workshop & Strategy First Long-Term Draft and List of Options R2E Mitigation Project xMasBreak Actions (2010/2011) Operation Study/Preparation of Actions Early relocations, equipment exchanges (TS) Time-Scale So Far R2E Mitigation Project Status & Outlook August 22 nd 2011 Mitigation Strategy 29 1 st Safety Critical 2 nd Shielding 3 rd Most Sensitive 4 th Remaining Immediate Relocation Fast & Global Improvement Highest Impact on Operation: (1)Relocation (2)Shielding (1)Relocation (2)Shielding (3)New Design R2E Mitigation Project Status & Outlook August 22 nd 2011 R2E Mitigation Project Status & Outlook R2E Global Strategy 30 Study Phase ( ) Calculations, Early Actions, Strategy Study Phase ( ) Calculations, Early Actions, Strategy Project Proposal (2010) Testing, Evaluation, Mitigation Plan Project Proposal (2010) Testing, Evaluation, Mitigation Plan R2E Mitigation Project ( ) Medium Term R&D For Long- Term EvaluationEvaluationLong-TermLong-Term Shielding & Relocation P1, P5, P7, P8 (+ small items) Collimation Betatron in IR3 Shielding & Relocation P1, P5, P7, P8 (+ small items) Collimation Betatron in IR3 PanicPanicEvaluationEvaluation StrugglingStruggling ResourcesResources Power- Converters Rad-Tol Solution SCLs Horizontal/ Vertical Links Power- Converters Rad-Tol Solution SCLs Horizontal/ Vertical Links Monitoring Analysis, Calibration Testing Component & System Tests Monitoring Analysis, Calibration Testing Component & System Tests CombinationRelocation Rad-Tol Equipment SCLs (Civil Engineering) CombinationRelocation Rad-Tol Equipment SCLs (Civil Engineering) 30 On hold Short-TermShort-Term Early Actions + Patches1, P1/P5/P7/P8 R2E Mitigation Project Status & Outlook August 22 nd 2011 R2E Next Steps 2011 Operation shows us: most critical equipment (mitigation measures integrated on-the-fly if possible) Evolution of radiation levels compared with expectations (possible additional weak-points) 2011/12 xMasBreak (and Technical Stops): Relocation of most critical elements Additional shielding of most critical areas 2012 Operation: Aim is that SEEs will not limit LHC performance Next long-shutdown: Relocation & Shielding for all critical areas Beyond: Power-Converters & Tunnel Equipment 31 R2E Mitigation Project Status & Outlook August 22 nd 2011 Backup 32 R2E Mitigation Project Status & Outlook August 22 nd 2011 Critical Area Overview (LHC Machine) Collimation Area (high losses possible at an early stage) Injection & Adjacent Areas: losses on collimators and dumps on TED High-Luminosity ExperimentsLHC Tunnel Electronics 33 LHCb: reaches early significant radiation levels R2E Mitigation Project Status & Outlook August 22 nd 2011 IR7 Performed Mitigation Actions Shielding installed: [ECRs: and ] RR73/77 (we gained ~factor 10 in radiation levels) UJ76 (we gained ~factor 10 for safe-room, 3-4 for the 1 st floor) Relocation of UPS from UJ76 (safe) [ECR: ] (safety critical at that time!) Relocation of fire control racks (safe) (possible impact on safety not excluded): UJ76 -> TZ76 [ECR ] Relocation of EN/EL control equipment (safe) RTU relocated from safe room in UJ76 -> TZ76 Relocation of fire detectors (possible impact on safety redundancy): Prepared for UJ76 and RR73/77 (see ECR ) Rack installation & cabling preparations: UJ76 -> TZ76 34 R2E Mitigation Project Status & Outlook August 22 nd 2011 Mitigation Actions Already Done Shielding installed: Ducts in P6 (RA63/UA63 and RA67/UA67) (gain ~factor 5-10) UJ22/23 and UJ88/87 (gain ~factor 10) US85 Safe-Room (gain ~factor 10) [ECR: ]ECR: Cryo-relocations/valve replacement in UX85 (safe) Relocation of fire control racks (possible impact on safety): US85, UJ56 [ECR: ] (safe)ECR: Relocation of fire detectors (possible impact on safety): US85 (other points prepared) [ECR: ] (safe)ECR: Relocation of EN/EL control equipment RTU safe room UJ56 (safe) Cabling preparations: P1, P7, P8 35 R2E Mitigation Project Status & Outlook August 22 nd 2011 Possible SEE Failures Observed in 2010 TE/EPC power supply burnout in UA87 Same effect observed in CNRAD SEE is very likely the cause (Streaming through Maze) TE/EPC power supply burnout in UA87 Same effect observed in CNRAD SEE is very likely the cause (Streaming through Maze) CRYO tunnel card SEE in 8L2 1 Fault in uFip (as observed in CNRAD 2010) SEE confirmed CRYO tunnel card SEE in 8L2 1 Fault in uFip (as observed in CNRAD 2010) SEE confirmed PXI power supply burnout in UJ16 To be confirmed by producer (comparison with CNRAD burnouts) SEE unlikely (early 2010 operation) PXI power supply burnout in UJ16 To be confirmed by producer (comparison with CNRAD burnouts) SEE unlikely (early 2010 operation) QPS Tunnel Card Failures (2x in 9L7 [ions], 2x in 8R8 [inj.], + others) ISO150 -> permanent PM trigger SEE confirmed (EMC has same effect) QPS Tunnel Card Failures (2x in 9L7 [ions], 2x in 8R8 [inj.], + others) ISO150 -> permanent PM trigger SEE confirmed (EMC has same effect) VAC power supply burn out In UA23 between maze and duct (TDI losses + TCDI losses ) SEE rather unlikely VAC power supply burn out In UA23 between maze and duct (TDI losses + TCDI losses ) SEE rather unlikely WIC crate failure in TI8 Observed in 2009 Known problem with moderate x-section WIC crate failure in TI8 Observed in 2009 Known problem with moderate x-section QPS tunnel Card Failures in 9R7 & 9L7 uFip communication lost (2x) SEE confirmed (seen in CNRAD) QPS tunnel Card Failures in 9R7 & 9L7 uFip communication lost (2x) SEE confirmed (seen in CNRAD) CONFIRMED or very LIKELY NOT CONFIRMED (unlikely) 36 R2E Mitigation Project Status & Outlook August 22 nd 2011 Failures 2011 [Mitigation] Collimation Control (3x, confirmed) [Soft + Relocation] Cryogenics UJ14/16/56/76 (several confirmed)[Relocation] Biometry UJ14/16 (2x, likely) [Operation + Rel.?] US85 PLCs (Cryo, 3x, confirmed) [Relocation] Power Converters (4x, TE/EPC) [New Develop.] UPS UJ56/US85?(unlikely, but possible) [Relocation] QPS Control UJ14/16 (few, corrector circuits) [Relocation] QPS ISO-150 (many, tunnel & shielded areas) [Soft + New Develop.] uFIP as used in QPS/Cryo (few times, tunnel + shielded areas) [Soft + New Develop.] Limitation-1: SEEs which are not SEEs? Limitation-2: hidden failures (not related to SEEs, caused by 3 rd party equipment with indirect impact, e.g. Ethernet Switches)? 37 R2E Mitigation Project Status & Outlook August 22 nd 2011 Power Converter Failures 38 TE/EPC R2E Mitigation Project Status & Outlook August 22 nd 2011 Minimize the number of converter types: Only the LHC60A-08V was specified for a radioactive environment ! 3 other converter types are part now of the radioactive sensitive areas! LHC POWER CONVERTERs LHC4..6kA-08V 1-Quadrant 200 Units LHC120A-10V 4-Quadrant 300 Units LHC600A-10V 4-Quadrant 400 Units LHC60A-08V 4-Quadrant 752 Units Units : Quantity in all machine (UA, RR, UJ, tunnel) V. Montabonnet, Y. Thurel 39 R2E Mitigation Project Status & Outlook August 22 nd 2011 IR7 Radiation: Today? 40 Combined Loss Analysis (BLMs & Injected/Dumped Intensities): IR3 lost in 2011: ~6E13 protons (per beam) IR7 lost in 2011: ~2E14 protons (per beam) Were a factor of ~50x below nominal (~1x10 16 p/b/y) Can nominal operation lead to relatively higher losses? Latest Weekly Report (www.cern.ch/r2e)www.cern.ch/r2e RR73/77: ~2-3x10 6 cm -2 UJ76: ~1x10 6 cm -2 Nominal Year: RR73/77: ~2x10 8 cm -2 UJ76: ~8x10 7 cm -2 A. Nordt, A. Thornton Scaling: x75 (Energy!) R2E Mitigation Project Status & Outlook August 22 nd 2011 Collimation Review 41 Performed mitigation options 2011/12 Operation UJ76 relocation Power-converter tests & R&D for RadTol Dev. Betatron collimation in IR3 very efficient not limited by IR7 next step (safety) high-importance to judge for RRs Would provide additional flexibility R2E Mitigation Project Status & Outlook August 22 nd 2011 RRs: Power-Converter R&D TE-EPC Planning for Rad Tolerant Converter Project Y. Thurel 42 Link: R2E Power Converter R&DR2E Power Converter R&D R2E Mitigation Project Status & Outlook August 22 nd 2011 RRs Additional/Alternative Options Betatron Collimation in IR3 No significant SEE impact for IR3 Low radiation levels in IR7 Additional flexibility Standard collimators to be added R&D for Superconducting Links 43 A. Ballarino et al. R. Assmann et al. R2E Mitigation Project Status & Outlook August 22 nd 2011 Summary (Protons) In6.02E+15 Dumped5.82E % Lost in Machine1.99E % Of Lost protons Collisions2.33E % Elsewhere1.76E % LM01LM02LM03 RM05RM04 RM03 RM06 RR77 RadMonDcumRth SEU (3V) Measured Beam contribution SEU (FLUKA) Expected Error [%] Stat. onlyExp./Mes. 6L7.7LM03S B L7.7LM02S B L7.7LM01S B1+B R7.7RM03S B1+B R7.7RM04S B R7.7RM05S B RR77.7RM06S B1+B BLM ratio IR7 / IR3 Ratio % Loss in IR7 TCSG.A6L7.B1 / TCSG.5L3.B TCSG.A6R7.B2 / TCSG.5R3.B IR7 FLUKA Application Benchmark >500m UJ76 TCP TCAP MBW TCS TCLA MQW Loss-Term (SixTrack [ R. Assmann et al. ]) FLUKA Calculations RR73 Assumptions Normalisation K. Roeed Operation R2E Mitigation Project Status & Outlook August 22 nd 2011 Radiation Levels 2010 Source terms, operational conditions as well as monitor readings have to be carefully evaluated 45 VERY GOOD AGREEMENT (given the underlying uncertainties) FLUKA Predictions Operation Assumptions Rad-Level Estimate Monitor Readings CalibrationAssumptions Final Values Tracking/Be amGas/etc. Threshold of monitors is ~10 6 ! (lower values only by trick [big uncertainties!]) R2E Mitigation Project Status & Outlook August 22 nd /2012/Nominal/+++ ??? 2010/2011/2012 Radiation Levels 46 Close to ThresholdCriticalDramatic?!!! Amazing !!! R2E Mitigation Project Status & Outlook August 22 nd 2011 IP8/TI8 Radiation Detectors 5RM05S 5RM06SPATL8711 PMIL8812 5RM07S PMIL8811 5RM08S Current location not known in full detail of the Radmon and RAMSES positions V. Boccone 47 Large Gradient: 4-5 orders of magnitude R2E Mitigation Project Status & Outlook August 22 nd 2011 TED Loss: RadMon Downstream 48 V. Boccone R2E Mitigation Project Status & Outlook August 22 nd 2011 R2E Mitigation Project Status & Outlook August 22 nd UJ87: Loss on TCDIH V. Boccone R2E Mitigation Project Status & Outlook August 22 nd 2011 Point-8 Application Benchmark Detector Measured dose ( Ratio (meas/simu) PMI8501 (UX85) PMI8511 (UX85) PAT8511 (US85) FLUKA/RAMSES benchmark FLUKA/RadMon benchmark DetectorRatio (FLUKA exp/measure) 8LE10S1.6 8LE07S2.0 8LE04S1.6 8LE08S2.2 FLUKA Simulations provide high energy hadron fluence, dose and 1 MeV Si equivalent in the LHCb cavern according to the Phase-2 shielding implementation proposed in the R2E Project UW85 US85 UX85 Very good agreement with PMIs and PATs RAMSES detectors RadMons set at 3V more difficult (at low count rates) Significant uncertainties to be considered (thermal neutron contribution, detector geometry, etc) Uncertainty at least a factor of 2 M. Calviani 50