SCT Operation in 2008 - an eventful year of commissioning and first running -

A. Ciocio LBNL ATLAS Meeting - Jan 2009 1

SCT Operation in 2008- an eventful year of commissioning and first running -

1. Early Commissioning and Detector Sign-off…………2

2. First SCT Run in ATLAS……………………………….4

3. Commissioning without cooling……………………….7

4. Barrel sector in SR1……………………………………10

5. Training………………………………………………….13

6. Preparation for Beam…………………………………..14

7. First Beam……………………………………………….17

8. Run with Cosmics ……………………………….….…25

9. Final Calibration …………………………………….…43

10. Schedule - plan for 09 ……………………………..…50


• Barrel signed-off (May) - 5 days commissioning until cooling heaters failure

• Cooling crises: heater repair, extension of pipes to make heaters accessible• Integration in ATLAS central of DAQ ,DCS, Monitoring and run with 4 SCT test modules• ROD and Power Supplies upgrades

Recap of 2007

Noise Occupancy 5.7 e-51700 ENC

All uniform across barrel

Barrel Defect channel %

Dead/noisy channels after SR1 tests 6712 0.206

2 modules lost after installation 2x1536 0.095

13 individual chips lost after installation 13x128 0.051

Total 11451 0.353

Number of defect channel / module (1module = 1536 channels)

1. Early Commissioning and detector sign-off

Avg chip noise vs chip number

Noise

Defect Channels

HV test


Endcap Sign-off

– Endcap A: Jan 9-20 2008

– Endcap C: Feb 8-11,2008

– One very leak loops: 13 modules • leak needs repair

– One “moderately” leaky loop • OK at low backpressure

Endcap A ENC (Temp = 25deg)Outers: 1707 eMiddles: 1653 eInners: 1149 eShort Middles: 1002 e

Endcap C ENC (Temp = 25deg)Outers: 1767 eMiddles: 1704 eInners: 1182 eShort Middles: 1013 e

Outers

MiddlesInners

Short Middles

Defect channel %

Endcap A 3960 0.26

Endcap C

+ 2 leaky loopsD9

4040

2x13 modules

0.266

2.6%

1. Early Commissioning and detector sign-off

February 15, 2008 we completed the sign-off of SCT endcaps,

officially marking the complete sign-off of whole SCT

NoiseDefect channels


SCT first RUN

• February 18 to 20, 2008 the SCT (barrel) pre-M6 run in ATLAS – Technical RUN mainly to verify integration and do first timing

in the ATLAS readout and control and prepare for the next ATLAS M6 combined run

– A chance to try global ATLAS integration with a large fraction of the real detector cooled and powered

– Used ROS monitoring to count module coincidences

for various timing offsets

2. First SCT RUN in ATLAS

First SCT pit cosmic on 20 Feb


M6 Run

• In M6 (March) first operation of SCT in combined ATLAS run

– TRT scintillator trigger + RPC– Operation was limited to Barrel by the number

of compressors available in the cooling plant– 3 barrel cooling loops off

(3x48 modules out of 2112)because running “warm” at high back pressure and these loops have small leaks

– Detector operation and Evaporative Cooling/Heater stable for whole period (24 hour operation)

– All RODs and HV/LV Power Supplies were available– One day standalone running to test ROD firmware upgrade– Online monitoring including DQMF– First check of the OFFLINE/Alignment algorithms– Few “noisy” module observed in M6 due to individual noisy strips not masked and

occasional HV trips

2. SCT First RUN2. First SCT RUN in ATLAS


Tracks reconstructed in SCT

• ID + RPC tracks

SCT residuals (No Alignment applied)~ 5000 tracksSCT residuals (GlobalChi2) ~65mCompares well with results from SR1

2. First SCT RUN in ATLAS

A. Ciocio LBNL ATLAS Meeting - Jan 2009 73. Commissioning without Cooling

Cooling Plant Failure May 1st, 2008

• 3 of 6 compressors damaged due to prolonged slippage of magnetic coupler between compressor section and motor

– In case of slippage for a long time, Eddy currents will heat inner metallic parts and damage (decompose, break) neighboring plastic parts.

– In addition excessive wear on piston rings was found

• As a result the C3F8 and plant was contaminated by both the incident and, over a longer period of time, by the wear of piston rings

– Subsequent analysis of samples indicated that the part of the plant in UX15 was not exposed to pollution coming from the accident.

– ~100-200 kg C3F8 were lost

– New C3F8 (1000kg) was received and additional 1000kg have been ordered

• The damaged compressors were repaired by Haug– The other 3 were cleaned and checked in-situ

• Implementation of the measures that would protect the system– An additional buffer tank was installed to prevent any loss of C3F8

– Compressors equipped with slippage sensors. The PLC will then stop a compressor if it starts to slip

– New mechanical filters needed to be installed, to protect the plant from the debris emitted from the piston rings and isolate the distribution racks (and the silicon detectors)

– Regular maintenance to replace piston rings (~1000hrs interval)


Crack in the crank cover (puts clean and dirty part of the

circuit in contact)

Burnt plastics, main source of pollution

Magnetic Coupling Motor 1450 RPM

Damaged parts

3. Commissioning without Cooling

A. Ciocio LBNL ATLAS Meeting - Jan 2009 93. Commissioning without Cooling

SCT Commissioning without Cooling: Off-Detector

• HV communication error (bad current readings) that was causing fake HV trips– Often occurs when card masked or interlocked off. Also occurs randomly, ~ 1/hour – For M6, auto recovery implemented as part of FSM– When channel is masked on to recover from the trip, neighbouring channels may start

oscillating– Power cycle all 8 channels of an HV board not possible during a run!– Had to suppress current trip in crate controller software (change thresh)

• Firmware upgrade for all boards to solve communication problem (May 19)

– Oscillation problem remains and will be solved next year with another major upgrade of the system

• Started to investigate “death of vcsels” as observed by pixel– TX vcsel array (12-way) sends optical clock and control signals to modules

• Situated in DAQ racks in USA15

– SCT barrel tests showed 5 of 1941 TX with no pin current and too low optical power– possibly related to ESD damage during production at the factory

• Since M6 had an SCT integration run in ATLAS readout ~ once/month to test new features and improvements of DAQ, DCS & monitoring chain

A. Ciocio LBNL ATLAS Meeting - Jan 2009 104. SCT Barrel Sector in SR1

SCT Barrel Sector in SR1

• Barrel sector refurbishment started at the beginning of March 2008

– 48 modules, 1 cooling loop, B3

• Cooling– Used the C3F8 cooling plant

• DAQ Software and Hardware• Detector backplane resistance tests• DCS Software Interlock and Power supply

– Environmental sensors:• 4NTCs for the cooling interlock• 10 NTCs for the mechanical and air temperature• 2 Xeritrons + 2 (HIH + NTC)s

for RH and dew point measurements

• Sector mechanics and service preparation• Reroute of cables for barrel and EC tests• Re-labeling • Fibres were disentangle• Harness installation • Plug-in• TEST

• SR1 space and cleanliness • Test box preparation

Started operation July 10

Huge involvement in the various areas of the detector construction and good opportunity for students and new post-docs to learn about SCT

Used for TRAINING and DAQ development


4. SCT Barrel Sector in SR1

SCT Barrel Sector in SR1

Harness testing in the box

Harness test before module mounting with loop back cards

Harness installed


Sector with cooling loop

•

4. SCT Barrel Sector in SR1

inlet/return side


Training

• SCT training started in May• By June 36 people took training courses in DCS and DAQ in ID SCR, Monitoring in ACR• Training shifts with H8 test box (4 modules)

• List of trainees• Started with residents (mostly student and post-docs)• List updated with information from GLs (in July) or directly from people

• Training Scheme and method to keep track of training progress• End of June started regular training shifts with Barrel Sector in SR1

https://twiki.cern.ch/twiki/bin/view/Atlas/SctSR1Tutorial• DCS and DAQ• 2 people/shift (4h)

• Regular Training Courses in DAQ, DCS, and DQM• 1 hour with instructors• Documentation and tutorial at:

https://twiki.cern.ch/twiki/bin/view/Atlas/SctCommissioningDocumentation

• Average time to complete training 2 weeks• Qualification overlap shift with experts going through a shift check list (later)• Training info, progress, links to documentation at:

https://twiki.cern.ch/twiki/bin/view/Atlas/SctTraining

• Started to use the ATLAS Shift Booking Tool OTP (August)

5. Training

A. Ciocio LBNL ATLAS Meeting - Jan 2009 146. Preparation for Beam

Commissioning before Beam (I)

• Cooling plant repair finished on July 14

• Needed to be able to cool down pixel for beam pipe bake-out

• No operation of SCT modules before end of bake-out

• Beam pipe bake-out successfully completed 29-31 July– This was a very tense operation - on critical path for LHC to open vacuum valves.

• ATLAS went to full shift mode on July 28

• Started of SCT commissioning with cooling on Aug 11

• Once we restarted, running the SCT colder than before, had control problems– Some loops only stable with modules powered up– Some loops also needed modules to be configured and clocking.– Restarting the system was taking long time but got better on synchronizing cooling, SCT

DCS and SCT DAQ, after many unwanted shutdowns (power cut, etc.)

A. Ciocio LBNL ATLAS Meeting - Jan 2009 156. Preparation for Beam

Commissioning before Beam (II)

• Frantic effort to get the detector into good shape before first beam– Commissioning cooling loops and basic module control on barrel and endcap– In parallel to Pixel commissioning – Started 24/7 mode (operational shifts and calibration)– Radical DAQ improvements needed completion– Intense calibration: Noise occupancy and equivalent noise charge good for most

modules– A few modules still untrimmed (EC-C) giving noisy spots

Calibration ENC noise on barrel is ~1560e- Noise Occupancy is ~4.4 x 10-5 at 150V Endcap ENC outer/middle ~1650e- Noise Occupancy outer/middle ~ 5 x10-5

Inner Modules much lower due to short strip length

• Starting Aug 29/30 had SCT on full power and started to adjust opto readout setting first on barrel then on endcaps


• Decided that it was safe to run the detector at 20V bias during first beam, with electronics on

– Threshold raised to 1.2fC.– Well below 100V limit above which implant damage has been observed– Lowest voltage possible where HV and DAQ stable– Hybrid experts consider LV on to be safer– Pixels stayed off

• Then we learned that first beams would be stopped just before ATLAS at the collimator

– Worries that many particles going parallel to barrel tracks could deposit even more charge (no simulation of this available)

– Reduced barrel HV to 5V and left it out of the DAQ– Endcaps set at 20V - with expected lower (and unmeasured) efficiency

Voltage setting

6. Preparation for Beam


September 10

• Run 87764 event 40050

7. First Beam


People clapping in the ACR

7. First Beam


People (and reporters) in the ACR

7. First Beam


People at the SCT desk in the ACR

7. First Beam

A. Ciocio LBNL ATLAS Meeting - Jan 2009 217. First Beam

First Beam “Splash” Event


Space points in EC in one “splash”

• SCT roughly timed in to crudely optimize hit occupancy in the central time bin

• SCT can time in ‘on the fly’ - invoke SCT busy during timing adjustments

• Endcaps running at 20V


SCT People in the ACR


More SCT People in the ACR


After September 19

• With no prospect of beam, but the full ID available with cooling for the first time, everyone was very positive to start commissioning with cosmics (right away!)

– TRT just completed change to Xenon active gas– Pixels detector operational for the first time

• Started Combined ATLAS Run until 27 October (almost 6 weeks!) Alternating calibration periods

Understand of how to time-in the various ATLAS triggers, and how to synchronize the SCT

Propagate configuration and calibration results to the offline Solved several problems (ROD, TX) and calibrated the detector the best we could Had several stops due to power cuts, interventions, incidents with cooling, but we always

recovered and improved our recovery procedures Shifts were filled, training in progress and people continued to be active and involved 24/7 Lots of cosmic events, with and without solenoid on

8. Run with Cosmics

RPC: mostly “100” hits trigger arrives slightly late by ~1BC

MBTS: mostly “001” or “010” hits trigger arrives slightly early


Number of SCT tracks

• Over 1 million SCT tracks

• ~250 thousand tracks with at least 1 pixel hit

0

200000

400000

600000

800000

1000000

1200000

90000 90500 91000 91500 92000 92500

Run Number

N SCT tracks

All SCTTracks

solenoid off

solenoid on

8. Run with Cosmics

200K solenoid off800K solenoid on


Cosmic event

• Cosmic ray event

• Solenoid on

8. Run with Cosmics

A. Ciocio LBNL ATLAS Meeting - Jan 2009 288. Run with Cosmics

“Data” is taken after BCID problem fixed•“Reference” from a long run, so many modules gave BCID errors by the end

Problems solved in October (I)

• SCT efficiency degraded over the course of a run– Much more obvious as ATLAS stability improved and we recorded many-hour long runs– Observed an increasing number of L1ID and BCID errors– By the end of some runs up to ~22% of modules showed errors– Also saw a decrease in the number of tracks over time (in a run).

• Synchronization errors– Once a module got out of synch, its efficiency was zero– John Joseph (arrived on Oct 15) traced the problem

to inconsistent treatment of timeouts in the FPGA code

...…..and FIXED IT!

Before

After


Problems solved in October (II)

• Already started to see signs of TX failures last July.

• The number of working TX channels had been gradually deteriorating at the rate of 2 or 3 per day, reaching ~5% dead channels at the end of October.

• Original intention was to replace the defective plugins from the batch of50 new plugins received from Taiwan, but the new plugins failed the QA at Cambridge

• Made the decision to replace TX plugins with existing (old) spares:

• Replaced 8 BOCs, 22 plugins in 17 different BOCs (completed on Nov 12)

• Work took 2 days to complete, 25 BOCs needed extraction

BPMChipBPMChip

VCSEL arrayVCSEL array

VCSEL converts electrical signal into light and sends it to the on-detector side.

BOC


SCT efficiencies from later runs

• More fixes for conditions

and holes on tracks algorithms

• Apply a track incidence angle cut

• Efficiency for barrel ~99%

• More to be understood, especially for the end caps

EC A EC C

Barrel


Data Quality Offline Web tools

• SCT bytestream check webpageNoise & Error Summaries


SCT Barrel Residuals

• SCT (Barrel) residuals ~20-30 m

8. Run with Cosmics


• Detector conditions– Tx replacements and fine opto-tuning

• From 200 modules to ~10• TX channel death rate still 1-2/day

– About 70 modules excluded (over 4088 - HV, LV, Ipin=0, Cooling, Misc.)• Including the 36 of ECC

– DAQ fixes• one ROD replaced• Calibration and Conditions DB

– Online monitoring• more histos• Stability

• Shifts– TDAQ Run Control tutorial (~10 people)– RPC (DCS) tutorial– RC+SL shift plan

• discussed among run coordinators• In addition to regular shifts• RC and SL shifts in OTP

– SCT Detector Operation• DAQ + DCS and DQM• DQM• On call: DAQ, DCS, DQM

8. Run with Cosmics

Preparation for ID Combined RUN (5 days - TXG WE)

Also: DSS tests as an exercise to prove that the detector and the cooling can be turned off under different circumstances


ID Combined Run List

8. Run with Cosmics

TRT only

TRT +RPC}

}

TRT +RPC

} TRT only

Sp

eci

al r

un

s

Recovery due to mixed-water cooling failure

Recovery due to power cut in US15

Run transition test

HV=50V

HV=20V

Bad configuration Runs

Good Run

{


Tracks and rates

8. Run with Cosmics

Runs with >300 time-out errors after power cut no good configuration


Run 96903

• Mostly “010”

followed by “011” hits

Barrel

ECA ECC

“010”

“011”

Time Bins

Summary Time Bins


Mean Noise Occupancy for all runs

8. Run with Cosmics


Noise Occupancy (Barrel) Run 96903

8. Run with Cosmics


8. Run with Cosmics

Hit Efficiency

Efficiency depends on layer

ECA ECC

Barrel


Special Runs at 50 V and 20 V

– ~0.5% drop in efficiency for 50 V– ~37% drop in efficiency for 20 V

8. Run with Cosmics

@ 50 V

@ 20 V(1.2 fC)

(barrel 4 and disk 2)

ECCECA Barrel


• Detector was operated by shifters and experts on call (and present)– DAQ, DCS, and DQM tasks (in OTP) over 3 shifts: 7h-15h, 15h-23h, 23h-7h

– The last week of running these three shift tasks were merged in one: • SCT Operation task

– Plus 2-3 experts on call shifts

– Plus 1 shifter for DQM (75% of the time)

– DQ checks Offline shift 7h-15h

– Overlap shifts • qualification • before first shift period

• After Oct 27 (back to commissioning)– DCS watch shifts 24h

– Calibration (DAQ expert) day-time shift

– Daily meeting at 9h15

• Nov 26 - Dec 1 ID Combined Run– SCT Operation + DQM shifts

• Training continued until ~end of November• SR1 was unavailable “only” for 3 weeks

for cooling plant maintenance in October

All about shifts information at:https://twiki.cern.ch/twiki/bin/view/Atlas/SctShifterInfo

Shifts

8. Run with Cosmics

>5 months

Daily meetings at 15h during cosmic run (including weekends) to discuss current issues between shifts and with data quality offline shifter. At 9h15 after Oct 27.


Full List 136

Trainees

Shifters 92

Experts 25

(Include experts)

Shifters statistics

• 30 residents at CERN

plus ~ 1/2 of experts residents

DCS 8

DAQ 10

DQM 9

SCT Institutes (34):Academia Sinica, Bergen, Birmingham, Cambridge, CERN, Cracow-AGH, Cracow-INP, Freiburg, Geneva, Glasgow, KEK, Kyoto, Lancaster, LBL, Liverpool, Ljubljana, Manchenster, MPI, NIKHEF, Okayama, Osaka, Oslo, Oxford, Prague-CU, Prague-CTU, QMW, RAL, Sheffield, Tsukuba, UC-Santa Cruz, UCL, Uppsala, Valencia, Winsconsin

(overlap)

0

10

20

30

40

50

60

ACR DCS DAQ DQM IDGen

QualifiedAlmost QualifiedCourse

SCT trained

> 2000 shifts

A. Ciocio LBNL ATLAS Meeting - Jan 2009 439. Final Calibration

Final Calibration in 2008

After ID combined run:• Full SCT ON

– Adjusted readout configuration• Latest list of number of modules in default readout configuration

– Response curve of SCT barrel and endcaps• Update with tuned thresholds based on gain curve measurement• Expected to improve noise occupancy

– Opto survey after end of calibration

• SCT sections ON to look at systematic dependencies of SCT Noise as reference for future studies

– Noise versus HV scan• Done on barrel side C and Endcap A

– Influence of FSI on noise (and leakage current)• Done on barrel at different power levels

– Noise versus cooling temperature: bp settings 3.0 (normal), 2.5, 2, 1.5, 1.2 bara

• Done on ~500 barrel and ~500 endcap modules

• DAQ improvements for calibration: – Speed (faster data archival and faster response from online GUI)

– Robustness (found/debugged reason of ROD failing to complete a scan due to errors)

• Faulty HV and LV boards replaced

SCT OFF December 12


• Barrel modules in readout 99.3% 14 of 2112 removed

• Endcap A modules in readout 98.7% 13 of 988 removed

• Endcap C modules in readout 95.3% 10+36 of 988 removed

98.9% 10 of 952 with loops 1C-TR and 9C-TR

removed from statistics

Modules excluded

9. Final Calibration

Modules in Default Readout


Noise versus bias voltage

• Obtain reference of noise dependence (detector capacitance) on bias voltage

• Measured ENC noise with 3ptgain scans

at 150, 120, 90, 70, 60, 50, 40, 30, 20 V bias voltage

Barrel module with 12cm strips Endcap Inner with ~6cm strip



Noise occupancy at different voltages

• noise occupancy scan (new thresholds) in calibration mode at 150, 50 & 20V


150V50V 20V

(x 10-5) 150V 50V 20V

Barrel 1.9 7.6 25.5

Endcap 3.5 ~11 ~35


Noise on barrel with FSI laser running

• Tested detector leakage current up to max 200mW (fibre 0)

• Tested barrel NO without and with 70mW (using chopper)– Using old thresholds and asynchronous L1A trigger

• Tests to be continued with FSI and L1A synchronousNo change in noise occupancy


70mW with 50Hz chopper FSIMean = 3.755 x 10-5

No FSIMean = 3.748 x 10-5


ENC noise in calibration


Barrel Noise ~ 1550 e-

Endcap noise ~ 1650 e- (outer/middle)and inner ~1100e- (950 short middle)


Noise Occupancy - New threshold setting

• Noise occupancy in calibration mode with new thresholds applied to chip configuration• New noise occupancy is ~2x 10-5

• ~ factor 2 improvement on barrel and less tail!


Previously ~4x 10-5

Old thresholds

Full Barrel

Rms of vt50 point shows modules are well trimmed

1/2 Barrel

Endcap A

Mean ~2


Schedule

• Redesign of pipes around the compressors to be sufficiently strong to avoid fatigue cracks induced by vibrations.

• All compressors sent back to company for this rework.• Refurbishment of distribution racks in UX15 (until end of April)• Cooling will be off and SCT powered down with exception of DCS/DSS and

environmental monitoring.• TDAQ related development work will resume mainly in SR1• Power supply tests

– HV– PP3 debugging– LV pin current measurement (19.01 -- 3 weeks)

• FSI tests• Interlock system

– Last connections in the DSS system (Laser alarms + ATLAS emergency off signals)– Replacing plug-ins– Work on ID sensors and repair a few damaged pins in the Type II cables

• Opto tests • Training in SR1 can resume as early as mid/end of January (3 weeks interrupt in February)

and then again from March

10. Schedule 2009

January/February


• Endcap disk in SR1– Work almost completed

• 3 rings, 33 modules

• DAQ, DCS, Services,etc.

– Completion with Cooling

in January

Schedule (SR1)

10. Schedule 2009

January/February

Disk 1 (spare NIKHEF)

one quadrant

Barrel sector and Endcap disk in SR1


• Spring schedule remarkably tight– Due to maintenance/ interventions (Jan. – Feb.)– ATLAS cooling maintenance week 7-8-9

• 3-4 days no electricity in same period

– New TDAQ/HLT releases– Sub-systems migration to new TDAQ– TDAQ technical run (March)– Barrel and BW closed (end of March)– Work on detector (till April)

• April– Run slices of ATLAS to do debugging of s/w and infrastructure

• May – ATLAS in better shape than ever, combined run for:

• L1Calo, L1Mu trigger latency uniformity• Offline commissioning, alignment, calibration

• May – June– recurrent combined runs with defined purpose

• July – semi-continuous to continuous running transition

Schedule from run coordination point of view

10. Schedule 2009

Silicon ID working again


Summary

• SCT detector sign-off, integration in ATLAS and first cosmics last spring, cooling failures, recovery, off-detector commissioning without cooling, preparation for beam, first beam

• SCT detector run with cosmics (late September-October) ~1 million tracks • SCT detector improvements

– TX replacement: from ~96% to ~99% modules in default readout configuration – Much lower number of readout errors: typically 4-5 of ~8000 links– System more reliable in readout (in particular during calibration scans)

• Successful ID Combined Run (end of November) ~ 800K tracks • Completed calibration in December

– New configuration serves as reference point for spring startup– New thresholds have improved noise occupancy – Looked at systematic dependencies of SCT Noise as reference for future studies

• Trained ~100 SCT people for operation• Built Barrel sector and Endcap disk (to be completed in Jan) in SR1

Looking forward to: - complete off-detector parts: DAQ, DCS, Databases, Mon, TX, Power Supply upgrades

- resume training in SR1- integrate in ATLAS after cooling is working again (May)

* prepare for the 2009 beam *

Documents

SCT Operation in 2008 - an eventful year of commissioning and first running -