Abbreviated list of April Global Run (AGR) achievements

• All: clock frequency scan• ECAL/RCT/L1: test (28 of 500) new oSLB oRM links• Strips: run with all FEDs & calibration constants at 15°C • HCAL: Commission new HO SiPMs with self- and DT-triggers

in bottom-only and top-only configurations• DT: Commission relocated sector collectors• RPC: First global run with new RE4 chambers• CSC: First global run with new ME4/2 chambers• PPD: First run with Prompt Calibration Loop on cosmics

17 Apr 2014 G. Rakness (UCLA) 1

A lot of new equipment successfully participated in this AGR

Some issues encountered (1/2)

• Communication is hard…– Shift crew DOCs

• Actions performed without telling the experts• DOCs talk directly to DAQ, DCS or Trigger shifter (skip Shift Leader)

– Shift crew shift crew• DCS shifter not always aware what should be on• DAQ/DCS shifter had parts of detectors in run but powered off

– DOCs DOCs• Trigger key did not always match the detector state/key

– Management Shift crew/DOCs• Actions performed without telling Run Field Managers• DTs removed from an overnight run• RC did not inform Tier-0 that run was continuing over the weekend

17 Apr 2014 G. Rakness (UCLA) 2

Some issues encountered (2/2)

• New hardware needs commissioning…– Shifters were busy overnights to keep runs going

• Why did we keep going out of sync?

– Timing between control signals need tuning since FPGAs on new CSCs need longer time to load (bigger)

– Spikes in upper trigger sectors which had not been run since collisions

• Old problems show up (and fixed) again…– Cables unplugged at Global Trigger– L1MuOpen triggers not included in Express Stream– miniDAQ data with bad menu caused crashes at Tier-0

17 Apr 2014 G. Rakness (UCLA) 3

G. Rakness (UCLA) 4

Power/cooling/elevator

• Sunday evening: power cut affected the cooling at p5 – Although water was still flowing, it was no longer being

cooled… and eventually initiated temperature alarms/automatic switch-off for several subsystems... • An elog which sort of describes what happened can be found at

http://cmsonline.cern.ch/cms-elog/816435• Discovered that technical incidents are kept track of in TECOPA,

not elog… see page 12 of https://twiki.cern.ch/twiki/pub/CMS/AprilGlobalRun2014/TECOPA_slides_20140411.pdf

• Elevator going underground cavern stopped… twice...– Wed: relay that tells door where it is w.r.t. opening broke– Thu: piece guiding the door fell out

17 Apr 2014

Action items• Make communication the procedure in the control room

– Between & within shift crews, DOCs, and management

• Follow up issue with gas– On both CERN gas group side and RPC side…

• Test out every new thing which is installed, even if it doesn't give good cosmic rays – E.g., DT links on top-half and calorimeter trigger

• Make sure monitoring paths have what we need– E.g., muons in cosmic rays, single objects w/beam

• Why do we keep going out of sync?– These are not difficult data taking conditions…

17 Apr 2014 G. Rakness (UCLA) 5

2014 Mid-Week Global Run (MWGR) schedule

17 Apr 2014 G. Rakness (UCLA) 6

https://twiki.cern.ch/twiki/bin/view/CMS/CentralShifts2014#Global_run_planning

Date Global run Notes/goals7 – 11 April April Global Run

2 – 4 July MWGR #1 DAQ2 + TTC

16 – 18 July MWGR #2 TCDS with subsystem partitions (try 1)

30 July – 1 August MWGR #3 TCDS with subsystem partitions (try 2)

13 – 15 August MWGR #4 TCDS full system

27 – 29 August MWGR #5 Try HLT as processing farm between “fills”

10 – 12 September MWGR #6 Remove CMSSW from AFS farm

24 – 26 September MWGR #7

8 – 10 October MWGR #8

30 October - 10 November (dates are placeholders, exact dates depend on TC schedule)

CRUZET present thinking to shoehorn this run in during the days when the HFs are being removed from their garages

25 November - 12 December (dates are placeholders, exact dates depend on TC schedule)

CRAFT present thinking is that this run would starts when the field reaches 3.8T

Intend to officially announce the dates for the MWGRs soon

Regarding worries about Trigger and Clock Distribution System (TCDS)

• Summary of concerns from subsystems about TCDS… – Upgrade not as transparent as they were led to believe

(some work involved)– The time schedule is late (installation in August)– It’s not going to work out of the box (little expertise left

in subsystems to commission)– https://indico.cern.ch/event/310364/

• Have discussed w/TCDS & HCAL, and will sit in when they interface at 904 to get a sense of what it takes

17 Apr 2014 G. Rakness (UCLA) 7

G. Rakness (UCLA) 8

To do• Sort out what is happening with shift administration

– Andrea won’t be extended past October… Tiziano has someone in mind to replace him but Austin doesn’t think that is a good job for him (I agree with Austin)

• Work out LHC/CMS handshake schedule– Technical Coordination, BRIL, …

• My items from RC to-do list– Clean up sources to "action list" from the meetings – Make sure that HF is fine as min-bias trigger for low lumi – Come up with best guess-timates for integrated and instantaneous

luminosity during LHC startup – Monitor TCDS developments/scheduling– Follow up problems w/ECAL laser calibration data xfer to Tier-0 – Follow up SCAL FED

17 Apr 2014