Coney - CM34 - Oct 2012 1
Operations Operations SummarySummary
L. Coney – UCRL. Coney – UCR
MICE CM34 – Oct 2012MICE CM34 – Oct 2012
2Coney - CM34 - Oct 2012
Online Updates – New Online Updates – New DAQDAQ
3Coney - CM34 - Oct 2012
DAQDAQ DAQDAQ
New DAQ in placeNew DAQ in place Implemented new acquisition library independent of Implemented new acquisition library independent of
DATE and EPICS – leads to flexibility and is easy to DATE and EPICS – leads to flexibility and is easy to useuse
Written simple applications for use in the control Written simple applications for use in the control room room
Have thin C++ wrapper of EPICS – flexible/easy – Have thin C++ wrapper of EPICS – flexible/easy – also independent of DATE and DAQ hardwarealso independent of DATE and DAQ hardware
Communication with CDB done by Controls & Communication with CDB done by Controls & Monitoring (Run Control)Monitoring (Run Control)
At end – interface with DATEAt end – interface with DATE New zero suppression of fADCsNew zero suppression of fADCs Very high DAQ rate now possibleVery high DAQ rate now possible New pulser trigger availableNew pulser trigger available
4Coney - CM34 - Oct 2012
DAQDAQ DAQ can now handle rates above the DAQ can now handle rates above the
initial 600 muons/spill goalinitial 600 muons/spill goal
New software/hardware system in New software/hardware system in place, debugged, tested, and place, debugged, tested, and functionalfunctional
All that remains on Yordan’s All that remains on Yordan’s Christmas list for the “PID” DAQ is Christmas list for the “PID” DAQ is the programmable FPGA logic for the programmable FPGA logic for the particle triggerthe particle trigger And the EMR and Trackers…And the EMR and Trackers…
5Coney - CM34 - Oct 2012
Online Systems Updates Online Systems Updates – C&M– C&M
Been very busy since GlasgowBeen very busy since Glasgow TargetTarget Daresbury C&MDaresbury C&M Spectrometer SolenoidSpectrometer Solenoid LH2 MonitoringLH2 Monitoring Run ControlRun Control RF TunersRF Tuners Auto-SMS notificationAuto-SMS notification More system monitoringMore system monitoring State MachinesState Machines
6Coney - CM34 - Oct 2012
C&M - TargetC&M - Target Rewritten Target C&M systemRewritten Target C&M system
Better EPICS complianceBetter EPICS compliance Web interfaceWeb interface CDB compatibleCDB compatible Online analysisOnline analysis Run priorityRun priority Root file outputRoot file output
Making similar changes to Tracker Making similar changes to Tracker systemsystem
7Coney - CM34 - Oct 2012
C&M C&M DLDL
SS standalone system being usedSS standalone system being used Being improved for final SS trainingBeing improved for final SS training
FC standalone readyFC standalone ready
8Coney - CM34 - Oct 2012
C&M C&M LH2 LH2
MonitoringMonitoring ArchivingArchiving Control expert onlyControl expert only
RF TunersRF Tuners Resonant frequency Resonant frequency
depend on depend on geometrygeometry
Change cavity Change cavity shapeshape
Controls system Controls system
9Coney - CM34 - Oct 2012
C&M – Run ControlC&M – Run Control Interfaces with:
DATE TargetMon State Machine Environment Standard operation Records all running parameters to CDB Records all meta-data at end of run Verifies readiness of subsystems Sets running parameters Tags run parameters
10Coney - CM34 - Oct 2012
Elog ChangesElog Changes Modifications to improve ease of use Modifications to improve ease of use
and understandability – see Henry and understandability – see Henry for input to new systemfor input to new system
11Coney - CM34 - Oct 2012
Step IV: Operations & Step IV: Operations & OnlineOnline
Moving toward Step IV runningMoving toward Step IV running Many configurations to test/runMany configurations to test/run Long term running/every opportunity takenLong term running/every opportunity taken All ISIS cycles – every day – Differs from past All ISIS cycles – every day – Differs from past
OperationsOperations Develop idea of not-so-remote control roomDevelop idea of not-so-remote control room Develop plan for staffing & commissioning of cooling Develop plan for staffing & commissioning of cooling
channelchannel
Develop robust system of OperationsDevelop robust system of Operations Includes all facets of Online Group (DAQ, C&M, Online Includes all facets of Online Group (DAQ, C&M, Online
Reconstruction/Analysis, Infrastructure)Reconstruction/Analysis, Infrastructure)
Goals – next phase of OperationsGoals – next phase of Operations FunctionalFunctional Safe/SecureSafe/Secure ReliableReliable Easy to useEasy to use
12Coney - CM34 - Oct 2012
Step IV: Operations & Step IV: Operations & OnlineOnline
FunctionalFunctional DAQ – incorporate all detectors – improve rateDAQ – incorporate all detectors – improve rate DAQ upgradesDAQ upgrades All C&M for each elementAll C&M for each element Alarm Handler values set appropriatelyAlarm Handler values set appropriately Add capability to Online Reco/AnalysisAdd capability to Online Reco/Analysis Automate run infrastructure – Run ControlAutomate run infrastructure – Run Control Automatic use – read/write to Configuration DatabaseAutomatic use – read/write to Configuration Database
Safe/SecureSafe/Secure Incorporated fully commissioned PPSIncorporated fully commissioned PPS Implemented formal shifter trainingImplemented formal shifter training Updated Operations/Online documentation & instructionsUpdated Operations/Online documentation & instructions Developing comprehensive list of safety-critical maintenanceDeveloping comprehensive list of safety-critical maintenance Updating safety paperwork – developing overall system for Updating safety paperwork – developing overall system for
MICEMICE Using new target controller with all necessary interlocks & Using new target controller with all necessary interlocks &
BPSBPS Online system security reviewed – improvements madeOnline system security reviewed – improvements made Access limited to micenetAccess limited to micenet
13Coney - CM34 - Oct 2012
Step IV: Operations & Step IV: Operations & OnlineOnline
Easy to useEasy to use Simplified Ops procedures – single key exchange ISISSimplified Ops procedures – single key exchange ISIS MICE control access to HallMICE control access to Hall Improving Online Reconstruction/Data Quality plotsImproving Online Reconstruction/Data Quality plots Need add Online Reco for KL/EMR/Accel AnalysisNeed add Online Reco for KL/EMR/Accel Analysis Run ControlRun Control Mature Alarm HandlerMature Alarm Handler
ReliableReliable New OS implemented – automatic updatesNew OS implemented – automatic updates Automatic/systematic backups in placeAutomatic/systematic backups in place New DAQ and C&M computers installedNew DAQ and C&M computers installed UPS coverage for critical systemsUPS coverage for critical systems Developing “stockroom” of spares – Developing “stockroom” of spares –
crates/computers/etccrates/computers/etc Automatic write to EPICS ArchiverAutomatic write to EPICS Archiver Extensive remote monitoring of Extensive remote monitoring of
equipment/computers/hardware/environment in EPICSequipment/computers/hardware/environment in EPICS Archive of all software – Launchpad repositoryArchive of all software – Launchpad repository
14Coney - CM34 - Oct 2012
Organization & Organization & PreparationPreparation
Micemine Run Plan worked well – Micemine Run Plan worked well – schedule & planschedule & plan
http://micewww.pp.rl.ac.uk/projects/http://micewww.pp.rl.ac.uk/projects/operations/wiki/RunPlan20121012operations/wiki/RunPlan20121012
15Coney - CM34 - Oct 2012
Need New/More ShiftersNeed New/More Shifters Recognized the need for training – now need Recognized the need for training – now need
to implement trainingto implement training Difficult to do while running – difficult to Difficult to do while running – difficult to
properly train w/o runningproperly train w/o running Need some training w/o beam but with Need some training w/o beam but with
equipmentequipment Develop training scheduleDevelop training schedule
Corollary – develop training updates when new Corollary – develop training updates when new equipment arriveequipment arrive
Need all MOMs able to lead/arrange trainingNeed all MOMs able to lead/arrange training Sign up more MICE – THIS MEANS YOU!Sign up more MICE – THIS MEANS YOU! Cannot exploit tiny group of (primarily UK-Cannot exploit tiny group of (primarily UK-
based) dependable individualsbased) dependable individuals This becomes even more true for Step IV This becomes even more true for Step IV
runningrunning
16Coney - CM34 - Oct 2012
Goals of RunningGoals of Running Test significant upgrades and changes to both the Test significant upgrades and changes to both the
Online Systems and MAUS software packages – NOT to Online Systems and MAUS software packages – NOT to take datatake data
Overall OnlineOverall Online New OS installed on all MLCR computersNew OS installed on all MLCR computers New master server, new account setups, new UPSsNew master server, new account setups, new UPSs
DAQDAQ Final live data test of new DAQ system – upgraded DATE & Final live data test of new DAQ system – upgraded DATE &
OSOS New interface with C&MNew interface with C&M New DAQ hardwareNew DAQ hardware
C&MC&M Full read/write from CDBFull read/write from CDB Overall system test of Step I Run ControlOverall system test of Step I Run Control Expert testing/feedback on Run ControlExpert testing/feedback on Run Control New HV user docsNew HV user docs
SoftwareSoftware Online test server setup functionalOnline test server setup functional New control room branch of MAUSNew control room branch of MAUS Improved TOF/CKOV online plotsImproved TOF/CKOV online plots
17Coney - CM34 - Oct 2012
Goals of RunningGoals of Running DAQ – Yordan’s talkDAQ – Yordan’s talk C&M – Pierrick’s talkC&M – Pierrick’s talk Software – Chris Rogers’ talk this Software – Chris Rogers’ talk this
morningmorning
Mixed results – confirm that we DO need Mixed results – confirm that we DO need regular runningregular running Partially successfulPartially successful Unfortunately – not smooth run – while this Unfortunately – not smooth run – while this
was expected due to the many changes, many was expected due to the many changes, many of the problems were not anticipatedof the problems were not anticipated
No real useful data until Sunday….although No real useful data until Sunday….although this was not the goal of the runthis was not the goal of the run
18Coney - CM34 - Oct 2012
Way Forward..Way Forward.. We are making progressWe are making progress
Previously we were focused on getting the basics going Previously we were focused on getting the basics going – Implementation of necessary functionality– Implementation of necessary functionality
Now – move focus to Now – move focus to reliabilityreliability Need to Need to KEEP THINGS WORKING!KEEP THINGS WORKING! Schedule regular 3-day weekend runsSchedule regular 3-day weekend runs MAUS – already using “release” system of MAUS – already using “release” system of
changeschanges C&M C&M
need do same with the controls and monitoring codeneed do same with the controls and monitoring code Stabilize applicationsStabilize applications Development computer/system separate from Development computer/system separate from
iocpc1/iocpc2iocpc1/iocpc2 Coordination between C&M developers Coordination between C&M developers mustmust happen happen
DAQ – staged upgrades – especially with new DAQ – staged upgrades – especially with new equipmentequipment
19Coney - CM34 - Oct 2012
MICE Step IVMICE Step IV Includes both Spectrometer Includes both Spectrometer
SolenoidsSolenoids Two trackers installed within the Two trackers installed within the
SS magnetsSS magnets One AFC (Focus Coil magnet & One AFC (Focus Coil magnet &
Absorber)Absorber)
20Coney - CM34 - Oct 2012
Step IV Ops – New Step IV Ops – New Equipment…Equipment…
We will have much more in the We will have much more in the Hall!Hall!
21Coney - CM34 - Oct 2012
Step IV – OperationsStep IV – Operations Continue:Continue:
Pre-run prepPre-run prep Further simplify runningFurther simplify running
Improve:Improve: Automate procedures Automate procedures fewer errors fewer errors Move from spreadsheet Move from spreadsheet only ConfigDB only ConfigDB Increase Online Reconstruction capabilitiesIncrease Online Reconstruction capabilities
Complete suite of detectors, Online analysis, Global Complete suite of detectors, Online analysis, Global recoreco
Reduce required number of shifters…Reduce required number of shifters…
New equipment arriving – plan for New equipment arriving – plan for commissioning, support & expertisecommissioning, support & expertise
Modify procedures to include SC magnets Modify procedures to include SC magnets and LH2 systemand LH2 system
Train shift personnelTrain shift personnel Develop plan to staff multitude of shiftsDevelop plan to staff multitude of shifts
22Coney - CM34 - Oct 2012
Step IV – OperationsStep IV – Operations Safety Safety
Need understand implications under STFC rules of Need understand implications under STFC rules of new equipmentnew equipment
Made arrangements to meet with Jane Vickers (ISIS Made arrangements to meet with Jane Vickers (ISIS safety) to begin preparation/planningsafety) to begin preparation/planning
Need understand from experts what each system Need understand from experts what each system entails/requires for operationentails/requires for operation
PPSPPS Need understand implementation of system with SC Need understand implementation of system with SC
magnets and LH2 systemmagnets and LH2 system Hall AccessHall Access
Limited – not every day – scheduled access onlyLimited – not every day – scheduled access only Increase efficiency of data-takingIncrease efficiency of data-taking Hall search dropped Hall search dropped twicetwice in 3 days last weekend. in 3 days last weekend.
Not acceptable.Not acceptable. Experts must be availableExperts must be available
23Coney - CM34 - Oct 2012
Operations OrganizationOperations Organization Overall Operations ManagerOverall Operations Manager
Local, familiar with STFC safety and operational Local, familiar with STFC safety and operational systemssystems
Provide oversight, link between MOMS, safety Provide oversight, link between MOMS, safety responsibility from project managerresponsibility from project manager
Monthly MOMsMonthly MOMs On call for 1 month – Daily running duties as nowOn call for 1 month – Daily running duties as now
MICE Shift personnelMICE Shift personnel Currently 2 shifters for ~8 hour shiftCurrently 2 shifters for ~8 hour shift Prefer to move to 1 MICE shifterPrefer to move to 1 MICE shifter
Current on-call experts – BLOC, SOC, TROCCurrent on-call experts – BLOC, SOC, TROC New system-specific experts – 3 per system – not New system-specific experts – 3 per system – not
necessarily current MICEnecessarily current MICE Magnet & cryo experts – maintenance of magnet Magnet & cryo experts – maintenance of magnet
systems, manage cryogens and vac pumping rigs & systems, manage cryogens and vac pumping rigs & instrumentation, cooling and powering expert, quench instrumentation, cooling and powering expert, quench behavior expertbehavior expert
LH2 expert – control engineering knowledge, safety LH2 expert – control engineering knowledge, safety proceduresprocedures
24Coney - CM34 - Oct 2012
Step IV OperationsStep IV Operations Given current MICE schedule for Step Given current MICE schedule for Step
IV equipment and installation…IV equipment and installation… We We mustmust be prepared for 24/7 running be prepared for 24/7 running
in order to get Step IV data before ISIS in order to get Step IV data before ISIS shutdown in August 2014shutdown in August 2014
Can our systems handle this?Can our systems handle this? Target ok? Magnets ok?Target ok? Magnets ok? DAQ ok? Controls ok? Datamover ok?DAQ ok? Controls ok? Datamover ok? Shift coverage ok?Shift coverage ok?
Another reason to limit Hall access.Another reason to limit Hall access.