Liaison Training Seminar

Brought to you by the BaBar/PEP-II Liaison Team

(First Version, Sep 2000 TIMeyer)

(2nd Version, Nov 2001 SSHertzbach)

2

Outline

• Job Description

• Backgrounds

• Your Toolbox– Mice, EPICS, and other clicks

• Some PEP-II information

• Tutorial– Hands-on time (?) at MCC or IR-2

• More details in the online shift manual– http://www.slac.stanford.edu/BFROOT/www/Detector/Backgrounds/liaisons/manual/

3

Being a Good Liaison• READ the online shift manual

– http://www.slac.stanford.edu/BFROOT/www/Detector/Backgrounds/liaisons/manual/

• SIGN UP for Liaison hypernews– http://babar-hn.slac.stanford.edu:5090/HyperNews/get/babar_pepii.html

• SIGN UP for Detector Ops hypernews (or read)– http://babar-hn.slac.stanford.edu:5090/HyperNews/get/DetOper.html

• VISIT two Liaison (2) shifts before your first for hands-on experience (visit IR-2 if needed)

• ATTEND this training workshop

• BE smart while on shift; DON’T pass the buck

• GIVE feedback to other Liaisons & Coordinator

4

Job Description - Intro• Shifts are taken at MCC, Bldg 005

• Attend PEP shift-change meeting 15 min before shift starts, as an observer.

• Answer x5121, call x5255 (IR-2)– Call/page experts when appropriate

• Use your toolbox to stay up-to-date– Make notes in main e-logbook, e.g., beam

conditions, changes, aborts; and diode calibrations

– See Liaison Shift Manual for details

• Bring next shifter up-to-date

5

PEP-II Hierarchy• Typically 3-4 PEP-II operators on a shift

– 1 “EOIC” – engineer operator in charge• This is the person with whom you should speak most often• They determine short term schedule• They know a LOT

– 1 or 2 for PEP-II– 1 for Linac, FFTB, other programs

• 1 Program Deputy (PD) who manages weekly goals– An accelerator physicists, and very friendly– Ask them technical question, but be prepared to

answer some in return!

6

Responsibilities

• Protect BaBar – from user errors

– from background conditions

– by monitoring protection systems & backgrounds

• Maximize integrated luminosity – 15 min. is 3% of shift 95% vs 98% eff.

• Communicate (BaBar PEP-II)

• Help understand and mitigate backgrounds– affects data quality

7

Why Protect BaBar?• Machine backgrounds at PEP-II are serious

– High because of unprecedented currents and luminosities (~90% of L1 rate is background)

• Consist primarily of – lost particles scattered by beam-gas

interactions which then shower into detector– Low angle radiative bhabhas which scatter off

machine elements into detector

• High backgrounds can cause– Detector deadtime / reco ineffeciencies– Radiation damage

8

Maximizing Luminosity• Communication

– always know BaBar needs and desires

– make best use of down time; coordinate BaBar and PEP-II

• Monitor topoffs (use AVLWT in FCT stripchart)– topoff starts at appropriate time?

– PEP hands beams to BaBar promptly?

– BaBar DAQ running?

• Anything else you can think of doing!

AVLWT

Trapped Event:Note elevated

L1 and DT

9

Communication• Liaison is primary source of information

– For BaBar about PEP-II

– For PEP-II about BaBar

• Responsible for – Coordinating downtime and repair work

• Unscheduled and short-term periods with no beam

• BBR or PEP won’t always TELL you; you need to ask

– Answering questions

– Condensing, distilling, and CONTROLLING information

– Writing it down in the e-logbook

• The Liaison makes BaBar and PEP-II a team

10

What PEP-II Will Ask• PEP-II really appreciates Liaison presence

– Backgrounds• Are they okay?• How are they measured?• What is their impact?

– Beam aborts• Who did it?• Why?

– Luminosity• What is it? How is it measured?• How much is it

– BaBar detector and physics• Why can’t we inject?• Anything else…

11

What IR-2 Will Ask

• IR2 wants to know, too– Are backgrounds okay?

– Backgrounds don’t look OK, WHY?

– What “just happened?”

– What’s going on over there?

– What happened to the beams?

– Who aborted the beam and why?

– What is taking so long? You’ll soon discover that true DIPLOMACY is required…answer the important questions without bothering too many people, and keep both sides synchronized and content.

12

Who caused the Beam Abort?• This is an important question whose LACK of an answer is

sure to irritate many, many people. LEARN HOW TO TELL.• Only the SVT diodes will abort the beam from the BaBar side

– Check the SVT Diode EPICS panel for a RED Trip light (more on this later)

• PEP-II keeps a running logfile of Beam Abort Triggers on a monitor on their side of the room

– Ask another Liaison if you don’t know it– Ask operators per “MPS CUD” and/or SLCX3– YES, you’ll have to move from your chair to read that display

• Key to messages you’ll see– SVT_H(L)ABT : SVT diode beam abort– EMC_H(L)INH : EMC_TOTL H(L)ER injection inhibit– BBRHVH(L)INH : BaBar !INJECTABLE injection inhibit– MCC_ABORT : PEP-II op big red button abort– KLYS_xxxxx : RF trip– BOAT : Bad Orbit Abort Trigger

• Don’t BLAME the operators; share, and work on it together

This is the only abort that BaBar can cause, and it should show up in BaBarSVT Background EPICS

13

Backgrounds at BaBar• High rate of backgrounds can threaten

– SVT, EMC, DCH, DRC, IFR, TRG

• Several subsystems have dedicated background monitoring and protection systems– It is the Liaison’s job to care for and feed them or

contact an expert when appropriate

– Different systems sensitive to different things• Sensors are often correlated, but not always!!

• When are backgrounds typically the highest? – Injection (SVT, DCH, IFR ramp down voltages)

– Especially at low currents, early in fill from scratch

14

Backgrounds Differ Daily• Plot shows bkgs

during StableBeams for 1 SVT diode as compared to best bkgs in July,2000

• Note that “ratio” decreased, but a LOT of noise, and even central value got worse in Aug. and Sep.!

Scrubbing of the vacuum August ROD

Vac leak

15

What To Do• What does high background mean?

– “Pain thresholds” are defined for each detector such that running within 5% of them is OKAY

– “Top of the stripchart” = pain threshold

• When backgrounds are high– Verify that it makes sense– Be sure that not just injection backgrounds– Observe what PEP-II operators are doing– If situation persist for more than 90 seconds,

ask operators nicely about it (be armed w/info!)

16

SVT Diodes: Intro• System of 12 PIN diodes & dedicated hardware

– Monitor and control radiation exposure

• 4 MID-plane diodes – Receive highest doses (5-10x top/btm diodes)– Are in a special ABORT circuit which aborts beam

• 8 TOP/BTM diodes (more precise)• Signal in mR/s (~5-20 mR/s)

– Requires exact knowledge of pedestal• Temperature sensitive!• Changes with radiation damage, too!

Biggest source of error is pedestal subtraction!BW-MID Pedestal 1200 nA, and 1 nA = 5 mRad/s

17

SVT Diodes: Summary Panel

Latched trip statusCan be reset under Details Trip History

See this menu for: Calibrations FastHistory Captains Panel

NOTE

18

SVT Diodes: Trips• Front end hardware offers “fast” protection

– Typically visible on stripcharts, but not always

– After “fast” trip, post-mortem buffer read out to disk which will flatline stripchart signal for 60 sec

• EPICS uses stripchart signals to implement 5-minute “Soft Abort” timer (typically 50 mR/s)– Whistles at MCC to warn

operators (unless disabled)

Easily affected by BAD calibration!

BW & FE have higher thresholds.

19

SVT Diodes: Calibration• Two independent systems (HW & SW)

• Both SVT diode calibrations are beam-off leakage current pedestal measurements.– Software: Dose rate (monitoring ) calibration for

• stripcharts, EPICS, and soft abort 5-minute timer• corrected in software (between calibrations) for

temperature & radiation damage, using elaborate model

– Hardware: Abort threshold calibration for• fast hardware aborts • corrected in hardware (between calibrations)

only for temperature, using linear model

20

SVT Diodes: Calibration• Monitoring(dose rate) signals

calibrated automatically, and continuously, if no beam.

• Front end ABORT hardware needs calibration 2 times per day MANUALLY by Liaison w/ NO BEAMS

– Takes 3 - 5 minutes (~1% of shift)

• More dose received, more often we need to calibrate

– Radiation damage HUGE issue

– BW:MID pedestal has gone from 1 nA 1100 nA !!

During daily BaBar Global Calibration is a guaranteed time to do this!!

No Liaison action required

21

SVT Diodes: Tips• Use the SVT stripcharts to watch bkg levels• Keep the SVT Diode EPICS panel open• TOP/BTM diodes MUCH more accurate, although less

sensitive– They are a good way to verify degree of bad bkgs

• All 4 MID diodes occasionally exhibit large oscillations/noise of +/- 100 mR/s for 3-8 min.– Cause known, fix unknown, but typically harmless (doesn’t trip)

• BW:MID and FE:MID have the highest radiation dose• BW:MID diode has unpredictable pedestal

– expect poor quality dose rates

• When in doubt, calibrate the diodes

22

DCH Currents and DCH Trips• DCH total current is the most precise signal for

measuring some types of bkg– Available on BKG stripchart

• Trips occur when HV segment exceeds its power supply current threshold for more than 5 seconds

• DCH trips cause inefficiency (and stress the chamber)

– Can tolerate 3-4 per shift (can, but should not have to)

– Often trips at top of fill; ops can fix this if occurs repeatedly

– Now (Oct. 2001) trips are infrequent!

2000

23

EMC Pin Diode• EMC worries about long-term integrated radiation

dose leading to light losses in crystals• EMC has placed several diode/CsI-crystal sensors on

the endcap• The sum of these sensors’ is used to inhibit injection

and cause EPICS alarms– Referred to as EMC_TOTL– Inhibits injection at ~0.7V– Most sensitive to poor HER injection

• Rare, but can suffer problems– Pedestal drift in which signal is >0.1V during NO BEAMS– Saturation beyond 1.0 V (rare?) is a sign of a system problem– Can bypass only in hardware at IR-2 (see Liaison instructions)

24

EMC Readout Diodes• EMC crystals are read out using biased

photosensitive diodes• Poor injection or high backgrounds can actually

trip the readout diodes’ bias supply– Mild history of power supply spike-trips, too

• Watch stripcharts on BKG panel– Colors carry no alarm value (just match color to plot

trace)

25

DRC Scalers• DRC phototube rates sensitive to LER tunes,

LUMI, and global orbit

• DRC has connected several phototubes to scalers– Some read out in EPICS

– Some read out into MCC

• High DRC ‘tube rates DAQ deadtime– Pain threshold is 200 kHz

– EPICS alarm and stripchart signals

Often excessive during injection and at top of fill

26

DRC CsI Sensors• DRC has added multiple diode/CsI packages on

beamline to understand source and distribution of backgrounds

• In EPICS – DRC SOB/CsI-Mon CsI-Monitors

• Available on BKG stripchart• Not mission critical, but a vast wealth of information

but need to learn how to make best use them.

27

TRG Rates and Deadtime• Fundamentally important

– ~90% of L1 rate is background – High L1 trigger rate incurs deadtime

• With L3 on 32 nodes, DT is OK up to L1 ~2 kHz• Even if DT is OK, want L1 1.5 kHz at lumi ~41033

• Available on FCT stripchart• EPICS alarms on DT• Well correlated with other bkg signals

– If isolated, probably BaBar problem

• FCT LUMI not always well calibrated wrt BaBar LUMI

28

Common Background Issues

None (has never tripped the beam)

NoiseSVT diodes oscillate

LER tunes, skews, collimators

LERHigh DRC rates

WaitDust (photoionized micron scale material drawn electrostatically into HER beam, typically vaporizes and disperses quickly, but can lead to sustained “trapped event”)

SVT spike or step change

Wait and re-calibrate when beams lost

Radiation damageSVT diode drifts upward during course of fill

LER if also high DRC

Trapped event if also high SVT

High L1 rates (chronic)

Wait , or

“shake beams”, or top-off, or

dump beams

Background spike or step change

High L1 rates (sudden)

SolutionCauseProblem

(Thanks to B. Murphy for contributions)

29

Common Beam Loss Reasons

Refill LER beam, usually keeping existing HER fill

1)RF trip (90%)

2)dI/dt

LER beam aborted

Dump LER beam (usually) and re-fill both from scratch

1)RF trip (90%)

2)dI/dt

HER beam aborted

Liaison may be asked to confirm BaBar abort.

Ops reset PR02 latch in SCP.

Liaison may be asked for “post mortem” history

1)BaBar abort (SVT diode, 90%)

2)Vacuum pipe temp

3)Magnet failure

Both beams aborted

SolutionCause*Problem

*Look for cause on white text screen near ceiling above LINAC desk in MCCIf you can’t find it, ask operators to point out “MPS CUD” display (SLCX3)

30

What You MUST Watch• Liaison Alarm Hander

– Launched from BaBarMainBKG

• 3 Liaison-specific stripcharts– Launched from BaBarMainBKG

• BaBarMain– BaBar State

– Injectable status

– Alarms

31

BaBar State Machine• State Machine mediates automated

activity between BaBar and PEP-II

• BaBar accepts requests from PEP-II for injection

• PEP-II declares StableBeams, after which BaBar can go Runnable

32

Your Toolbox• Liaison workspace/environment

• Starting EPICS

• Using EPICS

• Maintaining EPICS

• Using StripTool

• Responding to the Liaison Alarm Handler

33

Liaison Computing• Two computers available, many workspaces

• Sun/Solaris and NCD term, boot from IR-2

• Because Netscape and EPICS often color-conflict, we strongly recommend– Use NCDterm for EPICS only

• Username babarpep, on bbr-con13, password ******

– Use Sun/Solaris to run EPICS and Netscape• Username babarpep, on bbrmcc, password ******

• If you have time to e-mail, and do other work, use additional windows on Sun

34

EPICS• You MUST learn EPICS to survive• EPICS is the universal BaBar detector control

and monitoring environment– Hardware connected to VME single-board computers

(called IOCs)– Each IOC shares its knowledge across IR-2 local

ethernet in virtual EPICS environment– Users can connect with client displays, alarm

handlers, strip charts, UNIX tasks, etc.• Similar to web and browser with “links”

• MCC has been using EPICS longer than BaBar and they’re better at it

35

Starting EPICS• On the NCD (runs only EPICS)

– Open a window on <bbr-con13> as babarpep

– Type “startepics”

• On the Sun (runs EPICS and … )

– Left click the yellowyellow EPICS icon on the bar

– Color map problems can cause EPICS to crash• Close all stripcharts and other applications, restart EPICS• If crashes continue, restart the display manager

• You’ll get these windows:

36

EPICS & Mice• EPICS uses THREE mouse buttons

– LEFT click () (or in Liaison Shift Manual)• Activates buttons, switches, allows typing• Calls FIRST menu option by default

– RIGHT click () (or in Liaison Shift Manual)

• Reveals pull-down menu of sub-panels• Critical to realize this (you might never find

½ the panels in EPICS because you never RIGHT clicked and dragged!)

– CENTER click• Calls dmChan on the EPICS channel, returning a window

with the name of the channel and options for details– Use to learn name of a channel to add to a strip chart– Use to find the alarm limits for the channel– Close the dmChan window when finished with it

37

EPICS & Colors & Buttons

• As might be obvious, a lot of EPICS channels are color-coded to indicate alarm severity– WHITEWHITE = INVALID– RED = MAJOR– YELLOWYELLOW = MINOR– GREEN = NO_ALARM– On indicators, color-coding usually

represents ON/OFF (digital bit) status

• Often HELP button in upper right corner– LEFT click on this will open proprietary browser on

usually helpful and specific html file

38

Vital EPICS Panels• Always start from BaBarMain (ODC)

• SVT Background– Details Trip History

– Expert Controls Calibrate ABORT Thresholds

– Expert Controls AUTO Fast History

– Expert Controls 1-minute Diode Dose Rates *

– History Plots Monitoring Diode Dose Rates

• BKG (main background panel)– Abort Status SVT Diode Soft Abort 5-minute Timer *

• FCT FCTS Trigger Detail

• PEP State Machine

• LUM

39

Starting & Using Stripcharts• StripTool is an application bundled w/ EPICS• Allows scrolling time series plot of up to 10 EPICS

channels simultaneously– History starts accumulating when you START the tool

• Key features– LEFT clicking on side legend changes displayed y-axis

scale to that channel– RIGHT clicking and dragging in plotspace gives menu.

Use Controls Dialog to customize current chart:• change: scales, colors, line width, duration of time displayed;

add variables; etc.

– Coordinates of cursor in plotspace are displayed– Length of scroll-back history defined with the stripchart

40

StripTool Windows

41

Liaison Alarm Handler• Automated method of alerting user when

EPICS channels exceed predefined thresholds– Beeps and flashes

• Launched from BKG panel in EPICS– Click on Liaison ALH button launches a new session

• Liaison ALH is specific to Liaison– Channels in it are relevant

• LEFT click on main window gives detailed tree/hierarchy

42

Alarm Handler Tips• LEFT click on flashing square acknowledges and silences• Can always File Close to dismiss tree window

– This will NOT close ALH top window• Have Occupy-All-Workspaces (iconify & tuck away)• Open only ONE ALH (on one workstation)

Detailed breakdown

Tree hierarchy

Click G for helpClick P for EPICS panelClick Arrow for sub-tree

43

Clipping a Picture into the E-log• From bbrmcc on Sun, LEFT Click xv icon

(From babarpep on NCD <bbr-con13>, open an xterm window, type “xv &”)– RIGHT click in the new xv window that opens– Move the windows around so that they don’t block your view of what you

want to grab– LEFT click “Grab” in the xv controls window– Either

• LEFT click on the window you want to grab, or• Move the cross cursor to the corner of the area you want to grab; MIDDLE

click and drag open a selection rectangle. RELEASE when it is the right size.

– Hit “Save” in the xv controls window. – Enter a filename and save it.– Using the e-logbook running in Netscape on the Sun, start a new entry,

and click “More Lines” and then “Browse.”– Use the menus to select the file you just saved. You may need to edit the

filter or type the filename in by hand, as the directory contents are not refreshed immediately. Be sure you are in the correct directory.

– You’re DONE!

44

xv Windows

45

Some PEP-II Info

• See Liaison Shift Manual: further useful links: – ICHEP 2000 (Osaka) paper for PEP-II, and

1st year experience with backgrounds– BaBar NIM paper for detector overview, more

details on PEP-II and backgrounds, plus trigger– PEP-II regions map linked to drawings– and a few others

46

PEP-II Site Overview

47

PEP-II Regions

eHER

e

LER

48

Web Page to display PEP-II Regions

Web page allows you to click on region to display details of beamline

components in region.

http://www.slac.stanford.edu/accel/pepii/ring/regions_test.html(Steve Meyer)

49

PEP-II IR and BaBar in IR-2(North)(South)

50

PEP-II IR Details

HERFWD ~South

LER BKWD ~North

EAST

WEST

HER 9 GeV

HER 9 GeV

LER 3.1

GeV

LER 3.1

GeV

40 mrad

51

Our Tutorial• We’ll use

– Go to Liaison desk at MCC … or

– use IR-2 workstations to invoke EPICS just as in MCC

• We’ll practice– Finding panels

– Interpreting signals

– Answering questions (a quiz?!)

52

Tutorial Tasks• SSH into <bbr-con13> as babarpep

• Start EPICS– Are we taking data?

– What is the BaBar livetime percentage for this shift?

– How much logged lumi to date (by BaBar)?

– What is EMC_TOTL’s current value?

– Put BaBarMain in all workspaces (make the BaBarMain window appear in all desktop workspaces)

– Is PEP-II “respecting” BaBar? (find the BaBar State Machine page for help)

– What is the LER beam lifetime right now?

• Start the Liaison Alarm Handler– Is there an alarm currently? If so, why?

– How many conditions contribute to a PEP-II alarm?

• Start the 3 Liaison Stripcharts– What is the DCH total current pain threshold?

– What is the current L1 trigger rate?

• SVT Diodes– When were the abort thresholds last calibrated?

– What are the current SVT diode thresholds?

– When was the last dose calibration?

– By how much did the last dose rate calibration adjust the FE:TOP diode’s (stripchart) pedestal?

– Find the 5-minute timer / SoftAbort panel

• For the advanced– When will the HER beam current cause a YELLOW alarm?

– Add a TOP/BTM SVT diode to a stripchart

– What is the FE:TOP diode’s Total Integrated Dose?

– What is the BW:MID diodes’s pedestal current?

53

More Tutorial Questions

• The DRC scalers turn RED and the Alarm Handler beeps. What do you do?

• You look up and see that the beam currents are zero. What do you do?

• PEP-II trips a magnet supply and tells you that it will 20 minutes to bring it back up. What do you do?

54

• Liaison Shift Training web page

– http://www.slac.stanford.edu/BFROOT/www/Detector/Backgrounds/liaisons/manual/training.html

• This Liason Training Seminar

– linked to the above training page ( ppt, ps, pdf, html)

• Liaison Shift manual (also includes other links)

– http://www.slac.stanford.edu/BFROOT/www/Detector/Backgrounds/liaisons/manual/

• BaBar ICHEP 2000 (Osaka) paper (includes PEP-II and backgrounds)

– http://www.slac.stanford.edu/BFROOT/www/doc/public/conf_pubs/2000/babar-conf-0017.pdf

• BaBar 2001 NIM paper (Ch.2: Detector Overview, Ch.3: PEP-II and Backgrounds, Ch.11: Trigger, … )

– http://www.slac.stanford.edu/BFROOT/www/doc/public/journal_pubs/2001/pub_0108/index.html

• SVT Operations Manager's HomePage

– http://www.slac.stanford.edu/BFROOT/www/Detector/SVT/Operations/OpsManager.html

• Mike Wilson’s Home page (SVT Commissioner October 2001)

– http://www.slac.stanford.edu/~mwilson/svtrad/

• Thomas Schietinger's Background Page

– http://www.slac.stanford.edu/~schieti/background/

• Tim Meyer’s SVTRAD page

– http://www.slac.stanford.edu/BFROOT/www/Detector/SVT/Operations/SVTRAD/index.html

• Tim Meyer’s page on “SVTRAD Measurements of Radiation Backgrounds”

– http://www.slac.stanford.edu/~meyertim/svtrad/bkgs/

• SVT Detector Monitoring and Controls (This page may be a bit old, but seems to have lots of useful info)

– http://www.slac.stanford.edu/BFROOT/www/Detector/SVT/Operations/DetectorControl/DetectorControl.html

References