Hot Checkout System for Accelerator Operations at JLab Ken
Baggett (Team Leader) Theo Larrieu Ron Lauz Randy Michaud Ryan
Slominski Paul Vasilauskis
Slide 2
HCO Training Session Presentation Objectives: Define hot
checkout (HCO) Define HCO system roles and responsibilities Explain
the use of the new HCO software tool Detail component signoff
requirements and notifications Present checklist requirements for
HCO components Software Demonstration Q&A 2
Slide 3
Hot Check Out (HCO) Definition Hot Check Out: The process by
which all required hardware and software systems (old and new) are
tested for proper functionality and signed off as being ready for
beam operations. 3
Slide 4
Responsibilities Technicians/Engineers Work to verify operation
of components based on HCO checklist requirements. Sets status to
Checked. Group Leaders (or Group Supervisors) Work with PD to
determine and sign-off on system readiness. Sets status to Ready.
Geographic Integrators Responsible for the readiness of their
geographic area. Works with affected groups and Commissioning Team
to assure HCO compliance for beam destinations Management Program
Deputy, Operability, Geographic Integrators, and 12GeV Project
monitor/drive HCO progress based on beam destinations. Operations
Verifies component status based on beam destinations prior to
establishing beam for the Program Deputys accelerator program.
4
Slide 5
Getting to the HCO Tool https://accweb/hco/ Login required to
edit 5
Slide 6
Hierarchy of Categories The status of each node in the tree
structure is determined by its child nodes: The status of each
category is determined by its systems, each system status is
determined by its subsystems, each subsystem status is determined
by its components, and each component status is determined by its
responsible groups. 6
Slide 7
Status Options Not Ready Default setting at the beginning of
the HCO process. No HCO has been performed. Checked Component has
been verified by a Technician/Engineer as fully functional. Ready
Component has been verified by Group Leader as being ready for
operations. Masked - HCO was not completed on the component AND
determined to be not required for the scheduled program. Only
Operations Department Head has authority to Mask. 7
Slide 8
Overall Accelerator Readiness Screen Displaying all systems
status is primarily used by the PD and Operations to view overall
accelerator readiness. However, groups can view their systems
readiness by filtering. 8
Slide 9
Filtering by Destination Filtering helps get the list of items
to a manageable level. Filtering on Beam Destination displays all
the systems requiring HCO for beam operations to the selected
destination. 9
Slide 10
Filtering by Group Used primarily by groups ( Vacuum, RF, DC,
ICN, Alignment, etc.) to determine what components they are
responsible for determining readiness. 10
Slide 11
Filtering by Region and Group By further filtering by Region
the list narrows even more. Very useful when performing HCO in a
particular area of the accelerator. 11
Slide 12
Signoff Screen Must login to change status AND a Group and
Subsystem must be selected List of Subsystems is based on the
selected Group Optional Filtering Regions (several can be selected)
Status (Not Ready, Checked, Ready) Ready Turn (YES indicates
previous Groups have signed off and components Are waiting for your
signoff) Selected Group is outlined with a box to help eliminate
signoff errors 12
Slide 13
Signoff Screen (cont.) Three ways to select items for sign-off
Single click on the item Multiple use Ctrl click All Use symbol
THEN Push Edit Selected Choose desired status Enter a comment Push
Save. 13
Slide 14
Signoff Rules Users can only signoff on one group at a time A
status can only be set to Ready if it has already been set to
Checked Only authorized Group Leaders can set the status to Ready A
comment is required if a status change will cause a cascading
downgrade of other groups readiness 14
Slide 15
Group Order Setting status to Checked can occur in any order
but setting status to Ready MUST be done in the order listed. If a
group downgrades their Ready status for a component then all
downstream groups for that component will revert to Checked. If, in
the table below, MCA1L01B magnet is removed by the Installation
Group AND set to Not Ready, the downstream groups Vacuum,
Alignment, and DC Power will all automatically reset to Checked. At
that point each of the three Groups has the choice to reset their
status to Not Ready or leave it as Checked. 15
Slide 16
Daily e-mail Alerts The HCO system is set to automatically send
Group Leaders a report e-mail at the end of every day. Signoff
Progress Report group component status counts. Signoff Activity
Report activity in the past 24 hours including any cascaded
downgrades. AGAIN: The Group Leader is responsible to determine
what action is necessary for cascaded downgrades (reset their
status to Not Ready or leave it as Checked). Groups Leaders Signoff
Report indicates components waiting for the Group Leader to set to
Ready. 16
Slide 17
Checklists Checklists are NOT procedures but are used to
systematically verify system status. They are NOT a learning tool;
Thats what procedures are. Checklists come in various forms;
Read-Do Read a step and perform an action Do-Confirm Perform a set
of steps then document them as complete Data Taking When steps
require taking data. These checklists become permanent records and
must be recorded as such. Checklists can be developed by anyone but
are not considered Published until signed off by the appropriate
Group Leader. The master checklist is stored, and revisions
tracked, in the HCO system database. 17
Slide 18
Checklist Control Each checklist requires a digital signature
(pushing the sign and publish button) by an authorized Group Leader
before the checklist will be published to general users of the HCO
software. New revisions to an existing checklist require repetition
of the sign & publish process. Group Leaders can make changes
without unpublishing the document; others cannot. The Group Leader
must unpublish it before others can make changes. Then the Group
Leader must publish it again. New revisions must include a brief
comment indicating changes since the prior revision. Prior
revisions will be maintained in the database. The database will be
able to report checklists whose publication date is older than some
TBD threshold. Checklists beyond that date will not be accessible
until they have been reviewed and uploaded to the database. It is
the responsibility of the Group Leaders to maintain completed
checklists. 18
Slide 19
Checklists Tool Checklists are generated using the HCO
Checklist Tool. They can either be built using the tool or placed
there using the copy/cut and paste functions of other documents.
19
Slide 20
Example Checklist 20
Slide 21
Links, Reports, and Help Tabs Links Tab contains links to the
following: ABIL (Accelerator Bypassed Interlocks Log) e-Log
(electronic logbook) ATLis (Accelerator Task List) Others as
necessary Reports Tab can be used by Group Leaders to determine
their systems readiness. Help Tab presently includes contact
information for HCO assistance and a link to this training
document. 21
Slide 22
HCO Process Execution Initiation of the Program Verify ABIL by
System Program Deputy sets the initial beam destination HCO
software will populate a list of components that require HCO based
on the selected destination Group Leaders verify that the lists of
components are accurate and notifies the HCO Team if changes are
necessary (additions and/or deletions) Coordination of group needs
and desires HCO Team (PD) leads the daily 0800 meetings to Refine
the HCO plan for the day Communicate system status Resolve any
group and/or schedule conflicts 22
Slide 23
Schedule Training 8/07/13 (live session) Technicians/Engineers,
Group Leaders, Geographic Integrators, Operators, and Managers
Web-based training available 8/16/13 HCO software goes live 8/19/13
Call for ABIL compliance 8/27/13 - 9/13/13 Group Leaders ensure
remaining interlock bypasses are properly documented AND justified
Hot Checkout begins 9/16/13 PD is point of contact for HCO Daily
HCO meetings begin (0800) Geographical Integrators work with groups
and Commissioning Team to assure HCO compliance for selected beam
destination HCO Team will meet weekly during HCO to determine if
the process is advancing as expected and act accordingly HCO
Complete 10/29/13 23