View
225
Download
0
Category
Preview:
Citation preview
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
1/41
Presented by:
Terry Rodgers, CPE, CPMP
Vice President, Sustainable Operations
Primary Integration Solutions, Inc.
trodgers@primaryintegration.com
DATA CENTER O&M BEST
PRACTICES
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
2/41
www.primaryintegration.com
Data Center O&M Best PracticesPerspectives
Designing for Maintainability
Commissioning, Construction, Startup & TestingO&M Staffing and Organization
Operations & Maintenance Processes
Existing Facilities
Agenda
2
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
3/41
www.primaryintegration.com
Data Center O&M Best Practices
As the robustness & complexity of critical facilities has increased for
improved fault-tolerance and concurrent maintenance, the importanceof equally robust O&M practices has become apparent.
As much as 75% or more impact events where critical missions
have been compromised are associated with human Activity.
3
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
4/41
www.primaryintegration.com 4
A&E Programming, engineer & design, submittals & RFIs, CA
6 to 9 months of intense effort then some minor CA support
GC & Contractors Propose, purchase, construct, punch/commission9 to 15 months with minor follow-up effort
Mfgrs & Vendors Submittals/shop dwgs, assemble, startup & commission3 to 6 months and a 12 month warranty + SLAs
CxA Design, construction, startup, acceptance, training & documentation9 to 15 months plus 10 month pre-warranty return effort
Owner Design, construct, commission, transition, sustained operations15 months design/construction, 12 months transition, 24 yearsoperations
Perspectives
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
5/41
www.primaryintegration.com 5
The A&Es, GC & Contractors, Manufacturers and Vendors, andCommissioning Agents completion date (i.e., project close-out)
is the Owners start date (i.e., Day-1 of mission critical operations)
Dont let Sustainable Operations be an after-thought!
Perspectives
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
6/41
www.primaryintegration.com
Designing for Maintainability
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
7/41
www.primaryintegration.com
Include facility O&M considerations in the programming phase
Define facility O&M needs and expectations in the Basis-of-Design
Specify the associated O&M deliverables in the Construction Documents
Designing for Maintainability
7
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
8/41
www.primaryintegration.com
Include O&M considerations in the programming phase via the Owners Project Requirements(OPR) document
O&M Organization, Reporting Structure, & Budget
O&M Staffing (24x7 Coverage, Trades, Quals, etc.)
In-House, Outsourced, SLA, or a combination
Training Program (Day-1, New Hire, & Recurring)
O&M Programs & Processes
CMMS (new or expand existing)
Asset Management System (inventory control)
Configuration Management System
Comprehensive As-Built Documentation & Doc Control
Facilities Management space-planning (offices, command center, shops, etc.)
Designing for Maintainability
8
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
9/41
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
10/41
www.primaryintegration.com
CDs (Drwgs & Specs) should define O&M deliverables as part of the overall construction project
Equipment, System, & Space naming and color conventions
Signage, Bar-coding, RFID, etc.
A vs. B plants (mechanical & electrical) should be easily differentiated
Color code systems (mechanical, electrical, controls, critical vs. non-critical, etc.)
Valve and switch tagging
Drawings should include Valve and Switch schedules (tables)
Specifications should dictate size, material, location, ease-of-view, etc.
Maintenance and access requirements
Shown on drawings, permanently identified in field
Access panels with labels where covered/closed-in
Heavy equipment transport including truck/forklift routes, overhead rigging, pick-points
Emergency lighting (response vs. egress)
Storage and spare parts requirements
Designing for Maintainability
10
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
11/41
www.primaryintegration.com
Most (if not all) Data Centers are designed with the expectation they will be expanded or modifiedduring their lifespan
The design must accommodate this expectation
Appropriate means to isolate infrastructure between production from expansion
Avoid loss of redundancy schemes during construction
These situationsincrease the risk of impact events occurring and are when quality O&M
processes are most important
O&M staff well trained and practiced at reconfiguring systems and isolating equipment and
distribution paths
Updated and accurate as-built docs and operating procedures
Critical work authorization review and approval processes
Routine means to test/verify isolation points (valve leak tests, breaker tests, etc.)
Designing for Maintainability
11
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
12/41
www.primaryintegration.com
Specifications should include defining close-out documentation As-Built (vs. redlines) and exact content and formats
Redlines maintained throughout construction
Formal Record Drawings (As-Built) in hardcopy & electronic
Field verified for accuracy prior to acceptance
O&M Training details, deliverables, content, effort, and quality
Systems Operations & Maintenance Manuals (vs. Manufacturers O&M manuals)
Designing for Maintainability
12
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
13/41
www.primaryintegration.com
Systems Operations & Maintenance Manuals (SOMMs) should be organized by category(HVAC, FP, Security, BMS, etc.), system, & sub-system
Include for each system:
Descriptive narrative
Single-line diagrams (one for each operating mode and/or configuration)
Sequences-of-Operations, setpoints, alarm thresholds, safeties, etc. BMS points lists (DIs, DOs, AIs, AOs, etc.)
Standard Operating Procedures (Normal, Maint., Emergency, Recovery, etc.)
Approved submittals
Commissioning scripts including baseline data
Spare Part Lists (including on-site inventory)
Warranties
Service Level Agreements
Designing for Maintainability
13
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
14/41
www.primaryintegration.com
Specifications should define O&M Training Curriculum, Scope, & Content Venues (classroom, shop, on-the-job)
Instructor qualifications
Duration and times (i.e., shift coverage)
Testing and/or certifications
Safety considerations (OSHA, Arc Flash, PPE, etc.)
Operating procedures (normal, maintenance, emergency, & recovery modes)
Equipment level, Systems level, and Integrated Systems level scenarios
Maintenance procedures (inspections, preventive, predictive, corrective, replacement, etc.)
NOTE: Commissioning affords excellent training opportunities
Designing for Maintainability
14
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
15/41
www.primaryintegration.com
Commissioning, Construction, Startup & Testing
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
16/41
www.primaryintegration.com
The overarching goal is to deliver the site, staff, and processes totally constructed andprovisioned on Day-1 to support 24x7 mission uptime for the life of the facility.
Commissioning Process Design Phase
Commissioning Process Strategy
Use the Commissioning Process to Provision for O&M
Transition from Construction to Production
ASHRAE Guideline 0-2005 The Commissioning Process is an excellent reference and
user-friendly guide geared towards new construction. A sister guideline is currently
being developed for commissioning existing facilities.
Commissioning, Construction, Startup & Testing
16
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
17/41
www.primaryintegration.com
Commissioning process (Cx is not a procedure) Design Phase
Ensure the OPR captures construction QC, performance validation, and facility O&M
considerations
Ensure the BOD complies with the OPR and defines high-level requirements
Ensure the CDs comply with the BOD and include clear and concise specifications
Ensure the CDs capture the requirements and effort associated with Commissioning
(including documentation & training)
Develop the Commissioning Plan to ensure the built product can be operated,
maintained, and tested to validate performance
Commissioning, Construction, Startup & Testing
17
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
18/41
www.primaryintegration.com
Commissioning Process Strategy Identify issues and discrepancies as early as possible so they can be resolved with least
impact on schedule, resources, and costs
Factory Witness Tests identify issues prior to shipment
Field Component Verification identifies issues upon receipt and ensure proper
storage & care prior to installation Systems Construction Verification (progress inspections) identify issues prior to
start-up and functional testing
Site Acceptance Testing identifies issues at the component, equipment, and system
level before testing interfaces
Integrated Systems Testing identifies issues at system interfaces and facility-widelevel prior to going live.
Commissioning, Construction, Startup & Testing
18
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
19/41
www.primaryintegration.com
Use the Commissioning Process to provision for O&M Use the Commissioning design review process to ensure the OPR, BOD, and CDs capture
O&M needs
O&M staff participation in commissioning (FWTs, inspections, and testing) affords in-depth
ops training and establishes contacts with the EOR, mfgrs/vendors, and contractors
Submittals, Field Component Verification, equipment startup forms, and completed test
scripts can capture and document data for input to the CMMS, AMS, & CMS and spare
parts collection & inventory
Functional Tests and Integrated Systems Tests set the original baseline for performance
Commissioning, Construction, Startup & Testing
19
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
20/41
www.primaryintegration.com
Use the Commissioning Process to provision for O&M The Commissioning Process includes ensuring the built facility is thoroughly documented
System Operations & Maintenance Manuals should be developed concurrently with the
construction phase
SOMMs are a compilation of data and documents from:
OPR, BOD, and CDs
Record Drawings (As-Builts)
Approved Submittals
Final Commissioning Report & completed test scripts
Warranties, SLAs, etc.
Commissioning, Construction, Startup & Testing
20
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
21/41
www.primaryintegration.com
Use the Commissioning Process to provision for O&M Training Start with a comprehensive Training Plan
Develop a Training Program that is site and client specific
Embed training in the construction and acceptance testing phases
Utilize the site SOMMs to support and supplement training materials
Design training to match O&M staff position descriptions
Integrate project specific technical training with Corporate policy & procedures training
Training material should be editable so it can be updated and kept current
Training delivery should be recorded for future use (new-hires, remedial training, periodic
re-training, and refresher training prior to infrequently performed activities)
Commissioning, Construction, Startup & Testing
21
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
22/41
www.primaryintegration.com
O&M Staffing and Organization
Remember the following should be addressed during the programming
phase and should influence how the facility is designed and built!
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
23/41
www.primaryintegration.com
Use an Enterprise reporting structure Critical Facilities O&M should be a department within the IT division
Fundamentally eliminate the silos
Recognize the mutual support and dependence of IT and Facilities on each other
Combined goals drives collaboration and planning (sink or swim together)
Eliminate competition for resources (staff & $$$)
Combine NOC and Facilities Command Center operations into an Enterprise
Command & Control Center
Formalize and communicate load vs. capacity tracking, trending, and management
O&M Staffing & Organization
23
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
24/41
www.primaryintegration.com
Staff the O&M department to meet site specific needs Start with a site specific O&M Job Analysis (dont confuse jobs with positions)
Compile list of Jobs (tasks) required to O&M the site infrastructure
Differentiate between operations and maintenance tasks
Develop Job Descriptions that group tasks and assess what requisite skills and
knowledge are required
Identify all that pertain to critical facilities
Consider what qualifications, certifications, or other credentials are necessary where
appropriate
Decide how critical vs. non-critical aspects will be structured Decide how operations vs. maintenance aspects will be structured
O&M Staffing & Organization
24
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
25/41
www.primaryintegration.com
Staff the O&M department to meet site specific needs - continued Develop Position Descriptions
Position Descriptions describe the jobs and associated tasks, duties and
responsibilities expected for a specific staff position
Assemble with the following considerations:
Combined O&M or separate Operations from Maintenance
Jack-of-all-trades or Trade-specific (mechanic, electrician, controls tech, etc.)
True 24x7 coverage or Business Day + On-Call for off hours
Compile a list of duties and responsibilities based on the tasks and associated skills
and knowledge from the Job descriptions Decide how to staff each position (Direct employees, in-house contractors,
SLAs, or a combination)
O&M Staffing & Organization
25
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
26/41
www.primaryintegration.com
Staff the O&M department to meet site specific needs - continued Develop an Organization and Reporting Structure
Enterprise level organization
Facilities O&M level organization
Critical Facilities vs. Non-Critical Facilities
Operations with shift coverage
Maintenance broken down by trades
Electrical (Med Volt, Low Volt, Gensets, UPS, etc.)
Mechanical (Central Plant, HVAC, Chem Treatment, etc)
Monitoring & Controls (BMS, EPMS, calibrations, etc.) Vendors, Contractors, SLAs, etc.
Administration (CMMS, procurement, work approvals, etc.)
O&M Staffing & Organization
26
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
27/41
www.primaryintegration.com
Operations & Maintenance Processes
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
28/41
www.primaryintegration.com
Start with well defined Facilities Management mission Develop a mission or Vision statement
Create a Strategic Plan with well defined goals and tactical objectives
Develop programs and processes first, then supporting policies &
procedures
Operations & Maintenance Processes
28
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
29/41
www.primaryintegration.com
Establish Operations Program and Processes
Work authorization process (routine, PM, emergency, etc.)
Event notification & escalation protocols
Define Operations Key Performance Indicators (KPIs)
What defines operational excellence and how is it measured?
Critical vs. Non-critical boundary and operating protocols
Practice (dry-runs)
Double verification
Mimic boards or other real-time configuration status
Valve and switch position status tags
Lock-out/Tag-out and other Safety Programs
Operations & Maintenance Processes
29
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
30/41
www.primaryintegration.com
Establish Operations Procedures
Specific to each system
Normal, maintenance (configuration), emergency, and recovery procedures
Include pre-requisites and cautions
Work authorization requirements
Safety considerations (lock-out/tag-out, PPE, etc.)
BMS and alarm lock-outs
Verification of status of redundant or backup systems, etc.
Operations & Maintenance Processes
30
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
31/41
www.primaryintegration.com
Establish Operations Procedures - Continued
Provide step-by-step procedure including action to be taken, expected result (alarm,
indication, action, etc.), and means to collect resulting data (actual alarm, reading, action,
etc.).
Embed digital photos of equipment, components, and especially displays and control
panels
Annotate with arrows and relate to specific steps, actions, readings, etc.
Consider using a flowchart format
Include contingency and direction if expected does not match predicted
Include completion/closeout checklist to ensure all aspects of system are restored to
normal operating status (reactivate BMS, remove switch tags, lockouts, notify NOC, etc.)
Operations & Maintenance Processes
31
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
32/41
www.primaryintegration.com
Establish Maintenance Program and Processes
Define Maintenance Key Performance Indicators (KPIs)
What defines maintenance excellence and how is it measured?
Develop programs to manage assets, configurations, and documentation
Asset Management System (AMS)
Bar-code equipment and components
Computerize inventory, location, and who is authorized to pull stock
Include shelf-life, PMs (rotate shafts, etc.), and other storage considerations
Source, vendors, warranties, spec sheets, etc.
Operations & Maintenance Processes
32
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
33/41
www.primaryintegration.com
Establish Maintenance Processes - Continued
Configuration Management System (CMS)
Computerized database of deployed assets & interconnectivity
Sum loads and capacities to track & alarm at defined thresholds
Can be integrated with Work Flow processes so asset deployment, relocation, or
decommissioning automatically generates updates to CMS Can combine both IT, Network, and Facilities infrastructure
Utilize actual real-time metering (BMS, branch-circuit monitoring, Tivoli, HP Openview,
etc.) where available
Use eDiscovery for auto audits where possible
Set thresholds low enough to ensure redundancy and fail-over strategies remain valid
Operations & Maintenance Processes
33
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
34/41
www.primaryintegration.com
Establish Maintenance Processes - Continued
Computerized Maintenance Management System (CMMS)
Implementation tool for the Maintenance Program
Include install date, warranty duration, lifespan/replacement date
Provides input to develop and maintain 5-year & 10-year Capital Plans
Meet or exceed manufacturer recommendations
Stagger dates for work load leveling, seasonal considerations, & freeze windows
Tie to Maintenance Procedures
Separate daily, weekly, monthly, quarterly, etc
Develop logical ID convention that facilitates filtering, searching, and reporting ease
(from design phase!!!)
Operations & Maintenance Processes
34
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
35/41
www.primaryintegration.com
Establish Maintenance Processes - Continued
Types of Maintenance Programs
Corrective run to failure
Preventive routine based on frequency or run-hours
Predictive On-line condition monitoring with thresholds; extrapolate trends to predict
needs (predict failures) Vibration analysis (snap-shot or continuous via multiplex)
Thermography (IR Scans) and temperature sensors
Tribology (lubricant analysis)
Laser Alignment
Computerized Balancing Combination of all above
Input field notes and create equipment histories and review routinely
Operations & Maintenance Processes
35
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
36/41
www.primaryintegration.com
Establish Maintenance Procedures
Specific to each system and equipment
Separate procedure for daily, weekly, monthly, quarterly, etc.
Include pre-requisites and cautions (similar to Ops procedures)
Provide step-by-step procedure (similar to Ops procedures)
Include acceptance criteria, tolerances, settings, etc.
Include access considerations (ladder, confined space, PPE, key#)
Include ability to take field notes, comments, technician ID, etc.
Include completion checklist to ensure all aspects of system are restored to normal
operating status (reactivate BMS, remove switch tags, lockouts, etc)
Operations & Maintenance Processes
36
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
37/41
www.primaryintegration.com
Implement BIMS(Building Information Modeling System)
BIMS products are still relatively new to the industry
BIMS software uses databases to link electronic data, records, and documents including
AutoCAD drawings
BIMS includes virtual 3D models in layers such that you can simulate the construction and
identify conflicts, fouls, and sequence conflicts
BIMS can link AutoCAD objects to associated specs, quantities, suppliers/vendors,
submittals, warranties, inventory, ordering info, maintenance history, and almost any other
electronic record
BIMS can link CDs to CMMS, AMS, CMS, etc.
BIMS implementation, use, and upkeep requires significant pre-planning and resource
allocation
Operations & Maintenance Processes
37
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
38/41
www.primaryintegration.com
Existing Facilities
Build / improve programs to support management, staff, and infrastructure using the
Sustainable Operations Services (SOS) Methodology
The SOS Methodology:
1) Define the requirements and establish acceptance criteria
2) Evaluate / Assess actual conditions and develop a gap report3) Make recommendations to bridge the gap4) Build the needed program(s) based on client/site-specificBest Practices5) Build the associated processes and procedures6) Train management and staff on the program(s), processes, & procedures7) Implement the program(s), processes, and procedures
8) Test, verify, audit the results9) Reassess and apply Continuous Process Improvement
Operations & Maintenance Processes
38
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
39/41
www.primaryintegration.com 39
Data Center O&M Best Practices
As with Critical Facility design, there are few absolute right and wrong
solutions. The O&M Best Practice for a given facility depends upon
many (often conflicting or competing) variables.
What is most important, is plan ahead, select a solution, and then
execute in a high quality manner.
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
40/41
www.primaryintegration.com 40
Sustainable Operations
Data Centers will undergo change during their lifespan (expansions,
infrastructure upgrades, & technology refresh, etc.)
Industry Best Practices will change and evolve
Long-term Sustained Operations require programs with integral
performance measurement and Continuous Process Improvement
8/12/2019 7x24 Nor Cal OM Best Practices Presentation Rev4 9-1-2011
41/41
Recommended