DG 259 Fiber Optic Network Design Guidelines Issue No. 8

Contract: W-Issue No. 8
Deputy Vice President and Chief Communications Engineer
Issue Record
Formal Review
Intermediate Review
1 8/13/2001 Initial issue Baum 2 12/31/2006 Issue 2 Lin 3 12/31/2008 Issue 3 Lin 4 7/19/2013 Issue 4, added PSLAN, clarified RAACS Lin 5 9/2/2014 Issue 5, added COE Lin 6 11/26/2014 Issue 6, added seismic rqts, cabinet high
temp alarms Lin
7 4/22/2016 Issue 7, clarified PSLAN, limit vent grating use
Lin
Lin
Vice President and Deputy Chief Engineer
Fiber Optic Network Design Guidelines Issue 8 DG259 Page 2
Table of Contents 1 PURPOSE ................................................................................................................................. 3 2 REFERENCES............................................................................................................................ 3 3 TERMS AND DEFINITIONS ........................................................................................................... 4 4 NETWORK OVERVIEW ................................................................................................................ 6
4.5.1 Passenger Station LAN (PSLAN) ................................................................................14 4.5.2 Network Timing Protocol ..............................................................................................17 4.5.3 IP Addresses ...............................................................................................................18
1 Purpose This document has been prepared to provide guidance to engineers and designers to assure a uniform and consistent approach in the preparation of project design. The guideline is applicable to all NYCT Capital Program projects designed by in-house staff or by consultants. This document provides general design guidelines for fiber optic and network systems and equipment under the purview of Communications Engineering. The fiber optic and network systems are comprised of multiple technology layer networks, and are partitioned into multiple domains. This document provides an overview of the existing SONET/ATM network, and provides guidelines to the designer in supporting different design tasks to be performed under specific projects. See DG 250 (Communications), DG 258 (Traction Power), DG 302 (Fan Plants), and DG 303 (Pump Rooms) for details of specific applications that will use the network to carry information from remote / field locations to each application’s designated head-end location. During design, the following areas shall be considered:
• User needs/requirements (type and number of connections, connection termination end- points, traffic flow and capacity, service level, security requirements)
• Technology capabilities and constraints (reliability and availability) • Physical/space constraints (communications room space allocation, TTB punch-down
block spare, fiber distribution panel spare, cable tray extensions, existing cabinet space for mounting, space for new cabinets)
• Environmental constraints (heat dissipation and buildup, cooling of cabinet, humidity, airborne contaminants, vibration/shock, and abatement during construction) – specific requirements for cooling shall be provided by Mechanical Engineering. Designer shall coordinate with Mechanical Engineers for cooling requirements.
• Equipment power requirements (including source for power, available breaker positions on panel, cable routing, DC versus AC power, battery backup and duration for backup). Designer shall coordinate with Electrical Engineers for breaker and wire size details.
• Maintenance requirements (maintainability, operations complexity, integration with existing systems, remote troubleshooting)
• Training requirements for new system (maintenance training, operator training, administrator training, engineering training)
For any design, designer shall consider not only the capital expense of the project (installed-first cost or CAPEX), but also the recurring operational expense as a result of particular design decisions (on-going operational cost or OPEX). Designer shall consider both CAPEX and OPEX as factors in arriving at a final design, and attempt to minimize the total project life-cycle cost. 2 References The following documents serve as guidelines during the design and construction phases of projects.
• ANSI standards on SONET • Telcordia generic requirements on SONET, ATM, NEBS, Synchronization • IEEE standards on Ethernet • IETF RFCs on TCP/IP, routing protocols • IETF RFC 2544, “Benchmarking Methodology for Network Interconnect Devices”
• NYCTA Specification “General Provisions and Definitions”, TGN, Issue 7, November 19, 1999.
• NYCTA Specification “Wire and Cable Specifications: Coaxial and Antenna Cables”, RG Specification, Issue 5, February 2011.
• NYCTA specification “Wire and Cable Specifications: Telecommunications Wire and Cable”, TC specification, Issue 7, May 2009.
• NYCTA Specification “Data Cable Installation Guidelines”, TC Install, April 2002. • NYCTA Specification “Data Cable Testing Guidelines”, TC Test, April 2002. • NYCTA specification “”Fiber Optic Cables and Cords”, TO specification, Issue 3, October
2007. • NYCTA Specification “Fiber Optic Cable Placing Specification and Procedures”, TO
Placing, Issue 1, December 2000. • NYCTA Specification “Fiber Optic Testing Specification and Procedures”, TO Testing,
Issue 1, December 2000. • NYCT Guidelines for Single Mode Fiber Optic Cable Acceptance Test Procedures. • NYCT Guidelines for Central Office and Remote Telecommunications Equipment
grounding Requirements and Method, TG 601. • NYCT Guidelines for Telecommunication Network Equipment Installation, TG 801. • NYCTA Guidelines for Communications Rooms, Version 4.02 • DG100, Primary Design Documents Policy • DG101, Document Control of Contract Drawings • DG102, Contract Specifications • DG103, Design Calculations • DG105, Design Submissions • DG106, Drawing Control Procedure • DG107, Design Drawings • DG108, CADD Standard • DG120, Drafting Guidelines • DG121, Design Coordination • DG250, Communications Engineering Design Criteria and Guidelines • DG256, Power Substations Engineering Design Criteria and Guidelines • DG258, SCADA System Engineering Design Criteria and Guidelines • DG302, Critical Fan Plant Facilities Design Guidelines • DG303, Pump rooms Design Guidelines • DG312, Flood Resiliency Design Guidelines
3 Terms and Definitions
2W or 4W/TO 2/4 Wire / Transmit Only 4W/EM 4 Wire / Ear and Mouth ANC Access Node Cabinet ATM Asynchronous Transfer Mode ATS Automatic Train Supervision AWG American Wire Gauge BER Bit Error Rate BLSR Bidirectional Line Switched Ring C3 Command, Control, Communications Center CAI Customer Assistance Intercom CAT Category CBTC Communications Based Train Control
CCTV Closed Circuit TV CIR Committed Information Rate CIS Customer Information Screen COE Connection Oriented Ethernet CR Communications Room DDS Digital Data System DRI Dual-Node Ring Interconnections DS-n Digital Signal level n DWDM Dense Wavelength Division Multiplexing EBCS Emergency Booth Communications System EDR Electrical Distribution Room EIR Excess Information Rate EMC Electromagnetic Compatibility EMI Electromagnetic Interference EMS Element Management System EPL Ethernet Private Line EVC Ethernet Virtual Circuit FAT Factory Acceptance Testing FDB Fiber Distribution Box FDP Fiber Distribution Panel FO Fiber Optic FXO Foreign Exchange Office FXS Foreign Exchange Station HDSL High Bit-Rate Digital Subscriber Line HP Help Point IP Internet Protocol LAN Local Area Network MLT Multi-Link Trunking MMF Multi-Mode Fiber MOW Maintenance of Way NEBS Network Equipment Building System NEMA National Electrical Manufacturer’s Association NMS Network Management System OC-n Optical Carrier level n OCUDP Office Channel Unit Data Port OSPF Open Shortest Path First PA Public Address PBX Private Branch Exchange PCC Power Control Center PLAR Private Line Automatic Ringdown PLC Programmable Logic Controller PNNI Private Network Node Interface PoE Power over Ethernet PSLAN Passenger Station LAN PVC Permanent Virtual Circuit PWE Pseudowire Emulation RAACS Remote Auxiliary Alarm and Control System RCC Rail Control Center RIP Routing Information Protocol RMS Remote Monitoring System RSTP Rapid Spanning Tree Protocol SAT Site Acceptance Testing
SCADA Supervisory Control and Data Acquisition SCS Structured Cabling System ScTP Screen Twisted Pair SMF Single Mode Fiber SONET Synchronous Optical Network SPVC Soft Permanent Virtual Circuit STP Shielded Twisted Pair STP Spanning Tree Protocol STS-n Synchronous Transport Signal level n SVC Switched Virtual Circuit TDM Time Division Multiplex TDS Train Dispatch System TRS Train Radio System TTB Telephone Terminal Board UPSR Unidirectional Path Switched Ring UTP Unshielded Twisted Pair VLAN Virtual LAN VC Virtual Circuit VP Virtual Path VRRP Virtual Router Redundancy Protocol
4 Network Overview 4.1 DWDM Network The NYCT broadband communications network is a multi-layer/hierarchical network using DWDM, SONET, ATM and IP technologies. The DWDM network provides the ability to multiplex multiple high-speed signals onto a single fiber strand. DWDM can be viewed as a technology that provides virtualization of the fiber optic cable infrastructure. DWDM technology is used primarily to help alleviate fiber shortages by carrying multiple optical signals over a single fiber strand. Combinations of OC-48, OC-192, 10GBASE-X, and 1000BASE-X signals can be multiplexed and transported over different wavelengths. DWDM has two levels of multiplexing:
• Electrical multiplexing allows for grouping lower-rate signals onto a higher rate signal before carrying over a single wavelength, e.g., four (4) OC-48 signals combined onto a single OC-192 signal transported on a single wavelength
• Optical multiplexing allows for carrying multiple wavelengths over a single fiber strand. The current DWDM network provides for both electrical and optical multiplexing. This increases the fiber capacity by multiplexing multiple signals onto a higher rate signal, and then multiplexes this signal (along with other higher-rate signals) onto a fiber strand. No protection mechanism is currently enabled on NYCT’s DWDM system. Enabling the protection mechanisms is a matter of configuring the equipment (the current equipment supports protection switching 1+1 and 1:1). The DWDM equipment, which is the Fujitsu FLASHWAVE 7420, is installed at all sites where SONET equipment is installed. The FLASHWAVE 7420 supports multiple types of interfaces, including OC-3, OC-12, OC-48, OC-192, 1000BASE-X, 10GBASE-X. Designers may need to specify new modules to support the required interfaces. Designer shall coordinate with Operations Department for availability of spare interfaces or the need to procure new modules.
PBX7 MR120
Figure 1. Overall DWDM Configuration
The DWDM is physically set up to follow the SONET ring configuration; however DWDM is not using any ring-type protection mechanism, only the physical connectivity is in ring configuration. DWDM circuits (wavelengths) are configured as point-to-point circuits between adjacent equipment. Different wavelengths may be terminated at different sites. The wavelength that supports the SONET signal is terminated at each site. The wavelength that supports inter- agency electronic security communications (IESS project) is optically passed-through multiple sites and terminates at two main sites. Optical signal amplification may be required at intermediate sites in order to allow the pass-through wavelengths to travel across multiple sites. Amplifier placement is determined based on link engineering/calculation. Link engineering is required to be performed for future projects where DWDM wavelength is needed to support transport of signals. 4.2 SONET Network The SONET network provides a high-speed, highly robust infrastructure from which traffic from various applications may be transported. The SONET network is divided in two categories: the two (2) core SONET rings are designed to support the interconnection of major locations within the Communications Network (e.g., control centers, PBXs, etc.), while the four (4) edge SONET rings are designed to interconnect outlying regions to the major Control/Data Centers. To support the required level of reliability, the Core SONET Rings use the OC-48 2-fiber BLSR protection switching mechanism (recovery time between 50 ms to 100 ms), while the Edge SONET Rings use the OC-48 UPSR protection switching mechanism (recovery time less than 50 ms). The SONET Network Elements (NEs), which are the Fujitsu FLASHWAVE 4500, are deployed at select strategic locations across the Communications Network, to support the following TDM and data interfaces into the network: 10/100BASE-T, 1000BASE-X, DS1, DS3, STS-1, OC-3, OC-12, and OC-48. Presently, only TDM interfaces (OC-3, OC-48) are used on the FLASHWAVE 4500. All other interface types (including data interfaces) are envisioned to be supported by the ATM switch, Layer 2/3 switch, and channel bank equipment at each location. SONET provides direct interfaces only to the ATM switch. Designers may need to specify new modules to support the required interfaces. Designer shall coordinate with Operations Department for availability of spare interfaces or the need to procure new modules.
OC-48 OC-12
Basic SONET OC-3
DRI site
Figure 2. Overall SONET Ring Configuration
These SONET Rings are connected together via Dual-Node Ring Interconnections (DRI). DRI provides an automated mechanism for self-recovery in the event of failures of major connectivity sites. In addition to the DRI mechanism, the SONET Rings are also connected together via a shared-site interconnection (SSI). SSI provides a manual mechanism for recovery in the event of major failures of multiple connectivity sites. 4.3 Connection Oriented Ethernet (COE) Network The Connection Oriented Ethernet (COE) network provides a high capacity highly robust infrastructure from which traffic from various applications may be transported. COE network provides better efficiency and utilization for variable rate data traffic. The COE technology is based on Ethernet, with extensions that add deterministic and OAM features. These include ability to specify
• Specific routes for traffic to follow • Specific routes to re-route around failures • Fast failure detection and failure isolation • Fast recovery against failures
The COE network core sites mirror the (24) SONET sites. These sites are connected together using fiber strands from the same set of fiber cables used for SONET and DWDM networks.
RCC MR259
Figure 3. Overall COE Point-to-Point Configuration
The COE branch loops mirror the ATM branch loops, using fiber strands from the same set of fiber cables used for ATM branch loop networks. The main difference between COE and SONET/ATM is that the COE network is a flat point-to- point architecture with protection taking place on individual Ethernet Virtual Circuit (EVC). Each EVC is assigned a specified bandwidth profile based on the connection type and bandwidth requirements. All application traffic that traverses the core, branch and data center EVC’s will share the assigned bandwidth profile and be assigned to one of three classes: real-time, priority data and best effort. For stations, there will be three bandwidth profiles depending on the complexity of the station. In all cases, application traffic will be limited by the COE QOS policy in the bandwidth profile.
• Data Center to Data Center EVC bandwidth is 500M CIR/1500M EIR • Data Center to Core EVC bandwidth is 500M CIR/500M EIR • Core to Branch EVC bandwidth is 50M CIR/50M EIR (Small Station) • Core to Branch EVC bandwidth is 100M CIR/100M EIR (Medium Station) • Core to Branch EVC bandwidth is 150M CIR/150M EIR (Large Station) • NYCT to NYPD EVC bandwidth 150M CIR/150M EIR • Inter-Agency (C3) EVC bandwidth is 600M CIR/400M EIR
There is no longer the concept of ring protection (such as BLSR). The physical connection remains the same. The protection mechanism is based on ITU-T Recommendation G.8031 and Y.1731 for OAM. The COE network core sites are linked by Fujitsu Flashwave 9500 Reconfigurable Optical Add Drop Multiplexer (ROADM), using two wavelengths per link, each carrying 10 Gbps Ethernet
signal. Although there is a 10G interface used on the 9500 and CDS COE devices the bandwidth is actually lower per the EVC rules above. As capacity needs increase, additional wavelengths and EVC’s may be added. The FW9500 also has the ability to support 100Gbps. The COE branch loops are linked directly by Fujitsu Flashwave CDS, via fiber, carrying 10 Gbps Ethernet signal.
Figure 4. Core and Branch Loop COE Network
4.4 ATM Network An ATM network is built on top of this SONET infrastructure to support transport of traffic generated from each of the 468 passenger stations, as well as PBXs, data centers, and other transit facilities (bus depots, train yards, administrative buildings). The ATM network is designed as two PNNI signaling hierarchies. Lower level hierarchy is made up of passenger station ATMs (referred to as ATM Branch Loops). Top level hierarchy is made up of the network ATM switches (24 SONET locations). This hierarchical design reduces the ATM routing table to a manageable size. The ATM equipment supports the following TDM and data interfaces: DS1, DS3, STS-1, OC-3, OC-12, OC-481, 10/100BASE-T and 1000BASE-X Ethernet; however it is only configured with DS1 (CES and IMA), OC-3, OC-12 and 10/100BASE-T interfaces. In conjunction with channel banks, the following additional interfaces are supported: DS0s (2W-TO, 4W-TO, 4W-EM, OCUDP). In conjunction with the Layer 2 / Layer 3 switches, the ATM and IP equipment at each location will support 10/100BASE-T and 1000BASE-X interfaces.
1 Note that OC-3, OC-12 and OC-48 are carrying concatenated signals STS-3c, STS-12c and STS-48c, respectively.
DWDM
L2/L3
149th St. [390]
Times Sq [11]
AO8 (5/4), AO9 (13/11)
AO35 (6/5)
AO53
†
† AO18 (7/1) AO42 (6/4) AO59 (11/6) AO60 (9/2) AO65 (5/4) AO66 (6/1)
(1/5)
(1/5)
(2/9)
(4/14)
(2/8)
(3/14)
(4/18)
(3/12)
(1/0)
(1/3)
(1/3)
(1/5)
(3/11)
(x/y) x= Number of locations served y = Bandwidth in Mbps
AO29 (6/1) AO63 (10/2)
Station Branch Loops
Station Network Switches
Cisco MGX 8950
Cisco MGX 8950
Peer groups
Figure 7. ATM Network Logical Connectivity
The ATM network uses PVCs, SPVCs and SVCs to carry applications traffic, using PNNI to support dynamic connection setup and restoration. ATM restoration is on a per VP or per VC basis. In ideal case in a single hierarchy configuration, recovery takes ~ 50 – 100 ms per circuit/path (e.g., segment with 10 circuits may take 10 X 50 ms = 500 ms or ½ second to recover all circuits). However, NYCT’s ATM network is designed as two-level hierarchy (All network switches in one hierarchy, and passenger station switches in second hierarchy). Two- level hierarchy significantly increases recovery time, and may be up to several minutes for recovery. The channel banks are the Rad MegaPlex 2100H and ADTRAN Total Access 1500. Each station has one or more of both of these channel banks. The Rad channel bank is used for voice
over IP (VOIP) service, while the ADTRAN channel bank is used for traditional telephone service, including FXS/FXO, 2W/TO, 4W/TO, PLAR, DDS/OCUDP. 4.5 IP Network An IP layer network is built on top of the ATM and COE network to support the transport and communications needs for data applications. An IP router is installed at each of the 468 locations, configured with layer 3 functionality. The Cisco IE-5000, Cisco 3850, Cisco 3750, Alcatel-Lucent Omniswitch 6855 (OS6855), Omniswitch 6900 (OS6900) and the Omniswitch 10000 (OS10k) will be installed at various sites to provide layer 3 functions. The existing Nortel Ethernet Switch Router 1648T (which served the original layer 3 function) will provide layer 2 switching function at Passenger Stations that have migrated to COE. Nortel 1648T will provide layer 3 function for stations not yet migrated to COE. At Network Switch locations, a Nortel 1648T provides local service interface, while the Alcatel- Lucent OS6900 or OS10k provides the OSPF routing function. Cisco MGX 8950’s RPM-XF routing card provides routing function for stations not yet migrated to COE. The two OSPF networks (SONET-based and COE-based) interconnect together at several of the 24-core sites. The interconnect takes place between the RPM-XF module and the OS6900 or OS10k. At building facilities, the Nortel 5510, 5520, and Alcatel-Lucent Omniswitch 6850E (OS6850E) are also used. Nortel 5520/5530 and OS6850E are used when Power over Ethernet (PoE) support is required. Each Passenger Station’s switch/router is logically connected directly to two Network Switch location’s switches/routers. On the ATM network, this logical connection is supported via two diverse unprotected ATM PVCs. On the COE network, this logical connection is supported via two diverse protected (using G.8031 protection scheme) COE EVCs. Layer 3 redundancy is provided via OSPF routing protocol with custom configurations for protocol timing (hello timer of 3 seconds and dead interval timer of 9 seconds – this provides total IP reconvergence of approximately 9 seconds within each area). The IP equipment supports the following data interfaces into the network: 10/100BASE-T (copper and fiber-based interfaces) and 1000BASE- X Ethernet (copper and fiber-based interfaces).
SACNS IP COE IP
OS6900/10k
Passenger Station Router (eg. OS6855)
OSPF area A OSPF area B OSPF area 100+A OSPF area 100+B
ATM PVC link via ATM OC-12 loop (link unprotected)
ATM PVC link via SONET STS-3c (link unprotected)
COE EVC link via ROADM (link protected by G.8031)
COE EVC link via COE 10Gig loop (link protected by G.8031)
Direct 1Gig link
The IP network is divided into multiple areas. Major sites (those corresponding to the SONET locations) are grouped into the backbone area (Area 0). All other IP routers installed at passenger stations are grouped into separate areas based on the branch loop configuration, e.g., all IP routers that are part of a branch loop will be configured on the same area. All areas have a direct connection to the backbone area. The SACNS routed network is set up to support OSPF stub area at the ATM branch loop level. The COE-based routed network is set up to support OSPF not-so-stubby-area (NSSA) at the COE loop level. Application data traffic typically connects to two switches/routers for redundancy. The switch/router may be configured using Virtual Router Redundancy Protocol (VRRP), multi-link trunking (MLT) for layer 3 redundancy, Spanning Tree Protocol (STP), Rapid STP (RSTP), Multiple Spanning Tree Protocol (MSTP) for layer 2 redundancies. However, there are certain limitations to this configuration, e.g., to support VRRP both VRRP routers must belong to the same OSPF area. Designer shall validate network configuration for each project. 4.5.1 Passenger Station LAN (PSLAN) PSLAN is a local area network installed in a station. This LAN provides IP connection from devices directly onto the network. It is connected in a ring topology using fiber optic cables. PSLAN is created by installing Access Nodes (AN) throughout the station area. The Ethernet switches inside the AN are connected at optical gigabit rate in a ring topology. Because of physical limitations of stations, physical ring topologies are sometimes not possible; in such instances a collapsed ring topology would be used, i.e., diverse paths that form the ring may use fiber strands from the same cable. The ANs are installed to allow future planned devices to be within Ethernet standard without a PSLAN infrastructure, each application device farther than 300’ from the IP network would require dedicated fiber cable installation homerun to the CR. A single AN may not be sufficient to handle connection for all field devices. In such cases, an Application Node (ApN) will need to be installed next to the AN to provide additional port capacity. Additional ApNs will daisy-chain from the AN via the fiber port in a linear topology without forming a ring and may connect to other ApNs if there is need for additional port capacity. Limit up to (3) ApNs per AN. Each subway station contains a local IP router connected to the IP network. This local IP router provides the network access to all applications within the station, and supports the creation of separate VLANs for application traffic. In general, almost all new applications currently being installed or planned for installation are based on IP protocol. The PSLAN is a Layer 2 only network, i.e., no IP routing protocol, and supports Layer 2 features such as VLAN, STP or RSTP, and Power over Ethernet (POE). PSLAN equipment is high- temperature-rated (ruggedized) to operate in the uncontrolled environment of the station. Because this equipment is installed in the station, ANs and ApNs must be NEMA-4X rated to prevent water penetration from wash-down of station. All components in the cabinet must be high-temperature rated. The PSLAN Layer 2 switched network connects to a Layer 2/3 switch, typically in the Comm. Room. The Layer 2/3 switch provides Layer 3 routing function for the PSLAN.
PSLAN (Layer 2 only)
Layer 2 switch in AN
1000BASE-LX, VLAN
Layer 2 switch in Application Cabinet
Figure 9. PSLAN Logical Connectivity
Due to space constraints and to minimize initial capital cost for PSLAN installation, certain locations may not have adequate access ports on the PSLAN switch. Designers will need to survey each PSLAN configuration, determine whether or not access ports are sufficient. If not sufficient, Designers will need to extend the PSLAN AN to include additional ApNs for field device interconnect. Note that the power supply system inside the PSLAN AN and ApN is to be used strictly for network purpose, e.g., to power additional devices that may be added inside the AN or to power the Help Point units. 4.5.1.1 PSLAN Virtual LANs NYCT has assigned applications to specific VLANs as illustrated in the below table. This VLAN list is a snapshot of what is used at time of writing. Designers shall check with Network Operations Department for the latest VLAN assignments.
TYPE VLAN ID Description
Network Mgmt 11 For management of devices within CRs PACIS Phase 2 13 For PACIS phase 2 system under IRT stations
MTA-IT Layer 2 (temp) 15 Used to provide MTA-IT with a temporary connection across the Operational network
NYPD Video 22 For NYPD cameras located at a lateral site
CBDS 24 For devices serving computer based dispatch system
Help Point (HP) 25 For devices in Help Point units Do Not Use 26 Do not use. Shown for backwards compatibility Elevator Liftnet 27 For elevator LiftNet Time Clock & NTP 28 For NTP timing data and clocks with IP interface
EMD Video Retrieval 29 For devices in Central sites to access video for retrieval purposes
PACIS Phase 1 & Hybrid 30 For PACIS phase 1 system under B division stations (note devices are mainly on the enterprise network)
Mirage NAC 31 For management of NAC OTG 35 For OTG data (note most OTGs are on cellular
network)
Platform Edge Video 37 For video used by train conductors to view platform edge during train door closing
Do Not Use 41 Do not use. Shown for backwards compatibility ESS VOIP 42 For ESS intercoms IDS/IAC 43 For IDS/IAC devices IP Video 44 For IP cameras and video servers Fan Plant 52 For all fan plant application devices Signals 53 For all signals application devices Pump Room/Deepwell 54 For all pump room/deepwell application devices Substation/SCADA 56 For all substation application devices
BMS 61 For building management system devices (such as HVAC, utility metering)
Do Not Use 62 Do not use. Shown for backwards compatibility EA Recorder Server 63 For emergency alarm servers in Central sites
EA Recorder Network 64 For management of switches that supports EA servers
PBX/VOIP 71 For general purpose IP telephones Fire Alarm System (FAS) 72 For fire alarm controller EBCS 73 For emergency booth communications system FCALAN/NFPS 74 For New Fare Payment System devices PSLAN 81 For PSLAN switches and RMS Do Not Use 82 Do not use. Shown for backwards compatibility Do Not Use 83 Do not use. Shown for backwards compatibility
Radio 85 For devices support radio system (such as VHF simulcast)
Beacon 86 For IP devices supporting the Beacon CIS signage
Network Mgmt - Remote 91 Extension of VLAN 11 to remote sites (such as non-Signal switches installed in Relay Room)
Fan Plant Mgmt 92 For management of switches in fan plant
Signals Mgmt 93 For management of switches in Relay Room used to support Signals traffic (e.g., PLC)
Pump Room/Deepwell Mgmt 94 For management of switches in pump room and deepwell
Substation Mgmt 96 For management of switches in substation Do Not Use 141 Do not use. Shown for backwards compatibility
ESS VOIP EXT 142 For extension of VLAN42, only to be used when VLAN42 is already assigned but insufficient
IDS/IAC EXT 143 For extension of VLAN43, only to be used when VLAN43 is already assigned but insufficient
IP Video EXT 144 For extension of VLAN44, only to be used when VLAN44 is already assigned but insufficient
Signals EXT (include static route) 153 For extension of VLAN 53 and any static route
interconnect with Signal’s router
PSLAN EXT 181 For extension of VLAN81, only to be used when VLAN81 is already assigned but insufficient
MTA-IT Layer 2 Extension 600-699
For extension of MTA-IT network across Operational network, e.g., multiple layer 2 VLANs from stations consolidating in 130 LIV will require unique VLANs per station. One VLAN is assigned
for each station
NYPD Video 700-799
For NYPD cameras that are not in the same location as a lateral site, but will terminate at a lateral site. One VLAN is assigned for each station
Temporary Use 950-989 For IP devices installed for pilot/temporary test
Isolated Network 990-999 For IP devices that do not require network access, but needs local communications within a station. IP address assignment is still required
Table 1. PSLAN VLAN Assignments 4.5.2 Network Timing Protocol NYCT uses Network Time Protocol (NTP) to synchronize the time of a computer client or server to another server or reference time source. NTP provides client time accuracies within milliseconds on LANs, tens of milliseconds on WANs relative to a primary server synchronized to Universal Coordinated Time (UTC) via GPS. In the NYCT network three core sites have dedicated NTP servers (TP5K) synced to the GPS constellation. NYCT utilizes a hierarchical approach to serve time from the core to the passenger station (PS) routers and end devices. The TP5K are directly connected to the Core routers who act as client to receive time and servers to PS routers. PS routers and client end devices are configured to receive time from the primary and secondary core site routers as depicted below. For exact IP address of the NTP servers to utilize on a project the contractor shall make an official request and will test connectivity to ensure accurate time. In addition to the above core NTP servers, there are NYCT NTP Servers located at the PCC and 2020 Broadway (Brooklyn) available for time synchronization via the IP Network.
Figure 10. NTP Architecture
4.5.3 IP Addresses Each station has a reserved IP address range that is further subnetted into smaller ranges to support the various applications in the station. The IP address range is controlled by the EMD operational group. During design where Designer provides full design, Designer will submit Layer 1 and Layer 2 diagrams and a spreadsheet specifying the device count for each application/VLAN. Based on this, and coordinating with EMD and MOW, an IP address block will be assigned, with the following understanding:
• The IP address block is valid for the quantity of devices in the full design. If device count changes in any way during construction, the IP address block may be re-assigned. This may result in Contractor needing to re-configure devices with new/updated IP addresses.
• If a project with assigned IP address block is put on hold, Designer will notify EMD and MOW of such and have the IP address block released for other uses.
For projects that do not provide a full design, IP addresses are assigned to devices during construction stage, and requires Contractor to provide adequate information to allow the assignment of IP addresses. The process for the assignment of IP addresses is as follows:
• Contractor conducts site survey and submits detailed network drawings with a list of field devices and type, quantity, network switches and servers configuration to NYCT for review and approval
• Upon approval of these drawings, Contractor submits a request for IP allocation (e.g., IP address space allocation, quantity of devices for each requested subnet, VLAN ID, routing protocol) for all IP devices indicated on the network design drawings and the provided list. Note that all equipment that requires an IP assignment must be submitted together in a single request. This means that the Contractor’s Network Engineer or Systems Integration Engineer must coordinate among all different sub-Contractor groups to provide this consolidated request
• Contractor provides the IP list (in Excel spreadsheet format) for all IP enabled devices to NYCT for approval before proceeding with system configuration activities. A sample IP spreadsheet may be forwarded to the Contractor for their use as a sample
• Upon approval of the IP list, Contractor can then start to configure all subsystems and networking devices. Contractor shall submit for record configuration files for switches.
4.6 Remote Environmental Monitoring / Telemetry All network equipment, communications rooms and data cabinets are monitored for status and environmental conditions. The network status monitoring includes monitoring of minor, major and critical alarms (this is a general alarm that indicates one (or more) minor, major or critical alarm was triggered – it does not provide specific alarm detail). The environmental conditions monitoring include monitoring of water within the communications room, cabinet door opening indicator, room temperature and humidity sensors. The DPS NetGuardian device supports both network status and environmental condition monitoring in the communications rooms, with discrete and analog inputs, and control outputs. For environmental monitoring at right-of-way facilities (such as fan plants and substations), DPS NetGuardian DIN (smaller version of NetGuardian) is used. Monitoring information from each of the communications rooms are sent to a head-end server T/MON NOC (previously known as the IAM). 4.7 Element and Network Management Systems Operations, administration and management of the network is accomplished via various Element Management Systems (EMS) and an overall Network Management System (NMS). Each equipment reports to its respective EMS system; at the same time, equipment status is also reported to the NMS system. The current NMS system is the Teoco Netrac. The following EMSs are used:
• SONET, DWDM, COE: Fujitsu NetSmart • ATM: Seabridge Surpass hiD, Cisco WAN Manager, Marconi ServiceOn • IP: Nortel Optivity, HP NNMi, Cisco CiscoWorks LMS, Solarwinds, ALU OmniVista • Channel banks: Rad RADView • Telemetry: DPS IAM and T/MON • NMS: Teoco Netrac • Zhone CB (VHF Radio): Megasys
The broadband network will serve different applications needs, including, but are not limited to: telephony (including administrative telephones, critical/emergency telephones, HPI/SPI intercoms, etc.), SCADA monitoring data (from power substations, fan plants, signal relay rooms, pump rooms), PA/CIS, CCTV, train dispatch and radio systems, ATS and CBTC applications. The SONET/ATM system will enable, among other benefits, the phasing out of proprietary, legacy, and asynchronous network installations whose equipment have become manufacture
discontinued and thus would not allow future network augmentation or equipment life-cycle replacement. 5 General Network Design Guidelines The following guidelines shall be followed for any work affecting the communications network.
• Any network design or augmentation shall take into consideration both current and future application capacity needs, as well as other application-specific requirements, such as delay sensitivity and recovery requirements. It shall allow for easily expanding to support additional application requirements (e.g., by adding interface modules). Designer shall request network spare capacity from User Department. This shall include spare interface/port/slot capacity.
• At the beginning of project, Designer shall obtain user requirements for the application that is to be transported over the network. Information shall include:
o Types of services to be transported over the network; this includes description of whether it is TDM service (telephone, TDS, TRS, EBCS, mass-call), legacy and low-speed data service (serial data interface from RTU, telemetry), or high-speed data service (video, database replication)
o Service level requirements; this includes description of the capacity required for the service, burstiness of traffic, delay sensitivity (or latency) of traffic, disruption sensitivity of traffic. The following table provides a brief synopsis of types of services that can be supported by different layer networks:
Burstiness Latency Disruption
COE Constant and variable
ATM Constant and variable
Low Varies from 50 ms upwards depending on # of circuits and hierarchy
Ethernet Variable Varies from 10 ms upwards depending on congestion
Up to 40 sec (STP), less than 5 sec (RSTP)
IP Variable Varies from 10 ms upwards depending on congestion
Up to 9 sec (within area), up to 60 sec (multiple areas)
Table 2. Service Level Requirements
o Number and type of circuits for each application; this includes information on the number of end-devices that is to be installed under the project, the types of output interfaces supported by the devices
o End-points for the circuits; this includes information on the specific locations where the end-devices will be installed, any aggregation points prior to connecting to the network, the central/hand-off point at one (or more) central facility
o Specific locations where end-devices and network drop-off points will need to be installed
o Any security sensitivity requirements (e.g., need to install firewalls and network access control devices)
• During the Master Plan phase of a project, the Designer shall develop a high level design that includes cabinets, enclosures, and equipment. When reserving space for this communication equipment, the Designer shall submit the Communication Room Space
Request to reserve space for the communication equipment. The Master Plan sign-off shall be contingent on the following:
o If space is approved then MOW clearinghouse will return an approved submittal which will be included in the Master Plan by the DM. The CPM Design Manager, Designer, and MOW shall agree to the most appropriate resolution for mounting equipment
o If no space is available in the communication room, alternate space should be proposed by the MOW clearinghouse (i.e., new communication room proposed, cabinet/equipment on station platforms), in conjunction with DOS Stations, CPM Program Coordination and Communications Engineering
o If space outside of a communication room is proposed then the DM will include this proposal as part of the project scope
o If equipment cannot be determined during the Master Plan stage, the DM shall clearly include the exception in the Master Plan to indicate risk for possible space issues during the design stage
o If any other engineering discipline requires the placement of equipment (i.e., HVAC, electrical panels) inside a communication room it is required to go through the same process as outlined above during the Master Plan phase
• Designer shall follow additional resiliency requirements for projects categorized under Flood Resiliency as per Flood Design Guideline DG312:
o Cabinets, enclosures and pull-boxes shall be rated for NEMA-4X Cabinet air conditioner and fan shall not be used – all equipment shall be
rated for high temperature operation o Conduits shall be rated for wet environments o All room penetrations shall be sealed with water-tight sleeves o Conduits and pullboxes shall be installed as high as possible o All conduits shall be sealed to handle up to 22 ft. water-head pressure o Mounting should be above flood level; if this is not possible, clearly identify this
and ensure all components of design are rated for water immersion o Cables installed on messengers inside tunnels shall be mounted as high as
possible, as permitted by field conditions and in accordance with LLLE (Limiting Line of Line Equipment) requirements
o Equipment shall not be placed within or through vent bays or attach to vent gratings. Vent gratings must be cleared of all obstructions in order to allow installation of Mechanical Closure Devices and Deployable Vent Covers. Waivers need to be requested from MOW Chief Engineering Officer for any design where this is unavoidable.
• For right-of-way applications that require network redundancy, Designer shall include dual-homed connection for single remote site, and linear connection for multiple remote sites. This configuration shall be used in design for Signal Relay Rooms, Central Instrument Rooms, Power Substations, Fan/Vent Plants, yards, depots, or other applications where network redundant connection is a requirement.
o Signals systems shall be kept separate from other right-of-way facilities when
considering fiber cable sharing. That is, for signals projects install FO cable for signals systems only, while for other projects (fan plant, substations, etc.) install FO cable to support communications of all other right-of-way facilities along path of the FO cable (except signals facilities). This FO cable is considered right-of- way application structured cable.
o For deepwells and pumprooms, network redundancy shall be designed on a
case-by-base basis. In general deepwell and pumprooms may be designed with
a single FO connection to the nearest CR. In all cases, deepwell and pumprooms shall be designed with a secondary connection (over copper) using the Remote Auxiliary Alarm and Control System (RAACS), to be used for backup alarms. A deepwell project with multiple deepwells distributed along the subway tunnel right-of-way (spanning multiple stations) would require FO cable to be installed from station-to-station to support the localized deepwell communications. FO cable installation considerations also need to account for local communications of cluster of deepwells without the need to transport traffic over SONET/ATM network.
Primary CR Secondary CR
Additional
LAN #1 LAN #2
Optional link for redundancy
Additional
Sites
Dynamic routing protocol
RAACS for deepwell & pumprooms
Figure 11. Right-of-Way Applications Network Connectivity
o PSLAN Access Node (AN) and Application Node (ApN)s should be distributed
throughout station to provide maximum coverage for IP services as required under each project. Note that certain projects only require design of a partial PSLAN that covers only areas of the station necessary to support the project’s application devices.
Designer shall work with User Department to specify the station areas that are required to be covered.
Designer must ensure that measurements are performed such that any point of station has access to the cabinet without violating the distance limit. Because of cable slacks, routing around obstructions, vertical runs of cable/conduit, rule of thumb is to use 200’ as the point-to-point distance limit from the covered area to the PSLAN cabinet.
Designer must work with MOW Engineering, EMD, and Stations Clearinghouse to identify an approved cabinet mount location and mount type. Possible mount types include against a wall (either horizontal or vertical as approved by Stations Clearinghouse), on a column web or flange, and on a round column; other mount types may be possible as agreed with User department.
PSLAN AN/ApN is not to be installed inside a room/office/tower, but in the station environment. If network access point needs to be installed inside a room/office/tower, Designer will specify the wall-mount data cabinet as used for right-of-way facilities.
Power for PSLAN AN/ApN will be from Comm Room’s Benning power plant. A subpanel will need to be installed to provide the needed breaker positions for 120VAC, with lock-out/tag-out for each breaker. Each AN/ApN will be wired directly to a breaker. For the purpose of wire size, maximum allowed voltage drop of 10% may be used for sizing conductors for the AN/ApN – subject to minimum voltage being within the operating range of the equipment power supply, with minimum wire size no smaller than #12AWG.
• Note for stations where the PSLAN AN/ApN were installed under the Transit Wireless initiative, the AN/ApN includes a UPS/battery module. Extensions to AN/ApN network will stay consistent with equipment used, i.e., new AN/ApN will also include the UPS/battery module, and powered from local/nearby power source.
For locations with existing PSLAN: • Designer will survey existing AN or ApN to verify the current port
utilization. • Reach out to MOW and EMD to obtain information on already
reserved ports, and take these into account when determining free ports.
• Request approval from EMD for free ports to be used by project. Each Access Node in a Control Area should have the following ports
reserved: • (1) port for Booth/EBCS system • (1) port for NFPS/FCALAN system • (1) port for IP-based CIS system • (1) port on each switch for maintenance connection • (1) port on each switch for interconnection • (1) port for RMS • Optional (1) port for UPS (for the Access Node with built-in UPS) • (1) port reserved • NOTE this means that AN in control area with 16 total ports, 9
ports are reserved (10 ports reserved when UPS version used).
The remaining ports may be used for any application (7 usable ports – 6 usable when UPS version used)
• If project needs more than the above port quantity, then furnish and install additional ApNs that will connect (via fiber optic port) to the closest AN. Multiple ApNs will connect (via fiber optic port) to other ApNs using daisy chain/linear topology.
Each Access Node in a platform area should have the following ports reserved:
• (1) port for HP • (1) port for IP-based CIS system • (1) port on each switch for maintenance connection • (1) port on each switch for interconnection • (1) port for RMS • Optional (1) port for UPS (for the Access Node with built-in UPS) • (1) port reserved • NOTE this means that AN in platform area with 16 total ports, 8
ports are reserved (9 ports reserved when UPS version used). The remaining ports may be used for any application (8 usable ports – 7 usable when UPS version used)
Each AN in a combined control+platform area should have the following ports reserved:
• (1) port for Booth/EBCS system • (1) port for NFPS/FCALAN system • (1) port for HP • (1) port for IP-based CIS system • (1) port on each switch for maintenance connection • (1) port on each switch for interconnection • (1) port for RMS • Optional (1) port for UPS (for the Access Node with built-in UPS) • (1) reserved • NOTE this means that AN in control+platform area with 16 total
ports, 10 ports are reserved (11 ports reserved when UPS version used). The remaining ports may be used for any application (6 usable ports – 5 usable when UPS version used
Each ApN should have the following ports reserved: • (1) fiber port on the 1st ApN for interconnection with the upstream
AN (on subsequent downstream ApNs, the fiber port will connect back to the upstream ApN)
• (1) fiber port on the 1st ApN for interconnection with the downstream ApN (if additional ApN needed)
• (1) port on each switch for maintenance connection • (1) port on each switch for interconnection
• (1) port for RMS • Optional (1) port for UPS (for the Access Node with built-in UPS) • NOTE this means that ApN with 16 total ports, 5 ports are
reserved (6 ports reserved when UPS version used). The remaining ports may be used for any application (11 usable ports – 10 usable when UPS version used
• If project needs more than the above port quantity, then furnish and install additional ApNs that will connect to the closest AN
o NYPD may need to install a “lateral” to our station. A lateral is defined as an
NYPD network point-of-presence within NYCT’s facility. This lateral is typically a fiber cable installed from NYPD’s infrastructure into a station’s CR. Within the CR, NYPD requires rack space in an existing NYCT rack for: FDP, WAN switch, Firewall, LAN switch – total of 5RU of contiguous rack space. NYPD performs their own cable pull (including any conduit installs), cable termination, and equipment installation. Designer shall coordinate with NYPD to identify space needs, ensure AC power outlets are available for NYPD equipment, and notify EMD of such installations. NYCT’s IP network shall not be connected with NYPD’s IP network. Any interconnection shall be performed via a NYCT firewall to NYPD firewall direct connect, with prior approval from EMD and MTA-IT Security.
• Coordinate with the Operations Department to identify current availability for any
connection that requires support from existing network, and plan for equipment installation accordingly. For example, if a DDS circuit is required to transport serial data to an existing network location, then reservation of existing spare copper cable and reservation of slots/ports on existing equipment will be required. Note that Operations Department may not reserve copper circuits during the design phase. As such, copper circuit reservation will need to be made during construction. See Appendix B for example form to be filled out to reserve circuits.
• When furnishing and installing new equipment, preference shall be given to using existing product lines. This will help to reduce the impact on maintenance of equipment and operational costs. New equipment shall be used when the existing equipment is near end-of-life or a newer product with similar or better features are available and required; in such cases, integration of product to existing network management system, equipment spares, and training of operations personnel shall be included. Concurrence from both User and Operations Department is needed when specifying new equipment.
• The solution shall allow for ease of integration into the existing element and network management system, and support ease of management. Integration into existing element management systems may not be appropriate if a different manufacturer or later generation equipment is to be furnished. Integration into existing element and network management system may require updating and upgrading of the management system hardware platform to an up-to-date release to support new features/functions of the equipment.
o For all new equipment that cannot be supported by the existing element management system, a new Element Management System shall be furnished and installed to manage all the installed equipment. An EMS server shall be furnished and installed in one of the central data center’s server room plus an additional server at a backup data center’s server room, to be specified by NYCT operations department. The equipment monitoring and configuration capability shall also be integrated into the existing NMS system.
• Software development for integration of new equipment (or new release of existing product line) into existing network management system.
• In any design effort, care shall be taken to minimize disruption to existing traffic before, during and after installation and commissioning of the network. Preference shall be given to a design that minimizes disruption. Disruption is any interruption or required re-routing of existing traffic flow from its original intended route. NYCT in-house User and Operations department shall be involved with the coordination of this work.
• In any design effort, care shall be taken to minimize train service disruptions. Train service disruption includes any cable conduit routing that crosses one or more tracks. Preference shall be given to a design that minimizes disruption.
• The network design shall be based on using open standards for all components, e.g., device level specifications, interface level specifications, and protocol level specifications.
• The network design shall be based on using feature capabilities supported by existing product lines, and not on feature capabilities not yet supported by orderable products.
• In certain cases where installation of new equipment may be in existing cabinets or bays, existing equipment in the cabinet or bay shall not be disturbed. Close coordination with the appropriate operations department is required. If possible, consider the option of furnishing and installing a new cabinet/bay to install the new equipment. Design shall not require the relocation of existing/mounted equipment.
• Load calculations shall be performed for any equipment to be furnished and installed. Existing power plant may need to be augmented. See DG-250 for power plant design guidelines.
o Backup time required by the new service shall be considered when calculating the battery requirements. Designer shall request backup time from the user department.
o Considerations shall be given to the maximum power supplied to each Communications Room. If this is exceeded, then additional power feeds from EDRs may be needed. Coordination with Electrical/Power designer is required.
o Surveys may be needed to evaluate existing power plant utilization. See DG-250 for additional design guidelines.
• At locations where there is no Remote Monitoring Systems (RMS), a complete and operational RMS system to support various status and alarms of both the room and equipment shall be included. Section 19CR contains requirements for the RMS. The RMS design shall include the RMS equipment, alarm distribution block, and all sensors and contacts to support the required alarm points.
• Operations department shall be consulted throughout the design phase. For data equipment, Contractor will need to request IP addresses in order to connect equipment onto the network. This will require a completed LAN design, with information on exact number of hosts per LAN segment, number of VLANs, number of network segments, any specific configurations and protocols (such as priority). Designer shall ensure during construction phase, Contractor submits all necessary submittals to allow for IP address assignment. Below is a sample network drawing that shall be submitted by Contractor, along with approved network design documentation. Number of VLANs shall be kept to a minimum and shall be used only when necessary.
Figure 12. Example Layer 2 Network Configuration Required for IP Assignment
For projects that deal with new structures (such as new Seismic Design for nonstructural components (COMMUNICATIONS SYSTEMS)
1. Factors that govern the incorporation of seismic design requirements for communications systems are summarized as follows:
a. Structural Engineering’s determination of the Seismic Design Category per the Building Code of New York State (BCNYS).
b. Design Manager’s determination of the Occupancy Category of the facility/structure in coordination with the user department /facility’s owner and NYCT Code Compliance group.
c. Design Manager’s determination of the Importance Factor of communications system components in coordination with the user department /facility’s owner as per BCNYS and its reference standard ASCE 7 and based on: Occupancy Category of the facility/structure The communications system components that are required to function for
life-safety purposes after an earthquake. The communications system components that are needed for continued
operation of the facility after an earthquake. 2. If determined by the Design Manager that seismic design requirements for
communications system components is required, coordinate with the Design Manager to have the Seismic Design for communications system components prepared by either:
a. The contractor, during construction phase, provided that communications engineering design team prepares and includes into contract design documents performance requirements that require the contractor to be fully responsible for the seismic design and calculations, seismic hardware specifications and the seismic installation details for conduit and enclosures. OR
b. NYCT Structural Engineering in-House design in coordination with all involved design disciplines.
3. Seismic design submittals during construction phase, per item 2.a/2.b above will be reviewed by NYCT Structural Engineering in-House design team.
• When creating a design package, Designer shall always start with the master template
specifications and drawings. The following list provides a guide to the types of drawings that shall be produced:
o System Block Diagrams Signal Flow Block Diagram Overall Communications Systems Block Diagram Head-end/Central Communications Systems Block Diagram Remote Site Communications Systems Block Diagram Individual System Block Diagrams (e.g., Telephone, PA/CIS, CCTV)
o Schematic Diagrams and Key Plans Network Cable Layout Key Plan Structure Cable / Riser Diagram Fiber Distribution Panel Interconnection Diagram AC and DC Power Single Line Diagram Typical Equipment Installation Detail Individual System Layouts - consoles, workstations, desks, etc.
o Site Layouts Conduit and Cable Layout within Stations and between Stations (showing
ductbanks, messengers, manholes, conduits, splices, protection / arch bars). Obtain approval from station clearinghouse for any conduit installation within a station.
• The M:\stations-updated\ folder on the CAD network drive contains the most recent station drawings
Field Device / Equipment Layouts (speakers, cameras, communications outlets, etc.)
o Room Layouts Individual rooms: room layout of existing and new equipment footprints,
wall-mounted equipment, ground bars, conduit penetrations, cable trays, transition from conduit to cable trays. Request room layouts from MOW CR Clearinghouse. Submit the CR Clearinghouse Request Form to obtain approval from CR clearinghouse for any new equipment to be installed in CR.
FDP, TTB, Power Plant, Individual Systems, etc. o Equipment Layouts / Configurations
Power Plant Configurations Equipment Cabinet Layouts FDP / TTB Configurations
o Miscellaneous General Scope of Work Drawing General Notes Symbols and Abbreviations Relevant Standards drawings (including both “W” and “E” standard
drawings) • The I:\standards\comm folder on the CAD network drive contains
the most recent communications standards “W” drawings
• The I:\standards\electrical folder on the CAD network drive contains the most recent electrical standards “E” drawings (e.g., E-2041 on protection/arch bars, E-2-2- on conduit grounding)
o As-built drawings As-builts are included in design package as supplementary drawings for
information only. These are provided to guide the Contractor on types of as-built drawings required to be submitted.
o Standard drawing title block Standard title block is included in design package as a supplementary
drawing for information only. This is provided to guide the Contractor on the title block to use for all submittals
• Designer shall follow relevant Design Guidelines for creating specifications and drawings, such as DG 102, 103, 107 and 120.
6 Guidelines for Environmental Conditioning Networking equipment has a wide range of environments into which they may be installed and operated. In general, the network equipment to be furnished and installed does not have equivalent “industrial class” ratings, i.e., within the NYCT environment the equipment would require a controlled environment such as a shielded, air-conditioned cabinet. As such, the environmental conditions shall be well understood to be able to specify what necessary add-on is needed to meet the equipment’s requirements. Information such as temperature, humidity, EMI/EMC, airborne contaminant/steel dust, and vibration/shock needs to be specified in order to make Contractor aware of the issues. In general, a subway station will need to consider all these issues (especially temperature and humidity), while an elevated station may have lower requirement for temperature and humidity (but higher requirement for vibration/shock). All equipment to be furnished and installed within NYCT station environments shall have NEBS 3 rating; exceptions may be made with approval from operations department or if non-compliant equipment is already approved for use within NYCT property. All equipment shall be installed in environmentally conditioned cabinets.
• If the Communications Room is not environmentally controlled, then a sealed cabinet with cabinet air-conditioning shall be used to maintain temperature and prevent airborne contaminant.
o NEMA 12 rated cabinet may be used for communications rooms, relay rooms, and substations.
o NEMA 4X rated cabinet is required for pump rooms, deepwells and fan plants (for corrosion susceptibility).
• If the Communications Room is environmentally controlled, then a cabinet with fan units for air-flow circulation may be used to maintain air circulation.
• Open racks may be specified in environmentally conditioned rooms within building facilities (e.g., control centers, PBXs).
• If only ruggedized equipment is used inside a cabinet (, i.e., an equipment that can operate at -40°C minimum and operate at 75°C maximum), then cabinet air-conditioning is not required. Ensure all components are rated to the higher temperature inside the cabinet.
Sizing of air conditioners and cabinet fans are to be handled by Mechanical Engineers. Designer shall coordinate with Mechanical Engineers to provide air conditioner and cabinet fan sizing details. In case where the communications room does not have adequate space to install conditioned cabinet, hardened equipment is required to be specified, mounted into wall-mounted enclosures. NEMA rating for enclosure shall follow above guideline. Environmental testing is required to ensure the installed system meets the environment in which it is installed. This testing shall include cabinet testing, equipment testing, and cabling infrastructure testing. 7 Guidelines for Equipment and Interconnections Network design will involve new equipment to be specified, augmenting existing equipment to support new applications, and interconnection of the equipment to various other network components. These may include interconnection of the new equipment under a particular project, interconnection of the new equipment to the existing SONET/ATM equipment, and interconnection of the new equipment with the applications for which the network will be furnished and installed. Surveys shall be performed to determine the availability of network services, including spare interface ports, spare modules/slots, spare rack space, and spare room space for potential equipment and cabinet installations. Specification of the types of equipment will depend on the existing condition at the communications room, the existing equipment with which to interconnect (e.g., possible interface options), the capabilities of the servers and management systems at the data centers (e.g., management protocol to use), and the requirements for the applications (e.g., service level, delay characteristics, recovery requirements, bandwidth requirements). In addition, other considerations that need to be taken into account include any potential network bottlenecks that may be introduced. For systems projects where multiple applications require network support and significant data traffic need to be carried across the network, network simulation shall be performed to ensure no bottlenecks exist, using OPNET network simulation toolkit. 7.1 Specifying New Equipment In specifying new equipment, the following information shall be obtained prior to design:
• Evaluation of user requirements against equipment capabilities and capacity • Evaluation of feature and function of possible equipment to be used, including evaluation
of capacity; Contract specification shall include relevant features and functions required to support each project
• Ability of existing management system to manage the equipment; Contract specification shall include relevant requirements regarding work to be performed on the existing management system
• Evaluation of power requirements, heat dissipation, equipment physical dimensions, specific modules (e.g., power, control, interface); Contract specification shall include relevant requirements regarding power system
• Evaluation of interoperability of equipment with existing equipment; Contract specification shall include relevant requirements regarding interoperability testing
Equipment make and model shall be specified for projects that extend the current network and requires tight integration with existing network and management system. If existing equipment cannot support the user needs or the existing equipment is deemed to become discontinued by the manufacturer by the time of contract completion, then specification shall be performance- based. For performance-based specifications, requirements for the equipment shall explicitly require interoperability and integration with the existing system, and ability to integrate with existing network management system. Designer shall include requirements for NMS integration work. Training shall be included for installation, configuration / provisioning, monitoring, troubleshooting / repair of the equipment. Operations department shall be consulted on training requirements, e.g., number of personnel to be trained, type of training sessions to be held, number of classes required per training topic. Ensure that standard features and functions (as well as any explicit custom features) of the equipment are specified in the documentation. Network equipment maintained by TIS Security are not to be installed in Communications Rooms controlled by EMD, and vice versa. Designer will coordinate with the responsible group for room and space allocation for equipment under Contract. If an exception is made for co- location of equipment, ensure a signed memorandum is obtained from both groups agreeing to this exception. Network Access Control (NAC) security device is pre-approved to be installed in Communications Rooms. 7.2 Augmenting Existing Equipment For existing equipment, survey equipment to ensure that equipment is configured as per record drawings. If it is necessary to augment the existing equipment, consideration shall be given to ensure that any new modules to be added to the existing equipment will be compatible. This will require current version or release information of existing equipment, and compatibility of new modules from equipment manufacturer. If it is determined (in coordination with operations department) that existing modules have spare ports that may be used, request and receive confirmation (in writing) from operations department to reserve any slot/port to support the project. See Appendix B for sample form to be submitted for request. In case where existing equipment does not have spare ports on existing modules, but has spare slots for addition of new modules, survey to determine specific parts number for specific module needed for the project. Request and receive confirmation (in writing) from operations department to reserve empty slot for new module. See Appendix B for sample form to be submitted for request. Evaluate any need to install new cabling from module to termination panel (whether copper or fiber panel). Use correct rated cables. Consideration shall be given to cable routing from existing equipment to existing or new distribution panels. Design shall not result in fully utilizing all ports/capacity of the equipment. If support of application requires all available spare port/capacity, design shall specify new equipment for spare purposes. Designer shall verify new equipment will fit within the existing cabinet / rack. See guidelines on cabinet / rack below for further details. Comm Room Clearinghouse should be involved in the space planning and approval.
7.3 Guidelines for Specifying Network Interconnection In general, fiber optic network demarcation is either at the application FDP or at a TTB. Communications network design shall include design from the TTB or FDP at the remote site, across the network, and to the corresponding TTB or FDP at a head-end location (such as RCC, PCC or Sands St.).
Applications Applications
Figure 13. Fiber Optic Network Demarcation
Network equipment has various interface types. Designer shall first obtain information from users and application designers on specific requirements for application equipment interface (e.g., FXS telephone, CCTV encoder input via Ethernet). Specific network equipment shall be furnished and installed with the correct type and number of interfaces to terminate the connection. The following types of application traffic are common, as well as their physical interfaces:
• Telephone/voice o FXS (at the remote end – typically faces the application): requires one copper
pair on the CR TTB’s punch-down block; connects to a phone. o FXO (at the central office end – typically faces the PBX): requires one copper
pair on the TTB punch-down block; connects to a PBX. o 2W/TO and 4W/TO (at the remote and central office end): requires one copper
pair (for 2W/TO, or two pairs for 4W/TO) on the TTB punch-down block. o Ethernet interface from IP telephony device (Voice over IP) – this may be from
any IP-based telephone or intercom system • Legacy data
o DDS is the circuit at the end-device interface. o OCU-DP is the circuit at the channel bank interface used to terminate a DDS
circuit: requires two copper pairs for termination (typically RJ48S jack). • CCTV
o CCTV traffic may be carried over the network in two ways (note this assumes that CCTV traffic has been converted to IP): via T1 interface (requires two copper pairs of CAT3 or better; typically RJ48C or RJ48X jack) or via Ethernet interface (requires four copper pairs of CAT5e or better; typically RJ45 jack).
• PLC o PLC traffic may be carried over the network in two ways: via dial-up modem
(4W/TO circuit) or via Ethernet interface. • Workstation/PC
o Ethernet interface to the nearest switch Copper cable connections that are extended to termination points outside of the communications room shall be protected from over-current and over-voltage protection (e.g., using NEBS Level 3 Type 1 and Type 2). Applications that require protection includes any copper-based services, such as DS0, HDSL, or DDS connection, and therefore shall terminate in a protector block prior to connection to network equipment. Intra-building copper cable connections (such as Ethernet connection within underground station) may not require line protection. Fiber cable connections do not require protection. Copper cables that travel across high electric or magnetic field areas shall be shielded from interference. This is especially true for high data traffic as carried over Category cables (such as 10/100/1000BASE-T). As such, designers shall specify use of shielded twisted pair cables in facilities such as power substations, fan plants and relay rooms where cables are exposed outside of a shielded cabinet. Use of shielded twisted pair cable requires the use of shielded connectors and patch panels. Designers shall specify complete shielding solution, as well as grounding solution for the shield to ensure no ground loops are present. Note that for intra- building cabling, if shielded cables are used and properly grounded, then no line protection is necessary. Copper and fiber-cable interconnections are distance-limited by the transmitter’s output power, attenuation and noise across the cable infrastructure, and the receiver’s input sensitivity. Designers shall ensure that for each particular application and the associated media carrying the data, adequate link budget is included in the calculation to ensure acceptable signal availability.
• For existing networks, this includes specifying correct parameter settings for output power and input sensitivity (to be performed during construction in coordination between Contractor and in-house operations).
• For new networks, this includes calculating the overall attenuation of the cable path, and choosing the correct combination of transmitter and receiver pair to allow signal to be received at the receiver end.
7.4 Equipment Installation When adding new modules / cards to existing equipment, Designer shall ensure that modules to be added are compatible with equipment. This includes ensuring new modules would have compatible version or release #. To ensure this, Designer shall request, from the Operating Department, the current version information for existing equipment. If new equipment needs to be installed, surveys shall be performed to determine whether adequate space exists to install new equipment in the existing cabinet. Designer shall also perform heat dissipation calculations to ensure that the additional heat dissipated by the new equipment can be handled by the air conditioners. This shall be done in coordination with in- house operations department. Designer shall coordinate with the Operating Department to reserve space in existing cabinet for new equipment. This shall include reservation of one (or more) fuse/breaker positions for power of the equipment, and reservation of one (or more) positions on alarm and data terminal blocks for equipment connection. If existing data cabinet has no space, heat dissipation exceeds air conditioner’s handling ability, or no data cabinet exists, then surveys shall be performed to determine whether floor space exists to install a new data cabinet. Designer shall consider the overall footprint required for the
cabinet, taking into account the height of the room, placement of the new cabinet, overall door swing for the cabinet, possible accessibility issues for existing cabinets, possible impediments to room air flow, minimum access requirements for aisle space (minimum of 3’ clearance for aisle space). Designer shall submit request to Communications Room Clearinghouse (MOW Engineering SP&I) for cabinet space allocation (see Appendix A for sample form to submit). See section below for guidelines on types of cabinets to use. 7.5 Equipment Testing When installing new modules / cards to existing equipment, Designer shall ensure that modules to be added are compatible with equipment. This includes ensuring new modules would have compatible version or release #. To ensure this, Designer shall request from the Operating Department, the current version information for existing equipment. In lieu of FAT testing, certifications on compatibility and manufacturer testing of the modules are acceptable. Site acceptance testing will be required to ensure that the module does in fact operate with equipment, and that the module is not damaged during shipment or installation. When installing new equipment, equipment shall undergo both FAT testing and site acceptance testing. Specifications shall include detailed items for testing. FAT tests shall include component test, turn-up test, management flow test, equipment interconnection test, protocol test, performance / stress test, failure recovery test, interoperability test with existing equipment, and application interfacing test. For FAT interoperability test, vendor certification on full interoperability with NYCT’s existing equipment is adequate. Site acceptance tests shall include turn-up test, management flow test, BER test, equipment interconnection test, failure recovery test (failure as related to the new install), interoperability test, application interfacing test, alarm propagation test, overall end-to-end network and application tests. 7.6 Equipment Operation Training For new equipment not previously used within NYCT, Designer shall include provisions for training of the maintenance, operation and administration personnel. Training for engineering personnel may also be included. Designer shall request from Operations Department the number of personnel that will need training. Designer shall determine, with input from the Operations Department, the types of classes that will be required, such as:
• Installation training (such as equipment rack and stack, installation of modules, power-up of equipment),
• Operation and maintenance training (such as fault monitoring and reporting, configuration and provisioning of services, performance monitoring, enabling security features),
• Troubleshooting training (such as isolation and localization of failures, tracing of problems, testing of equipment and modules, repair of equipment and modules).
• Management system training (such as setting of alarm thresholds, configuration of alarm traps, root cause analysis methods)
For furnish and install of existing equipment, training shall not be included as part of contract. 8 Guidelines for Cabinet and Bay
Cabinets shall be used in all station Communications Room environments. Cabinets are used to protect equipment against communications room’s harsh environment, e.g., high temperature, high humidity, dripping water, steel dust, and electromagnetic interference. Cabinets may also be used in environmentally conditioned rooms where equipment security is a concern; in this case, cabinet is used primarily to protect unauthorized access to equipment. Racks shall be used in environmentally conditioned communications rooms within administrative buildings. Use of cabinet and racks shall be coordinated with the Operations Department. 8.1 New Cabinet and Rack Types of cabinet to install shall be determined by the room space limitations and types of equipment to be installed in the cabinet. The following guidelines shall be followed for cabinets, and included in each project specification:
• Cabinets shall be sized to allow for equipment mounting up to 23”. Smaller 19” mounting frames may be used where space constraint is a concern. If 19” is used, Designer shall ensure all equipment will fit in the smaller mounting frame.
• Free-standing cabinets may be used where space permits (data centers). This shall allow for both front and rear access for equipment maintenance purposes.
• Where free-standing cabinets are not possible due to space constraints, then a front- access swing-frame or pull-out and pivot, cabinet shall be used.
• Where height is constrained, custom height cabinets shall be specified, as determined by project requirements. Typical heights may be 84” or 72”. Note cabinets shall be specified as 84” unless ceiling height prohibits.
• In certain cases, wall-mounted cabinets may be required. In such cases, a front-access wall-mounted cabinet shall be used, with dual hinges to allow for rear access.
• When sizing cabinets, ensure equipment layout is considered, including equipment protrusions (front and rear), bending radius for power and data cable connection to equipment, turning radius for swing frame and pull-out and pivot frame, required free space to open cabinet doors (take into account protrusion of air conditioners), required working space for cable bend radius during pulls.
• Cabinets shall be EMI shielded. • Cabinets shall be NEMA-12 rated.
o NEMA-4X rating is required where cabinets are subjected to hose-directed water and requires protection against corrosion. Typically cabinets installed in pump rooms, deepwells and fan plants shall have NEMA-4X rating.
o NEMA-3R rating is required if installing batteries inside the cabinet. • Cabinets shall be configured with the following:
o Ground bars, UPS (if protected AC power is unavailable), environment alarms for over-heat sensors and power cut-off relays, door open indicators, enclosure lighting, 120VAC convenience outlets, 120VAC power outlets, fuse / breaker panels, patch panels and terminal blocks (for data lines and voice lines respectively), alarm terminal blocks, door pockets for drawing, door lock with pad-lock handles.
• Cabinets shall have provisions for handling temperatures that exceed maximum equipment operating temperature as follows:
o Cabinet shall be equipped to automatically and without operator intervention trigger the equipment to gracefully shutdown, as per the equipment manufacturer operational feature, in case of: Power loss and depletion of backup power. Inside cabinet temperature reaches preset High Temperature.
o Cabinet shall be equipped to automatically and without operator intervention trigger the equipment to start up once the: Power is restored and Inside cabinet temperature reached preset Operating Temperature.
o Right-of-way data cabinets (e.g. Deep Wells, Pump Rooms, Fan Plants and substations) shall be designed to support ruggedized equipment and be equipped to alarm at preset high temperature. No power shut-down is required at preset high temperature.
• Ensure that the cabinet has adequate space for mounting all equipment. • In environmentally conditioned rooms, fans shall be installed in cabinets. Cabinet shall
be non-insulated. EMI shielding for cabinet shall still be specified. Designer shall coordinate with Mechanical Engineers for requirements on cabinet fans.
• Cabinets shall have installed air conditioners in non-conditioned rooms. Cabinet shall be insulated when installed in underground stations, and non-insulated when installed in above-ground stations. Air conditioners shall be sized to maintain an internal cabinet temperature not to exceed any equipment installed in the cabinet. Designer shall coordinate with Mechanical Engineers for requirements on conditioning of the cabinet.
• Air conditioners shall be connected to backup power, providing as much running time as equipment. Multiple air conditioners may need to be included to ensure equipment is adequately cooled in case of failure of power or of one air conditioner. Designer shall calculate the maximum heat dissipation of the equipment to be installed in a cabinet to determine the number of air conditioners that need to be connected to backup power.
• All openings into the cabinet (e.g., conduit terminations) shall be sealed to prevent dust and water penetration.
Types of rack to install shall be determined by the room space limitations and types of equipment to be installed in the rack. The following guidelines shall be followed for racks:
• Racks shall be sized to allow for equipment mounting up to 23”. Smaller 19” mounting frames may be used where space constraint is a concern. If 19” is used, Designer shall ensure all equipment will fit in the smaller mounting frame.
• Free-standing racks shall be used to allow for both front and rear access for equipment maintenance purposes.
• Where height is constrained, custom height rack shall be specified, as determined by project requirements. Typical heights may be 84” or 72”. Note racks shall be specified as 84” unless ceiling height prohibits.
• Racks shall be configured with the following: o Ground bars, UPS (if protected AC power is unavailable), environment alarms for
over-heat sensors and power cut-off relays, 120VAC convenience outlets, 120VAC power outlets, fuse / breaker panels, patch panels and terminal blocks (for data lines and voice lines respectively), alarm terminal blocks.
• Ensure that the rack has adequate space for mounting all equipment. 8.2 Cabinet and Rack Installation
Designer shall have verified all space requirements for cabinets and racks