IEEE 1588 PTP on Cisco Nexus 3100 Platform and 9000 … 1588 PTP on Cisco Nexus 3100 Platform and 9000 Series ... 12 Networking Configuration and Best Practices with Cisco Nexus 3100

© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 1 of 30

White Paper

IEEE 1588 PTP on Cisco Nexus 3100 Platform and 9000 Series

Switches


Contents

What You Will Learn ................................................................................................................................................ 3

Introduction .............................................................................................................................................................. 3 The Challenge ....................................................................................................................................................... 3 The Solution .......................................................................................................................................................... 4

PTP Concepts .......................................................................................................................................................... 4 PTP Clock Types .................................................................................................................................................. 4 PTP Topology ....................................................................................................................................................... 4 PTP Algorithm ....................................................................................................................................................... 5 Hardware and Software Timestamping ................................................................................................................. 7

End-to-End Data Center PTP Deployment with Cisco Nexus 3100 Platform and 9000 Series .......................... 8 Topology and Components ................................................................................................................................... 8

Grandmaster Clock .......................................................................................................................................... 9

PTP-Enabled Cisco Nexus 3100 Platform and 9000 Series Data Center Switches ......................................... 9

Nexus 3100 Platform and 9000 Series PTP Architecture ................................................................................... 11 Cisco Nexus 3100 Platform and 9000 Series PTP Packet Walk ......................................................................... 12

Networking Configuration and Best Practices with Cisco Nexus 3100 Platform and 9000 Series ................. 13 Cisco Nexus 3100 Platform and 9000 Series PTP Configuration ....................................................................... 13

Mandatory Configuration ................................................................................................................................ 13

Optional Configuration .................................................................................................................................... 13

Cisco Nexus 3100 Platform and 9000 Series PTP Configuration Validation ....................................................... 14

Data Center PTP Performance with Cisco Nexus 3100 Platform and 9000 Series ........................................... 16 PTP Performance Measurement Definitions and Concepts ................................................................................ 16 Cisco Nexus 3100 Platform and 9000 Series Performance Measurement Methodology and Equipment ........... 18 Cisco Nexus 3164Q PTP Performance ............................................................................................................... 19 Cisco Nexus 9396PX PTP Performance ............................................................................................................. 20 Cisco Nexus 9332PQ PTP Performance ............................................................................................................ 21 Cisco Nexus 9504 with X9636PQ Line Card PTP Performance ......................................................................... 22

Cisco Nexus 9000 Series ERSPAN and PTP Timestamping .............................................................................. 24 Concept of ERSPAN ........................................................................................................................................... 24 ERSPAN Support on the Cisco Nexus 9000 Series ............................................................................................ 26 ERSPAN Packet Format on Cisco Nexus 9000 Series ....................................................................................... 26 ERSPAN with Time-Stamping Configuration on Cisco Nexus 9000 Series ........................................................ 28 ERSPAN Granularity and Marker Packet ............................................................................................................ 29

Conclusion ............................................................................................................................................................. 30

For More Information ............................................................................................................................................. 30


What You Will Learn

This document describes how to enable a highly accurate timing solution that can provide sub-microsecond

accuracy for today’s data center network and financial trading applications by using IEEE 1588-2008 Precision

Time Protocol (PTP) Version 2 (PTPv2) on the Cisco Nexus® 3100 platform switches and Cisco Nexus 9000 Series

Switches. PTP is a distributed timing synchronization protocol with nanosecond accuracy for packet networks.

This document explains the challenges that today’s network and applications are facing and why PTP is needed to

provide sub-microsecond accuracy and how the protocol works. It explains PTP concepts and compares hardware

and software timestamping. It also presents the PTP features that the Cisco Nexus 3100 platform and 9000 Series

support, including PTP timestamping in Encapsulated Remote SPAN (ERSPAN). It provides PTP best practices

using the Cisco Nexus 3100 platform and 9000 Series. The document also describes the PTP accuracy that the

Cisco Nexus 3100 platform and 9000 Series can achieve and explains how the measurements are performed.

Introduction

Accurate and precise timing information is critical to today’s data center networks and financial trading applications.

Network and system administrators need the visibility to see exactly what is happening on the network and when

each event occurs. Application developers and administrators need to correlate various event logs with processes

and applications in a very large and complex computing environment. Compliance and digital forensics also require

that every data transaction be precisely timestamped. A fundamental requirement for today’s data network is a

reliable, accurate, and deployable time-synchronization protocol so that accurate timing information can be

provided to all the relevant elements of the data communications network, including routers, switches, servers, and

applications.

The Challenge

Traditionally, Network Time Protocol (NTP) has been used to provide millisecond-level timing in packet-based

networks. However, millisecond accuracy is no longer adequate because of the reasons mentioned in the previous

section.

Today, to gain visibility into exactly what happens in each process and in the server and switch, organizations need

a timing synchronization protocol that can provide microsecond-level details across the whole network.

The Global Positioning System (GPS) can provide accuracy of +/-100 nanoseconds (ns), but it needs a dedicated

medium to distribute the signal to the end user, which means that every device in the network needs either a BNC

with IRIG-B or some other serial interface to receive the GPS timing information from a separate network. This

requirement makes GPS impossible to deploy in a data center network, in which even the smallest server farm has

hundreds or thousands of servers, as well as routers, switches, and other network elements that don’t have

dedicated special timing protocol interfaces. Practically, organizations need a packet-based solution that is precise

and easy to implement and manage.


The Solution

IEEE 1588-2008 PTPv2 is a timing solution for today’s data center and financial applications. It provides the

following benefits:

● Spatially localized systems with options for larger systems

● Packet-based timing distribution and synchronization

● Nanosecond to sub-microsecond accuracy

● Easy management and maintenance, with few administrative operations

● Capability to manage redundant and fault-tolerant systems

● Low-cost and low-resource-use solution that works well for both high-end and low-end devices

PTP’s accuracy comes from the hardware support for PTP in the switch and server network interface cards (NICs).

It allows the protocol to accurately compensate for message delays and variation across the network. This

hardware support for PTP enables the protocol to achieve nanosecond accuracy.

PTP Concepts

This section presents some of the main concepts of PTP.

PTP Clock Types

The main PTP clock types are summarized here.

● Grandmaster clock (GM): A grandmaster clock is the highest-ranking clock within its PTP domain and is the

primary reference source for all other PTP elements.

● Slave clock: A slave clock receives the time information from a master clock by synchronizing itself with the

master clock. It does not redistribute the time to another clock. In the data center, servers are typically PTP

slave clocks.

● Ordinary clock: An ordinary clock is a PTP clock with a single PTP port. It can be a master clock

(grandmaster) or a slave clock.

● Boundary clock (BC): A boundary clock is the intermediary device between a PTP grandmaster and its PTP

slave clients. It has multiple PTP ports in a domain and maintains the time scale used in the domain.

Different ports on the boundary clock can be master ports or slave ports. The boundary clock terminates the

PTP flow, recovers the clock and timestamp, and regenerates the PTP flow. A slave port recovers the clock

and master ports to regenerate the PTP packets.

Transparent clock (TC): A transparent clock measures the time needed for a PTP event message to transit

the device and then compensates for the packet delay.

PTP Topology

The Best Master Clock Algorithm (BMCA) is used to select the master clock on each link, and it ultimately selects

the grandmaster clock for the whole PTP domain. It runs locally on each port of the ordinary and boundary clocks

to compare the local data sets with the received data from Announce messages to select the best clock on the link.

Figure 1 shows an example of a PTP topology in a data center.


Figure 1. PTP Topology in a Data Center

In an IEEE 1588-2008 PTPv2 network, the master-slave hierarchical clock topology needs to be established in a

PTP domain before clock synchronization occurs. This tree-like topology is similar to the spanning-tree topology,

and the grandmaster clock is the most accurate clock in this hierarchical system, so every PTP slave clock

synchronizes with it. In the PTP network, PTP-enabled ports of the ordinary and boundary clocks examine the

contents of all PTP Announce messages received on the port, and then each port runs an independent PTP-state

machine to determine the port status. Using BMCA, Announce messages, and the data sets associated with the

ordinary or boundary clock, the PTP port can be determined to be in one of the following three states:

● Master: The port is the source of time on the path served by the port. A clock having a master port becomes

the master clock (not the grandmaster) for its downstream PTP nodes.

● Slave: The port synchronizes with the device on the path on the port that is in the master state.

● Passive: The port is not the master on the path, nor does it synchronize with a master. This state prevents

timing loops at the PTP level.

Figure 1 shows that when a network has multiple master clocks - for example, because a new grandmaster clock is

added to the system - eventually only one is selected as the grandmaster clock, and it becomes the root of the

master-slave topology. The port on boundary clock 1 connected to master clock 2 will transit to a passive state,

and no master-slave relationship will be established between those two clocks. The dashed line in the figure

indicates that the master-slave relationship between two boundary clocks is not formed, and that one of the ports is

in a passive state as determined by the port-state machine

PTP Algorithm

At a high-level, PTP exchanges packets that contain a timestamp that represents the current time to which the

clock of the receiving device needs to be adjusted. For example, assume that the PTP source sends a PTP

message advertising a time of 1:00:00 p.m. However, this message takes time to reach its destination. For

example, if the PTP packet takes 1 second to reach its source, it will be 1:00:01 p.m. when the source receives a

PTP packet indicating that the time is 1:00:00 p.m. How does PTP compensate for the network latency?

The synchronization is achieved through a series of messages exchanged between master and slave clocks, as

shown in Figure 2.

1. The master clock sends a Sync message. The time that the Sync message leaves the master is timestamped

as t1, which can be embedded in the Sync message itself (one-step operation) or sent in the Follow_Up

message (two-step operation).

2. The slave receives the Sync message; t2 is the time that the slave receives the Sync message.


3. The slave sends the Delay_Req message, which is timestamped as t3 when it leaves the slave and

timestamped as t4 when the master receives it.

4. The master responds with a Delay_Resp message that contains time-stamp t4.

Figure 2. Clock Synchronization Process

In the example in Figure 3, the master-time value is initially 100 seconds, and the slave-time value is 80 seconds.

The slave time needs to be adjusted to match the master time.

Figure 3. PTP Message Exchange Example


The clock offset (the difference between the master and slave clocks) is calculated as follows:

Offset = t2 - t1 - meanPathDelay

The link delay from the master clock to the slave clock is t2 - t1.

The link delay from the slave clock to the master clock is t4 - t3.

IEEE 1588-2008 assumes that the path delay between the master and slave clocks is symmetrical, so the mean

path delay is calculated as follows:

meanPathDelay = ((t2 - t1) + (t4 - t3)) / 2

You can use this formula to calculate the offset between the master and slave clocks in the preceding example:

Mean Path Delay = ((t2 - t1) + (t4 - t3)) / 2

= (-18 + 22) / 2

= 2

Offset = t2 - t1 - Mean Path Delay

= 82 - 100 - 2

= -20

So in this case, the slave clock is 20 seconds behind the master clock and will adjust its clock to +20 seconds.

A PTP device that implements this algorithm is known as a two-step clock because the time information is provided

by Sync and Follow_Up messages.

Another version of the algorithm, the one-step operation, sends the timestamp in the Sync message instead of

using a separate Follow_Up message. The one-step PTP can be more precise, but can be more complicated to

implement in hardware. This complexity mainly derives from the need to update the timestamp in real time and to

modify the checksum.

The advantage of one-step PTP is that it prevents increased load on both the link and the CPU.

Hardware and Software Timestamping

The objective of PTP implementation on a switch is to provide PTP timing signals to the connected servers so that

the system clocks can be synchronized accurately.

In a software PTP implementation, the CPU timestamps the packets. This solution is easier to design and

implement and is also cost effective compared to hardware timestamping. But it is less precise: packets need to go

to the CPU to be timestamped, and the CPU is running the whole operating system, which includes many

processes in addition to the PTP process. Even if the PTP process is given the highest priority, software PTP

cannot be as precise as hardware PTP.

In a hardware PTP implementation, the capture and insertion of timestamps in PTP packets is performed by the

application-specific integrated circuit (ASIC) for a switch or by the NIC for a server. This process ideally is

performed at the physical (PHY) level. For ingress, the best time for this process is right after packets have been

received on the wire. For egress, the best time is right before packets are serialized on the wire, and after packets

leave the buffer, so that the timestamp precision is not affected by potential buffer use due to congestion or speed

mismatch.


End-to-End Data Center PTP Deployment with Cisco Nexus 3100 Platform and 9000 Series

Topology and Components

Because PTP is based on Ethernet, it can run over the existing data center architecture and doesn’t usually require

a redesign of the data center. Figure 4 shows a typical leaf-and-spine data center design with PTP components.

Figure 4. Data Center PTP Deployment

The components for this end-to-end data center PTP deployment include the grandmaster clock and Cisco Nexus

switches.


Grandmaster Clock

The grandmaster clock typically is dedicated, specific equipment. This device usually is connected to GPS, which

provides an accurate time source input. The master clock then generates PTP packets over the network.

The Symmetricom TimeProvider 5000 is an example of a PTP grandmaster clock (Figure 5).

Figure 5. Symmetricom TimeProvider 5000 PTP Grandmaster Clock

PTP-Enabled Cisco Nexus 3100 Platform and 9000 Series Data Center Switches

The Cisco Nexus 3100 platform switches are compact 1- and 2-rack-unit (1RU and 2RU) switches for top-of-rack

(ToR) data center deployments or end-of-row (EoR) deployments. They offer wire-rate Layer 2 and 3 switching with

the data center-class Cisco® NX-OS Software operating system.

The Cisco Nexus 9000 Series Switches consist of Cisco Nexus 9500 platform modular switches and Cisco Nexus

9300 platform fixed-configuration switches. Cisco Nexus 9000 Series Switches can run in Cisco Application Centric

Infrastructure (ACI) mode or Cisco NX-OS mode.

In ACI mode, Cisco Nexus 9000 Series Switches, when used in combination with the Cisco Application Policy

Infrastructure Controller (APIC), provide an application-centric infrastructure.

In NX-OS mode, Cisco Nexus 9000 Series Switches function as classic switches. Equipped with enhanced Cisco

NX-OS Software as the operating system, Cisco Nexus 9000 Series Switches provide network connectivity through

traditional means but with exceptional performance and enhanced network resiliency and programmatic

automation functions.

This document focuses on PTP on the Cisco Nexus 9000 Series in NX-OS mode.

As of Cisco NX-OS Release 7.0(3)I7(1), PTP is supported on all Nexus 3100 and 9000 Series Switches and Line

Cards, except the N9K-X9408PC-CFP2 line card, the N9K-M4PC-CFP2 uplink module, and the 9500-R chassis.

Figure 6. Cisco Nexus 3164Q


Figure 7. Cisco Nexus 9396PX

Figure 8. Cisco Nexus 9332PQ

Figure 9. Cisco Nexus 9500 Chassis with X9636PQ Line Card


PTP is supported on the Cisco Nexus 3100 platform and 9000 Series Ethernet switches starting from Cisco NX-OS

Release 7.0(3)I1(1). It includes the following features:

● IEEE 1588-2008 PTPv2 standard

● Two-step boundary clock

● User Datagram Protocol (UDP) over IPv4 multicast using multicast address 224.0.1.129 as defined in the

IEEE 1588 standard

● Hardware-assisted PTP implementation

● Effective handling of network congestion by processing and forwarding PTP messages with higher priority

by default; no need for another step to configure additional quality of service (QoS)

● Support for ERSPAN with ERSPAN Header Version 3, which includes a timestamp

● Support for PTP on both Layer 2 and Layer 3 interfaces

Nexus 3100 Platform and 9000 Series PTP Architecture

The Cisco Nexus 3100 platform and 9000 Series support PTP operations with hardware assistance. The

forwarding ASIC timestamps the PTP packets in both the ingress and egress directions in hardware.

Figure 10 shows the Cisco Nexus 3100 platform and 9000 Series PTP architecture.

Figure 10. Cisco Nexus 3100 Platform and 9000 Series PTP Architecture


On the Cisco Nexus 9300 platform switches, the PTP hardware clock is in the Application Leaf Engine (ALE) and

ALE-2 ASICs. On the Cisco Nexus 3100 platform switches, it is in a separate field-programmable gate array

(FPGA). On the Cisco Nexus 9500 platform modular switches, it is in an FGPA present on the supervisor card that

all line cards use.

Cisco Nexus 3100 Platform and 9000 Series PTP Packet Walk

Two scenarios are possible: one in which a front-panel port of the Cisco Nexus 3100 platform or 9000 Series

Switch is a PTP slave, and one in which the port is a PTP master.

When a front-panel port of the Cisco Nexus 3100 platform or 9000 Series Switch is a PTP slave, the switch is

receiving time information from a PTP master and correcting its own clock. The packet flow is as follows:

1. The switch receives a PTP Sync message from the PTP master on the slave port. The forwarding ASIC has a

clock synchronized to the PTP hardware clock. The forwarding ASIC records the time of arrival of the Sync

packet. It adds this timestamp to the packet using an internal header and forwards it to the CPU. The PTP

software process running on the CPU receives the packet and stores the timestamp. The timestamp is t2 in

Figure 2.

2. The PTP master sends a PTP Follow_Up message to the switch. The message contains the t1 timestamp that

was recorded by the master when it sent the PTP Sync packet. The Cisco Nexus 3100 platform or 9000 Series

Switch PTP software process receives this packet and stores its timestamp. The PTP software process now

has t1 and t2.

3. The Cisco Nexus 3100 platform or 9000 Series Switch sends a Delay_Req message to the master. The

timestamp of departure is recorded when the Delay_Req messages leaves the forwarding ASIC. The PTP

software process stores this timestamp, which is t3. The PTP software process now has t1, t2, and t3.

4. The PTP master records when it received the Delay_Req message. This timestamp is t4. It sends a

Delay_Resp message containing t4 to the Cisco Nexus 3100 platform or 9000 Series Switch slave. The PTP

software process now has t1, t2, t3, and t4.

5. The PTP software on the Cisco Nexus 3100 platform or 9000 Series Switch computes the offset from t1, t2, t3,

and t4 based on the formula described in the “PTP Algorithm” section earlier in this document. The PTP

hardware clock of the Cisco Nexus 3100 platform or 9000 Series Switch is adjusted accordingly. This

adjustment is why a device operating as a PTP boundary clock performs what is called clock recovery.

This concludes the PTP message exchange on a PTP slave port of the Cisco Nexus 3100 platform or 9000 Series

Switch. This message exchange is continuously performed between the Cisco Nexus 3100 platform or 9000 Series

Switch slave port and the master port to which it is connected, so the clock of the switch is always synchronized

with the master.

When a front-panel port of the Cisco Nexus 3100 platform or 9000 Series Switch is a PTP master, the switch is

providing the time to a slave device that is synchronized to it. The packet flow is as follows:

1. The Cisco Nexus 3100 platform or 9000 Series Switch sends a PTP Sync message to the slave. Using its PTP

hardware clock, it records the timestamp t1, which indicates when the packet left the forwarding ASIC.

2. The t1 timestamp is sent by the Cisco Nexus 3100 platform or 9000 Series Switch to the slave in a PTP

Follow_Up packet.

3. The Cisco Nexus 3100 platform or 9000 Series Switch receives a PTP Delay_Req message from the slave. It

records the timestamp t4, which indicates when the packet was received.


4. The Cisco Nexus 3100 platform or 9000 Series Switch sends t4 to the slave in a PTP Delay_Resp packet.

5. The slave device now can adjust its own clock based on the timestamps it received from the master and

recorded. At no point does the Cisco Nexus 3100 platform or 9000 Series Switch adjust its own clock, because

this PTP message exchange takes place on a master port.

The Cisco Nexus 3100 platform or 9000 Series Switch PTP implementation is not affected by congestion or

buffering. The switch adds the timestamps at the PHY level, when the packet is ready to be serialized and after it

has been buffered. This approach results in very high PTP precision, as discussed in the section “Data Center PTP

Performance with Cisco Nexus 3100 Platform and 9000 Series” later in this document.

Networking Configuration and Best Practices with Cisco Nexus 3100 Platform

and 9000 Series

Cisco Nexus 3100 Platform and 9000 Series PTP Configuration

This section describes the most important PTP configuration commands on the Cisco Nexus 3100 platform and

9000 Series. A comprehensive guide to configuration can be found in the system management configuration guide

available at http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-

x/system_management/configuration/guide/b_Cisco_Nexus_9000_Series_NX-

OS_System_Management_Configuration_Guide_7x/b_Cisco_Nexus_9000_Series_NX-

OS_System_Management_Configuration_Guide_7x_chapter_0100.html.

Mandatory Configuration

The commands shown here need to be present for PTP to operate on the Cisco Nexus 3100 platform and 9000

Series.

Global Configuration Commands

switch(config)# feature ptp

This command enables PTP on the switch.

switch(config)# ptp source <ip>

This command specifies the source IP address for the PTP packets generated by the switch.

Interface Configuration Commands

switch(config)# interface Ethernet slot/port

switch(config-if)# ptp

These commands enable PTP on a port. The Cisco Nexus 3100 platform or 9000 Series Switch is a boundary

clock, so it has both master and slave ports. There is no configuration difference between a master port and a

slave port. They are both configured with the ptp option, and BMCA determines whether the port is a PTP slave or

master port.

Optional Configuration

The commands listed here are optional.

switch(config)# clock protocol ptp

This command configures the switch so that it uses PTP to update the system calendar. This configuration keeps

the switch’s clock synchronized with PTP. If you don’t enable this command, the switch still will propagate the PTP

clock on its master ports. However, the time source will be the Cisco Nexus local clock.

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/system_management/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_System_Management_Configuration_Guide_7x/b_Cisco_Nexus_9000_Series_NX-OS_System_Management_Configuration_Guide_7x_chapter_0100.html





switch(config)# interface ethernet slot/port

switch(config-if)# sync interval value

These commands configure the PTP Sync message packet rate on an interface. If a value is not specified, the

default is 0.

The PTP logSyncInterval (sync interval) value represents the number of PTP Sync message packets sent on the

interface per second (Table 1).

Table 1. PTP logSyncInterval Values

logSyncInterval Value Number of PTP Sync Messages per Second

-3 8

-2 4

-1 2

0 1

1 1 every 2 seconds

You should leave the default sync interval value set to 0 unless you have a specific need for greater accuracy.

Cisco Nexus 3100 Platform and 9000 Series PTP Configuration Validation

The commands shown here can be used to validate PTP on the Cisco Nexus 3100 platform and 9000 Series

Switches.

switch# show clock

This command can be used to verify that the switch clock is synchronized with the grandmaster clock. You cannot

verify precise accuracy with this command-line interface (CLI) command, but you can at least verify that the time of

day matches the grandmaster, if clock protocol ptp was configured.

switch# show ptp clock

This command displays the properties of the local clock, including the clock identity.

This is an example of show ptp clock output:

switch# show ptp clock

PTP Device Type: Boundary clock

Clock Identity : 88:f0:31:ff:fe:2a:fa:e1

Clock Domain: 0

Number of PTP ports: 3

Priority1 : 196

Priority2 : 255

Clock Quality:

Class : 248

Accuracy : 254

Offset (log variance) : 65535

Offset From Master : 0

Mean Path Delay : 0

Steps removed : 0

Local clock time:Wed Jan 28 17:56:19 2015

9396px-nd1#


switch# show ptp parent

This command displays the properties of the PTP parent. It is useful for verifying the parent clock’s identity.

This is an example of show ptp parent output:

switch# show ptp parent

PTP PARENT PROPERTIES

Parent Clock:

Parent Clock Identity: 00:14:01:ff:fe:00:00:01

Parent Port Number: 1

Observed Parent Offset (log variance): N/A

Observed Parent Clock Phase Change Rate: N/A

Grandmaster Clock:

Grandmaster Clock Identity: 00:14:01:ff:fe:00:00:01

Grandmaster Clock Quality:

Class: 6

Accuracy: 35

Offset (log variance): 0

Priority1: 1

Priority2: 1

switch# show ptp brief

This command displays the PTP state of all interfaces. A PTP port can be in one of the following three states:

● Master: The port is the source of time on the path served by the port.

● Slave: The port synchronizes with the device on the path on the port that is in the master state.

● Disabled: PTP is not enabled on this port.

● Passive: The port is not the master on the path, nor does it synchronize with a master.

Because the Cisco Nexus 3100 platform or 9000 Series Switch is a PTP boundary clock and supports only one

PTP domain, the switch can have only one slave port. If the switch has one slave port already, a second port

connected to a second grandmaster will be in the passive state. When the first grandmaster or the first slave port

fails, BMCA will move the previously passive port to a slave state. With this process, grandmaster redundancy can

be achieved.

This is an example of show ptp brief output:

switch# sh ptp brief

PTP port status

-----------------------

Port State


------- --------------

Eth1/1 Slave

Eth1/2 Master

Eth1/3 Master

Eth1/4 Master

Eth1/5 Master

Eth1/6 Master

...

Switch#

switch# show ptp corrections

This CLI command displays the last few PTP corrections.

Here is an example of show ptp corrections output:

PTP past corrections

---------------------------------------------------------

Slave Port SUP Time Correction(ns) MeanPath Delay(ns)

---------- ----------------------------------------------

Eth1/46 Mon Dec 23 09:52:11 2013 48581 -1 293

Eth1/46 Mon Dec 23 09:52:12 2013 49318 3 297

Eth1/46 Mon Dec 23 09:52:13 2013 49193 -8 297

Eth1/46 Mon Dec 23 09:52:14 2013 49208 12 298

Eth1/46 Mon Dec 23 09:52:15 2013 48625 -3 298

Eth1/46 Mon Dec 23 09:52:16 2013 47607 -13 295

Eth1/46 Mon Dec 23 09:52:17 2013 49091 0 295

Eth1/46 Mon Dec 23 09:52:18 2013 47961 2 295

Eth1/46 Mon Dec 23 09:52:19 2013 48005 -1 295

Eth1/46 Mon Dec 23 09:52:20 2013 48350 0 296

Eth1/46 Mon Dec 23 09:52:21 2013 48507 -5 292

Eth1/46 Mon Dec 23 09:52:22 2013 48105 2 292

Eth1/46 Mon Dec 23 09:52:23 2013 48188 12 301

Eth1/46 Mon Dec 23 09:52:24 2013 48021 6 301

Eth1/46 Mon Dec 23 09:52:25 2013 48239 -12 296

Data Center PTP Performance with Cisco Nexus 3100 Platform and 9000 Series

PTP Performance Measurement Definitions and Concepts

This section describes essential definitions and concepts used to characterize PTP performance.

The offset is the difference between the times on two clocks. In the following test results, it is the measure of how

accurately the slave synchronizes with the master clock. This measurement indicates the amount of inaccuracy

brought by the Cisco Nexus 3100 platform or 9000 Series Switch as a boundary clock. Less is better.


The mean path delay is the average time taken by PTP frames to travel between the master and the slave. This

measurement does not indicate the performance or accuracy of the switch or servers. A small mean path delay is

useful for obtaining baseline results. A large mean path delay with a lot of jitter is representative of a complex data

center with buffering and latency spikes, control protocols running, a high rate of traffic, etc. This measure can be

interesting in a real-world PTP performance test in a processing-intensive environment.

The standard deviation indicates the amount of variation or dispersion from the average. A low standard deviation

indicates that the data points tend to be very close to the mean (also called the expected value); a high standard

deviation indicates that the data points are spread out over a large range of values.

In the rest of this document, the following definitions are used to characterize PTP performance:

● Average offset

● Offset standard deviation

● Minimum and maximum offset peak

● Mean path delay

These values are measured by external tools using the same stable clock reference.

The average offset is often close to 0 because the positive and negative clock offsets equalize each other when

calculating an average.

Minimum and maximum offset peaks are useful; however, those values alone do not indicate how often those

peaks were reached. Therefore, the offset standard deviation is important in addition to the average offset and

offset peaks. Another way to see the offset standard deviation is through the jitter.

These four data points together with the mean path delay provide a very good view of a device’s PTP performance.

Figure 11 is an example of an offset graph showing the average offset, offset standard deviation, and minimum and

maximum offset peaks. The vertical axis is the offset value, and the horizontal axis is the set of PTP clients.


Figure 11. Sample Graph of Offset Values

Cisco Nexus 3100 Platform and 9000 Series Performance Measurement Methodology

and Equipment

The test described here uses a traffic generator from Spirent for PTP message generation, master and slave

simulation, and background traffic. It uses a Spirent 9RU chassis (SPT-9000) with HyperMetrics CV 8 x 10 Gigabit

Ethernet Enhanced Small Form Factor (SFP+) cards. Spirent Release 4.33.0086 is used.

The Spirent 9RU chassis is connected to a Symmetricom TimeProvider 5000 grandmaster clock. Spirent receives

the clock from a timing interface. The Symmetricom TimeProvider 5000 grandmaster gets its time source from an

embedded GPS receiver.

The Spirent 9RU chassis has one master port connected to the Cisco Nexus switch being tested. The rest of ports

on the Spirent 9RU chassis are PTP slave ports getting the clock from the Cisco Nexus PTP master ports, which

redistribute the clock from the Spirent 9RU chassis. Therefore, Spirent can compute the offset between the clock

sent from its master port and the clock received on its slave ports. This result indicates the accuracy of the Cisco

Nexus switch under test.


Figure 12 shows the performance measurement methodology.

Figure 12. Performance Measurement Methodology

The Cisco Nexus 3100 platform and 9000 Series use Cisco NX-OS Release 7.0(3)I1(1).

For all the tests, the PTP configuration is the default on the Cisco Nexus switches. In particular, the PTP sync

interval parameter is left at the default value of 0.

Cisco Nexus 3164Q PTP Performance

The Cisco Nexus 3164Q is configured in 4 x 10-Gbps breakout mode, with 44 physical 10-Gbps ports connected to

Spirent. This configuration results in 43 PTP slaves. In addition, 212 emulated PTP slaves are configured in

Spirent. This configuration amounts to a total of 255 PTP slaves to which the Cisco Nexus 3164Q needs to provide

the time.


Figure 13 shows the Cisco Nexus 3164Q performance topology with Spirent.

Figure 13. Cisco Nexus 3164Q Performance Topology with Spirent

After 6 hours, the results as averages of all ports are:

● Average offset: 4 ns

● Offset standard deviation: 50 ns

● Minimum offset peak: -300 ns

● Maximum offset peak: 302 ns

● Average mean path delay: 110 ns

The average offset is very close to 0. The offset standard deviation of 50 ns shows that during the duration of the

test, the average offset values were clustered closely around the mean. Therefore, the Cisco Nexus 3164Q

introduces an extremely small offset in the PTP client clock synchronization process even with a larger number of

PTP slaves.

Cisco Nexus 9396PX PTP Performance

The Cisco Nexus 39396PX is configured with 44 physical 10-Gbps ports connected to Spirent. This configuration

results in 43 PTP slaves. In addition, 4 emulated PTP slaves are configured in Spirent. This configuration amounts

to a total of 47 PTP slaves to which the Cisco Nexus 9396PX needs to provide the time.


Figure 14 shows the Cisco Nexus 9396PX performance topology with Spirent.

Figure 14. Cisco Nexus 9396PX Performance Topology with Spirent







Cisco Nexus 9332PQ PTP Performance

The Cisco Nexus 9332PQ is configured in 4 x 10-Gbps breakout mode, with 44 physical 10-Gbps ports connected

to Spirent. This configuration results in 43 PTP slaves. In addition, 84 emulated PTP slaves are configured in

Spirent. This configuration amounts to a total of 127 PTP slaves to which the cisco Nexus 9332PQ needs to

provide the time.


Figure 15 shows the Cisco Nexus 9332PQ performance topology with Spirent.

Figure 15. Cisco Nexus 9332PQ Performance Topology with Spirent


● Average offset: -5 ns





Cisco Nexus 9504 with X9636PQ Line Card PTP Performance

The Cisco Nexus X9636PQ line card on a Cisco Nexus 9500 platform switch is configured in 4 x 10-Gbps breakout

mode, with 44 physical 10-Gbps ports connected to Spirent. This configuration results in 43 PTP slaves. In

addition, 99 emulated PTP slaves are configured in Spirent. This configuration amounts to a total of 143 PTP

slaves to which the Cisco Nexus X9636PQ line card needs to provide the time.


Figure 16 shows the Cisco Nexus X9636PQ performance topology with Spirent. In the figure, the line card is

connected to a Cisco Nexus 9504 Switch.

Figure 16. Cisco Nexus X9636PQ Performance Topology with Spirent







The average offset is almost 0 ns. It always stayed close to this average (standard deviation is 9 ns). Even with a

large number of PTP slaves, the Cisco Nexus 9500 with a Cisco Nexus X9636PQ line card maintains a very high

PTP accuracy.


Cisco Nexus 9000 Series ERSPAN and PTP Timestamping

Concept of ERSPAN

ERSPAN transports mirrored traffic over an IP network. The traffic is encapsulated at the source router and is

transferred across the network. The packet is decapsulated at the destination router and then sent to the

destination interface.

ERSPAN consists of an ERSPAN source session, routable ERSPAN generic routing encapsulation (GRE)-

encapsulated traffic, and an ERSPAN destination session. You separately configure ERSPAN source sessions and

destination sessions on different switches. Figure17 shows an ERSPAN topology.

Figure 17. ERSPAN Topology

The Cisco Nexus 9000 Series supports ERSPAN Type III, which contains timestamp information. It can be used to

calculate packet latency between different parts of the network.

Figure 18 shows a typical example of latency monitoring in a data center. The goal is to determine the latency

between switch A and switch B.

To monitor the latency, an ERSPAN source session is configured on each switch. The source port for this session

is the ingress port of the flow for which the latency is monitored. The destination IP address of the session is the IP

address of a server running a packet-capture analyzer such as Wireshark. The two switches have their clocks

synchronized using PTP, so the timestamps recorded in the ERSPAN header have the same time reference. On

the analyzer server, the packets from both switches are received (because the same ERSPAN destination IP

address was configured).

With this information, you can easily compute the delta between the ERSPAN timestamps of the packets received

to obtain the network latency.


Figure 18 shows how to configure ERSPAN and PTP to perform latency monitoring in a typical network.

Figure 18. Latency Monitoring Using ERSPAN and PTP

The important point to note is that an ERSPAN destination session is not configured on any switch in this scenario.

An ERSPAN destination decapsulates the GRE and ERSPAN headers from the packet. Therefore, the timestamp

would be lost. Instead, Wireshark is used, and it can read the ERSPAN header and display the timestamp.

Figure 19 shows an example of a Wireshark capture of an ERSPAN packet. The timestamp is highlighted in red.


Figure 19. Wireshark ERSPAN Capture

ERSPAN Support on the Cisco Nexus 9000 Series

Cisco Nexus 9300 platform switches equipped with ALE or ALE-2 ASICs support the ERSPAN Type III header. As

of Cisco NX-OS Release 7.0(3)I1(1), the switches are the Cisco Nexus 93128TX, 9396PX, 9396TX, 9372PX,

9372TX, and 9332PQ.

The Cisco Nexus 3100 and 9500 platforms do not support ERSPANv3 headers in spanned copy of the packets,

and therefore cannot carry a timestamp in monitored packets. On those switches, packets are encapsulated as

GRE-tunnel packets. The ERSPAN header is not added to the packet.

ERSPAN Packet Format on Cisco Nexus 9000 Series

The ERSPAN Type III packet format includes an additional 12- or 20-byte header on top of the GRE header.

Therefore, it adds a total of up to 62 bytes to the original frame length: 14 (MAC) + 20 (IPv4, or 40 for IPv6) + 8

(GRE) + 20 (ERSPAN).

The 20-byte ERSPAN header includes a 32-bit time-stamp field. The ERSPAN header starts at the 42nd

byte in the

ERSPAN packet. The time-stamp field is bytes 46 to 49 in the ERSPAN packet.


Here is a summary of the ERSPAN Type III packet format:

ERSPAN Type III header (12 octets)

0 1 2 3

0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| Ver | VLAN | COS |BSO|T| Session ID |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| Timestamp |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| SGT |P| FT | Hw ID |D|Gra|O|

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Platform Specific SubHeader (8 octets, optional)

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| Platf ID | Platform Specific Info |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| Platform Specific Info |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

The composite header in the preceding format summary is immediately followed by the original mirrored frame and

then by the standard 4-octet Ethernet cyclic redundancy check (CRC).

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| Original Mirrored Frame |

| ... |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| CRC |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Table 2 provides descriptions of each field in the ERSPAN Type III packet header.

Table 2. ERSPAN Type III Packet Header Fields

Field Length Definition

Header Fields in Common with ERSPAN Type II

Ver 4 ERSPAN encapsulation format version

Type II: 0x1. Type: III 0x2. Version 0x0 is obsolete.

VLAN 12 VLAN of the frame monitored by an ERSPAN source session. For the ingress monitor, this will be the original source VLAN, and for the egress monitor this will be the destination VLAN.

Cos 3 Class of service (CoS) of the monitored frame. Ingress or egress CoS value depending on the monitor type.

BSO 2 2-bit value indicating the integrity of the payload carried by ERSPAN.

00: Payload is a good frame with no error, or unknown integrity.

11: Payload is a bad frame with CRC or alignment error.

01: Payload is a short frame.

10: Payload is an oversized frame.

T 1 Indicates that encapsulated frame is truncated. Encapsulated packet exceeds the maximum transmission unit (MTU) settings for an ERSPAN source session.

Session# 10 Identification associated with each ERSPAN source session. Identifies the stream to be decapsulated at the destination.


Field Length Definition

Fields Specific to ERSPAN Type III Header

Timestamp 32 32-bit timestamp. Wraps approximately 4 billion time units depending on the time-stamp granularity specified.

SGT 16 Security group tag of the monitored frame.

P 1 This bit indicates that the payload is a protocol frame, or Bridge Protocol Data Unit (BPDU) frame.

FrameType 5 00000 expected.

Hardware ID 6 Unique identifier of an ERSPAN engine.

Direction 1 Original frame subjected to SPAN in ingress or egress. Ingress = 0 and egress = 1.

Timestamp Granularity

2 00: Granularity = 100 ms (mandatory default)

01: Granularity = 100 ns

10: 1588 PTP 11 = 1 ns

Optional SubHeader

1 O flag indicates whether or not the optional subheader is present.

When O == 0b ERSPAN, payload starts after the O flag.

When O == 1b ERSPAN, payload starts 8 bytes after the O flag.

Supplemental Info 64 Optional information.

ERSPAN with Time-Stamping Configuration on Cisco Nexus 9000 Series

The ERSPAN configuration is described in the configuration guide available at

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-



OS_System_Management_Configuration_Guide_7x_chapter_010001.html.

The ERSPAN Type III header contains a 32-bit timestamp. This timestamp represents the time of day since

January 1, 1970. The CLI option required to enable the ERSPAN timestamp is header-type 3 in the ERSPAN

session configuration.

A sample ERSPAN configuration with header-type 3 is shown here:

switch(config)# monitor session 1 type erspan-source

switch(config-erspan-src)# vrf default

switch(config-erspan-src)# source interface Ethernet1/30 rx

switch(config-erspan-src)# destination ip 192.168.8.1

switch(config-erspan-src)# header-type 3

switch(config-erspan-src)# no shut

The ERSPAN Type III header is added to the packets only if the ERSPAN destination IP address route is resolved

through an uplink interface (that is, an interface connected to an ALE or ALE-2 ASIC). On the Cisco Nexus

93128TX, 9396PX, and 9396TX, the uplinks are all the interfaces on slot 2. On the Cisco Nexus 9372PX and

9372TX, the uplinks are e1/49 to e1/54. On the Cisco Nexus 9332PQ, the uplinks are e1/27 to e1/32.






ERSPAN Granularity and Marker Packet

The resolution of the ERSPAN timestamp is 100 picoseconds (ps). Therefore, the 32-bit time-stamp field in the

ERSPAN header will wrap around every 0.43 second. This is not enough to represent the full time of day, which

requires 64 bits. Consequently, Cisco NX-OS offers the capability to periodically send a marker packet that

contains the full Coordinated Universal Time (UTC) timestamp. The ERSPAN timestamp and the marker packet

time value combined provide the capability to recover the full value of the ERSPAN timestamp. Cisco NX-OS

sends 10 marker packets per second.

The marker packet is carried in a UDP packet with destination port 8880. It uses the same source and destination

IP addresses as the ERSPAN session.

A sample ERSPAN configuration with marker-packet capabilities is shown here:

switch(config)# monitor session 1 type erspan-source

switch(config-erspan-src)# marker-packet

Table 3 shows the marker packet format.

Table 3. Marker Packet Format

Field Position (Byte: Bit)

Length (Bits) Definition

Align 16 The 2-byte align bit is inserted to align the rest of the packet with a 4-byte boundary. The value is 0xFF to indicate the beginning of the marker packet.

Version 4 Version number. Default is Version 1.

Type 4 The type of marker packet. Default is 0.

ssid 8 The session ID of the ERSPAN source session.

granularity 8 The granularity of the 3- bit hardware timestamp.

The value is 0x100, which represents a 100-ps granularity.

Utc_offset 8 The UTC offset between the ASIC clock and the CPU UTC clock. Default is 0; currently set to 0.

timestamp 32 The 32-bit ASIC hardware timestamp.

UTC sec 32 The upper 32 bits of the UTC timestamp from the CPU clock of the Cisco Nexus 9000 Series Switch. This value is the base for the UTC recovery of the ERSPAN header time-stamp field.

UTC usec 32 The lower 32 bits of the UTC timestamp from the CPU clock of the Cisco Nexus 9000 Series Switch.

Sequence 32 The sequence number of the marker packet.

Reserved 32 Reserved for future use.

Signature 32 A value of 0xA5A5A5A5.

The ERSPAN timestamping is performed in hardware. The timestamps are recorded when the ERSPAN-monitored

packets arrive at the ALE or ALE-2 ASIC.

Note that the PTP timestamp and the ERSPAN timestamp mean different things:

● The PTP timestamp is the timestamp used in PTP control packets that allow PTP to operate and

synchronize clocks over the network.

● The ERSPAN timestamp is the timestamp in the ERSPAN header. It is separate from the PTP timestamp.

ERSPAN timestamping can actually work without PTP enabled on the switch. In such a case, the ERSPAN

timestamp would be derived from the local oscillator of the switch and would not be synchronized with any

source. This is an acceptable solution for basic latency monitoring using just one switch because it still

shows the delta between the arrival times of packets in the switch.


Conclusion

IEEE 1588-2008 PTPv2 provides a reliable, highly accurate distributed time synchronization solution for data

centers that require nanosecond or sub-microsecond accuracy. PTP is easy to implement with very little

administrative effort and can tolerate network and clock failure with built-in fault-tolerant mechanisms.

The Cisco Nexus 3100 platform and 9000 Series Switches have an extremely precise PTP implementation. With a

large number of simultaneous PTP slaves, the Cisco Nexus 3100 platform and 9000 Series PTP accuracy stays

under 50 nanoseconds.

This solution can be combined with a software PTP implementation on the server to achieve microsecond

accuracy. Hardware PTP on the server can also be used if sub-microsecond accuracy is required.

PTP in combination with ERSPAN on the Cisco Nexus 9000 Series provides a very powerful and easy-to-use

solution to perform latency monitoring in the data center.

For More Information

For more information about IEEE 1588-2008 Precision Time Protocol Version 2, see the IEEE Standard for a

Precision Clock Synchronization Protocol for Networked Measurement and Control Systems at

http://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?reload=true&punumber=4579757.

For more information about the Cisco Nexus 9000 Series Switches, see the detailed product information at the

product homepage at http://www.cisco.com/c/en/us/products/switches/nexus-9000-series-switches/index.html.

The configuration guides for PTP and ERSPAN on the Cisco Nexus 3100 platform and 9000 Series can be found

at:

● http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-



OS_System_Management_Configuration_Guide_7x_chapter_0100.html

● http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-



OS_System_Management_Configuration_Guide_7x_chapter_010001.html

Printed in USA C11-733921-02 09/17

http://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?reload=true&punumber=4579757

http://www.cisco.com/c/en/us/products/switches/nexus-9000-series-switches/index.html









Documents

IEEE 1588 PTP on Cisco Nexus 3100 Platform and 9000 … 1588 PTP on Cisco Nexus 3100 Platform and 9000 Series ... 12 Networking Configuration and Best Practices with Cisco Nexus 3100