Timing Distribution System (TDS)

9 April, 2010

Greg Deuerling

Rick Kwarciany

Neal Wilcer

NOvA Timing Distribution System Features

• Distributes system clock and synchronizes time stamp registers of all DAQ devices on detector.

• All timing system links are CAT5e cables except Near Detector MTDU to STDU1 (need fiber to go the 200 or so meters).

• Configurable and monitorable via DCS.• DCMs have two clock system interfaces, so a complete

redundant clock system can be deployed if desired.• Can correct for cable delays using TOF/2 method if

desired.

April 2010 Collaboration Meeting

TDS to DAQ System

GPS TDU

MTDU

STDU

STDU

April 2010 Collaboration Meeting

Master Timing Unit

April 2010 Collaboration Meeting

Slave Timing Unit

April 2010 Collaboration Meeting

Some Functional Details

• Timing system’s main connection to DCS is to MTDU via Ethernet.

• Timing critical commands are transmitted to system from MTDU through timing links.

• Timing critical events are synchronized with SYNC signal on timing links (generated by MTDU).

• STDU can be queried for status via Ethernet, but cannot inject commands or SYNCs.

• MTDU and STDU firmware remotely loadable via Ethernet.

April 2010 Collaboration Meeting

On System Startup…

• MTDU put into “Delay Learn” mode.– Transmits a SYNC pulse to slaves.– Measures time of round trip via ECHO signal.– All STDUs and DCMs do the same.

• MTDU synchronizes timing registers in all DAQ devices.– Transmits Timing Register Preset command.– Issues SYNC to start Timing Counter on all devices on the same clock cycle.

• MTDU issues command(s) to set mode in all DCMs and FEBs.– FEB and DCM control registers write accessible via timing link.

• MTDU periodically issues a Timing Register Verify command as a sanity check.

April 2010 Collaboration Meeting

MTDU

April 2010 Collaboration Meeting

GPS Module

April 2010 Collaboration Meeting

TOF/2 (Time of Flight / 2)

All TDS’s TriggerSimultaneously

TDB Time Line

Incident TDB 1

Reflected TDB 1

TDB-1 to DCM Cmd

Incident TDB 8

Trigger TDS 8

TDB-8 to DCM Cmd

Incident TDB 12

Trigger TDS 12

TDB-12 to DCM Cmd

Reflected TDB 8

Reflected TDB 12

Incident TDB 14

Reflected TDB 14

TDB-14 to DCM Cmd

April 2010 Collaboration Meeting

Schedule & Summary

• We have two assembled prototype TDU modules (STDU and MTDU use same board). Hardware is working.

• We have four GPS units and four antennas.• We expect to have two assembled CPU daughter boards and

two more TDU modules by end of month.• For NDOS we will need:

– One Master Timing unit.– One CPU daughter board.– One GPS module and antenna.– 1 Slave Timing Distribution unit (two would be nice).

• We expect to have firmware working that will be functional but may not be 100% feature complete.

April 2010 Collaboration Meeting

Timing Distribution System (TDS)

Questions?

April 2010 Collaboration Meeting

Supporting Notes

• Following are notes of the above slides

July 21-23, 2009 Director's CD-3b Review

Master Timer

• Incorporates a common view GPS trained clock generator with a time-stamp counter.

• Capable of decoding Fermilab beam timing signals (RF-Clk, BSync, TClk, MDat).

• Capable of broadcasting time-of-day and time-stamp to Far Detector (100Mb ethernet).

• Provides a USB slave port (alternate computer connection).

• Provides repetitive timing & control packets to downstream distribution boxes.

• Measures & stores roundtrip time-of-flight of timing commands.

July 21-23, 2009 Director's CD-3b Review

Timing Distribution Unit

• Coherent timing along the backbone of the detector.– Timing is referenced to the far-end cable loopback.– Synchronization is within +/- 1/2 clock cycle (16.834Mhz).

• Synchronization is based on the roundtrip time-of-flight of timing packets.– Digital equivalent of the time-of-flight of a reflection on a coax cable

using a Time Domain Reflectometer.– Time-of-flight measurements are stored for later readout.

• Propagation delays are self compensated– No need to match cable pair lengths (xmt & rcv)

• Distributes Clock, Timing & Control commands to DCMs

July 21-23, 2009 Director's CD-3b Review

Timing Distribution System (old design)

0013XX0012XX 0009XX 0002XX 0001XX

FPGA W/PLL

Based Serdes

Master Timer GPS Clock Generator

163.84MhzVCXO

PLL Based Serdes

PLL Based Serdes

PLL Based Serdes

FPGA w/ FPGA w/ FPGA w/

NOvA Global Timing DistributionDistribution

Unit 1Distribution

Unit 2Distribution

Unit 8Distribution

Unit 9

Incident Timing Packet

Return Timing Packet

FPGA w/ 4 internal Serdes

Serdes163.84-Mb

Serdes163.84-Mb

Serdes163.84-Mb

GobalUpstream

GobalDownstream

NOvA Local Timing to DCMs

Serdes163.84-Mb

GPS Antenna(Outside)

`

Loopback

DCM24DCM23DCM2DCM1

July 21-23, 2009 Director's CD-3b Review