W.Skulski University of Rochester, July/10/2008 1 Wojtek Skulski University of Rochester The Universe and the FPGA Digital Pulse Processing for Dark Matter

W.Skulski University of Rochester, July/10/2008 1

Wojtek Skulski

University of Rochester

The Universe

and the FPGA

Digital Pulse Processing

for Dark Matter Search


• Wojtek Skulski (University of Rochester and SkuTek Instrumentation).

• Frank Wolfs (University of Rochester).

• Eryk Druszkiewicz (University of Rochester).

• Large Underground Xenon Detector Collaboration (LUX).

Contributors


• Introduction: Dark Matter Search.

• Pulses from the Dark Matter LnXe detector.

• Overview of signal sampling and Digital Pulse Processing.

• Why do we need a trigger?

• Overview of a few trigger architectures.

• Self-triggered digital DAQ becomes Digital Trigger.

• Present status of LUX Digital Trigger.

• Conclusion.

Outline


Digital Pulse Processing for Dark Matter Search

• Outputs from the Dark Matter LnXe detector are digitized using flash A/D

converters with 12 or 14 bit resolution @ up to 100 MSPS.

• Pulses due to radiation and/or DM events are detected in real time.

• Waveforms containing events of interest are recorded for analysis.

• Electronics has to provide the following:

• Low-noise analog front end which receives the signals from the phototubes.

• Flash A/D converters, 12 or 14 bit @ up to 100 MSPS.

• Field-programmable gate arrays which detect the pulses.

• Data readout from the FPGA and archiving for offline analysis.

• GUI for diagnostic data display and control of the experiment.

• Rochester group is responsible for designing the LUX Trigger System.


History of digitizer development by the author•2002. The first single-channel digitizer DDC-1: 12 bits @ 48 MSPS.

•2003. The first 8-channel digitizer DDC-8: 10 bits @ 40 MSPS.

Originally developed for PHOBOS experiment (advanced trigger). Unfortunately, PHOBOS

was discontinued before the DDC-8 could be used.

•2004. DDC-8/XLM: a minor modification of DDC-8.

•2006. DDC-8 Pro: 12/14 bits @ 64 MSPS.

Developed for student labs. Became the first iteration of the LUXcore trigger: a single, 8-

channel board serving 8 phototubes.

• 2007. DDC-8 LUX: 12/14 bits @ up to 80 MSPS + System Connector.

More than 8 phototubes need to be served multiple boards are needed. System Connector

was developed to link together multiple DPP boards. The first version of the Event Builder

was developed as the “receiving end” for System Connectors.

• 2008. DDC-8 DSP: 12/14 bits @ up to 125 MSPS and a large FPGA.

All previous digitizers used flat-pack FPGAs, whose capacity was exceed by our DSP

algorithms. A transition to the BGA package was made (finally).

• 2008/2009. A new version of the Event Builder with a BGA FPGA is also planned.


A few facts concerning

Dark Matter Search


The biggest mystery: where is almost Everything?

• Most of the Universe is missing from the books…

• Ordinary matter accounts for only 5% of the Universe.

Source: Connecting Quarks with the Cosmos, The National Academies Press, p.86.

We are here


The 1st smoking gun: galactic rotation is too fast.

• Gravitational pull reveals more matter than we can see.


Prediction basedon visible matter.

Observation.

Orbital velocity.

Distance from the center.

Rotation curve of the Andromeda galaxy.


The 2nd smoking gun: large-scale gravitational lensing.

• Light from distant sources is deflected by clusters of galaxies.• Visible mass cannot account for the observed lensing pattern.

• Reconstructed mass distribution shows mass between galaxies.


Reconstructed mass distribution.Observed lensing.


What is the Dark Matter composed of?

• Nobody knows, but there are candidates predicted by the theory …

• Axions: light particles that may explain CP violation.

• Neutralinos: heavy particles predicted by SUSY.

• The neutralino is neutral, weakly interacting, and as massive as an atom of gold.

• Very rarely it will bounce off an ordinary nucleus and produce some ionization.

• Our experiment will attempt to detect neutralinos deep underground where the

background from cosmic rays is very low.

• We will use a two-phase liquid xenon (LNXe) detector named Large Underground

Xenon detector (LUX).


Detectors for Dark Matter Search


Underground low-background laboratory

Cosmic particles

stopped by 1 km

of rock.

Dark Matter

particles

penetrate freely.

NB: LUX will be

at DUSEL, but here

I am showing Boulby.


LUX detector prototype


LUX detector consists of many channels

Liquid Xe

144 phototubes

Each phototube

requires an independent

ADC and the

data-processing channel

Gas Xe


The principle of 2-phase xenon detector

Figure from: J.T.White, Dark Matter 2002. http://www.physics.ucla.edu/hep/DarkMatter/dmtalks.htm

HVGas inlet

Quartz PMT

Grids

S2

S1

1.5cm

2.5 cm

Figure from: T.J.Sumner et. al., http://astro.ic.ac.uk/Research/Gal_DM_Search/report.html

gas

liquid

HV

S1: scintillation in liquid Xe.S2: electroluminescence in gas Xe.


Explanation of signal from a 2-phase xenon detector


Electron drifttime in LnXe.

Primary pulsefrom the liquid Xe.

Amplified pulsefrom the gas Xe.

Time flows this way


Signal processing in a single channel LnXe detector


Primary scintillationin liquid phase.

Secondary scintillationin gas phase(electroluminescence).

• Extract the areas under S1, S2, and the separation time between the S1 and S2.

• Time-stamp the data in order to correlate pulses in different channels.


Multi-channel signal processing

144 phototubes

• Each phototube is connected to an independent ADC and

the data-processing channel, which extracts S1, S2, and

the time interval between S1 and S2.

• The distribution of light among the PMTs tells where the

interaction happened within the volume.

• Channels are not independent. They are correlated. In

addition to processing individual channels, the correlated

inter-channel processing is also necessary.

• The data acquisition system has a fairly advanced

architecture explained in the next section.


Electronics for Dark Matter Search


Digital Pulse Processing (DPP)

• The pulse-processing electronics can be either traditional analog, or digital. The

latter has advantages over the former: higher integration, more flexibility, and

lower cost. It also has a slight disadvantage: it needs to be programmed.

• Outputs from the Dark Matter LnXe detector are digitized using flash A/D

converters with 12 or 14 bits @ several tens MSPS (e.g., 64 or 100 MSPS).

• Pulses are detected in real time Digital Pulse Processing has to be implemented.

• Electronics has to provide the following:

• Low-noise analog front end which receives the signals from the phototubes.

• Flash A/D converters, 12 or 14 bits @ several tens of MSPS.

• Field-programmable gate arrays which detect the pulses with DPP algorithms.

• Data readout from the FPGA and archiving for offline analysis.

• GUI for diagnostic data display and control of the experiment.


Analog input stage

Nyquistfilter ADC

Samplerate

processor

Waveformmemory

Eventrate

processorAnalog signal input

Samplingclock

Gain and offsetcontrol

Optionalexternal trigger

in/out

analog digital

Trigger Individualchannel triggeroutput

Pulse informationoutput

Functional diagram of a single DPP channel DPP = Digital Pulse Processing


Functional diagram of a multichannel DPP board

Board-levelevent

processor

(Formatting,

compression, etc.)

Single channel

Board-widetriggerlogic

ADC

Slowcontrol

To eventbuilder

One DPP boardAnalog

Single channelADC Analog



From trigger subsystem

Digital interface:readout,

monitoring,and setup

To slowcontrol


Event-builder

DPPboard

Triggersystem

Slowcontrolmonitor

Recording

Functional diagram of a multiboard DPP system

DPPboard

DPPboard

DPPboard

Signalsfrom

detectors

DPPboard

Network

Subset ofsignals

fromdetectors


Which data is interesting?


Useful data

• Select useful data (so-called events) and reject baseline data.

• Typical rejection ratio is larger than 1000.

Baseline, not useful Not useful

Time flows this way


Estimated rates of “interesting” data samples

Board-levelevent

processor

(Formatting,

compression, etc.)

Single channelADC

To eventbuilder

Analog




Digital interface:readout,

monitoring,and setup

Assumptions: 8 channels, 14-bit @ 64 MHz200 s ADC trace per event (contains only the “interesting” samples)1000 events per second, and 200 s fully recorded per eventThe period of “200 s” covers the electron drift time in LUX detector

112 megabytes / second per channel

22.4 kilobytes / second per channel

448 kilobytes / second per DPP board

One DPP board8 channels


ADC14 bit

Samplerate

processor

Waveformmemory

Eventrate

processor

Samplingclock 64 MHz

Trigger

Why is an FPGA necessary?

112 megabytes / second per channel

Times 8 channels 896 megabyte / second (each board)

FPGA is the only device which can continously process such data rates.


Event-builder

DPPboard

Triggersystem

Recording

Why is trigger necessary?

DPPboard

DPPboard

DPPboard

Signalsfrom

detectors

DPPboard

Unrestricted data rate 896 megabyte / second (each board)

Such data rates can be neither managed nor recorded.

Trigger subsystem pre-selects

only “good data”

to be recorded.

Pre-selected data rate

becomes manageable

and can be recorded.

Subset ofsignals

fromdetectors


Limitations of the backplane architecture

and the solution:

point-to point fast serial links


Limitations of the VME backplane readout

Full event data (full waveforms)20,000 16-bit words per channel

Trigger data4 * 16-bit

Trigger data event builder. 144 channels * 8 bytes = 1152 bytes = 29 s (MBLT).

Waveforms event builder. 144 channels * 40 kbytes = 5.760 Mbytes = 144 ms (MBLT).

It means that the VME system can read only 7 full events per second, using 32-bit MBLT

protocol, or 14 events per second using the 2eVME protocol.

The limitations of the PCI readout will be roughly similar .

• Assume 200 sec of waveform memory per channel.

• 14-bit ADC means 15 bits (because of the ADC overflow bit).

• For simplicity let’s say 1 sample = 16 bits.

• Full waveform is 12,800 @ 64 MSPS, or 20,000 @ 100 MSPS (one ADC channel).

• Trigger data means (pulse area, pulse width, time stamp) = four 16-bit words = 8 bytes.

• If using VME interface, then two types of data transfer cycle are available:

MBLT transfer, 4 bytes (32 bits) per transfer 40 MB/s = 40 bytes / s

2eVME transfer, 8 bytes (64 bits) per transfer 80 MB/s = 80 bytes / s


Implications of the backplane performance estimate

The VME system can read no more than 7 full events per second, using the 32-bit MBLT

transfer @ 40 MB/s. (No more than 14 full events/s using the 2eVME protocol @ 80 MB/s).

Moreover, these rates will be achieved at 100% deadtime, what is not a good situation.

What can we do?

1. Use the trigger to pre-scale the event rate to only 7 “interesting events per second”.

2. Do not read full waveforms. Read only the pulses, and skip the quiescent baseline.

3. Compress the waveforms to reduce the transfer rate.

4. Use faster transfer rate.

Ad 1. Now we see, why the trigger is of such importance in this project.

Ad 2. “Baseline suppression” seems unavoidable in this situation.

Ad 3. Real-time compression requires appropriate FPGA resources.

Ad 4. A point-to-point serial data link offers MUCH HIGHER data transfer rate than VME.


Backplane switching currents may induce noise

An additional consideration is digital noise which may be injected into the sensitive analog

inputs by large switching currents inherent in the single-ended backplane such as VME or

PCI. Both are old-style single-ended bus interfaces involving large transient currents. In such

high-current environment it may not be possible to attain millivolt noise level. It means, that

during the measurement the VME interface has to be kept inactive. The measurement cannot

be restarted until the VME readout is over. The VME readout period is defining the dead time

of the DAQ system.

Vendors of commercial VME or PCI digitizers claim that the above does not happen.

However, in an experiment which is pushing the detection limits towards low-amplitude

signals, it is prudent to verify whether or not digital switching currents are indeed harmless.

A radical solution is to use low-noise standard such as USB-2 or HDMI, which employ Low-

Voltage Differential Signalling (LVDS). Only 3.5 mA per link is being switched in a

differential mode, which avoids inducing noise in nearby circuits.


Fast serial link fast trigger decision, but similar event readout

• Do not use a backplane such as VME or PCI. Use differential point-to-point data links.

• HDMI: four differential pairs carry up to four DDR data streams from each DPP board to the

event builder. Very high data rate can be pushed through a short HDMI cable.

• Flat-pack FPGA packages limit the signaling rate to 200 Mbps. (Implies limitation: 200

Mbits/s * 4 links = 100 Mbytes/s per cable.)

• BGA packages allow the signaling rate of 622 Mbps. (Implies limitation: 622 Mbits/s * 4

links = 311 Mbytes/s per cable.)

• 100 MB/s from each board means a huge improvement over a backplane readout, because

all links operate in parallel.

• The most dramatic improvement is when transferring short trigger packets from the DPP

boards to the Event Builder, because the trigger information can fit into the on-chip FPGA

memory at the receiving end. The trigger readout rate is dramatically improved.

• There is not enough memory in the receiving Event Builder FPGA to accept the entire

waveforms from all DPP boards. They need to be transferred out at the USB-2 rate, which is

similar to the backplane rate. Therefore, the full event readout rate is not improved.


Bottlenecks of the digital pulse processing

system


Three bottlenecks of the digital recording systemA typical digital system has three bottlenecks, which have to be tackled in system design.

1. Raw Data Bottleneck.

The rate at which a digital system can produce raw data is staggering. For example, a

modest system of 100 channels, 14 bits @ 100 MSPS produces 17.5 gigabytes per second.

No matter, how large are on-board waveforms memories, they will quickly overflow with

raw data. A digital system must either provide a) method to offload the waveforms

memories at the same rate, at which they are being filled, or b) it must limit the rate of

data production to a more manageable level.

2. Data Recording Bottleneck.

Data must be recorded. A typical recording medium (a disk or a tape) can take roughly

100 MB/s. A disparity between #1 and #2 is roughly 103.

3. Analysis Bottleneck.

Data must be analyzed. If we record at 100 MB/s, then we are recording 1 GB every ten

seconds. Analysis can take 10x more CPU effort than recording (optimistically).

Therefore, improving the recording rate (bottleneck #2) will exacerbate the analysis glut.


How to remedy the three bottlenecks?Some digital DAQ systems take only occasional records of data, followed by long periods of

inactivity (e.g., a digital oscilloscope which is recording occasional pulses). In such cases, the

average data rate is low, and there is no problem. However, in some cases the DAQ has to

work full time, all the time. In such cases we have to tackle the aforementioned bottlenecks.

Raw Data Bottleneck can be addressed with real-time data reduction (i.e., digital pulse

processing) which extracts only the relevant pulse characteristics, such as amplitude, duration,

and pulse shape parameters. Rather than storing the full waveform, we can store only a few

numbers. The reduction factor can be 10x or more. Even larger reduction can be achieved,

when pulses are infrequent, separated by long stretches of uninteresting baseline. The baseline

does not have to be recorded at all.

Data Recording Bottleneck cannot be improved, because any improvement in this area will

impact the next bottleneck. (Whatever is recorded, has to be analysed.)

Analysis Bottleneck can be eased if the pulse data have already been preprocessed.

Real-time pulse processing is the key to designing a large digital DAQ system.


What is needed to implement real-time pulse processing?

Real-time pulse processing is the key to achieving the data reduction necessary in a large

digital DAQ system. There are three key ingredients of real-time digital pulse processing.

Adequate raw processing power needs be present. One has to employ FPGAs on the digitizer

boards because only the FPGAs provide parallel operation on parallel sample streams. Both

FPGAs and digital signal processors (DSPs) may be used on the downstream boards,

depending on requirements of a particular DAQ system.

Pulse processing algorithms have to be implemented in the FPGA fabric. The FPGA

implementation of even simple algorithms (e.g., pulse detection) can be tricky. It requires

expertise, which is not always available in physics community.

Confidence, that the digital pulse processing is working and it is safe. Physicists are very

reluctant when it comes to discarding the data. Data reduction does lead to discarding the data,

and therefore it is viewed with suspicion. From the safety point of view, the prefered solution

is to record every sample, and then to analyze the recorded data from disk. Such a preference

leads to bottlenecks mentioned on the previous slide.


Can we trust real-time pulse processing?

Physicists are very reluctant to discard the data. Digital pulse processing leads to discarding

the data. It is therefore a valid question, whether real-time digital pulse processing is a good

idea?

• Pulse processing needs to be adequately developed, understood, and tested in order to be

reliable.

• The critical portion of waveforms can be recorded in order to perform offline crosschecks

with the DPP results.

• In the good-old-days the analog pulse processing was performed all the time. We did not

worry, that by doing so we were discarding good data, because there were no “good data”

to be recorded other than the output from our shaping amplifiers. Nowadays we need to

realize, that “shaping amplifiers” could as well be named real-time analog filters. Such

analog filters are neither better nor worse than the digital filters. If we used to trust the

former, we should also accept the latter. Whether the filter is digital or analog, it will be

equally trustworthy, when it is properly designed.

• Digital pulse processing is necessary. It needs to be accepted after it is understood.


How is trigger implemented?


What is trigger doing?

Trigger subsystem pre-selects all interesting events, a representative

sample of background events, and also some noise.

A strobe signal (a pulse) is sent to DAQ to trigger its operation,

such that the accepted data can be digitized and recorded.

Triggersystem

Interesting eventsBackground eventsNoise

From detectorStrobesignalto DAQ


How is trigger implemented?

• Trigger is such an important topic, that we need to look at the implementation

details of the trigger.

• Three different ways of implementing the trigger:

1. Analog computer built from off-the-shelf NIM1) modules.

2. Small DPP system processing a subset of the detector signals.

3. The main DPP system also performs the triggering function (i.e., the DAQ

system is self-triggered).

1) Nuclear Instruments and Methods modules (NIM) are shown later.


Event-builder

DPPboard

Triggersystem

Recording

How is trigger connected to the DAQ?

DPPboard

DPPboard

DPPboard

Signalsfrom

detectors

DPPboard

The role of the trigger is to pre-select “interesting events”.

Strobe signal (a pulse) to the DAQ.Subset ofsignals

fromdetectors

DAQ


Very many LEMO cables

Five crates full ofNIM modules

Trigger 1: analog computer built from NIM modulesAnalog signals processed in analog domain


Pros and cons of NIM implementation

Advantages:

• Knowledge about programmable logic is not needed.

• Very little training required to work on a NIM system. (Every physicist knows how to use an

oscilloscope and how to adjust trimming potentiometers.)

• Practically unlimited number of analog inputs. (However, system complexity grows very

quickly with the number of analog inputs, as shown in the previous slide.)

Disadvantages:

• Tedious to document. The only documentation are oscilloscope screen shots and a written

record in a logbook.

• Very difficult to reproduce the settings.

• Flaky operation because of many knobs, switches, and connectors.

• Tuning cannot be performed remotely. continued...


The main advantages of NIM implementation

• Very little learning is required to operate the equipment.

• Signals in NIM electronics can be examined anytime with oscilloscope. No planning is

required concerning signal diagnostics.

• Very wide bandwidth and fast rise time of pulses (approx. 1 ns).

• Timing is adjusted in analog domain. According to common wisdom, analog means

infinitely fine time adjustments between pulses. (But in practice the fine time adjustment is

very tricky.)

• Timing can be measured and adjusted to a few tens of picoseconds.

Why is the antiquated analog approach still being used?


DDC-8 Pro (LUXcore trigger system)

Trigger 2: a single Digital Pulse Processor

Analog signals digitized and then processed in digital domain

8 analog signals from phototubes

NIM OUT, 2 lines

NIM IN, 2 linesfrom optional NIManalog computer

The NIM OUT pulse is sent to DAQ system

FPGASpartan3-400performs digitalprocessing in real time


Pros and cons of a single DPP solution

Advantages:

• Tuning can be performed remotely.

• Trigger can be documented in full by saving all the settings and configuration files.

• Settings can be reproduced exactly by reloading configuration files.

• Mechanical knobs, switches, and connectors are eliminated reliable operation.

Disadvantages:

• Only a few out of many phototubes can participate in the trigger decision.

• Knowledge about programmable logic is needed to develop or modify any DPP system.

Very few physicists have working knowledge about programmable logic.


This architecture will be used for the LUX trigger

Trigger 3: a few Digital Pulse Processors

Decision pulsesent to DAQ system

A few front-end boards

A single Level-2 decision board

Fast uni-directional data links


Pros and cons of the several DPP solution

Advantages:

• (A few advantages similar to the previous case of a single DPP board.)

• Much larger number of inputs.

Disadvantages:

• (A few disadvantages similar to the previous case of a single DPP board.)

• This architecture is in fact the same as the DAQ architecture. Plainly speaking, this is a

small and less powerful DAQ system. Why invest time and effort in trigger-only solution?

Why not design and build a COMPLETE self-triggered DAQ instead? Why bother having

two PARTIAL systems rather than one COMPLETE system?


Trigger 4: self-triggered COMPLETE DAQ system

Recording

Front-end DPP boards Level-2 event builders

Fast bidirectional data links.(Bidirectional in order to send the trigger decision back to the front-end boards.)

Level-3 event builder


Self-triggered DAQ system: how it works.

Recording

Front-end DPP boards Level-2 event builders Level-3 event builder

Full event data (full waveforms)Trigger data

1. All front-end boards send “trigger data” downstream to the central event builder.

2. The event builder makes the trigger decision (accept, reject) and sends it back.

3. If the decision is “accept”, then the front-end boards send the waveforms.

The communication is carried over very fast, point-to-point, bidirectional data links.


Pros and cons of the self-triggered DAQ system

Advantages:

• Merging the DAQ with the trigger will avoid duplication of effort.

• All phototubes can participate in trigger decision.

Disadvantages:

• New boards need to be developed (done in 2008!). Commercial boards do not provide

sufficient level of “system integration” to build such a system.

• This architecture is quite advanced. If DPP looks like black magic, then the new advanced

trigger/DAQ system will look even more like black magic.


Inter-board Data Link

(System Connector)


Recording

Fast bidirectional data linkswith ~gigabit/s performance.

The role of the System Connector

The point-to-point data links are dedicated. Each front-end board can offload its data

with gigabit/s performance without waiting for other boards.

• Each DPP board has its own link to the back-end Event Builder.

• The link provides all the facilities for system integration.

• Therefore, the link will be named System Connector.


Physical standard of the System Connector

HDMI Type A or Type C connector/cable:

19 pins total: 7 single-ended, 4 differential pairs, 4 ground pins

4 differential pairs (3 data + clock), nominally 340 MHz per pair (category 2 HDMI cable)

USB for comparison

The Fast Data Link cable and connectors have to be properly designed for fast data

transfer. A home-made ribbon cable does not guarantee sufficient signal integrity. The

video-oriented HDMI cable provides good signal integrity and it is relatively cheap.

One meter cable $13, PCB socket $5.61

http://en.wikipedia.org/wiki/Image:HDMI-CloseupOfPlugs-vs-USB.jpg


System Connector wire diagram (nominal)

DPP board Event Builder4 differential pairs340 MHz per pair (nominal)

7 wires for slow control signals

1

2

3

4

The HDMI wiring table will be used as guidance. In our application the wires

will be used differently because we are transmitting arbitrary digital data rather

than video data. Most important: we are using the connector/cable compliant

to its electrical standard. Therefore, signal integrity is guaranteed.


Fast data links and RocketIO

• The newest high-performance Virtex5 is equipped with several RocketIO transceivers,

which provide multi-gigabit signaling rates per one differential pair. It is therefore a valid

question, whether or not RocketIO should be used to implement the Fast Data Link?

• I believe it is not a good idea to use RocketIO in this case. RocketIO requires special

stripline design techniques, as well as connectors of higher performance than HDMI. Rather

than using RocketIO, gigabit rates can be achieved by several (up to four in case of HDMI)

differential pairs with lower signaling rate, as well as inexpensive HDTV-grade connectors.

• Low-cost Spartan-3 series FPGAs do not support RocketIO.

• I may consider RocketIO in the future versions of my boards.


System clocking

and time stamping


ADC clocking

• In order to correlate the waveforms, all ADCs must be clocked off a single clock source. The

clock signal is distributed to all DPP boards and connected to ADCs and to the FPGAs.

• The clock must be high quality: stable frequency and low jitter, as required by the ADC

specs.

• The simplest solution: the ADC clock is distributed at its actual frequency (e.g., 64 MHz or

100 MHz). A more complex solution is to multiply the clock. In case it is multiplied on-

board, then the multiplication circuit must be of high quality (e.g., a phase-locked loop). The

Digital Clock Managers provided by the FPGAs may not meet the ADC specs.

• The ADC clock is best distributed using its own shielded cable in order to avoid noise pickup

from adjacent signals. Backplane clocks are to be avoided.

• Hopefully, clock quality will be good enough when it is distributed with LEMO cables and

NIM fan-in/fan-out. This needs to be tested.

• In case the single-ended clock quality is not sufficient, differential circuits must be used.

Mini-USB is a possible candidate differential cable/connector (used in a non-standard way).


Time stamping• In addition to the ADC clock, a separate time-stamp clock may be used in the system. This

clock provides a universal time base for the entire system, especially in those cases, where

some DPP boards operate at a different ADC frequency from other boards. (E.g., commercial

DPP boards from various vendors are used in addition to our DPP boards.)

• The time-stamping clock needs not be of very high quality. Backplane clocks or wire bundles

are OK. The System Connector is also OK to distribute this clock.

• The time stamping clock can be distributed at a lower frequency and multiplied on-board,

using the Digital Clock Manager (DCM) provided by the FPGAs. The DCM timing jitter will

not impact time-stamping. (The DCM should not be used for ADC clocking, or used only

with caution and after verifying that it meets the ADC specs).

• A logic signal “reset time stamp” is used to synchronously start time stamping counters in all

DPP boards. This signal may be distributed over the System Connector, or it can use its own

cable. Effectively this signal assumes the role of “start a new run”.

Two signals are sufficient to implement time stamping: the clock and the reset.


Present status of

Rochester electronics

for Dark Matter Search


Spy channel out64 MHz * 12 bits

FX2 USB-2,control anddata readout

FPGASpartan3-400400,000 gates

ADC, 8 channels, 12 or 14 bits @ 64 MHz 768 or 896 megabytes/s

Analog signal IN8 channels, +/- 1 Volt

NIM OUT, 2 lines

USB connector

NIM IN, 2 lines

Diag LED display

The first DPP board: DDC-8 Pro (limited to 8 phototubes)

Developed in 2006 and well tested.

JTAG

RS-232


DDC-8 LUX prototype manufactured in 2007 and used now

FX2 USB-2

FPGA: Spartan3E: 500,000 gates

Active Filters

ADC up to 80 MHz

8 channel inputs2*NIM in

2*NIM out

USB connector

Pulse out(spy channel)

Power

Power

Passive Filters

JTAG

HDMI connector

CLK

2*NIM inused for

time-stamping

RS-232


DDC-8 DSP prototype manufactured in July 2008

63

FX2 USB-2

FPGA: Spartan3A DSP: 3.4M gates

Active Filters

ADC up to 125 MHz

8 channel inputs2*NIM in

2*NIM out

USB connector

pulse out (spy channel)

Power

Passive Filters

JTAG

HDMI connector

CLK

2*NIM inused for

time-stamping

RS-232


The Trigger Event Builder

Spartan3E500,000 gates

FX2 USB-2

The Event Builder is a “supervisor” board developed for the LUX trigger.

Revision 0 was manufactured in December 2007.

Eight System Connectorsconnected to DDC’s

Up to 100 MB/s per link

USB 2.0up to ~40 MB/s

Clock IN * 1NIM IN * 5NIM OUT * 4

NB: the photo shows an unpopulated PCB. The actual board was manufactured and tested,but I did not take a photo.


Present status of the Rochester Digital Trigger (Jul/2008)

• The digitizer is finalized: 8 channels, 14 bits @ up to 125 MSPS, FPGA with 3.4M gates,

USB-2, and a fast System Connector transferring data to the downstream Event Builder.

• Minor corrections of the DDC-8 DSP Rev. 0 are possible. Major changes are not planned.

• Noise performance of the previous editions of DDC-8 was excellent. I expect similar

great performance from the new DDC-8 DSP.

• The testing and porting firmware will start in July 2008 and continue through Fall 2008.

• The Revision 0 of the Event Builder (EB) is finished and tested.

• This revision of the EB still uses a small FPGA with 0.5M gates.

• LUX trigger firmware most likely will fit into this FPGA.

• Firmware development and system integration will continue through Fall/2008.

• A new EB version with a large FPGA is planned. The urgency of this board depends on the

performance and limitations of the Rev. 0 Event Builder.


Software and firmware for the

self-triggered DAQ system


Approaching system integration• Previous slides showed the components of the self-triggered DAQ system: the boards and

the history of their development. As of July/2008 the components are mostly finished.

• The digitizer DDC-8 DSP is finalized. Testing will be done in Summer and Fall 2008. After

the tests I plan to make as many digitizers as needed, subject to available funding.

• The Event Builder Rev. 0 would benefit from minor layout improvements. The present

prototype EB board will be used in Fall 2008 to develop the LUX trigger system. We will

need a few EB spares, and therefore I plan to implement minor corrections to the EB Rev. 0

before making a few copies of it.

• On a longer time scale, a more powerful EB will be designed (with a larger FPGA). The

details of the new EB will be decided after the first version of LUX trigger is working.

• The system integration consists of cabling (a minor issue) and of software development.

Single-board software is well developed. Multi-board software is in planning stages.

• The digitizer FW/SW is well advanced (waveform capture, readout, etc.). It is being used.

• The EB firmware/software development has recently started.


SoftwareData readout, control panels, the GUI, spectrum displays, etc, are all extremely well

advanced over many years of digitizer development (since 2002). As an example, I am

showing a screenshot from circa 2004. This software is a very robust basis for further

development. We can show many more such sleek pictures.


Digitizer development system

This photo shows the digitizer development system in Spring 2004. Since then

significant progress was made in hardware, software, and firmware.


Digitizer performance


Noise performance

There are many aspects of this project which I am not going to even mention in this last

section, such as digital filter design and their sensitivity to pulses of various shapes, trigger

latencies, or readout speed. All this is ongoing and well advanced, but also very detailed and to

a large extent specific to the LUX experiment.

In this last section I want to show the noise performance of my designs, which turns out to be

excellent. It should be obvious to an informed reader, that low noise is the key to anything

else. The noise is affecting trigger sensitivity, its ability to detect small pulses, as well as

energy resolution in spectroscopic applications.

After seven years of development I have built the device, whose performance is adequate for

serious scientific work. The work is continuing towards applications of the digitizer in one of

the most demanding experiments.

Many applications are possible in addition to LUX. The DDC is a general-purpose digitizer

equipped with a powerful on-board pulse processing engine, which can execute a variety of

application-specific algorithms.


Low noise low thresholdI designed the analog part of my digitizers in 2003 and I have been using it with very few

modifications since then. Its noise performance is excellent. As an example, I am showing -

ray histograms collected with a 1” by 1” NaI(Tl) detector with the original 8-channel DDC-8. A

very low threshold ~5keV will be very important for LUX.

Counts

Energy (keV)0.0 100.0 200.0 300.0 400.0 500.0 600.0 700.0 800.0

0.0

200.0

400.0

600.0

800.0Energy

Small NaI(Tl)137 Cs

662 keV

33 keV, Ba X-ray

Compton back-scatter

Counts

Energy (keV)0.0 200.0 400.0 600.0 800.0 1.0E+3 1200.0 1400.0

0.0

100.0

200.0

300.0

400.0Energy

Small NaI(Tl)60 Co

1332 keV1173

Compton

back-scatter


Low noise of the waveforms high sensitivity

The waveform was collected with a low-noise signal source connected to the 12-bit version of

the digitizer DDC-8 Pro. The noise in this waveform is dominated by the flash A/D chip, and

therefore it can hardly be any better. Very low sample-to-sample noise seen in these data will

be very important for pulse detection in LUX.

RMS noise = 0.214 mV

600

500

400

300

200

100

0C

ount

s-17 -16 -15 -14 -13 -12 -11

ADC value (0.488 mV per unit)

DDC-8 Pro, 12 bits @ 64 MSPS Projected noise distribution

RMS noise = 0.214 mV

-16

-15

-14

-13

-12

0.48

8 m

V p

er u

nit

500400300200100

Samples (15.625 ns per sample)

DDC-8 Pro, 12 bits @ 64 MSPS


Summary• Digital Pulse Processing was explained.

• Several trigger architectures were presented .

• A novel fully Digital Trigger is composed of a few digitizer boards and one Event Builder.

This architecture will be used for LUX trigger.

• The same approach can be used to build a complete self-triggered DAQ, which can be

expanded to hundreds of channels by cascading Event Builders. Such a DAQ does not need a

separate trigger. It can trigger itself by quickly transferring short “trigger packets” to one

central location, which is then issuing a system-wide trigger.

• DDC-8 LUX board was manufactured in December 2007. Parameters: 12/14 bits @ up to 80

MSPS, FPGA with 0.5 million gates, external clocking and time stamping, several NIM in and

out connectors, and a System Connector. It is being used now to develop the Digital Trigger.

• DDC-8 DSP board was manufactured in July 2008. Similar to the LUX board, but with 3.4

million gates, and sampling up to 125 MSPS. It will replace the LUX board in Fall/2008.

• The Event Builder Revision 0 was manufactured in December 2007. It will be used to integrate

the LUX Digital Trigger. A more powerful Event Builder is planned.

Documents

W.Skulski University of Rochester, July/10/2008 1 Wojtek Skulski University of Rochester The Universe and the FPGA Digital Pulse Processing for Dark Matter