Online DAQ System: From Detector to Tape

June, 2000 NIU Workshop

DD

Online DAQ System: From Detector to Tape

T. YasudaFermilab


DD

• Overview• Hardware• Control System• Primary/Secondary Data

Path• DAQ Applications• DAQ in action• Conclusions


DD Overview

• The DØ DAQ system is divided into two components:

• Trigger system• Level 1 hardware trigger

components• Level 2 specialized processors• Level 3 crate readout and

software trigger components

• Online or Host system• Detector controls• Data Logging• Monitoring• Control room applications


DD Overview

• DAQ Architecture– Event data rate and

operational redundancy achieved by a high degree of parallelism

• Level 3• Host

– Capability for multi-user, multi-stream operation

• With central resource configuration manager

– Network-centric Host design


DD DAQ Components

DetectorDetector

FCCFCC

UNIXServers

NT Level 3

LinuxPCs

ControlRoomPCs

Controls Trigger and Readout


DD Hardware Description

• 3 Compaq/Digital Alpha Servers– d0ola: Alpha Server 4000, 1

processor, 466MHz, 500 MB memory

– d0olb: Alpha Server 4000, 2 processors, 600 MHz, 500 MB memory

– d0olc: Spec out by Aug 3. Probably Alpha Server ES40, 4 processors, 667 MHz

– Clustered / redundant– 500 GB shared RAID disks for online

apps and database (Mirrored)– 500+ GB local ‘data buffer’ disks,

fiber channel based (40M/sec)

Alpha ES40667 MHz

Intel III733 MHz

SGI 22002x400, R12k

SPECint2000 413 336 334

SPECfp2000 500 243 328


DD Hardware Description

• Linux/NT nodes– Buying 6 nodes with dual PIII, 600

MHz, 500MB memory, 2 graphics cards

– 3 Linux nodes and 3 NT nodes exist– Will run Vmware on Linux nodes

• Control system embedded 68Ks and PowerPCs (VxWorks)

• Network – Cisco 6509 GB Ethernet switch for

all FCH nodes– Satellite 100 MB switches in MCH– GB fiber to FCC

• Security– Access control filter to online

machines– Kerberos authenticated ssh session

only


DD Control & Monitoring

DetectorDetector

Readout CrateReadout CrateControls CrateControls Crate

Trigger and Readout

UNIX ServersControl Room PCs

EPICS Clients: Low Voltage High Voltage Rack Monitor 1553 Devices SMT Monitor FT Monitor etc

EPICS Clients: Low Voltage High Voltage Rack Monitor 1553 Devices SMT Monitor FT Monitor etc

Controls

Ethernet

1553

ORACLEORACLE

HardwareDatabase

EPICS DB GeneratorEPICS DB Generator

Vertical Interconnect


DD Control System

• Built upon EPICS control system– A ‘standard’ toolkit upon which

we’ve built DØ extensions– Lots of user-community supplied

tools

• ORACLE Hardware Database– Extract EPICS db from ORACLE– Web-based and batch interfaces

• Hardware Control– Low Voltage, High Voltage, etc

dedicated GUI applications

• Downloading – Registers, Pedestals, etc

• Significant Event (alarm) System

• Interface to Accelerator and Cryogenics controls systems


DD Status of Control System

• Calorimeter– Preamp PS, BLS PS, ADC PS

control exist– Pulser control (in progress)

• SMT– EIPCS records for Sequencer,

Sequencer control, VRB, VRBC, Emulator (in progress) exist and used in the test stands

• Muon– Tested communication using

1553 for PDT, MDT, SRC cards

• FPD– Used RM support to control

motors


DD Status of Control System

• Luminosity– Scalers and FE processing

results communicated to Accelerator via ACNET

• Cryo– Communicated with DMAX

system

• Common Tools– generic 1553 support– generic VME support– HV used in SMT, Muon, Lum

• V1 running for months, V2 work starting

– Diagnostic support for busses– Standard operator interface

(GUI)


DD Hardware Database

• Describes Control aspects of electronics

• Based on ORACLE– 2 instances of the database

( dev, user testing)

• Web based interface for entering, modifying, deleting records

• Python script for batch entries exists

• Calorimeter records in the database


DD Hardware Database


DD Significant Event (Alarm) System

• System to detect alarm conditions and state changes in the DAQ system– Server with DAQ components as

clients– COOR sends alarm and run control

messages– CR, DL, DD send alarm messages– Version 1 Display exists– Working on version 2 Display

(Summer student)– Need to integrate EPICS alarms into

the system (Fall 2000)• on IOC EPICS alarms -> ITC client• ITC client sends alarms to Server on

host• EPICS Alarm Handler can be used for

now


DD Significant Event System

Significant Event Server

Fault Watcher

Archiver

HV Control

Heartbeat

Display

Front-End

Run Control(COOR)

Process PeriodicHeartbeat

RunSuspend

FF F F

F

F Filter

SE Message

Filtered Message


DD Configuration & Run Control

DetectorDetector

L1, L2, TCCL1, L2, TCC

L3 SupervisorL3 Supervisor

L3 VRCL3 VRC

Readout CrateReadout Crate

L3 FilterL3 Filter

FCCFCC

Data Cable

Data Cable

Controls CrateControls Crate

UNIX Servers

NT Level 3

Ethernet

Ethernet

Linux PCs

Control Room PCs

Controls Trigger and Readout

Run ControlClient

Run ControlClient

ComicsComics

DSMDSM

COORCOORCollector / RouterCollector / Router

Data LoggerData Logger

Disk

Data DistributorData Distributor EXAMINEEXAMINE

RIPRIP

1553

Vertical Interconnect


DD Software Description

• Configuration Management and Run Control– Coordination (COOR)– User interface (TAKER)– Download manager (COMICS)

• Primary event path– DAQ State Manager (DSM)– Collector / Router– Data Logger– Event metadata manager

(SAM)– Event data manager (enstore)

• Secondary event path– Secondary DAQ Supervisor– Data Merger


DD Software Description

• Event monitoring– Data Distributor– Analysis applications (EXAMINE)

• DAQ Monitoring– Client/Server access to DAQ flow

statistics, trigger rates, etc

• Detector Monitoring– Front End active & parasitic

monitors

• Calibration– Client/Server interface to

database• Infrastructure

– Databases (ORACLE)– Task-to-task communication

(ITC)


DD Secondary DAQ Data Flow

Controls/Readout Crates

1553 BusVME Bus

FCCFCC

Linux PCsUNIX Server

EXAMINEEXAMINE

UNIX Server

Collector / RouterCollector / Router


RIPRIP

Disk


DataLogger

DataLogger

Disk

DataDistributor

DataDistributor

RIPRIP

Data MergeData Merge

Control Room PCs

EPICSCA Client

EPICSCA Client

Monitor GUIMonitor GUI

ExamineExamine

Examine GUI

Examine GUI

ITC

DetectorDetector

CA Link

EPICSCA Server

EPICSCA Server Detector DAQDetector DAQ

SharedData


DD Secondary DAQ System

• Alternative data path• Mainly used for monitoring

and calibration• Takes advantage of powerful

front-end processors• Uses the same data path as

the primary path after Data Merger


DD DAQ Monitor

• Monitors the status of DAQ subsystems (L1/2, CR, DL, DD)

• Collects statistics information from the subsystems

• C++ itc Server with python Display clients


DD DAQ Monitor


DD Event Monitoring:EXAMINE

• Samples and reconstructs event based on Stream IDs and trigger IDs

• Clients of Distributor– Network and file event transfer

modes work– Calorimeter EXAMINE

• used for preamp testing

– CFT EXAMINE• Getting ready for raw data

unpacking• MC packed data?

– SMT EXAMINE• used for SiDet data

– Muon EXAMINE• used for commissioning


DD Event Monitoring: EXAMINE

DetectorDetector

L1, L2, TCCL1, L2, TCC

L3 SupervisorL3 Supervisor

L3 VRCL3 VRC

Readout CrateReadout Crate

L3 FilterL3 Filter



Disk

Data Cable

Data Cable

UNIX Servers

NT Level 3Ethernet

Linux PCsControl Room PCs

Data DistributorData Distributor

EXAMINEEXAMINE

Express LineExpress LineROOT ClientROOT Client

ROOT ClientROOT Client

Trigger and Readout


DD Event Monitoring:EXAMINE

– Need:• L3 EXAMINE• Vertex EXAMINE• Preshower EXAMINE

– Planned improvements• Histoscope -> Root after NIU

workshop• on-the-fly histogram• e-browser• Framework improvement

– name server for accessing only the histograms


DD Online Event Display


DD Online Calibration

• Perform electronics calibration of sub-detectors and insert results to ORACLE database

• COOR controlled via Taker• Common server and database

interface for all sub-detectors• Calibration results transmitted

as special event messages through DAQ paths

• Current status– Successfully ran SMT calibration

at 1% test stand and NW test stand


DD Calibration Manager

Taker

COOR

COMICS

CratesCrates

CalibrationManager

CalibrationDatabase

Calib. DataProcessor

DatabaseInterface

Configure

Request start run

Request download

DownloadStart run

End run

Data

PedestalsGains

PedestalsGains

ComparisonResults

DatabaseAccess

End calib.

Start run, End run

Validator

Calib ManagerDisplay


DD Online DAQ in Action

• Electronics/DAQ Commissioning– 2 VRB crates with 10 cards

each, synchronized with SCL from TFW to L3

– L1 muon crate– 1 Muon Scint crate with 2

MRCs– 1 Muon PDT crate with 1 MRC– 1 Calorimeter crate– Combinations of 2 systems

done• but not with MCH2+MCH3

– 2 simultaneous runs done• 3 simultaneous runs require one

more L3 node or script runner



• SMT Test Stands– 1% and NW test stands

• 1 HDI, 1 Sequencer, 1 VRB, 1 VRBC, 1 VBD

• Download done with COMICS and database

– 10% test stand• 3 HDIs, a few Interface Boards, a

few Sequencers, a few VRBs, 1 VRBC, 1 1553 controler, 1 VBD + L3

• Download done by spread sheet for now

– Databases (Electronics) exist for all three stands

– Calibration run performed at 1% and NW test stands



• Commissioning Run– Two detectors installed for the

upcoming Commissioning Run• Run I Luminosity scintillation

counters

• Forward Proton Detector

– Both detectors will be read out using the Run II Online system.

– Data will be transferred to and from the Accelerator Controls System via the EPICS/ACNET Gateway.


DD Conclusions

• All of the DAQ components exist and function.

• Improvements are implemented daily following user suggestions.

• We have been intimately involved in daily commissioning activities for the past few months.

• Bring in your sub-detectors!!

Documents

Online DAQ System: From Detector to Tape