Addressing the Software and Addressing the Software and Hardware Trade-Offs of an Hardware Trade-Offs of an
Embedded Distributed Embedded Distributed SystemSystem
The Case of the ANTARES Data Acquisition and Detector Control
S. Anvar, CEA Saclay DapniaOn behalf of the Antares collaboration
2
RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
http://antares.in2p3.fr
A Neutrino TelescopeA Neutrino Telescope
3
RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
The Antares CollaborationThe Antares Collaboration
Institutes Particle physics Astronomy Sea science
4
RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
ConstraintsConstraints
Environmental constraints Baseline data rate of ~100 kHz / PMT Data rate fluctuations ~2MHz / PMT due to bioluminescence Strong uncertainties on environment parameters Reduced space and thermal conductivity …
Reliability (MTBF) Maximize simplicity Minimize consumption Minimize number of components …
Deployment constraints High cost of submarine cables Limits on string cable rigidity or sea operations …
5
RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
Design principleDesign principle
Stringcontrol
Junctionbox
Electro-mechanicalcable
Electro-opticalcable (50 km)
Detectors
Slow controlDataClockEnergy
Processingnodes
Acquisitionnodes
OffshoreOnshore12 strings25 detecting nodes per
string (every 12.5m)3 optical modules (PMTs)
per acquisition node2 "ARS" digitizing chips per
optical module
Real time processing: Offshore: readout 1800
ARS chips spread out over 30 000 000 m3 @ 2500m underwater
Onshore: perform real time trigger computations on incoming data.
6
RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
Each offshore acquisition node: one DAQ/SC Each offshore acquisition node: one DAQ/SC BoardBoard
ARS readout/SCLocal trigger
106 gatesXilinx FPGA
ARS readout/SCLocal trigger
106 gatesXilinx FPGA
Real-timeembedded CPU
PPC MPC860VxWorks
Real-timeembedded CPU
PPC MPC860VxWorks
SDRAMMemory
64 MB
SDRAMMemory
64 MB
DAQ/SC Board
Achieved power consumption of ~5W including optical Ethernet tranceiver laser
Ethernet 100
Serial links(RS485, RS232)
7
RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
DAQ/SC BoardARS chip
The ARS chip and the DAQ/SC boardThe ARS chip and the DAQ/SC board
8
RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
Network Topology (1)Network Topology (1)
Cable rigidityCable diameter
Connection complexity
Each
LC
M h
as it
s ow
n I/O
dat
a flo
w(S
low
Con
trol
& P
hysi
cs D
ata)
SCM
Sectors
M
SCM
M
To Shore(DWDM)
100 Mb/s
1 G
b/s
9
RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
Network Topology (2)Network Topology (2)
10
RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
Offshore switch Board
Allayer chip
The offshore switch boardThe offshore switch board
11
RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
The electronic crateThe electronic crate
12
RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
Used as data concentratormany input ports one output port
Network Topology (3)Network Topology (3)
Solution barrel shifting synchronized by global clock
PC PC PC PC PC PC PC
ON-SHORE SWITCH: (in: 60x1000, out: 20x1000)
SECTOR SWITCH (in: 5x100, out: 1x1000) SECTOR SWITCH (in: 5x100, out: 1x1000)
N N N N N N N N N N
risks of congestion intelligent flow control in node
~15 Mb/s
Instant ~6 Gb/s
13
RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
Massively Distributed and Heterogeneous Massively Distributed and Heterogeneous SystemSystem
Final System: A Network of ~320 Nodes ~300 Offshore Nodes (MPC860 + VxWorks) ~20 Onshore Nodes (PC + Linux)
A number of distributed applications running in parallel and communicating DAQ L2 Trigger Slow Control / Acoustic positioning acquisition L1 trigger ? Run control
14
RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
Software TechnologiesSoftware Technologies
Programming Languages C/C++ for offshore software components C++ / Java for onshore processing Java / C++ for onshore display
Distribution Multiple messaging and data servers (ControlHost) TCP/IP based VxWorks Zero-copy TCP/IP (~25Mb/s ~38 Mb/s per node) Offshore network traffic control (barrel shifting)
Data and configuration storage Oracle database Root files
Design & Development Object Paradigm UML (Unified Modeling Language) + Extensions Model-driven engineering
15
RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
Interrupt – shared memory communicationInterrupt – shared memory communication
ARS readout(FPGA)
intrHandler(RTOS)
frameSendertask (RTOS)
Semaphore(RTOS)
Data Memory(SDRAM)
write
read andwrite
Actual frameformatting andsending throughTCP/IP socket
16
RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
Interrupt – shared memory communicationInterrupt – shared memory communicationseen as asynchronous method callseen as asynchronous method call
ARS_Read
+fillBuffers(id:int)
frameSender
1
6
FrameSender
+acq(in frameID:int)
acq(in frameID:int)
:ARS_Readout :FrameSender
6
<<fpga>>AR_Node
readout
Interrupt - SharedMem
1
<<RTOS>>FS_Node
framer
FrameSenderARS_Readout
<<reside>><<reside>>
asynchronous method call
reference to design pattern
17
RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
Applying the design pattern: model Applying the design pattern: model transformationtransformation
ARS_Read
+fillBuffers(id:int)
frameSender
1
6
FrameSender
+acq(in x:Frame)
Implemented in VHDLImplemented in VHDL
IntrSharedMemProxy
+doIntr(in x:PrmRefType)
FrameSenderSkeleton
+accept(in x:Frame)
IntrSharedMemSkeleton
#run()#installInterrupt()+accept(in pr:PrmRefType)
PrmRefType
{bind(Frame)}
RealTime::Thread
#run() reusablesoftware
1
18
RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
Widespread beliefWidespread belief
Performance goes against abstraction & Performance goes against abstraction & flexibilityflexibility
Performance goes against abstraction & Performance goes against abstraction & flexibilityflexibility
Wrong if the issue is explicitly addressedWrong if the issue is explicitly addressed
ActuallyActuallyAbstraction Abstraction modularity modularity specific optimization specific optimization perf. perf.
Wrong if the issue is explicitly addressedWrong if the issue is explicitly addressed
ActuallyActuallyAbstraction Abstraction modularity modularity specific optimization specific optimization perf. perf.
19
RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
Evolutions (1)Evolutions (1)
ARS readout/SCLocal trigger
106 gatesXilinx FPGA
ARS readout/SCLocal trigger
106 gatesXilinx FPGA
Real-timeembedded CPU
PPC MPC860VxWorks
Real-timeembedded CPU
PPC MPC860VxWorks
SDRAMMemory
64 MB
SDRAMMemory
64 MB
DAQ/SC Board
Ethernet 100
Serial links(RS485, RS232)
ARS readout/SCLocal trigger
Real-timeembedded CPU
SystemOn Chip
Xilinx Virtex II Pro?Xilinx Virtex IV
ARS readout/SCLocal trigger
Real-timeembedded CPU
SystemOn Chip
Xilinx Virtex II Pro?Xilinx Virtex IV
SDRAMMemory
64 MB
SDRAMMemory
64 MB
DAQ/SC Board
Ethernet 100
Serial links(RS485, RS232)
Lower consumptionReduced number of components
increased reliability
20
RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
Evolutions (2)Evolutions (2)
Present scheme: « all data to shore »Unexpected variable baseline rates: ~100 400 kHz instead of ~70
21
RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
Evolutions (2)Evolutions (2)
Present scheme: « all data to shore »Unexpected variable baseline rates: ~100 400 kHz instead of ~100
Software L1 trigger Reduced L1 partial data sent from offshore to central L1 workstation Central L1 computes trigger with global vision of detector Trigger decision broadcasted through software Expected gain in data rate: x1000
From 5 to 30 cm/s (!)From 5 to 30 cm/s (!)
22
RT2005 – S. Anvar – CEA Dapnia – Adressing the HW and SW trade-offs of an embedded distributed system: the case of the Antares DAQ and detector control
ConclusionsConclusions
Slow controlDataClockEnergy
HEP DAQ and trigger techniques High data rates Very weak signal Highly distributed system Soft real time constraints …
Challenges unusual for HEP No known beam (calibration) Very embedded system Very hostile environment Uncontrolled factors (biolum) …
Both Robust and Flexible Design Allows adaptation Allows for remote evolution …