Embed Size (px)
Citation preview
The Effelsberg Direct Digitization Project
Radionet Workshop,
September 2020
Amit Bansod, Jan Behrend, Ewan Barr, Niklas Esser, Mathias Heininger, Sakthi Priya Sathyanarayanan, Tobias Winchen, Jason Wu
Contact: [email protected]
2
Direct Digitization
Current and future receivers with integrated digitizer/packetizer unit
UDP Ethernet output (40GbE / 100 GbE/ ...) Increased sensitivity Improved stability RFI mitigation techniques Reduction of hardware complexity ...
Increase science / € Reduced maintenance COTS components (network, computing) Adaptability to new / rare science cases:
» real time transients with machine learning,
» detection of extreme energy particles
3
Data Transport
Standard Ethernet data backbone (Infiniband, RoCE)
Data streamed in multicast groups, client subscribe to group
RobustnessScalability
SPEAD2 Protocol (VDIF)
Digitizer / Packetizer
Data Broker
DataClient
DataClient
DataClient
DataClient
DataClient
4
Processing Clients
COTS Server:MKRECV / MKSEND programs for
network traffics (w. ibverbs)
72+ Gbit/s w. < 0.1% loss PSRDADA ring bufferPSRDADACPP framework
https://github.com/MPIfR-BDG/psrdada_cpp
Processing on GPUs w. CUDA
FGPA processing boards: (ROACH2/SKARAB)
PCIe mounted FPGAs under investigation
Data Client
MK
RE
CV
MK
SE
ND
/V
DIF
SE
ND
AP
P
DA
DA
DA
DA
5
Client Control via KATCP
FPGA
Digitizer / Packetizer
FPGA ControllerFPGA Controller
KATCP Server(Python)
- configure / start / stop
Data Client
MK
RE
CV
MK
SE
ND
/V
DIF
SE
ND
AP
P
DA
DA
DA
DA
Docker Container
KATCP Server(Python)
- configure / start / stop
FPGA ControllerDigitizer/Packetizer Controller
KATCP Server(Python)
- configure / start / stop
All clients encapsulated inside docker containerFlexibilityReproducibility
6
Uniform State Machine for all ComponentsE
rro
r
Idle
Streaming Ready
Set
Running
Pa
nic
Configured
Measurement Stop
Measurement Prepare
Measurement Start
Configure
Deconfigure
Intermediate states during execution (Configuring, capture starting, …)Intermediate states may also result in error state
Capture Start
7
Control Framework: MPIKAT
Abstract Pipeline Implements server with
default katcp commands Provides standard sets
of sensors Unified commandline args System to check provided config
options
Utility Library: Ip address handling Subprocess management Management of dada buffer Numa inspection ...
Source Repository:
https://gitlab.mpcdf.mpg.de/mpifr-bdg/mpikat
Abstract Interface (EDDPipeline)
SpectrometerPipeine
MasterController
SkarabController
PulsarPipeline
...
8
Master Controller
Err
or
Idle
Streaming Ready
Set
Running
Pan
ic
Configured
Unprovisioned ProvisionDeprovision Extends state machine:
`unprovisioned’ Provisioning:
Boot up of individual container on per observation basis
Connect container to requested data streams
Relay commands to individual components
9
Provisioning
?
All container in central docker registry Container launch + build using ansible
(execution manually or by master controller) Semi-automatized allocation of resources:
Why not kubernetes?Not (yet?) NUMA awareRDMA capabilities unclearOverhead vs. Performance ?Manual control of components may be necessaryAnsible used to set up core system
- hosts: gpu_server[0] roles: - pulsar_pipeline- hosts: gpu_server[1] roles: - spectrometer_pipeline
https://gitlab.mpcdf.mpg.de/mpifr-bdg/edd_provisioningRepository:
10
Telescope Connection
MasterController
Eff. FitsWriterInterface
Eff. ScpiInterface
Eff. StatusServer
State DB(Redis)
DockerRegistry
Effelsberg Configuration
Digitizer / PacketizerHigh Speed Data
Observation Data
Control
Mosty telescope – agnostic architecture
© N
. Ta
cken
, M
PIfR
EDD backend
Telescope Control
11
User Frontend
12
Internal Monitoring
Central Logging Monitoring Dashboards
13
Currently Available Data Products
Spectroscopy:High resolution FFT Spectrometer (up to 32M Channels, 2 GHz Input
Bandwidth)» Gated, allows for high speed noise diode switching
Full Stokes Spectrometer (up to 512k Channels, 1 GHz Input Bandwidth))» Gated, allows for high speed noise diode switching
PFB on GPU (6 GHz Bandwidth)PFB on FPGA (Skarab, Roach2)
Basic VLBI Mode2 (4) bit downconversion of baseband + VDIF output
Pulsar PipelinesPulsar timingPulsar search
...
14
Conclusion
EDD: A versatile backend for new generation direct digitization receivers using modern software architecture
Commissioning observations with Effelsberg next week To be deployed at
TNRT 40m,MPG-SKA Prototype Dish ...
Software FOSS:https://gitlab.mpcdf.mpg.de/mpifr-bdg
Contact: [email protected]
