Upload
penelope-hood
View
220
Download
0
Tags:
Embed Size (px)
Citation preview
Workshop in ALMA Logs
Prepared by Juan Pablo Gil – Arturo HoffstadtADC Software GroupALMA Observatory
March, 2015
ALMA Hardware Layout
• 66 Antenna
• ~45 device per antenna
• AOSTiming
• Photonic References
• 64 ant-correlator
• 16-antenna correlator
ALMA Software Layout
• ALMA Software is built upon ACS
• Different Subsystems
• CONTROL• Antennas• Time reference
• SCHEDULING
• DATACAPTURER
• BLCORR (64 ant)
• ACACORR (16 ant)
• TELCAL
• QUICKLOOK
• EXECUTIVE
• ARCHIVE
ACS Deployment
• Component
• Actual code that controls one physical device, or does one specific task
• Eg. IFproc, DTX, ARCHIVE_CORR_BULKRECEIVER_NT, CONTROL_MASTER
• Container
• Logic grouping of components.
• Has PATH/containerName
• Eg. CONTROL/DV01/cppContainer has IFProc, DTX
• Host
• Computer which runs containers.
• Eg. dv01-abm runs CONTROL/DV01/cppContainer
• xx-abm, gasXX, cob-cpn-xx, lo-lmc-x, etc
dv01-abm
CONTROL/DV01/cppContainer
Other containers…
IFProc ADC
Mount WCA
ALMA Logging
• Human-readable description of one action at one specific time
• ~30 GB of every day log
• Container Logs:
• Coarse grained (Info)
• Local to machine
• Stored in plain text files
• XML Logs:
• Fine grained (Debug)
• System wide
• Stored in XML files
Introducing high performance distributed logging service for ACS - Avarias et all, 2010
XML Log Example
• A typical XML log entry
<Debug
TimeStamp="2014-06-07T17:14:50.658" File="WVRImpl.cpp” Line="665” Routine="void WVRImpl::publishWVRData(const Control::WVRData#, ACS::Time)” Host="pm01-abm" Process="CONTROL/PM01/cppContainer" Thread="CONTROL/PM01/WVRMonitorThread" Context="” SourceObject="CONTROL/PM01/WVR" Audience="Developer” >
<![CDATA[Sucessfully published the following WVR event. Antenna: PM01 Mid-point of data: 17:14:49.920 Temperatures: 284.069, 284.029, 284.105, 284.014 deg. K. Sampled every 1.152 seconds. Event is valid. Data monitored at: 17:14:50.496]]>
</Debug>
• Log Levels: TRACE, DEBUG, INFO, NOTICE, WARNING, ERROR, EMERGENCY, …
• 2.5 millions of XML entries in 4 hours of observatory operation.
• Valuable information hidden in Debug and below.
• Main tool: jlog
Container Log Example
• Subset of XML log.
• Bird of Eye view of operation (By default, only stores up to INFO level)
• Generated at container level as text log line.
2015-03-19T11:52:07.546 [CONTROL/DV10/FrontEnd/ACD - void ACDImpl::setCalibrationDeviceBandInternal(CalibrationDeviceMod::CalibrationDevice, ReceiverBandMod::ReceiverBand)] Setting Hot load in front of band 4Vector Size: 1Starttime: 1e4ce2e5e639700Band: 3Freq: 7.68268e+10Caldevice 22015-03-19T11:52:07.631 [CONTROL/DV10/FrontEnd - void FrontEndImpl::lockFrontEndInternal(Control::FrontEnd::SubscanInformation, bool, bool)] Receiver band was already locked at 1.537186e+11ACD Movement: Time: e602f8 To: 3 Set As: 2ACDCSV,e602f8,3,22015-03-19T11:52:15.461 [CONTROL/DV10/cppContainer-GL - void PositionStreamConsumer::processData(Control::MountStatusData)] At 11:52:15.461 got the last needed data with a timestamp of 11:52:15.360. The delay is 0.101 seconds.2015-03-19T11:52:16.253 [CONTROL/DV10/MountController - virtual void Control::MountControllerImpl::abortDataCollection()] Cleaning up past data collection that started at 11:51:54.000 and was due to stop at 11:52:15.360. Time now is 11:52:16.253
What is not an ALMA Log
• It is not observational data. It is behavior description.
• Linux messages: dmesg, /var/log/messages
• Mount circular buffers
• BL CORR internal coding
• Engineering scripts
• HVAC shutdowns
• Utility Modules
Where are the Logs?
• Container logs
• /alma/logs/{hostname}
• XML Logs
• /mnt/gas02/data1/AcsLogs-8.1/
• Don´t be afraid of the name!
• log2015-03-20T19:42:07.557_2015-03-20T20:03:30.984.xml
• Other places to search
• Computing Logs: http://computing-logs.aiv.alma.cl
• Kibana (beta): http://hypatia.osf.alma.cl:5601
Useful locations at AOS
CONTROL
• /alma/logs/gas01/CONTROL/ACC/javaContainer
• /alma/logs/gas01/CONTROL/ACC/pythonContainer
Antennae
• /alma/logs/ANT-abm/CONTROL/ANT/cppContainer
Photonic References
• /alma/logs/lo-lmc-1/CONTROL/LMC1/cppContainer
• /alma/logs/lo-lmc-2/CONTROL/LMC2/cppContainer
BL Corr
• /alma/logs/cob-cdpm-b/CORR/CDPMaster/cppContainer
• /alma/logs/cob-cc/CORR/CCC/cppContainer
ACA Corr
• /alma/logs/coj-cc-1/ACACORR/OBSERVATION_CONTROL/cppContainer
• /alma/logs/coj-cc-1/ACACORR/CMD_TRANSFER/cppContainer
• /alma/logs/coj-cpm-1/ACACORR/CDPM/cppContainer
Thank you!
The Atacama Large Millimeter/submillimeter Array (ALMA), an international astronomy facility, is a partner- ship of Europe, North America and East Asia in cooperation with the Republic of Chile. ALMA is funded in Europe by the European Organization for Astronomical Research in the Southern Hemisphere (ESO), in North America by the U.S. National Science Foundation (NSF) in cooperation with the National Research Council of Canada (NRC) and the National Science Council of Taiwan (NSC) and in East Asia by the National Institutes of Natural Sciences (NINS) of Japan in cooperation with the Academia Sinica (AS) in Taiwan. ALMA construc- tion and operations are led on behalf of Europe by ESO, on behalf of North America by the National Radio Astronomy Observatory (NRAO), which is managed by Associated Universities, Inc. (AUI) and on behalf of East Asia by the National Astronomical Observatory of Japan (NAOJ). The Joint ALMA Observatory (JAO) provides the unified leadership and management of the construction, commissioning and operation of ALMA.