Open Source Linux Based Distributed "Real-Time" Environmental Measuring System - DEMS

Bojan Černač

Abstract

The Slovenian Environmental Agency – SEA has recently upgraded a network of over 281 hydrological and meteorological automatic measuring stations within the framework of the BOBER (Better Observation for Better Environmental Response) project. The purpose of the project was to increase the capacity of an integrated system for assessing and studying various quantities related to the water cycle from various aspects. DEMS - Distributed "real-time" Environmental Measuring System with X-DEG communication platform have been designed to meet the requirements for optimization of surveillance and management of measuring networks with consideration of WMO guidelines. The key benefits include distributed "real-time" data acquisition over various communication channels, interval data processing, local data archiving, data dissemination and visualization. Furthermore, a repository oriented traceable software package and documentation management enables transparent application upgrading. Platform independent software ensures portability, including low power solutions. A built-in

debugger/monitor and an information and validation system provide improvement of measurement traceability and data quality assurance. The system supports modular large-scale sensors and instrument integration provided by software and hardware modularity and standardization that result in flexibility and a robust optimized design for end user solutions. The paper provides an overview of the DEMS/X-DEG solution related to a software/hardware concept, network management, measurement traceability and data quality assurance.

DEMS - Distributed Environmental Measuring System

The most important objectives and requirements for the real time environmental measuring system are showed in Table 1. DEMS (Distributed Environmental Measuring System) with X-DEG (Distributed Environmental Gateway), as a basic platform is primarily built for continuous environmental monitoring and providing and delivering high-quality environmental data in real time.

WMO CIMO / TECO

Standardization and optimization

of measuring networks and procedures

Measurement traceability

Remote measuring networks and data management

The quality of measuring systems

operation: Reliability, Accuracy, Stability,

Efficiency, Operability, Recoverability, Fault Tolerance

Security

Data quality assurance

Modularity, Portability,

Table 1: Objectives and requirements for measuring system design

DEMS/X-DEG features distributed "real-time" data acquisition over various communication channels, interval data processing, local archiving, customizable data reporting and data dissemination, visualization and alert services. Furthermore, a repository oriented software package and documentation management enables transparent, traceable application upgrading. Platform independent software ensures portability. A built-in debugger/monitor and an information and validation system provide improvement of measurement traceability and data quality assurance. The modular large-scale sensors and instruments (for example airports, air

quality and water quality monitoring) integration is provided by software and hardware hierarchic modularity and standardization. This results in flexible and optimized robust design for end user solutions and cost effective and easy upgrading. DEMS with X-DEG communication platform as standard building unit covers various types of measuring systems (including low power application). This leads to standardization of measuring information system, understandability of the system and consequently relatively transparent, easy and efficient maintenance management. Table 2 shows basic features of DEMS/X-DEG.

DATA

Data acquisition and validation

Calibration Sensor status processing Interval data processing and validation

data archive RAM

Backup file oriented interval data archive

(CF/SD media) Meta data

IPC validation (Inter-process communication) Real-time data distribution Real-time clock

synchronization

SYSTEM

application modules

(JMD, sensord) Xml configuration Topology and system resources analyzer

Debugger Alarm services Modular, distributed design

Software portability

Security Low power design

SERVICES

GIS supported »plug & play«

web visualization TCLI standard user interface Remote software installation SVN / ipkg

System validation / testing procedures Redmine project server Wiki project server

Table 2: DEMS/X-DEG features

The DEMS network is implemented with integrated virtual servers for managing distributed X-DEG measuring systems and traceable software, hardware and documentation management (Redmine, SVN - Software Versioning and Revision Control System, Wiki, Visualization server). It

supports central data base, meta data information system and servers for managing interval data transfer (SFTP - Secure File Transfer Protocol / FTP - File Transfer Protocol / SCP - Secure Copy Protocol) from X-DEG remote measuring locations and data visualization (Figure 1).

Figure 1: DEMS/ X-DEG – measuring system structure

X-DEG design

The characteristics of the measuring system are strongly related with built in software services and functionalities that are implemented at the level of communication platform. X-DEG has been designed as high performance communication platform with distributed modular concept of hardware and software. Functionality is based on embedded Linux operating system where from the user point of view sensor kernels, called sensord (sensor daemon) modules are running independently. Figure 2 shows X-DEG optimal hardware and software architecture. Each sensord can be linked with physical UART or TCP / IP communication channel. It manages the

connected sensor and performs data acquisition. Sensord as measuring system constitutive software element ensures design of hierarchic tree and intelligent node modular structure. Job manager daemon called JMD is used as gateway for coordination, sensord performance supervising and data dissemination. It also assures interval data archiving and periodical data transmission to remote data center. In the goal to achieve remote measuring networks and data management the platform is designed with appropriate interfaces and supported by appropriate services.

Figure 2: X-DEG software/hardware architecture

Figure 3: X-DEG connectivity

Figure 3 shows the implementation of environmental measuring system at the level of sensors and instruments connectivity and integration into measuring network. Possible supported connectivity structures are:

- UART on device, - serial device server, - TCP/IP, - multi-CPU device.

Implementation of X-DEG measuring system

X-DEG supports 16 communication channels RS232/RS485/RS422 for digital sensors and data acquisition remote IO modules connectivity. Data acquisition remote IO modules are used for measuring analogue physical quantities such as current, U/I and resistance signals. Fast connect RJ45 plug-ins are integrated with UPS system for

sensor electronics supply. X-DEG wiring is standardized to the junction box level (for example A2-1). Sensors are connected via junction boxes regarding documented wiring schemes. Connections from junction box to the RJ45 connection field are unified regardless connection sensor type. Typical implementation of X-DEG meteorological measuring system is showed on

Figure 4: X-DEG meteorological measuring system implementation

X-DEG application services

X-DEG application software is designed in the modular hierarchic tree structure. Constitutive elements are independent binary programs sensord - sensor daemon, JMD - Job Manager Daemon and associated generic xml files. Applications are running on Linux OS. Software portability is assured by use of POSIX standard in GlibC library as

API (Application Programming Interface) between user space and kernel space. Connectivity between modules is implemented via TCP/IP socket server. All sensord kernels are identical regarding implemented functional services. The only difference is at the level of data polling and interval data processing which depends on the connected sensor/instrument. The typical size of single module (sensord, JMD) does not exceed 500KB. The entire

application for more complex measurement systems with a full sensors configuration is typically less than 10MB. Consequently, the installation procedures are shorter and suitable for use in networks with lower capacity of communication channels. Small

size software is suitable for use on less powerful processors and low power architectures. Sensord and JMD software models with implemented application services are shown on the Figure 5 and Figure 6.

Figure 5: sensord software model

Figure 6: JMD software model

X-DEG sensord features

- WD - WatchDog service,

- TCLI – Telnet Command Line Server, - E/A – event/alarm service, - DBG - log/debugger service,

- IPC - Inter-process communication service,

- DR - Data Report Service, - DP - Data Processing Service –

calibration, interval data & status processing, validation,

- HTA, HPA – historical instant (ta) and interval (pa) data,

- MDA - Meta Data Archive, - MDDA - Memory Dump Data

Archive, - XML configuration.

X-DEG JMD features

- WD - WatchDog service (Linux OS), - TCLI – Telnet Command Line server, - FTPas - FTP auto synchronized

service, - TCLIsf - Telnet Command Line

Interpreter sensord forwarding, - DBG - log/debugger service, - IDP - Interval Data Polling, - MDA - Meta Data Archive, - IDFA - Interval Data File Archive, - XML configuration

X-DEG has a built-in access authorized, standardized Telnet Command Line Interpreter (TCLI) server in every daemon, JMD-job manager daemon and sensor daemon. It represents a powerful mechanism for local and remote supervision and measuring system management. Separate sensord modules are available at predefined telnet ports. TCLI implemented commands are documented at the application level as shown below.

The following basic functions are implemented:

- get – gets various archive data reports (instant and interval data, system topology, meta data),

- set – system functional operation settings, meta data administration, data channel/device enable/disable,

- calib – calibration, - sensor – xml configuration overview,

data validity and status decoding, sampling and processing time settings,

- log – system operational monitoring and debugging.

hmp155d(60002)> Terminal Help: get - main data acquisition commands ('get' or 'get help') set - management and configuration commands ('set' or 'set help') calib - calibration commands - available in calib mode ('calib' or 'calib help') sensor - 'sensor' or 'sensor help' for sensor commands help log - log commands ('log' or 'log help') mode - mode change commands ('mode' or 'mode help') exit - exits current mode (from config, manage, calib or debug mode) - disconnects (from normal mode) help - this help date - get current date and time version - application version version info ¸ - application IPKG info watch - application driver thread watchdog counter value clear - clear screen <ESC> - to cancel the command output <Backspace> - to delete last char <Up>/<Down> - scroll command history compress.gzip - enable GZip compression on the reply stream (more info 'compress help') uncompress - disable compression on the reply stream

As a basis of structure hierarchy and functional operation of measuring system, X-DEG uses generic xml configuration of the application software (sensord and JMD modules). Configuration is performed at initialization of the system. Xml files enable transparent topology and communication

settings overview, meta data and reports format administration. Traceable editing and separate settings management is ensured. Sensor integration is performed by generic addition of individual xml segments named “SensorModules” (Figure 7).

jmd.xml

|-- system.h.xml

`-- terminal_users.h.xml

`-- wmt702d.xml

….. |-- wmt702d.h.xml

…..`-- wmt702d _terminal_users.h.xml

`-- hmp155d.xml

…..|-- hmp155d.h.xml

…..`-- hmp155d _terminal_users.h.xml

Figure 7: X-DEG hierarchic xml configuration

Data acquisition of sensor values, calibration, instant and interval data processing and data validation is implemented at the level of sensord. Figure

8 shows data processing sequence. Some typical supported calculations, report formats and data quality control functionalities are presented in Table 3.

Data Validation, Calculations and Reports

Data quality control - Upper / lower warning/error limits,

- Step change validation, - Sensor status indication/validation, - Service indication.

Statistical calculations

- Average value over set interval, - Minimum, maximum, terminal values over set interval, - Standard deviation, - Cumulative values.

Other calculations - wind: average speed, average direction, speed gust, direction

gust, speed lull, direction lull, peak, minimum, scalar speed, scalar direction, stability direction, status, validity,

- accumulated precipitation: sum Tp, sum 1st 5min, sum 2nd 5min, sum 12h , bucket level, bucket percent, bucket difference,

- precipitation intensity: 1st 5min average, 2nd 5min average, 1st 5min maximum, 2nd 5min maximum,

- snow height: 1st 5min average, 2nd 5min average, - sun and precipitation duration: minute status, 1st 5min status,

2nd 5min status, Tp min status, 24h min status, - Various service parameters processing

Default reporting formats - Instant values, validity, sensor status/validity decoding

- Interval values , validity, sensor status/validity decoding - Meta data report - Debug log

Table 3: Sensord data processing features

Figure 8: hmp155d data processing diagram

Conclusions

DEMS Environmental Measuring System provides modular, distributed "real-time" data acquisition, end user oriented interval data processing, target data archiving and data back-up, powerful data dissemination and visualization. Fast - real time response of the system and access to X-DEG archive data is generally related to communication channels capacity and communication protocol optimization at the transport and application layer. This kind of performance results in very responsive data transfer over fixed (ADSL) and also over mobile network services (GPRS/HSDPA) and meets the need of real-time visualization for maintenance purposes and real-time data dissemination to support various prognostic models. Secondly availability of station data at all stages of the data acquisition and processing sequence, enables on-line software validation and consequently leads to improvement of reliability, accuracy, continuity and consistency of data collection and general improvement of monitoring and quality control systems (Figure 8). Furthermore data quality can be improved with additional mathematical discriminant function analysis on the sensord level. A repository oriented, traceable software package and documentation management of DEMS network and each target X-DEG location, provides transparent and cost effective application upgrading. An integrated project management feature provides issue tracking, bug-tracking tool and support of version control systems to maintain current and historical versions of master source code, station software packages and full X-DEG supporting

documentation. Platform independent software ensures portability, including low power platforms. Each X-DEG measuring system has a built-in debugger/monitor, an information and validation mechanism, which results in improvement of measurement traceability and data quality assurance. It supports hierarchic distributed large-scale sensors and analyzers integration provided by software and hardware modularity, and standardization that result in flexibility and a robust optimized and cost effective design for end user solutions. X-DEG measuring systems are field-tested and in operation at 285 locations of hydrological and meteorological Slovenian national measuring network (Figure 9, Figure 10). The system was also tested and installed at some air quality measuring sites. Production of installed measuring systems DEMS / X-DEG was performed in SEA laboratory. Future development will be focused on the implementation of newer processor architectures. The X-DEG software design allows us to implement measurement systems at the level of the SoM (System on Module). What is more Linux OS based SoM provides standardized powerful small sized and low cost solution adequate for applications for distribute oriented networks with remote installation support. Such approach provides customized solutions not only on the field of environmental measurements but also on other fields where the need for remote measurements, for example energy management, drinking water treatment plants, automation in residential areas etc.

Figure 9: SEA automatic measuring network after the implementation of BOBER project

Figure 10: X-DEG meteorological, hydrological and mobile air quality station

References

[1] World Meteorological Organization, »Guide to Meteorological Instruments and Methods of Observation«, WMO-No. 8, Seventh edition, 2008

[2] Bojan Černač, METEOMET Follow up of the task 3.1.6: Validation of AWS software as in accordance to the assessment of the questionnaire answers and requirements of task 3.6.3, 2014

[3] Jovan Bojkovski, Gaber Begeš, Janko Drnovšek, Domen Hudoklin, Jože Knez, Drago Groselj, Bojan Černač:, METEOMET Deliverable Report on task 3.1.6: Report on assessment of the need for validation of data-logging software used in AWS, ENV07 MeteoMet, 2013

[4] World Meteorological Organization, »Guidelines on Quality Control Procedures for Data from Automatic Weather Stations«, Submitted by Igor Zahumenský (Slovakia), Geneva, Switzerland, 13-16 October 2004

[5] Distributed Systems Department of Computer Science University of Crete. Available at: http://www.csd.uoc.gr/~hy556/material.html.

[6] James A. Langbridge, Professional Embedded ARM Development, Published by John Wiley & Sons, Inc.10475 Crosspoint Boulevard Indianapolis, IN 46256, Indianapolis, Indiana, 2014

[7] Steve Furber, ARM System-on-chip arhitecture, 2nd Edition, 2008

[8] Moxa Inc, UC-8410/8416/8418/8430 LX User’s Manual, 2011

[9] Intel, Intel® IXP425 Network Processor, 2006

[10] Daniel J. Barett, Richard E. Silverman, Robert G. Byrnes, SSH, The Secure Shell, The Definitive Guide, Second Edition, May 2005

[11] Bill O. Gallmeister, POSIX.4 Programming for the Real World, 103 Morris Street, Suite A Sebastopol, CA 9547: O'Reilly & Associates, Inc., 1995

[12] Karim Yaghmour, Jon Masters, Gilad Ben-Yossef, and Philippe Gerum, Building Embedded Linux Systems, Second Edition, United States of America, 2008

[13] Toradex AG, NXP®/Freescale i.MX 6

Computer on Module - Colibri iMX6, https://www.toradex.com/computer-on-modules/colibri-arm-family/nxp-freescale-imx6, 2016