MANTIS · MANTIS D1.2 . Consolidated State-of-the-Art of Sensor-based Proactive Maintenance ... which enable high quality speech recognition and also very good speech synthetization

Cyber Physical System based Proactive Collaborative Maintenance

MANTIS

D1.2 Consolidated State-of-the-Art of Sensor-based Proactive Maintenance

Appendix 16:

Existing devices and interface types related to HMI

Work Package WP1 - Service platform architecture requirement definition. Scenarios and use cases descriptions

Version 1.6

Contractual Date of Delivery 30/04/2016

Actual Date of Delivery 03/06/2016

Dissemination Level Public

Responsible Erkki Jantunen

Contributors Juha Valtonen (LUAS) Lidia Godoy (ACCIONA), Rafael Socorro (ACCIONA), Daniel Vladuši č (XLAB), Gregor Papa (JSI), Mikel Viguera (FARR), Arja Kotkansalo (LUAS), Ville Rauhala (LUAS), Antti Niemelš (LUAS), špela Poklukar (JSI), Quinten Marcelis (ILIAS)

D1.2 Consolidated State-of-the-Art of Sensor-based Proactive Maintenance 662189 – MANTIS – ECSEL-2014-1

The MANTIS consortium consists of:

Num. Short Name Legal Name Role Country 1 MGEP Mondragon Goi Eskola Politeknikoa J.M.A. S.Coop. CO ES 2 MONDRAGON Mondragon Corporacion Cooperativa S.Coop. BEN ES 3 IKERLAN Ikerlan S.Coop. BEN ES 4 TEKNIKER Fundacion Tekniker BEN ES 5 FARR Fagor Arrasate S.Coop. BEN ES 5.1 KONIKER Koniker S.Coop. TP ES 6 GOIZPER Goizper S.Coop. BEN ES 7 ACCIONA Acciona Infraestructuras S.A. BEN ES 8 MSI Mondragon Sistemas De Informacion S.Coop. BEN ES 9 VTT Teknologian Tutkimuskeskus VTT Oy BEN FI 10 LUAS Lapin Ammattikorkeakoulu Oy BEN FI 11 NOME Nome Oy BEN FI 12 FORTUM Fortum Power And Heat Oy BEN FI 13 SQ Solteq Oyj BEN FI 14 WAPICE Wapice Oy BEN FI 15 AAU Aalborg Universitet BEN DK 16 DANFOSS Danfoss A/S BEN DK 17 UNIV Universal Foundation A/S BEN DK 18 HGE Hg Electric A/S BEN DK 19 VESTAS Vestas Wind Systems A/S BEN DK 20 SIRRIS Sirris Het Collectief Centrum Van De Technologische Industrie BEN BE 21 ILIAS Ilias Solutions Nv BEN BE 22 ATLAS Atlas Copco Airpower Nv BEN BE 23 3E 3e Nv BEN BE 24 PCL Philips Consumer Lifestyle B.V. BEN NL 25 PHC Philips Medical Systems Nederland B.V. BEN NL 26 PHILIPS Philips Electronics Nederland B.V. BEN NL 27 S&T Science and Technology B.V. BEN NL 28 TU/E Technische Universiteit Eindhoven BEN NL 29 RUG Rijksuniversiteit Groningen BEN NL 30 UNINOVA UNINOVA - Instituto de Desenvolvimento de Novas Tecnologias BEN PT 31 ISEP Instituto Superior de Engenharia do Porto BEN PT 32 INESC Instituto de Engenharia de Sistemas e Computadores do Porto BEN PT 33 ADIRA ADIRA - Metal Forming Solutions S.A. BEN PT 34 ASTS Ansaldo STS S.p.A. BEN IT 35 CINI Consorzio Interuniversitario Nazionale per l’Informatica BEN IT 36 AIT Austrial Institute of Technology GmbH BEN AT 37 HBM Hottinger Baldwni Messtechnik GmbH BEN AT 38 INNOTEC Innovative Technology and Science Limited BEN UK 39 AITIA AITIA International Inc. BEN HU 40 BME Budaperst University of Technology and Economics BEN HU 41 JSI Josef Stefan Institute BEN SI 42 XLAB XLAB d.o.o. BEN SI 43 FHG Fraunhofer Institute for Experimental Software Engineering IESE BEN DE 44 M2X M2Xpert GmbH & Co KG BEN DE 45 STILL STILL GMBH BEN DE 46 BOSCH Robert Bosch GMbH BEN DE 47 LIEBHERR Liebherr-Hydraulikbagger GmbH BEN DE

ii http://www.mantis-project.eu MANTIS

662189 – MANTIS – ECSEL-2014-1 D1.2 Consolidated State-of-the-Art of Sensor-based Proactive Maintenance

Document Revisions & Quality Assurance

Revisions:

Version Date By Overview

0.1 24.8.2015 Lidia Godoy (ACCIONA) First draft

0.2 25.8.2015 Juha Valtonen added new chapter

0.3 31.8.2015 Juha Valtonen Document template inserted

0.4 10.9.2015 Gregor Papa Added new chapter

0.5 21.9.2015 Mikel Viguera (FARR) HMI in press machine

0.9 28.9.2015 Juha Valtonen (LUAS) Combined all contributions together to this document

1.0 29.9.2015 Juha Valtonen(LUAS) Checked by V. Rauhala

Abstract updated

1.1 06.10.2015 Riku Salokangas Added contractual date of delivery etc.

1.2 21.01.2016 Gregor Papa Combined chapters from appendix 17 and 18 to this document

1.3 24.03.2016 Juha Valtonen, Arja Kotkansalo

Added contributors and standards

1.4 18.4.2016 špela Poklukar Added some additional standards

1.5 27.04.2016 Quinten Marcelis (ILIAS) Other mobile devices

1.6 02/06/2016 Mikel Muxika (MGEP) Format correction

Deliverable info update

http://www.mantis-project.eu iii MANTIS


Abstract

This appendix provides an overview of existing devices and interface types related to HMI. Interface section of this document focuses on natural user interfaces (NUI) where interaction is done with human abilities such as touch, vision, voice and motion. In these few years, touch has become a very popular interaction method especially in mobile devices. Devices section introduces common everyday mobile devices which are still rarely used in industry compared to the stationary PCs. Especially maintenance workers would greatly benefit from these mobile devices and of their ability to bring lots of helpful technical information to the field in a small package.

Later sections have two example cases where HMI of a pultrusion line and a press machine are presented.

iv http://www.mantis-project.eu MANTIS


Table of Contents

Introduction ............................................................................................................................... 2 Natural user interfaces ................................................................................................................ 3

2.1 Automation ......................................................................................................................................... 4 Mobile HMI devices .................................................................................................................... 5

3.1 Smartphones ..................................................................................................................................... 6

3.2 Tablets ............................................................................................................................................... 8

3.3 Other mobile devices ....................................................................................................................... 10 PC HMI Devices ...................................................................................................................... 11

4.1 Desktop PC ...................................................................................................................................... 12

4.2 Industrial PCs .................................................................................................................................. 13 Measurement devices ................................................................................................................ 14 Equipment control panel ........................................................................................................... 17 Control room ........................................................................................................................... 19 Existing devices and interface types within Pultrusion Line ......................................................... 20

8.1 Main Control panel ........................................................................................................................... 21

8.2 Resistances state ............................................................................................................................ 23 8.3 Printing panel ................................................................................................................................... 24

8.4 Presses speed ................................................................................................................................. 25

8.5 Visualization techniques involved in Pultrusion Line ....................................................................... 26 Existing devices and interface types within Press machine .......................................................... 27

9.1 Human Machine Interface ................................................................................................................ 28 9.1.1 Main Menu ................................................................................................................................... 28

9.1.2 Press parameters ........................................................................................................................ 29

9.1.3 Die Control ................................................................................................................................... 30

9.1.4 Synoptic ....................................................................................................................................... 31

9.1.5 Press Maintenance ...................................................................................................................... 32 9.2 Panel buttons ................................................................................................................................... 33 Conclusions .......................................................................................................................... 34

References ...................................................................................................................................... 35 List of annexes ............................................................................................................................... 38 Annex A1. ........................................................................................................................................ 1

http://www.mantis-project.eu 1 MANTIS


Introduction

Based on the high-level review of HMIs [1], the HMI field is divided into five big categories: Acoustics (sound), Optics (light), Bionics, Motion and Tactile (touch).

Acoustics, sounds or audio is mostly used for voice input (from a man to machine) or auditory status/response from machine to man. The majority of main languages already have built modules which enable high quality speech recognition and also very good speech synthetization. Furthermore, if the domain is highly specialized (e.g., medicine) the speech recognition works almost flawlessly.

The Optics is mostly related to the Computer Vision, which is quite successfully used for motion tracking, detection of objects, detection and recognition of motions, which roughly translates into gesture recognition and consequently, use. It can be implemented through camera or other specialized sensors š usually lasers, LEDšs. However, detection of other states is possible, such as sentiment analysis, which works especially well when coupled with audio and video [2].

For the purpose of this review wešre skipping the Bionics category. While it is immensely attractive and interesting, it has not reached a degree of maturity that would be used in cases where accuracy is of utmost importance.

The Motion detection, interpretation and use is one of the most used categories. It can developed using a plethora of sensors, gyroscopes, accelerometers, cameras, etc. It is thus most widely used technology - mouse, 3D mouse, mobile phone motion and actions, to name the most prevalent ones.

Finally, the tactile category, which is, in essence also the one used most widely š computer keyboards are the simplest example. However, other uses have emerged š e.g. smart textiles [3], or even virtual skin [4].

2 http://www.mantis-project.eu MANTIS


Natural user interfaces

Researchers interested in open discovery of natural user interfaces are gathered under the NUI Group [5]. They define natural user interface as a computer interaction methodology which focuses on human abilities such as touch, vision, voice, motion and higher cognitive functions such as expression, perception and recall. A natural user interface seeks to harness the power of a much wider breadth of communication modalities which leverage skills people gain through traditional physical interaction [6].

Another definition of NUI and the corresponding examples can be found at the TechTargetšs IT encyclopaedia [7]: A natural user interface (NUI) is a system for human-computer interaction that the user operates through intuitive actions related to natural, everyday human behaviour.

A NUI may be operated in a number of different ways, depending on the purpose and user requirements. Some NUIs rely on intermediary devices for interaction but more advanced NUIs are either invisible to the user or so unobtrusive that they quickly seem invisible.

Some examples and applications of natural user interfaces [7]:

• Touch screen interfaces let users interact with controls and applications more intuitively than a cursor-based interface because it is more direct š instead of moving a cursor to select a file and clicking to open it, for example, the user touches a graphic representation of the file to open it. Smartphones and tablets typically enable touch input. Touch is being adapted for non-screen applications as well š for example, touch interface called šskinputš that allows users to interact by tapping their own skin [8].

• Gesture recognition systems track user motions and translate those movements to instructions. Nintendo Wii and PlayStation Move motion gaming systems work through controller-based accelerometers and gyroscopes to sense tilting, rotation and acceleration. A more intuitive type of NUI is outfitted with a camera and software in the device that recognizes specific gestures and translates them to actions. Microsoftšs Kinect, for example, is a motion sensor for the Xbox 360 gaming console that allows users to interact through body motions, gestures and spoken commands. Kinect recognizes individual playersš bodies and voices. Gesture recognition can also be used to interact with computers.

• Speech recognition allows users to interact with a system through spoken commands. The system identifies spoken words and phrases and converts them to a machine-readable format for interaction. Speech recognition applications include call routing, speech-to-text and hands-free computer and mobile phone operation. Speech recognition is also sometimes used to interact with embedded systems.

• Gaze-tracking interfaces allow users to guide a system through eye movements. In March 2011, Lenovo announced that they had produced the first eye-controlled laptop. The Lenovo system combines an infrared light source with a camera to catch reflective glints from the useršs eyes. Software calculates the area of the screen being looked at and uses that information for input.

• Brain-machine interfaces read neural signals and use programs to translate those signals into action. BCI can make it possible for someone who is paralyzed to operate a computer, motorized wheelchair or prosthetic limb through thought alone.

Lengthier survey of NUIs and their connection with machine learning supported contextual multi-touch HMI interaction can be found in Blašica Study [9].

A concrete example of the current NUI is a feature, enabling multi-device support. These are the features such as OS X Lion Resume, which lets users pick up where they left off their applications, along with their user interfaces. Chrome-to-phone enables users to send links from their Chrome desktop browser to App on their Android device. Chrome-to-mobile sends pages from your computeršs Chrome browser to the Chrome browser running on your mobile device. Firefox, Chrome both synchronize bookmarks, tabs, and web history between desktop and mobile clients.



2.1 Automation

In order for NUIs, modern HMI/HCI to work, an underlying layer needs to reason about the inputs and possible outputs, given the context, tasks and the user. This is a natural task for machine learning systems, which are used for such tasks (an example is a ShoeBox application [10], used for automatic recognition of peoplešs affinity towards certain images).

This example shows a way to use machine learning as an underlying way for the intelligent HMI and thus, application. However, as Moustakis and Hermman say [11], implementation of machine learning (ML) in human-computer interaction (HCI) work is not trivial. They made a survey on professionals and academics, specialized in HCI, asking them about use of ML in their HCI work. Only one third has used ML.

However, given the expanding power of networking and computing power, these two fields are coming together [12]. The most recent examples of this trend are the autonomous cars, which are being developed by all the major automotive or even software companies (Google, Uber, etc.).



Mobile HMI devices

In this last decade, mobile smart devices have become a part of everyday life. Smartphones are a common thing that almost everyone carries around. Tablets are replacing the need for a desktop computer for some people. Part of the enterprise world has already adopted the benefits that smart devices have to offer. Heavy industry in other hand has much slower adoption rate and are still following far behind.

Mobile devices in industrial sector has been mostly used in remote diagnostics and maintenance visualization in SCADA solutions. Now there are few mobile HMI solutions on the market and it is certain that we will see more in these few years. One thing preventing the rise of these solutions is the security concerns. Mobile access may not be tempting enough to justify possible loss of income caused by the new system. Though development costs are coming down for mobile device/PLC interfaces. At first the implementations are probably much simpler than current traditional HMI, offering only notifications and reports. [13]

There are two different solutions to mobile access. First is local control where mobile device is connected to a Wi-Fi router which is connected to the controlling units. This is the more secure way but it can be only accessed in the range of the router. Second way is remote control where mobile device connects to a VPN server. This server is then connected to the controlling units. Because this connection can be made from anywhere, heavy security is needed to prevent harmful connection attempts.

Figure 1. An example mobile HMI



3.1 Smartphones

Smartphone is a phone which has advanced operating system with a graphical interface and an internet connection. Smartphones have become an invaluable tool in business world by offering e-mail, calendar and other important internet requiring applications where ever you go.

Current smartphones are capable of running almost any type of software. Mainly the only restrictions are the limitations of the screen size and touch controls. Processing power only limits use of heavier applications, but those are rarely needed on a small mobile device. Alongside with normal mobile telecommunications, smartphones offer Wi-Fi and Bluetooth communications.

There is two different ways to implement applications on mobile device. Native applications are developed for a particular mobile. Those applications needs to be installed to the mobile device. Web applications are accessed via mobile devicešs browser and the actual software is running on a server. Both methods have their advantages and disadvantages. From user interface standpoint, they both are on an equal playing field. The actual developing process is a bit easier for the web application. Native applications require their own SDK for each platform and use different native programming language. This increases workload if application has to support multiple platforms. Because web applications run on a browser, developer can use any framework and most browsers will support it. Native applications on the other hand are much faster and are preferred if heavier processing is required. Also if the application has to use hardware like accelerometer or camera on the mobile device, native application is required.

Maintenance with smartphone

With different applications operator can handle the maintenance through the Smartphone or Tablet. Operator can handle e.g. fault messages, work orders or acknowledge received orders with the apps. Also worked time can directly be registered in the application. [14] If the user doesnšt have an Internet connection when using the application, the data will be stored in the internal database. When the mobile device has Internet connection again it will automatically synchronize with the Computerized Maintenance Management System (CMMS). Many applications have also the possibility to add pictures. Operator can take a photo with the camera on the smartphone (Figure 2) or tablet and add to the fault message or work order (direct registration). A picture says more than a thousand words. GPS functionalities are also commonly used (Figure 3). Users can find the objects with stored GPS coordinates. [15]

Figure 2 Documentation of maintenance is important, and sometimes a picture is worth a thousand words. [16]



Figure 3 Example of application opportune; Internet connection, addition of image and GPS option [15]



3.2 Tablets

Market has a wide variety of tablets for different needs. Their usual size range is 7-11 inches. Three common operating systems are Android, IOS and Windows. For industry there are also variety of rugged tablets. Rugged tablets have higher IP rating compared to consumer tablets. Depending tablets IP rating, it can withstand dust, liquids and drop from certain heights. These are necessary protections in harsh industrial environments. Rugged tablets tend to be heavier due to higher IP rating and many of them have low quality displays compared to consumer ones. Rugged and other business model tablets usually have possibility to add optional interfaces and equipment that might not be available in normal consumer products. Some of these features are:

• Hot swap battery,

• Barcode reader

• LAN port

• Smart card reader.

There is also security options available. Some of these security features are:

• TPM

• Security lock

• Password security.

Figure 4. Rugged Windows tablet by Panasonic [17]

Application development process for the tablets are quite similar to the smartphones. Main difference is that the Windows tablets can run the same software which is used on the desktop computers running Windows. Though user interfaces are rarely developed in such way that desktop application would be user-friendly on a tablet device.

Maintenance with tablet

Tablet and a smart phone are very similar in usage (Figure 5). They share many common traits and have similar features. A tablet has larger screen real-estate and sometimes higher resolutions. These benefits are utilized in maintenance related applications by providing more information on screen. Depending on the tablet, the operating system is the same and thus software developed in, for instance, Android can be directly installed onto a tablet with minimal software changes while retaining the same basic user interface interaction.



Figure 5 Different tablet version to the mobile workers [18]

Users/Maintenance staff can access tablets watching all types of documentation e.g. PI-diagram (Process and Instrument Diagram) with the database information can also be utilized efficiently in daily plant operation and maintenance. This kind of function is exist but users donšt use in common. The production systems technical hierarchy must be arranged. Functional location or equipment position code identifies the given position in a production process and specifies its location. Also integration into CMMS like SAP, JDE, Maximo is achievable and the visual appearance is equal to PCs.



3.3 Other mobile devices

For specific tasks, special purpose mobile tools are available. Either as standalone mobile device, or as extension of one or more mobile platforms. These devices can overlap with measurement devices, equipment control panels or are constructed for reading and processing system data on a local level. If it is not always possible to transmit all (raw) data to the cloud all the time, how can the maintenance operator access it?

One of the best known examples are car engines. Because of limited communication possibilities, only processed readings can be transmitted to the cloud and only at certain points in time. A technician needs an (E)OBD2 reader to diagnose the engine or vehicle in more detail. This data can then be correlated with the other data within the Mantis scope to give the technician a rich interface and insight into the problem at hand.

Figure 6 Left: (E)OBD2 mobile reading device, Right: wireless (E)OBD2 reader for mobile platforms



PC HMI Devices

PCs are popular platform for HMI in industry. It is usually a set of desktops in control room or industrial PC located outside of normal office conditions. PC is able to directly communicate with the control units if required interfaces are found in the setup. Controlling can be done with several different protocols depending on the controlling unit. Some of these protocols are:

• Profibus/Profinet

• Modbus (RTU/ASCII/TCP)

• Ethernet/IP

• EtherCAT

PC is a flexible platform to build custom HMI solutions. It offers enough processing power to build more complex and graphics-intensive applications. PC has a great number of different HMI solutions available on the market.

Industry, production and maintenance personnel with regard to data collection are typically performed in such a way that information is reported on a shared desktop computer. Production personnel usually use the computers in the control room and maintenance staff at the workshop or storage computers. [19, p. 47]

Personalized terminal can be a traditional desktop computer or laptop. Often, the maintenance organization, electrical and automation technicians already have laptops is use, which are intended, for example, operation of PLCs and electric drives. Maintenance workers need computers every day, at least for entering work hours and failures, faults and handling work orders. Maintenance workers need computers, in addition to recording of data, in exploitation of information available.

Personal computers allow access to information systems such as ERP or other databases whenever needed. Access to work instructions, technical drawings and information are provided, however access to automation systems are often restricted. A personal computer can be customized for user needs. Required applications can be installed and possible user interfaces can be customized to user needs. [19]



4.1 Desktop PC

PC is a highly customizable device. It offers use of many different peripherals and software. A mouse, keyboard and monitor is usually found in every setup. Currently basic devices are connected to PC via USB port which are nowadays always provided by the motherboard. More exotic devices can be connected via expansion cards which are usually connected to motherboards PCI slot.

Different versions of Windows are commonly used as an operating system in desktop PCs. There are still many old Windows versions in use. This is because some old software might not have working version for the latest Windows. Also if migration to a newer version brings operation to a temporary halt, price tag might be too much in a process where each hour of downtime costs lots of money.

Figure 7. An example desktop



4.2 Industrial PCs

Industrial PCs are meant for places where normal PCs canšt operate reliably. Usually IPCs are used for process control or data acquisition. To withstand harsh conditions, casings are rugged and they have filtering for electromagnetic interference. Usually PCs are also shock/vibration resistant and they can operate in higher temperatures. Expansion slots provide possibility to add analog and digital I/O which are not provided normally. Panel PC is one type of industrial where a display is included in same enclosure. These displays usually come with touch screen, but normal keys are also included with some models.

Industrial PCs usually run an embedded version of Windows. This version of Windows allows users to customize the OS to only include the components that are necessary.

Siemens is one example provider of industrial PCs. They offer variety of industrial rack, box and panel PCs for manufacturing and process automation.

Figure 8. Panel PC by Siemens



Measurement devices

Wireless sensors and measurement modules minimizes the time spent close to the measured machine. Wireless sensors works as an on-line system field terminal for e.g. Metso DNA. Sensors operates alongside tablet computers. Sensors have built-in memory and can be left in the field (Figure 8).

Figure 9 Route based condition measurement and analysis [20]

Wireless industrial tablet identifies machinery with potential problems that require scheduled servicing and alerts to risk of breakdowns. The wireless industrial tablet is route-based condition monitoring data collector and analyzer with IP65 and MIL-STD-810G certification for operation virtually anywhere. Industrial tablet computers needs analyzer software that includes route planning, route and off-route measurements as well as tools for vibration analysis. Industrial tablet uses touch-screen user interface for basic or advanced analysis. [20, p. 7]

In Figure 9 is an online measuring system for the continuous monitoring of machine and process parameters on a decentralized basis. The system is directly accessible via Ethernet and Web interface. [21]

Figure 10 Online measuring system



An automated link from the manufacturing level through to the control level is possible using this system. Sensor measure conventional values, such as temperature, pressure, and operating hours. It also provides additional vibration-based parameters. [21]

The automatic alarm threshold adjustment allows a reliable alarm system. An alarm is indicated immediately by an LED on the device. The alarm can be transmitted to the control station by means of interfaces. Application can convert any smartphone into an alarm receiver on a WLAN network, Figure 10. [21]

Figure 11 Smartphone as alarm receiver

In operation and maintenance there are several condition monitoring devices which has a display panels. Next, an overview of some the most commonly used devices which are used by maintenance operators.

Vibration

By using analysis equipment and software, the individual vibration signals are separated and displayed in a manner that defines the magnitude of vibration and frequency. In the Figure 11 is one example of hand-held machine vibration collectors (analysers). The analyser is a portable computer for collection, storage and review of machine condition and process data. Wireless monitoring technology allows on-line systems for continuous machine surveillance and protection.

Figure 12 One example of hand-held machine vibration collectors and machine condition advisor [22]



Thermometers and Thermal Cameras

Thermal imaging cameras are an accessory tools in maintenance to determine electrical and mechanical condition. By discovering hot-spots with the thermal imaging camera, preventive action can be taken. The thermal imaging camera is a non-contact instrument which is able to scan and visualize the temperature distribution of entire surfaces of machinery or electrical equipment. When mechanical components become worn and less efficient, the heat will increase. The data of the thermal sources can be collected by the thermal imaging camera. By periodically comparing readings from a thermal imaging camera with a machinešs temperature signature under normal operating conditions, can detect a multitude of different failures. [23]

Thermal imaging cameras are quite used in industrial environments for predictive maintenance inspections and it is the growing predictive maintenance technology on the market today. [24] In the Figure 12 is shown condition monitoring products, infrared and non-contact thermometer.

Figure 13 Condition monitoring products infrared and non-contact thermometer [25]



Equipment control panel

A local operator interface terminal is a choice for displaying and analysing data in a defined location or area (Figure 13). Even if we have all the information in control rooms we still need to have access to certain information from the field next to machine. The controller/OIT is provided by an equipment manufacturer or system integrator, data logging/trending, and alarming capabilities will be built-in and thus readily available. For many applications, the point where visualization and analytical performance really gains traction is when software on a PC platform taps into historical data.

Some of the most well-known commercially available business databases include Oracle Database, Microsoft SQL Server, Microsoft Access, SAP Sybase, and IBM DB2. These software packages are used throughout industry for maintaining all sorts of data, and they offer a platform for querying and analysing information.

The software offer their own visualization and reporting tools, and most PC-based HMI packages include these and other data analysis features. Operators can trend various data points over chosen time ranges, and effectively perform their own investigation of the available data. [26]

Figure 14 Particle monitor, which provide an image of the degree of contamination in the hydraulic system. [27]

For data coming from an embedded controller or a PLC, a local operator interface terminal (OIT), which is a dedicated device as opposed to a typical PC, is a good choice for displaying data and alarms for the related equipment or area (Figure 14).



Figure 15 Operator Interfaces attached to devises (HMIšs) [28]



Control room

According to Boring, Richard, Hugo and Dudenhoeffer [29] the connection between HCI and control room design is important. The control room is always in the presence of the operator, which monitors and controls the production process via a computer. The operator is always at the computer and he is usually the one who reported the production organization notifications systems. [19, p. 47]

Figure 16 Control room view (consolidation with one HMI)

Most plants have a mix of control systems (picture 15). It could be distributed control systems (DCS) from different suppliers or former systems and also different PLC solutions for smaller or separate process areas. A wide range of systems can include graphic elements, faceplates, alarm and event handling etc. enabling a common look and choice of operation features and navigation for all connected systems. Quite many systems provide interfaces for Manufacturing Operations Management (MOM), Manufacturing Execution Systems (MES) or Enterprise Resource Planning (ERP) system to read and write data to the control system. [30]

Data input originally came from limited human-based measurements, observations and manual data entryšbut most data today is automatically generated by sensors, with the trend of field sensors becoming more numerous and smarter with more reporting capabilities continuing to grow [26]

The production of information has an extensive meaning. For example, the control rooms, automation systems accumulates information automatically process, while, for example, in the logbook interactions are recorded by the people themselves. Information on the interactions diary can also be classified in part by, for example, headings, under which people record their data. It is important to think carefully about what kind of information you want to gather, to the received information would be useful, for example, the perspective of leadership. [31]



Existing devices and interface types within Pultrusion Line

Existing means for controlling and visualizing parameters in pultrusion line, are based on know the state of functional conditions of the machine through devices located in several panels. It is a machine without complex developments related to user interfaces as show the following figures. In general, the pultrusion process is automatic work, but there is also the possibility to controlling in manual way some parts of the process. Within pultrusion line the existing panels, located in different points, can be divided into four groups:

• Main control panel

• Resistances state

• Printing panel

• Hydraulic presses speed



8.1 Main Control panel

This panel involves the main functions of pultrusion line. Depending on the desire or required technological degree of HMI, it can be found several options as the following pictures (figure 16, figure 17) shown. Generally the visualization panel is composed of:

• A certain number of digital indicators representing the value of current temperature sensors located in several points within the machine (mainly located in the mold) and the setting temperature of these points. Setting temperature can be modified by the buttons located in the display indicator in order to control the resistances temperature.

• Cooling section: Indicating fan operation for press oil cooling through led devices and also there exists the possibility to activate/deactivate by buttons.

• Pump hydraulic press section: Operating mode of Pump hydraulic presses (run, stop) through buttons and led indicators.

• Clamp section: Operating movement of presses (run, back, stop)

Figure 17. Main control panel

Figure 18. Main control panel with touchscreen

The figure 3 shows an advanced version of the panel due to the implementing touchscreen which allow to visualize different parameters in different ways (e.g. plotted, data tables) through a main menu through different options.

Figure 19. Touchscreen main control panel



This main menu presents a useful submenu dedicated to monitoring in summary form hydraulic system parameters such as: hydraulic press 1 speed (cm/min), hydraulic press 2 speed (cm/min), maximum pulling force (KN), stop in every (min), stop for (s), cutting length (cm), oil tank temp (°C), total length (cm), clean mold.



8.2 Resistances state

Through analog receivers, resistances state panel shows the amperage level of different resistances located around the mold of the pultrusion line stay. They can be indicating possible failures related to disconnected or decrease power. Consequently, it may cause severe structural damage in the pieces due to bad cured material.

Figure 20. Amperage resistances panel



8.3 Printing panel

The functionality of this panel is to select messages which are displayed in the piece during pultrusion process.

Figure 21. Printing panel



8.4 Presses speed

Figure 6 digital indicator show jointly hydraulic presses speed average, but this data is not very accurate regarding meter tolerance ± 1cm/min.

Figure 22. Presses speed



8.5 Visualization techniques involved in Pultrusion Line

The visualization techniques are based on led, analog and digital indicators.

• Led indicators in order to know: o Hydraulic press: Run/Stop pump status o Press 1 oil: Run/Stop cooling status o Press 2 oil: Run/Stop cooling status o Clamp press 1: Run/Back/Stop press status o Clamp press 2: Run/Back/Stop press status

Figure 23 Led indicators

• Digital indicators in order to know: o Value of current temperature sensors and resistances setting temperature. o Hydraulic presses speed average

Figure 24 Digital indicators

• Analog indicators: o In order to indicate the amperage level of different resistances around the mold located

in pultrusion line.

Figure 25 Analog indicators



Existing devices and interface types within Press machine

The latest press lines of FAGOR ARRASATE are equipped with a display and control made by a TP1500 panel together with the TIA-Portal SCADA software of SIEMENS and with a specialized development of FAGOR. The use case machine (FAGOR ARRASATE 400 Tn Servo Press) selected for MANTIS is equipped with an older display and control.

The programmable logic controller (PLC) which model is Step7 351-2DP, together with the TP that performs the tasks of visualization and machine control (Human Machine Interface), are connected through an Ethernet network. This makes the whole system easy and powerful control. It has also a RJ45 Ethernet connector for coupling the programming equipment to the network, whereby the system is conveniently accessible to maintenance personnel.

The whole information of the press is centralized in the display panel and control. The Human Machine Interface system (control and display panel) is located in one of the column of the press and as resume has the following functions:

Installation diagnosis. Different screens are used to identify those conditions that do not allow starting the machine normally. This intuitive method facilitates a quick detection of unfulfilled conditions. The display of šFaultš represents those faults that occur in the machine, making it easy to quickly solve for maintenance purposes.

Machine parameters visualization. Through the monitor, some machine parameters are displayed:

Remote control. For each of the major operations to be performed, there is a screen with the correct sequence of steps to execute, represented as a sequence of events.

Tool data. The system can store the parameters of the machine related to each tool as "Tool program". In this way, the machine can be adjusted for each tool change.

Production data. The system displays the total number of strokes that have run the machine, the number of parts produced from tool change and from each shift change.



9.1 Human Machine Interface

The press has a monitor with a series of touchscreens that facilitate the implementation of the machine. This system provides information of the installation; it allows visualizing the machine parameters and facilitates its modification and the execution of certain operations.

At the bottom right of the screen the following buttons are displayed:

Data Save.

Jump to the previous screen.

Alarms screen.

General Synoptic screen

This key allows us to leave the home screen.

At the top of the screen next information is displayed:

• Degrees

• Speed

• Recipe Name

• Date and Time

• Title of the displayed screen

Figure 26 Main menu screen top display

9.1.1 Main Menu

This is the first screen that appears automatically when the machine is turned on.



Figure 27 Main menu screen

This screen allow access to the following elements:

• Press

• Transfer

• Collisions

• Synoptics

• Production

• Die

• Alarms

• Maintenance

Following, some of the most important elements are shown:

9.1.2 Press parameters

On this screen the actual measurements of the programmable data of the press are displayed and can be edited for vary the measures.



Figure 28. Press parameters screen

9.1.3 Die Control

This is a Pressšs die control general screen display

Figure 29. Die control screen This screen shows the general visualization of the pressšs die controls and the physical situation of the press.



9.1.4 Synoptic

The synoptics indicates that the status is when is not correct and indicates when is correct.

Figure 30. Synoptic screen Pressing the Network diagnostic, the status of each slave Profibus, Ethernet and Pilz is indicated.

Figure 31. Profibus, Ethernet and Pilz diagnostic screens



9.1.5 Press Maintenance

This screen displays the maintenance status. It helps to reference encoders.

Figure 32. Press maintenance screen



9.2 Panel buttons

The physical press panel buttons are located together with the digital display in one of the column of the press. They have similar functions as the press parameters digital panel.

Figure 33. Physical press panel buttons All the buttons pressed on the physical panel are visualized as well on the touchscreen digital panel.

Figure 34. Press panel buttons screen



Conclusions

New technology offers lots of different interaction methods for HMI to use in the future. Different natural user interfaces are one examples of these methods. Touch screen is already commonly used in different devices, especially in mobile devices. Gesture control will probably see use when combined with augmented reality devices. Other methods could see limited use because they are hard to implement in to industrial setting or donšt offer any real benefits when compared to other methods.

Todayšs mobile devices are advanced enough to be used as a tool in maintenance work. Smartphone is a device which almost every worker already has and carries around. This already would make it great tool in the field. Tablets would be better for work where screen size is an issue. This would include things like reading datasheets and inspecting 3D models. But due to their size, they would be mostly used on pre-planned jobs and not being carried around all the time.

Desktop PCs are probably going to keep their status in control rooms and off-site environment where mobility has no real advantage and bigger screen size is required. Also work which requires writing longer reports are preferred to do with a keyboard.



References

[1] J. Cannan and H. Hu, "Technical Report: CES-508," School of Computer Science & Electronic Engineering University of Essex, [Online]. Available: http://cswww.essex.ac.uk/staff/hhu/Papers/CES-508%20HMI-Survey.pdf. [Accessed August 2015].

[2] V. P. Rosas, R. Mihalcea and L.-P. Morency, "Multimodal Sentiment Analysis of Spanish Online Videos," [Online]. Available: http://ict.usc.edu/pubs/Multimodal%20Sentiment%20Analysis%20of%20Spanish%20Online%20Videos.pdf. [Accessed August 2015].

[3] D. Marculescu, R. Marculescu and S. P. a. Jayaraman, "Ready To Ware," 2003. [Online].

[4] H. Raffle, H. I. and J. Tichenor, "Super Cilia Skin: A Textural Interface," 2004. [Online].

[5] "The NUI Group website," [Online]. Available: http://nuigroup.com/go/lite. [Accessed August 2015].

[6] "The NUI Group website, FAQ," [Online]. Available: http://nuigroup.com/faq. [Accessed August 2015].

[7] "TechTarget, IT encyclopedia, NUI definition," [Online]. Available: http://whatis.techtarget.com/definition/natural-user-interface-NUI. [Accessed August 2015].

[8] "Skinput," [Online]. Available: https://en.wikipedia.org/wiki/Skinput. [Accessed August 2015].

[9] B. Blažica, "The inherent context awareness of natural user interfaces: a case study on multitouch

displays, PhD study," Jožef Stefan International Postgraduate School, 2013. [Online].

[10]

B. Blažica, D. Vladušič and D. Mladenić, " ShoeBox: A Natural Way of Organizing Pictures

According to User’s Affinities, Human-Computer Interaction. Towards Mobile and Intelligent

Interaction Environments Volume 6763 of the series Lecture Notes in Computer Science pp 519-524," [Online].

[11]

V. S. Moustakis and J. Herrmann, "Where Do Machine Learning and Human Computer Interaction Meet?," Applied artificial intelligence Journal, October 1997. [Online].

[12]

J. Grudin, "AI and HCI: Two Fields Divided by a Common Focus," [Online]. Available: http://research.microsoft.com/pubs/138574/AIMagazine.pdf. [Accessed August 2015].

[13]

M. Pantoleano, "HMI and Mobile Devices: On the menu for your plant?," [Online]. Available: http://www.industrial-ip.org/community/blog/hmi-and-mobile-devices.

[14]

A. Siimes, Ed., Mobiilisovellus kšynnissšpidon tarpeisiin, Lapland University of Applied Sienceses, 2014.



[15]

"Plant Information - Proficy Mobile," Novotek, 2015. [Online]. Available: https://www.novotek.com/en/products/plant-information/proficy-mobile. [Accessed 10 08 2015].

[16]

"Toiminnanohjaus taskussa," M-Technology Oy, 2014. [Online]. Available: https://toiminnanohjaustaskussa.wordpress.com/. [Accessed 12 8 2015].

[17]

"Samsung Thoughpad FZ-M1," [Online]. Available: http://business.panasonic.co.uk/computer-product/toughpad/FZ-M1.

[18]

Panasonic, "Realising the gift of time -Enterprise and goverment tablet solutions," 2014.

[19]

V. Rauhala, "Kšynnissšpidon tiedonkeruun tehostaminen, KÄYNTI š Kšynnissšpidon tiedonhallinta," 2013.

[20]

Metso, "Results automation," 2014.

[21]

Schaeffler, "FAG SmartCheck, Machinery monitoring for every machine".

[22]

SKF, "Vibration measurement tools," [Online]. Available: http://www.skf.com/pk/products/condition-monitoring/basic-condition-monitoring-products/vibration-measurement-tools/index.html. [Accessed 1 9 2015].

[23]

FLIR, "Thermal imaging guidebook for industrial applications," FLIR Systems AB, 2011.

[24]

C. Maras, "Thermal Imaging Cameras a Great Tool for Predictive Maintenance Inspections," Maintworld, no. 2, 2013.

[25]

SKF, "Basic condition monitoring products," [Online]. Available: http://www.skf.com/pk/products/condition-monitoring/basic-condition-monitoring-products/index.html. [Accessed 1 9 2015].

[26]

Advantech, "IoT and Big Data Combine Forces," 2015.

[27]

O. Off-Highway, "Fluid power systems & components," 2015. [Online]. Available: http://www.oemoffhighway.com/product/11293138/enm-co-series-t39-lcd-hour-meter-and-up-counter. [Accessed 29 8 2015].

[28]

Siemens, "Operator Control and Monitoring Systems," 2015.

[29]

R. L. Boring, J. Hugo, C. M. Richard and D. D. Dudenhoeffer, "SIGšThe Role of Human-Computer Interaction in Next-Generation Control Rooms," Idaho National Laboratory, 2005.

[30 "Control systems connectivity and modernization," ABB, 2014. [Online]. Available:



] http://new.abb.com/control-systems/system-800xa/control-systems-connectivity-control-room-consolidation-modernization. [Accessed 10 08 2015].

[31]

J. Tarvainen, "Tiedon jalostaminen kšynnissšpidon tarpeisiin KÄYNTI - kšynnissšpidon tiedonhallinta," 2013.



List of annexes

Annex A1. The relevant standards for Appendix 16, concerning WP5, HMI design and development

Task 5.4 - Intelligent Human-Machine Interface design


662189 – MANTIS – ECSEL-2014-1

D1.2 Preliminary State-of-the-Art of Sensor-based Proactive Maintenance Appendix 16: Existing devices and interface types related to HMI

Annex A1.

Standards related to appendix 16: Existing devices and interface types related to HMI

Standard Organization

Number Title Publishing Year

Work Package

Task

ANSI/HFES

100 Human Factors Engineering of Computer Workstations

2007 WP5 Task 5.4

ANSI/ISA 101.01 Human Machine Interfaces for Process Automation Systems

2015 WP5 Task 5.4

ISO 9241-16 Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 16: Direct manipulation dialogues

1999 WP5 Task 5.4

ISO 9241-6 Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 6: Guidance on the work environment

1999 WP5 Task 5.4

ISO 9241-5 Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 5: Workstation layout and postural requirements

1998 WP5 Task 5.4

ISO 9241-6 Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 6: Guidance on the work environment

1999 WP5 Task 5.4

ISO 9241-12 Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 12: Presentation of information

1998 WP5 Task 5.4

ISO 9241-400 Ergonomics of human--system interaction -- Part 400: Principles and requirements for physical input devices

2007 WP5 Task 5.4

ISO 9241-15 Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 15: Command dialogues

1997 WP5 Task 5.4

ISO 9241-143 Ergonomics of human-system interaction -- Part 143: Forms

2012 WP5 Task 5.4

ISO 9241-14 Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 14: Menu dialogues

1997 WP5 Task 5.4

ISO 9241-2 Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 2: Guidance on task requirements

1992 WP5 Task 5.4

ISO 9241-1 Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 1: General introduction

1997 WP5 Task 5.4



ISO 9241-110 Ergonomics of human-system interaction -- Part 110: Dialogue principles

2006 WP5 Task 5.4

ISO 9241-210 Ergonomics of human-system interaction -- Part 210: Human-centred design for interactive systems

2010 WP5 Task 5.4

ISO 9241-11 Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 11: Guidance on usability

1998 WP5 Task 5.4

SFS-EN 894-1 Safety of machinery. Ergonomics requirements for the design of displays and control actuators. Part 1: General principles for human interactions with displays and control actuators

2009 WP5 Task 5.4

SFS-EN 894-2 Safety of machinery. Ergonomics requirements for the design of displays and control actuators. Part 2: Displays

2009 WP5 Task 5.4

SFS-EN 894-3 Safety of machinery. Ergonomics requirements for the design of displays and control actuators. Part 3: Control actuators

2009 WP5 Task 5.4

SFS-EN 894-4 Safety of machinery. Ergonomics requirements for the design of displays and control actuators. Part 4: Location and arrangement of displays and control actuators

2010 WP5 Task 5.4

UNI 10652? Maintenance - Appraisal and evaluation of the goods condition

2009 WP5 Task 5.4

CENELEC 50459-1 Railway applications - Communication, signalling and processing systems - European Rail Traffic Management System - Driver-Machine Interface - Part 1: General principles for the presentation of ERTMS/ETCS/GSM-R information

2005 WP5 T5.4

CENELEC 50459-2 Railway applications - Communication, signalling and processing systems - European Rail Traffic Management System - Driver-Machine Interface - Part 2: Ergonomic arrangements of ERTMS/ETCS information

2005 WP5 T5.4

CENELEC 50459-3 Railway applications - Communication, signalling and processing systems - European Rail Traffic Management System - Driver-Machine Interface - Part 3: Ergonomic arrangements of

2005 WP5 T5.4


662189 – MANTIS – ECSEL-2014-1

D1.2 Preliminary State-of-the-Art of Sensor-based Proactive Maintenance Appendix 16: Existing devices and interface types related to HMI

ERTMS/GSM-R information

CENELEC 50459-4 Railway applications - Communication, signalling and processing systems - European Rail Traffic Management System - Driver-Machine Interface - Part 4: Data entry for the ERTMS/ETCS/GSM-R systems

2005 WP5 T5.4

CENELEC 50459-5 Railway applications - Communication, signalling and processing systems - European Rail Traffic Management System - Driver-Machine Interface - Part 4: Symbols

2005 WP5 T5.4

CENELEC 50459-6 Railway applications - Communication, signalling and processing systems - European Rail Traffic Management System - Driver-Machine Interface - Part 4: Audible information

2005 WP5 T5.4

ETSI EN 301 515 Global System for Mobile communication (GSM); Requirements for GSM operation on railways

2005 WP5 T5.4

ETSI TR 102 281 Railways Telecommunications (RT); Global System for Mobile communications (GSM); Detailed requirements for GSM operation on Railways

2013 WP5 T5.4


Documents

MANTIS · MANTIS D1.2 . Consolidated State-of-the-Art of Sensor-based Proactive Maintenance ... which enable high quality speech recognition and also very good speech synthetization