Improving the Failure Understanding in Embedded Systems ... · 1Information Technology Department, Mutah University, Jordan 2Computer Graphics and HCI Group,3Software Engineering:DependabilityGroup

http://dx.doi.org/10.14236/ewic/HCI2016.32

Improving the Failure Understanding inEmbedded Systems through Interactive Visual

Support: An Evaluation Study

Ragaad AlTarawneh1, Shah Rukh Humayoun2, Achim Ebert2, Peter Liggesmeyer3

[email protected], {humayoun, ebert, liggesmeyer}@cs.uni-kl.de1Information Technology Department, Mutah University, Jordan

2Computer Graphics and HCI Group, 3Software Engineering: Dependability GroupUniversity of Kaiserslautern, Germany

Effectively analyzing failure mechanisms is critical for ensuring safety and reliability aspects in embeddedsystems. This requires close collaborations between the system engineers who design the system andthe safety experts who analyze the failures. To support this process, we present an interactive visualenvironment, called ESSAVis++, that helps in extracting the required safety information about the failuremechanisms in the underlying embedded system. In this paper, we focus mainly on the conducted userevaluation study to compare between ESSAVis++ and ESSaRel (a tool used in industry) in terms of accuracy,efficiency, and user acceptance. The goal was to determine the suitability and effectiveness of a commoninteractive visual environment that would be used by the system engineers and safety experts during thesafety analysis process in embedded systems. Results indicate an increase in participants’ efficiency andaccuracy levels using the ESSAVis++ platform, due to the visual support and provided interaction options.We also observed a high acceptance rate among the participants towards ESSAVis++, mainly due to itsintuitive visualizations.

Embedded Systems; Safety Aspect Visualization; Information Visualization; User Evaluation.

1. INTRODUCTION

Embedded systems are essential of many criticalsystems that are ubiquitous in our modern lives,such as smart cards, cars, airplanes, washingmachines, etc. They are classified as electronicdevices that incorporate a computer system in theirimplementations. Normally, they are composed oftwo types of components (i.e., software componentsand hardware components), which collaborate witheach other through a set of interfaces [Lee andSeshia (2010)]. This all makes them have highlycomplex structures. Understanding such complexsystems requires close collaboration between thedifferent groups of responsible engineers andexperts who design, develop, and maintain thesesystems. The main challenge in such collaborationsis the divergence between the perspectives ofthe different participated groups regarding theunderlying embedded system. Some engineers(e.g., system engineers) focus mainly on thestructural relations among the system components,while others (e.g., safety experts) focus mostly on thefailure relations between the system components.

Maintaining these systems to ensure safe situationsrequires intensive study of all those critical situationsthat may bring them to undesired states [Kaiseret al. (2003)]. One of the main technique usedfor detecting the possible failures in embeddedsystems is the Fault Tree Analysis (FTA) [Kaiseret al. (2003)] technique. Safety experts (who arenormally responsible for detecting or studying thesepossible failures) use FTA to trace all the possiblereasons for each specific hazard. After buildingthe FTA model, they start analyzing the safetyscenarios to identify all safety-critical componentsin order to maintain them and thus keep thesystem in a safe mode. The result of this analysisprocess helps the system engineers (who arenormally responsible for maintaining all of thesystem components) to fix the overall system inorder to make it safer for the end users. Thisiterative interaction and collaboration between thesafety experts and the system engineers supportsthe maintenance process of embedded systems.However, this interaction and collaboration betweenthe two groups is affected by differences in theinput data format used by each group. Consequently,

c© AlTarawneh et al. Published byBCS Learning and Development Ltd. 1Proceedings of British HCI 2016 Conference Fusion, Bournemouth, UK

Improving the Failure Understanding in Embedded Systems through Interactive Visual Support: An Evaluation StudyAlTarawneh • Humayoun • Ebert • Liggesmeyer

the overall speed of this maintenance process isimpacted by this gap.

Targeting the above-mentioned concern, we aim atproviding one common interactive visual environ-ment to enhance the understanding of system en-gineers and safety experts participating in the safetyanalysis process of embedded systems. The goal ofthis proposed platform is to reduce the gap betweendifferent participating groups and to accelerate themaintenance process with better efficiency and ac-curacy. For this, we developed a visual interactive2Dplus3D environment called ESSAVis++, which isan enhanced version of our previously developedtool, called ESSAVis (Embedded Systems SafetyAspects Visualizer) [AlTarawneh et al. (2013, 2012)].ESSAVis++ has been designed to overcome thelimitations of ESSAVis, e.g., by allowing the usersto navigate at different level-of-detail on-demand inthe target safety abstract graph representation orenabling the users to change the configuration ofthe scene on-demand in order to get the additionalinformation about the required system component.

In this paper, first we provide an overview of theESSAVis++ platform and show how it overcomes thelimitations of ESSAVis. Then we focus mainly on auser evaluation study that we conducted to checkthe feasibility and effectiveness of this platform. Ouruser evaluation study was conducted with real endusers (i.e., safety experts and system engineers)as well as with other users having a backgroundin engineering and computer science. This userevaluation study was dedicated to the comparisonbetween the ESSAVis++ platform and the ESSaReltool [Kaiser et al. (2007)] (which has previouslybeen used in industry, as mentioned in CESARProject Report, p. 43 [CESAR Project (2010)]), interms of accuracy and efficiency regarding analysesof the safety aspects of failures occurring in anembedded system. We also considered the trend ofuser acceptance of such platforms. The goal of thestudy was to determine the suitability of ESSAVis++as a future visual collaborative platform for the safetyanalysis process in embedded systems. The resultsof this study show increases in the participants’efficiency and accuracy levels using ESSAVis++compared to ESSaRel, mainly due to the visualsupport and the interaction facilities provided. Wealso observed a high acceptance rate among theparticipants towards ESSAVis++ due to its intuitivevisual support. This opens the doors of thinkingof new possible directions how to enrich thesafety analysis process of complex systems usinginteractive visual environments.

The remainder of the paper is structured as follows:In Section 2, we provide an overview of the related

work. In Section 3, we present the ESSAVis++platform. In Section 4, we explain the settings of ouruser study. In Section 5, we present and discuss thestudy results. Finally, we conclude in Section 6.

2. RELATED WORK

In order to understand system failures, it isimportant to depict the relations between differentsystem parts. In most cases, this is done via agraphical representation of the safety scenario underconsideration. Therefore, many people have alreadytried to visualize this in 2D representations via a treestructure using the fault tree analysis (FTA) [Kaiseret al. (2003)] approach, which is a widely usedtechnique for modeling safety scenarios of complexsystems. Many tools have been developed to helpthe safety experts in modeling the system status.Examples of such work are ESSaRel in [Kaiser et al.(2007)], UWG3 in [Kaiser et al. (2003)], and CeciliaOCAS in [Bieber et al. (2004)]. In these tools, a 2Drepresentation is used to depict the FT using thenode-link diagram style. Normally, these tools usedifferent geometrical shapes to represent the faulttree elements, e.g., simple primitives to representcomponent types (like cycles for basic events andsmall rectangles for the gate information), simplearrows to represent the failure relations betweenthe system parts, etc. Additionally, color and textcues are also used to provide any extra information,such as the gate type or the probability values. Allthese tools were designed to help model the faultmechanism with limited analytical options. Moreover,they lack in providing an overview of the failuremechanism. That is why it is a relatively hard taskto trace the failure between any two points usingsuch tools. Among these, only ESSaRel provides atextual description of some of the safety aspects ofthe FT (e.g., the set of minimal cut sets), which isunreadable in most cases due to the large size ofthe data and the file format.

Visualizing safety aspects of embedded systemsis comparatively a new field. As the complexity ofthe underlying embedded system is increased thesize of corresponding fault tree is also increased.Information visualization can play an important rolein speeding up the system understanding process,as it helps safety expert and engineers in findingthe important information about the underlyingembedded system in shorter time. Also, it helpsin reducing errors made by human in searchingrelevant information. Further, it can provide aplatform for visually integrating the different parts of acomplex system. Few tools have also been proposedwith some visual support to help safety expertsin analyzing the safety aspects of large systems.For example, the PLFaultCAT tool [Dehlinger and

2


Lutz (2006)] was designed for analyzing the safetyaspects of software systems. Another tool exampleis by Yang et al. [Yang et al. (2012)], wherethey proposed a visualization system for supportingsafety experts in identifying proper safe solutions forthe system in visual form. In [Kumar et al. (2009)],the authors proposed an interesting visualization-based tool, called SViT, which aims at helping home-owners assess the current safety level of their homesand finds the causes for this level.

Compare to previous tools, ESSAVis++ visualizesthe safety aspects of embedded systems using a2Dplus3D metaphor, i.e., integrating the 2D repre-sentation of the failure mechanism of the underlyingsystem with the system 3D representation. Further,it visually integrates the hardware parts and theabstract safety views, which is useful in conveyingthe whole story to the engineers and safety experts.

3. THE ESSAVIS++ PLATFORM

ESSAVis++ is an extended version of the ESSAVisplatform, which previously was presented in [Al-Tarawneh et al. (2013, 2012)]. The aim is to visualizethe safety aspects of embedded systems using a2Dplus3D metaphor, i.e., integrating the 2D repre-sentation of the failure mechanism of the underly-ing system with the system 3D representation. ES-SAVis++ enhances the ESSAVis platform by provid-ing many new interaction options and visualizationsfor a better and more accurate understanding of thefailure mechanisms. In compared to ESSAVis, fewof the additional features provided by ESSAVis++are: 1) explosion of the system 3D model using aslider bar, 2) using different graph layout techniquesaccording to the situation for visualizing the safetyabstract graph model rather than relying on justthe orthogonal layout algorithm, 3) allowing usersto navigate at different level-of-detail on-demand inthe targeted safety abstract graph representationthrough the expanding and contracting options at-tached to compound nodes, 4) additional interactiveoptions to allow users changing the configurationof the scene on-demand, 5) allowing users to getadditional information on-demand through a textualinformation box, and 6) visualizing software metricsinformation of the selected components.

The key point about the ESSAVis++ platform isits ability to combine different data sets and thenprovide seamless integration between them. Afterintegrating the input data sets into different visualrepresentations, ESSAVis++ allows users to interactwith it through a number of interaction facilities.ESSAVis++ combines the two main views (i.e., theabstract graph representation and the system 3Dmodel) to describe the safety status of the underlying

Figure 1: The two views: the system 3D model view on theleft-side and the abstract graph representation view on theright-side. Here, the 3D model of RAVON robot [Proetzsch(2010)] is aligned in a layered fashion together with theabstract representation of the safety scenario.

embedded system (see Fig. 1). Here, each viewrepresents the safety status from a different usergroup’s perspective. The integration between thesetwo views (see Fig. ??) allows engineers and safetyexperts to understand the whole safety situation ofthe underlying system easily and more accurately.

The main input to ESSAVis++ is CAD representa-tions of the system model, which it uses to constructthe system’s 3D model. Additionally, ESSAVis++reads a safety scenario modeled as a fault treemodel. Then it calculates the relevant safety infor-mation to help the safety experts defining the setof critical components. After this, ESSAVis++ mapsthe input safety information to the actual systemcomponents. Then it arranges the set of componentsto prepare it for the visualization process.

For visualizing the abstract representation of thesafety scenario, ESSAVis++ converts the CFT modelinput into a compound graph model, which isa special graph with two relations types (i.e.,the structural relation and the adjacency relation)between its nodes [Raitner (2005)]. Then it uses oneof the suitable layout algorithms (e.g., the orthogonallayout or the Sugiyama layout; see Hermanet al. (2000) for more details about graph layoutalgorithms) for the final representation. It usesthe OGDF framework1 for calculating the finalgraph representation (see Fig. ??). In order tonavigate through the resulting compound graphrepresentation of the safety scenario, ESAAVis++provides the facility to expand or contract anyof the compound nodes on demand through afocus+context viewing mechanism. The selectedcomponent is then expanded to show its internalstructure, in addition to the failure relations between

1http://www.ogdf.net/

3


Figure 2: ESSAVis++ platform overview.

the sub-components within it. This helps to navigatethe failure mechanism structure in a way that reflectsboth types of relations (i.e., the structural and thefailure relations in the graph view), see Fig. 2b.

ESSAVis++ constructs an interactive system 3Dmodel using the given CAD model and providesa number of interaction options for picking andselecting one or more components of this 3D model.It also provides the facility to expand the 3D modelon demand using the explosion-slider interactionoption (see Fig. ??). ESSAVis++ synchronizes thetwo views (i.e., the abstract representation view andthe system 3D model view) in such a way that if usersselect one node in the graph representation, thecorresponding component in the system 3D modelis also highlighted (see Fig. ??). ESSAVis++ usesa novel way to integrate between the different viewsusing different styles, e.g., the side-to-side view orthe layered view.

To visualize data, ESSAVis++ uses the Vruipackage [Kreylos (2012)], which is a toolkitfor providing a virtual reality (VR) developmentenvironment. Vrui works on many different hardwareplatforms; therefore, it is possible to render theapplication in many different environments (e.g., 3Ddisplays, PowerWall displays, or CAVE systems).

This feature increases the scalability and portabilityof ESSAVis++ on multiple environments.

4. THE USER EVALUATION STUDY

The aim of our conducted user study was to evaluatethe feasibility and effectiveness of ESSAVis++ as acommon interactive visual platform to be used bysystem engineers and safety experts for the safetyanalysis process of embedded systems. For thispurpose, we designed and conducted a controlledexperiment with real end users (i.e., safety engineersand system engineers), as well as other users witha background in computer science or engineering inorder to check its effectiveness in general.

In our study, most of the tasks were designedfrom the safety experts’ perspectives in orderto check whether ESSAVis++ helps them inanalyzing the safety scenario faster than throughthe traditional tools. For this, we decided to evaluateour ESSAVis++ platform against the ESSaReltool [Kaiser et al. (2007)]. ESSaRel was designedto support the modeling process of safety scenarios,alongside the provision of analyzing the createdsafety scenario. Due to this difference, we designedour study to include only those options that were

4


provided by both platforms for analyzing the safetyscenario.

4.1. The ESSaRel Tool

ESSaRel (Ebedded Systems Safety and ReliabilityAnalyser) is a tool that was developed to providea flexible dependability integration framework forembedded systems. It can compose and integrateanalysis models with different characteristics, e.g.,Fault Tree (FT), Component Fault Tree (CFT),and state/event fault trees [Essarel Tool (2014)].ESSaRel is considered a powerful tool for editing thesafety model; however, it lacks in showing an overallview of the safety scenario. This makes the processof tracing a failure rather difficult. ESSaRel helpssafety experts in “modeling” the CFT model throughan editor. It offers a multi-view feature to presentthe multi-level concept in the CFT model. Therefore,the internal structure of a specified component ishidden on the upper level. Users can select aspecific component, which results in showing theinternal structure of the component. Although thisfacility helps safety experts to modify the internalstructure of the required component; however, thismakes it difficult to understand the overall systemstructure because the new view does not preservethe relations between the current level and the upperlevel.

4.2. Study Goals and Hypotheses

The goal of our user study was to analyzeESSAVis++ and ESSaRel for the purpose ofcomparing their accuracy, efficiency, and useracceptance in analyzing fault tree models from oneof the real end users’ (i.e., the safety experts)perspective. In order to compare the results for thetwo platforms, we identified a set of metrics bymeans of a GQM model [Basili et al. (1994)]:

• Metrics for accuracy: We measure and comparethe percentage of correct answers for each task.

• Metrics for efficiency: We measure and comparethe time needed for answering each task.

• Metrics for user acceptance: We collect theparticipants’ feedbacks through closed-endedquestionnaires and compare it.

Based on these metrics, we specified the followinghypotheses:

• Hypothesis 1:

– Null Hypothesis:Accuracy(ESSAVis++) < Accuracy(ESSaRel)

– Alternative Hypothesis:Accuracy(ESSAVis++) � Accuracy(ESSaRel)

• Hypothesis 2:

– Null Hypothesis:Efficiency(ESSAVis++) < Efficiency(ESSaRel)

– Alternative Hypothesis:Efficiency(ESSAVis++) � Efficiency(ESSaRel)

• Hypothesis 3:

– Null Hypothesis:User Acceptance(ESSAVis++) < User Accep-tance(ESSaRel)

– Alternative Hypothesis:User Acceptance(ESSAVis++) � User Accep-tance(ESSaRel)

4.3. Study Scenario and Tasks

Based on the defined system model, an expertin safety analysis from the Software EngineeringResearch Group: Dependability of the Universityof Kaiserslautern defined a safety scenario andmodeled the corresponding safety models. Thedefined safety scenario we used in our studydescribes one of the possible situations where theRAVON robot [Proetzsch (2010)] can hit other solidobjects. The components of this possible hazardcomprise 48 compound components in the RAVONstructure with 70 basic events that can appear indifferent locations in the FT model simultaneously.However, due to the test constraints, we decided toinclude only part of the scenario. The scenario weused in our test consisted of 25 nodes with more than33 basic events. It is important to note that the safetyscenario was the same in both tests, with differentnames in order to avoid any learning effects. In total,three tasks were defined with the help of the expertin safety analysis:

• Task 1: Find the required safety information for theunderlying safety scenario. This included:

– Task 1.1: Find the Minimal Cut Set (MCS)information (Task 1.1.1). Then sort the foundMCS information according to their sizes (Task1.1.2) and their probability values (Task 1.1.3).

– Task 1.2: Find the number of basic events inthe root component.

• Task 2: Find the most critical components usingthe safety information extracted from the Task 1.

• Task 3: Find the failure path and the logical rela-tion between the two pre-specified components.

4.4. Study Design and Execution

The study was designed as a controlled experimentunder laboratory conditions with a maximumtimeframe of two hours per participant. To reducelearning effects, we randomized the participants for

5


each sub-group in such a way that half of theparticipants ran the experiment first on ESSAVis++and then on ESSaRel, while the other half did itthe opposite way. In the experiment, 25 participantswith different backgrounds participated. We dividedthe participants into two groups: the real endusers group and the other users group. The realend users group included 6 safety experts withat least three years of industry experience fromthe Fraunhofer Institute for Experimental SoftwareEngineering (Fraunhofer IESE), who deal withembedded systems in their routine work. Therewere also 4 system engineers in the real enduses group, who had experience of developing andmaintaining robots. The other users group includedsoftware engineers and visualization experts (4 and11 respectively) from the same university. Out of25 participants, 21 were males. The age of theparticipants ranged from 20 years to 42 years, with amean of 28.8 years.

Before performing tests with all participants, weconducted a pilot study with a safety professionalfrom the Fraunhofer IESE, who was also involvedthroughout the evaluation process. The settings werefinalized after changes were made in the initialsettings according to this pilot study feedback.

To assure that the participants had a basic levelof knowledge about the two platforms, a 30-minutetutorial on ESSAVis++ and a 30-minute tutorialon ESSaRel were conducted. We allowed all theparticipants to ask further questions during thetraining session and during the test in case ofany technical issues. At the end of each test, weasked each participant to fill in two questionnaireforms: a closed-ended questionnaires form witheight questions based on a Likert scale (scaled from1 to 5 to show the degree of agreement where 1meant “Strongly Disagree” and 5 meant “StronglyAgree’, while the sixth option was “Don’t know”),and an open-ended questionnaires form with threegeneral questions. Each test lasted for approximatelyone hour per participant, excluding the training timeand the time for filling in the questionnaires.

The experiment was performed using a desktopenvironment, with the participants seated in frontof the computer. They interacted with the platformwith a normal mouse and a keyboard. We allowedthem to try the target platform prior to the test andencouraged them to ask for more details about anyinteraction technique during the test.

5. RESULTS AND DISCUSSIONS

In this section, we describe the results of our studyand analyze these collected results for different

evaluation tasks. For each task, we compute andreport the accuracy of the results (in terms of numberof correct answers / number of right answers) andthe time needed to complete the task (in seconds).Except for the open-ended questionnaires, weperformed a descriptive statistic analysis to providequantitative statistical information about the relevantmetrics.

We show the study overall results in Table 1and Table 2. In these tables, we also divided theparticipants into two groups (i.e., the real end usersgroup and the other users group, as mentionedin the previous section). This group categorizationhelps to understand the behavior of the participants’performance according to their background. Figure 3and Figure 4 provides the results of Table 1 in line-chart form to give a quick comparison overview ofparticipants’ performance and accuracy results ofthe both platforms, i.e., ESSAVis++ and ESSaRel.

First, we performed a descriptive analysis of thecollected data. Then we tested the data for normalitywith a Shapiro-Wilk’s test [Shapiro and Wilk (1965)].Since the data were not normally distributed, weran a Median test for independent samples andone-tailed Wilcoxon Signed-Rank test for dependentsamples with a significant level of 0.05 to test ourhypotheses. Table 2 provides descriptive statisticson the analysis together with the results of theone-tailed Wilcoxon Signed-Rank test. In the caseof the feedback from the closed-ended questions,we performed the Mann-Whitney U test to comparestatistically the positive responses (i.e., the numberof 4s and 5s) of participants for the both tools.

In Task 1.1.1, participants were asked to calculatethe Minimal Cut Set (MCS) information relatedto the current safety scenario, which is importantfrom the safety experts’ perspective in order toassess the safety criticality of the underlying system.MCSs provide information about the safety-criticalcomponents in the scenario and the critical safetypath between these critical components. That iswhy both platforms (i.e., ESSAVis++ and ESSaRel)provide this information due to its importance in theanalysis process. In this task, 19 participants spentless time with ESSAVis++ whereas 5 participantsspent less time with ESSaRel. However, the analysisper participant category did not contradict thegeneral results. ESSAVis++ provides this informationin an intuitive manner as it presents this informationin one of its menus, called the Safety Informationmenu. Users can trigger this option simply byinteracting with the ESSAVis++ Main menu by right-clicking the mouse button. They do not need tocalculate it manually, as it is calculated automaticallyfor the root component. In the case of ESSaRel,

6


Table 1: Participants’ average performance and accuracy

Table 2: Statistical information to show the difference between ESSAVis++ and ESSaRel

users need to perform a relatively longer process.They get the MCS information via three steps: (1)select the option “MCS Calculation” option from the“Analyze” menu; (2) select the required component,which is the root component in our case; and then (3)ESSaRel shows the MCS information in a separatewindow through a table representation.

In Task 1.1.2, participants were asked to calculatethe most critical MCS information in the givensafety scenario according to the size parameter. TheMCS is considered to be critical if it has a lowernumber of basic events. In our test, an MCS wasconsidered critical if it had fewer than three basicevents. ESSAVis++ sorts the MCS list accordingto the size parameter, while ESSaRel shows thesize information for each entry in the MCS menuwithout any sorting option. In this task, 8 participantsprovided more accurate results with ESSAVis++,5 participants with ESSaRel, while the remaining11 participants’ accuracy was the same in bothcases. Participants were able to accomplish this taskaccurately in both tools; however, the average timewas comparably less in ESSAVis++ (i.e., 42 sec.)compared to ESSaRel (i.e., 107 sec.) mainly due to

the fact that ESSAVis++ automatically sorts the MCSmenu entries.

Task 1.1.3 was to order the pieces of critical MCSinformation according to the probability value foreach one. The MCS is also considered to be criticalif it has a high probability value of failing. In ourtest, we specified > 1 ∗ 10−12 as the probabilitythreshold. Again, ESSAVis++ provides an option tosort the entries list according to their probabilityvalues, while in ESSaRel, users need to calculatethis information manually. In this case, participantswere able to calculate this information in ESSAVis++easily and with high accuracy (i.e., 94% accuracy),while in ESSaRel, many participants decided to skipthis task due to the required tedious calculations.Moreover, the time required for accomplishing thistask in ESSAVis++ (i.e., 39 sec.) was much less thanin ESSaRel (i.e., 740 sec.).

In Task 1.2, participants were asked to find thenumber of basic events in the root component ofthe given safety scenario. In this case, participants’understanding of the task differed, as some ofthem calculated the number of basic events on the

7


Figure 3: Participants’ average performance (i.e., efficiency) in seconds for all tasks

Figure 4: Participants’ average accuracy in percentage for all tasks

root-component level only while others calculatedthis information recursively level by level over thewhole tree. Therefore, we obtained different answersdepending on the participants’ understanding.ESSAVis++ provides this information directly throughthe safety-information menu; whereas in ESSaRel,users need to traverse the entire safety scenariohierarchy and then sum up the number of basicevents in each component one by one. Due to thedifferences in understanding, we received differentresponses, which is why we cannot determineaccurately which platform performed better. Theaccuracy value was approximately similar for bothplatforms (i.e., 88% for ESSAvis++ and 87% forESSaRel). However, the time required to completethis task using ESSAVis++ (i.e., 75 sec.) was lesscompared to using ESSaRel (i.e., 193 sec.). Thisis due to the visual support and the interactiontechniques provided by ESSAVis++, as it allowsexpanding and closing the required componentwithout losing the overall context.

In Task 2, participants were asked to identifythe safety-critical components based on the MCSinformation obtained from Task 1. We defined thecritical components to be the ones attached to oneof the critical MCSs in the system. In this task,13 participants provided more accurate results withESSAVis++, 5 participants provided more accurateresults with ESSaRel, while the accuracy resultsof the remaining 6 participants were equal in bothcases. The overall accuracy value for ESSAVis++was 82%, and for ESSaRel was 67%. However,on average participants used a little more time to

complete this task on ESSAVis++, i.e., an averageof 200 seconds on ESSAVis++ compared to anaverage of 181.5 seconds on ESSaRel.

In Task 3, participants were asked to find the logicalrelations between the two pre-specified componentsas well as the failure propagation path between thesetwo components. With regard to finding the failurepath between these two components, 4 participantswere more accurate in ESSAVis++, 2 participantswere more accurate in ESSaRel, while the remainingparticipants’ accuracy level was the same in bothcases. In order to find the logical relation betweenthe two pre-specified components, participants wereasked to navigate through the failure hierarchy.ESSAVis++ keeps the gate information but doesnot provide an explicit representation of it, whileESSaRel provides it explicitly and shows it throughsymbols. In this case, 5 participants were moreaccurate with ESSAVis++, 4 participants were moreaccurate with ESSaRel, and the remaining 15participants’ accuracy level was the same in bothcases. The overall average accuracy on ESSAVis++(i.e., 92%) was higher compared to ESSaRel (i.e.,86%). However, the real end-users group accuracywas 100% on ESSaRel compared to 97% onESSAVis++, contradict to the other users groupwhere the accuracy was 77% on ESSaRel comparedto 88% on ESSAVis++ (see Table 1). This is mainlydue to the reason that the real end-users grouphad the experience of using ESSaRel in theirworking life routine. In the case of time required tocomplete this task, on average participants spentless time on ESSAVis++ (i.e., 75 sec.) compared

8


to ESSaRel (i.e., 193 sec.). Even though the realend-users group accuracy was a little better inESSaRel; however, on average they spent much lesstime on ESSAVis++ (i.e., 33.5 sec.) compared toESSaRel (i.e., 81 sec.) to complete this task. Thisis because ESSAVis++ provides animations to showthe failure relations between different components.Also, as mentioned earlier, ESSAVIS++ providesbetter interaction options for expanding and closingthe required component in real time.

In Table 2, we provide descriptive statistics onthe analysis together with the results of one-tailedWilcoxon Signed-Rank test. We performed WilcoxonSigned-Rank test for each individual group as wellon all the participants’ results together.

First, we check the participants’ accuracy resultsthrough the one-tailed Wilcoxon Signed-Rank testwith a significant level of 0.05 to test statisticallyour Hypothesis 1. In the case of the real end-usersgroup, we get a Z-value : −1.618080 and a p-value :0.105645. In this case, the W -value is 1.5 while thecritical value of W for N = 5 at p � 0.05 is 0;therefore, the result is not significant at p � 0.05.When we perform the test on the other users group,we get a Z-value : −1.460593 and a p-value :0.144127. With the W -value = 1 and the critical valueof W for N = 4 at p � 0.05, it is statistically notpossible to validate the test due to the reason thatthe total number of pairs with no ties (i.e., N = 4) isnot large enough to perform this test. However, whenwe perform the test on all the participants’ accuracyresults then we get a Z-value : −2.022600 and a p-value : 0.043114. In this overall case, the W -value is0 while the critical value of W for N = 5 at p � 0.05is 0; therefore, the result is significant at p � 0.05and the null hypothesis is rejected. This also provesour Hypothesis 1, which states that ESSAvis++ ismore accurate or at least the same accurate as ofthe ESSaRel.

We also check the participants’ efficiency resultsthrough the one-tailed Wilcoxon Signed-Rank testwith a significant level of 0.05 to test statistically ourHypothesis 2. When we perform one-tailed WilcoxonSigned-Rank test on the efficiency of the real end-users group, we get a Z-value : −1.5724 and a p-value : 0.115851. In this case, the W -value is 3 whilethe critical value of W for N = 6 at p � 0.05 is 2;therefore, the result is not significant at p � 0.05. Inthe case of performing one-tailed Wilcoxon Signed-Rank test on the other users group, we get a Z-value : −1.991741 and a p-value : 0.046399. In thiscase, the W -value is 1 while the critical value of Wfor N = 6 at p � 0.05 is 2; therefore, the result issignificant at p � 0.05. When we consider all theparticipants, then we get a Z-value : −1.991741 and

a p-value : 0.046399. In this overall case, the W -value is 1 while the critical value of W for N = 6 atp � 0.05 is 2; therefore, the result is significant at p �0.05 and the null hypothesis is rejected. This provesour Hypothesis 2, which states that ESSAVis++ ismore efficient or at least the same efficient as of theESSaRel.

From the feedback of the closed questionnaires (seeFig. 5), we observed that most of the participants(i.e., 22 participants) gave credit to ESSAVis++compared to ESSaRel (i.e., 3 participants), dueto the interaction facilities and the visual supportprovided by ESSAVis++. The participants whopreferred ESSaRel mentioned that the ESSAVis++interaction approach is a bit confusing and verynew for them, making it difficult to adapt to it. Forthese participants, the interaction methods providedby ESSaRel were more “intuitive”. These participantsmainly had a software engineering background,and we observed that the main reason for theirpreference for ESSaRel was their experience withsimilar tools as ESSaRel. In the feedback, themajority of participants chose ESSAVis++ as thepreferred platform for working in the future. Only2 out of 25 participants voted for the ESSaRelplatform. We found that these two participants weremore familiar with the kind of interfaces provided byESSaRel.

In the case of participants’ feedback in the closed-ended questions, we perform the Mann-Whitney Utest to compare statistically the positive responses(i.e., the number of 4s and 5s) of participants for theboth tools. In the case of the participants’ responsefor the number of 4s, we get a Z-value : 3.0456 anda U -value : 2.5. In applying the one-tailed Mann-Whitney U test, we get a p-value : 0.00114 whilethe critical value of U at p � 0.05 is 15; therefore,the result is significant and the null hypothesis isrejected. In applying the two-tailed Mann-Whitney Utest, we get a p-value : 0.00228 while the criticalvalue of U at p � 0.05 is 13; therefore, the result issignificant. In the case of the participants’ responsefor the number of 5s, we get a Z-value : 3.3082 and aU -value : 0. In applying the one-tailed Mann-WhitneyU test, we get a p-value : 0.00047 while the criticalvalue of U at p � 0.05 is 15; therefore, the resultis significant and the null hypothesis is rejected. Inapplying the two-tailed Mann-Whitney U test, we geta p-value : 0.00094 while the critical value of U atp � 0.05 is 13; therefore, the result is significant.When combining the both positive responses (i.e.,4s and 5s), we get the similar results as in the caseof only 5s. This proves our Hypothesis 3, whichstates that ESSAVis++ is more accepted by usersor at least gets the same acceptance from users asof the ESSaRel.

9


Figure 5: Participants’ feedback for ESSAVis++ and ESSaRel platforms for the closed-ended questionnaires. The feedbackfor both platforms was collected separately after a participant finished the work on each tool.

We conclude from the above discussions thatall these results’ findings do not contradict thehypotheses of the study. We can also conclude thatthe visual support and the interaction techniqueshelp users to understand the safety scenario moreaccurately and more swiftly. Moreover, we observedthat the platform that automates the backgroundcalculations really helps the users to trust theanalysis result more than the platform where theseusers need to perform the required calculationsmanually. We can trace this observation fromthe acceptance ratio of ESSAVis++ compared toESSaRel. Also, many participants mentioned duringthe test that ESSAVis++ is more intuitive comparedto ESSaRel because of the interaction options,the visual support, and the different highlightingtechniques provided by ESSAVis++. Especially thesafety experts, who belong to the real end users ofESSAVis++ and work with ESSaRel in their dailywork routines, appreciated that ESSAVis++ showsthe multi-views of the related components in thesafety scenario.

As in every study, there are some limitations forour study, too. First, the study was designed toinclude only those tasks that can be performedusing both platforms. We avoided evaluating thosetasks that cannot be performed by either of the twoplatforms in order to maintain consistency in theresults.

6. CONCLUSION

To support the safety analysis process in embeddedsystems, we presented our ESSAVis++ visualplatform. The focus of this paper was on the user

evaluation study that we conducted to measure thefeasibility and effectiveness of a common interactivevisual platform to be used by the real end users (i.e.,system engineers and safety experts). We presentedthe study results in terms of accuracy, efficiency,and acceptance ratio by comparing ESSAVis++ withESSaRel.

The results of our user evaluation study show thatESSAVis++ provided a relatively higher accuracylevel than ESSaRel. However, ESSAVis++ outper-formed ESSaRel in efficiency due to many aspects,such as the interaction techniques provided, the vi-sual support, and the synchronization of the abstractview of the failure in 2D with the system 3D model.Moreover, the participants’ acceptance level for ES-SAVis++ was significantly better than for ESSaRel,again due to the visual support and the interactionfacilities provided.

In the future, we intend to provide a visualizationfacility in ESSAVis++ for viewing the softwarearchitectural aspects of the underlying embeddedsystem. We also plan to synchronize this intendedvisualization with the two visualization forms alreadyprovided. One possible future direction is to findan intuitive approach to synchronization betweendifferent data sets in such a way that the platformwill respond to its users in real time. Moreover, wealso intend to perform user studies to collect users’reactions toward the idea of integrating betweendifferent data sets into one scene as well asmeasuring their comprehension level for this kind ofdesign decisions.

10


7. ACKNOWLEDGEMENT

We would like to thank Max Steiner and DavideTaibi who helped us in executing the user evaluationstudy. The main work was done when the first author(Ragaad AlTarawneh) was a PhD student in theComputer Graphics and HCI Group at the Universityof Kaiserslautern.

REFERENCES

AlTarawneh, R., Bauer, J., Keller, P., and Ebert,A. (2013). Essavis: A 2dplus3d visual platformfor speeding up the maintenance process ofembedded systems. In Proceedings of the 27thInternational BCS Human Computer InteractionConference, BCS-HCI ’13, pages 43:1–43:6,Swinton, UK, UK. British Computer Society.

AlTarawneh, R., Bauer, J., Keller, P., Ebert, A., andLiggesmeyer, P. (2012). Essavis: A framework tovisualize safety aspects in embedded systems. InSIGRAD 2012, pages 59–66.

Basili, V. R., Caldiera, G., and Rombach, H. D.(1994). The goal question metric approach. InEncyclopedia of Software Engineering. Wiley.

Bieber, P., Bougnol, C., Castel, C., p. Heckmann,J., Kehren, C., and Seguin, C. (2004). Safetyassessment with altarica - lessons learnt basedon two aircraft system studies. In 18th IFIP WorldComputer Congress, Topical Day on New Methodsfor Avionics Certification, page 26.

CESAR Project (2010). “cesar project report”.“www.cesarproject.eu”.

Dehlinger, J. and Lutz, R. R. (2006). Plfaultcat:A product-line software fault tree analysis tool.Automated Software Engineering, 13(1):169–193.

Essarel Tool (2014). Embedded systems sadety andreliability analyer. Software Engineering ResearchGroup: Dependability, University of Kaiserslautern.“http://essarel.de/”.

Herman, I., Melancon, G., and Marshall, M. S.(2000). Graph visualization and navigationin information visualization: A survey. IEEETransactions on Visualization and ComputerGraphics, 6(1):24–43.

Kaiser, B., Gramlich, C., and Forster, M. (2007).State/event fault treesa safety analysis modelfor software-controlled systems. ReliabilityEngineering System Safety, 92(11):1521–1537.

Kaiser, B., Liggesmeyer, P., and Mckel, O. (2003).A new component concept for fault trees.Reproduction, 33:37–46.

Kreylos, O. (2012). Vrui virtual reality toolkit.“http://idav.ucdavis.edu/∼okreylos/ResDev/Vrui/index.html”.

Kumar, P., Subramanian, N., and Zhang, K.(2009). Savit: Technique for visualization ofdigital home safety. ACIS International Con-ference on Computer and Information Science,pages 1120–1125.

Lee, E. A. and Seshia, S. A. (2010). Introduction toEmbedded Systems - A Cyber Physical SystemsApproach. Lee and Seshia, 1 edition.

Proetzsch, M. (2010). Development Process forComplex Behavior-Based Robot Control Systems.RRLab Dissertations. Verlag Dr. Hut. ISBN: 978-3-86853-626-3.

Raitner, M. (2005). Visual navigation of compoundgraphs. In Pach, J., editor, Graph Drawing, volume3383 of Lecture Notes in Computer Science,pages 403–413. Springer Berlin Heidelberg.

Shapiro, S. S. and Wilk, M. B. (1965). An analysisof variance test for normality (complete samples).Biometrika, 52(3/4):591–611.

Yang, Y., Keller, P., Livnat, Y., Liggesmeyer, P.,and Hagen, H. (2012). Improving safety-criticalsystems by visual analysis. Dagstuhl Follow-Upseries.

11

Documents

Improving the Failure Understanding in Embedded Systems ... · 1Information Technology Department, Mutah University, Jordan 2Computer Graphics and HCI Group,3Software Engineering:DependabilityGroup