Linköpings universitetSE–581 83 Linköping

+46 13 28 10 00 , www.liu.se

Linköping University | Department of Computer and Information ScienceMaster thesis, 30 ECTS | Datateknik

2018 | LIU-IDA/LITH-EX-A--18/031--SE

Improving AR visualizationwith Kalman filtering andhorizon-based orientation– To prevent boats to run aground at sea

Förbättring av AR-visualisering med Kalmanfiltrering och ho-risontbaserad orientering - för att förhindra båtar att gå pågrund

Pontus Hero Ek

Supervisor : Jonas WallgrenExaminer : Ola Leifler

External supervisor : Tobias Larsson

Copyright

The publishers will keep this document online on the Internet – or its possible replacement– for a period of 25 years starting from the date of publication barring exceptional circum-stances. The online availability of the document implies permanent permission for anyone toread, to download, or to print out single copies for his/hers own use and to use it unchangedfor non-commercial research and educational purpose. Subsequent transfers of copyrightcannot revoke this permission. All other uses of the document are conditional upon the con-sent of the copyright owner. The publisher has taken technical and administrative measuresto assure authenticity, security and accessibility. According to intellectual property law theauthor has the right to be mentioned when his/her work is accessed as described above andto be protected against infringement. For additional information about the Linköping Uni-versity Electronic Press and its procedures for publication and for assurance of documentintegrity, please refer to its www home page: http://www.ep.liu.se/.

c© Pontus Hero Ek

Abstract

This thesis researched the possibility of improving the compass of smartphones as theearth’s magnetic field is not strong and is easily disturbed, either by the environment ortechnology. The compass is used in Augmented Reality (AR) when the AR visualizationshould correspond to a position on earth. The issue lies in oscillating input values to thecompass that reduces the AR experience.

To improve the AR experience without the use of external equipment, this work tried toboth filter the incoming values with a Kalman filter and to know the direction by capturingan image with a horizon that was image processed. The Kalman filter achieved a reductionin incoming disturbances and the horizon was matched against a panorama image thatwas generated from 3D data. The thesis starts off with requirements and contents of ARand goes through the different approaches that begins with a LAS point cloud and ends inmatching horizons with normalized cross-correlation.

This thesis furthermore measures performance and battery drainage of the built ap-plication on three different smartphones that are nearly a year apart each. Drift was alsomeasured as it is a common issue if there is no earthly orientation to correct itself unto,for instance the magnetometer. This showed that these methods can be used on OnePlus2, Samsung Galaxy S7, and Samsung Galaxy S8, there is a steady performance and effi-ciency increase in each generation and that ARCore causes less drift. Furthermore thisthesis shows the difference between a compass and a local orientation with an offset.

The application that was made focused to work at sea but it was also tested on build-ings with good results. The application also underwent usability tests that showed that theapplied functionalities improved the AR-experience. The conclusion shows that it is possi-ble to improve the orientation of smartphones. Albeit it can go wrong sometimes which iswhy this thesis also presents two ways to indicate that the heading is off.

Acknowledgments

I would first like to thank my supervisor Jonas Wallgren at Linköping University. The feed-back, answers, and proofreading improved this thesis with guidance that guided me in theright direction. I would also like to thank Ola Leifler and Tobias Larsson for their supportduring the thesis.

Finally, I must express my sincere appreciation for the help from Jonas Ekskog that helpedwith valuable inputs for the application and thesis.

v

Contents

Abstract iii

Acknowledgments v

Contents vi

List of Figures viii

List of Tables ix

1 Introduction 11.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Aim . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.3 Research questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.4 Delimitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Background 52.1 The application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.2 Geographical data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.3 Data from geodata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.4 Combitech . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3 Theory 73.1 Augmented reality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73.2 Horizon-based orientation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103.3 Filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133.4 Sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153.5 Usability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

4 Method 214.1 Pre-study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214.2 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214.3 Graphical design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224.4 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224.5 Measurable effects with the implementations . . . . . . . . . . . . . . . . . . . . 284.6 Usability tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

5 Results 315.1 Pre-study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315.2 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315.3 Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315.4 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325.5 Measurable effects with the implementations . . . . . . . . . . . . . . . . . . . . 39

vi

5.6 Usability tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

6 Discussion 436.1 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 436.2 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 466.3 The work in a wider context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

7 Conclusion 497.1 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

Bibliography 51

A Appendix A 57A.1 Kalman filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57A.2 Quaternions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

B Appendix B 61B.1 Test plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61B.2 Questionnaire . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

vii

List of Figures

3.1 The architecture of AR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83.2 Reality-Virtuality Continuum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93.3 Example of a histogram from Otsu’s threshold method . . . . . . . . . . . . . . . . 113.4 How prediction and correction works . . . . . . . . . . . . . . . . . . . . . . . . . . 143.5 Safety classification and its subattributes . . . . . . . . . . . . . . . . . . . . . . . . . 173.6 Subjective satisfaction classification and its subattributes . . . . . . . . . . . . . . . 18

4.1 Extraction process from 3D data to a panorama image . . . . . . . . . . . . . . . . . 244.2 Rotations of the smartphone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254.3 Kalman filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264.4 Understanding ARCore with its local perception . . . . . . . . . . . . . . . . . . . . 27

5.1 View in Unity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325.2 View from smartphone with a rendered horizon . . . . . . . . . . . . . . . . . . . . 325.3 Original image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335.4 Image after Otsu’s method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335.5 Generated horizon curve . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345.6 Overlaid horizon detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345.7 Generated panorama from mesh . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345.8 More results from rooftop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355.9 Results from boat images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355.10 Panorama where the boat images were taken . . . . . . . . . . . . . . . . . . . . . . 355.11 Image where the generated horizon were too high . . . . . . . . . . . . . . . . . . . 365.12 Image where the generated horizon were too low . . . . . . . . . . . . . . . . . . . 365.13 Overlaid horizon detection on buildings . . . . . . . . . . . . . . . . . . . . . . . . . 365.14 Panorama for house image 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375.15 Kalman filter result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375.16 How the filter handles disturbances . . . . . . . . . . . . . . . . . . . . . . . . . . . 385.17 Kalman filter on a low-pass filter input . . . . . . . . . . . . . . . . . . . . . . . . . . 385.18 World orientation with vessel rotation and local orientations . . . . . . . . . . . . . 395.19 Drift in Android sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

viii

List of Tables

3.1 Average kernel filter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103.2 A 1x5 kernel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123.3 Edge detection kernel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

5.1 Frames per second for the application with different smartphones . . . . . . . . . . 395.2 Allocated time for the horizon detection on Samsung Galaxy S7 with different

resolutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405.3 Allocated time for the horizon detection on the three different smartphones with

a height of 120 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405.4 Battery consumption for different smartphones . . . . . . . . . . . . . . . . . . . . . 405.5 Average answer from the participants . . . . . . . . . . . . . . . . . . . . . . . . . . 41

A.1 Quaternion multiplication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

ix

1 Introduction

This chapter presents the motivation and aim behind this thesis. Furthermore, this chapterlists the research questions that will be answered to arrive at the aim. Lastly, there will besome comments on the restrictions in this thesis in the delimitations section.

1.1 Motivation

There is an ongoing breakthrough [43] with technologies that illustrate information. Two ofthese technologies are augmented reality (AR) and virtual reality (VR). Their new way ofvisualizing information may be a solution to handle the ever increasing amount and differenttype of information that is available. AR can be used in many industries and have recentlybeen used mostly in prototyping [43]. AR is commonly thought of an assistant that visualizeinformation in real-time, making it easier to make something correct. A fascinating idea ofAR is to use it to save lives [47].

The idea of displaying a real-time augmented experience is not a recent discovery. How-ever, since cell phones have been improving in performance and increasing the number andquality of sensors, a new area of usage for AR has bloomed. The idea of using AR in smart-phones is something that is quite new as it started to make sense when smartphones withcameras and high computational power came in 2007, such as the iPhone [23]. Smartphoneshave been improving in performance at a high rate but is the performance nowadays stillimproving at a high rate?

In the last couple of years there has been a decrease in demand for performance of smart-phones. The speed of improvement that smartphones has, will maybe receive an additionalboost when mobile virtual reality, artificial intelligence, and 4k screens [52] have had its break-through. Some argue that we are in the breakthrough right now.

Combitech answers this by investing in AR related projects and a problem that has oc-curred in these projects is that the magnetometer is easily disturbed. This leads to a sufferingAR experience when the AR visualization elements is displaced and stutters. This occurswhen a correct heading of the smarthpone is required as there is a need for a position in thesmartphone to correlate to the positions of objects in the real world.

Combitech has before collaborated with Thomas Porathe which is why it was obvious tofollow his work [42] to explore AR in the marine environment. His work is based in Norway

1

1. INTRODUCTION

which has a high boat density and difficult fluctuations on the seafloor which leads to manyaccidents.

The statistics at sea show that there have been around 32 deaths per year over 2007 to2016 [6]. The same report illustrates that approximately 20% of the accidents with a deadlyoutcome comes from boats running aground. Furthermore, the statistics from Redningssel-skapet, Norway’s rescue association at sea, illustrated that leisure boat accidents have in-creased in recent years. There were 249 leisure boat accidents in 2015 followed by 261 in 2016and 330 in 2017 [5]. One cause for this was illustrated with a survey by Gjensidige, a Norwe-gian insurance company. They noted that 40% of leisure boat drivers at sea in a section in thethe middle of Norway could not understand nautical charts [45].

Therefore, AR at sea is a great area to experiment with for the purpose of reducing theaccidents. This thesis focus on improving the placement of an AR view that illustrate a red no-go area [42] when the phone is directed at sea where the water is shallower than three meters.The issue is that a smartphone’s orientation is lousy because its magnetometer is not accurateenough which result in fluctuations of the AR visualization elements. The inaccuracy liesin that the magnet field around the earth is not so strong and therefore the magnetometeris easily disturbed. The magnetometer gives lousy results if for instance, the smartphone isnear equipment that disturbs the signal such as a boat hull or a phone-case that has a smallmagnet to keep it closed.

This displacement of the no-go area reduces the user’s trust in the application as thisdisturbance will cause the AR-visualization on the screen to be offset. This problem withthe magnetometer occurs when there is a requirement for better precision of the location,meaning that the world coordinates in latitude and longitude correspond to the same area inthe phone. Therefore, other common AR applications do not suffer from this problem as theAR view does not correspond to a specific location.

1.2 Aim

This thesis aims to research and select an appropriate approach to increase the accuracy ofAR. This research is meant to contribute to the field of AR to further improve the accuracy ofAR on a smartphone, particularly when it is hard to receive correct input to the smartphonefrom its sensors. Two main approaches will be evaluated separately and the integration ofthe approaches will be evaluated on the smartphone’s perceived orientation with a usabilitytest.

There have been other studies in the AR field that estimate the orientation of the smart-phone for AR applications. One of the most promising recent approaches to horizon-basedorientation is from a joint study by NASA and Stanford University [8]. Their approach is be-lieved to obtain a more correct horizon, but there is no indication of how long it took for thehorizon to be detected. Therefore the difference in conditions further reduced the inclinationto use their method as they used a slow movement vehicle with an independent Graphicsprocessing unit (GPU). There is no information about which GPU was used so the processingresources was assumed to not be available on a handheld device.

Another approach to improve an AR visualization is to filter input of the orientation fromthe sensors in a smartphone. The Kalman filter is a filter that has been proposed in ear-lier studies to reduce the impact of disturbances in signals. A previous article tested how aKalman filter estimated three rotations [61]. The Kalman filter in the article only followed astep function and reacts similar to a low-pass filter, but still illustrates a useful scenario ofusing a Kalman filter for lowering the impact of disturbances on rotations.

This aim can specifically be traced back to a paper released in 2012 by Michael Geryautzand Dieter Schmalstieg [21]. When the potential of AR applications are discussed it is ob-served that:

2

1.3. Research questions

The most significant technical challenge on mobile devices is measuring the camera’s pose (that is, itsposition and orientation) in relation to relevant objects in the environment.

The more specific aim is to improve the accuracy on the application and it has been de-cided that two main approaches will be researched:

• Improve the accuracy by the use of horizon-based orientation to extract the horizonwith image processing techniques. This horizon is then matched against a renderedhorizon from 3D data.

• Filter data which seem faulty in the input.

The thesis further includes a test, to see the changes caused by the implemented algo-rithms. This test will compare the accuracy of the original approach with the modified ap-proach in this thesis. This test will be a visualization of the generated mesh from the pointcloud set from Lantmäteriet [30], a Swedish authority for property formation and geographi-cal information. The mesh illustrates the surface of the environment and this will be overlaidon the camera to know that we have the correct information of where we are.

Furthermore, battery consumption, frames per second, and time for the matching of hori-zon will be recorded in order to see the consumption of the implementations. The outcomeof this thesis is an application that achieves a high accuracy with the help of the three ap-proaches that can be seen in the list above. The added functionality will also be evaluated bya usability standpoint on how the application can more easily be trusted.

1.3 Research questions

This thesis answer four research questions:

1. Can horizon-based orientation be realized so that the generated horizon matchesagainst a 3D mesh with the correct heading? If the heading from the matching is in-correct, how can the application and user know?

2. How does the heading of the smartphone’s magnetometer compare to a heading that isaltered with a correctly applied filter?

3. How will the implementations affect the battery-life and the CPU usage of the smart-phone?

4. How will the application with all the approaches integrated compare against the com-mon orientation vector with magnetometer in Android Studio in usability?

1.4 Delimitations

This thesis only looks at a stand alone smartphone. There will not be any other equipmentinvolved, such as a solution with an external compass.

3

2 Background

This chapter involves a background to the application and information regarding the basicelements in this thesis.

This thesis focuses on how to improve the accuracy when the AR in a smartphone isrendered at sea. The application is motivated by an increasing number of accidents at seaand how a smartphone’s orientation can be improved.

2.1 The application

The application is a prototype on how a solution that uses AR can look like in order to reducecasualties at sea. The focus of this thesis is specifically on how to improve the smartphone’saccuracy of its rotation compared to the real world. The idea behind the application originatesfrom Thomas Porathe’s article: 3-D Nautical Charts and Safe Navigation [42]. This articlefurther presents how the cognitive workload would be reduced with a navigation systemthat can be displayed in real time with red no-go areas.

This thesis will improve the visualization of red no-go areas that would be marked if thebottom of the sea is shallower than three meters. The augmented image on the smartphoneshould display the bottom of the sea correctly. How it is evaluated for the image to be correctcan be read in the method chapter.

2.1.1 Requirements

The application has a target group that will use this application in situations where it is im-portant that the user feel secure and can trust the application. The correctness of the visual-izations is therefore important due to the sense of security it contributes to the user.

2.2 Geographical data

The coordinate system that is used is the SWEREF 99 TM together with the height system ofRH 2000 which comes from Lantmäteriet [30]. They are Sweden’s official reference systemsfor the plane and the height system respectively. The data has an accuracy of 0.1m on a hard-surfaced area and readings display that the average standard error on a hard-surfaced area is

5

2. BACKGROUND

0.05m. The standard error in plane is on average 0.25m, however the accuracy of height in asingle laser point can differ more if the terrain is inclined.

2.3 Data from geodata

The data from Lantmäteriet came as a LAS dataset, which had to be converted to an OBJ fileformat that can be accessed by Unity. LAS is a data format that contains a point cloud. Itis an industry-standard binary format for storing airborne lidar data [59]. Lidar stands forLight Detection and Ranging and is a technique for measuring the distance to the surface ofa distant object [7]. Lidar measures the distance to an object by illuminating the object witha light source and determine the time it takes between the emission and the detection of thereflection. The approach of lidar is the same type of approach as radar technology and is anoptical remote sensing technique.

2.4 Combitech

Combitech is a technical consult firm that operates in Scandinavia. Combitech has the intentof being at the front in recent technologies. Therefore they have a department called RealityLabs that is researching and experimenting on new technologies. The short projects thatexperiment with the new technologies have the aim of both attracting and testing the newtechnology towards customers. AR is a very hot topic in their section and they have oftenworked with AR in the last year.

6

3 Theory

A horizon in this thesis is found by first transforming an image to grayscale and blurringthe image. Afterwards every pixel in the image is segmented into either black or white.Where the black pixels are the ground and the white pixels are the sky. Then the horizonis proposed to be where the white pixels meets the black pixels, starting from the top. Thechapter is divided into five sections which are: augmented reality, horizon-based orientation,filter, sensors and usability.

3.1 Augmented reality

In previous decades augmented reality has been something from science fiction but has nowbecome known worldwide. The popularity of AR may be a result from enthusiasts of sciencefiction. However there exist some great features that AR have, such as visualization. Withthe help of a great visualization tool, the future can look brightly on the idea of visualizationin education [50], medicine [47] and in the industry [43].

Augmented reality was discussed decades ago and got its definition in 1994 [36]. Thefirst prototype of a hand-held device comes from an article in 2003 [57]. The study testedand saw that it was possible to make a portable AR prior to smartphones. The study used aHewlett-Packard PocketPc and a 320x240 camera to create a road map application that hada 3D compass that displayed where to go. This can be said to be the first mobile augmentedreality (MAR) which is defined as the augmented reality in mobile devices where you canaccess the AR experience anywhere you are [39]. MAR has been going very well becausesmartphones have much location awareness with the help of its sensors - for instance ac-celerometer, gyroscope and GPS.

According to Singh and Singh, AR is defined as a view of the real, physical world thatinclude additional information that enhance the view [51]. The augmentations is the extradata that is incorporated in the view. The information can consist of any data type such astext, 3D object or video.

The AR architecture that is meant for the smartphone is illustrated in figure 3.1 and hasits original from [51] but have been modified by the author.

One can see that first there is an observed reality which takes a scene into a reality sensor(camera). This image is later passed into a trigger matcher (pattern) with metadata such asgeographical identification. The metadata is then passed along to an augmentation selector.

7

3. THEORY

Figure 3.1: The architecture of AR

This augmentation selector also obtains additional data that is required and creates the aug-mented image that should be outputted. Lastly the augmented image and the original imageis combined into the presented reality. In some contexts this procedure is simplified into:sensors to processing and then processing to display.

This architecture is simply the foundation that is required to present an AR view. What iseven more important is an AR experience that is to one’s satisfaction. To achieve an adequateAR view, there are five characteristics that can be evaluated.

1. Firstly there is a demand for high-quality sensors that interacts with the environment,therefore the sensors that are involved can be looked into further.

2. Trigger matching and image augmentation is required for the scene to be understood,this in order to apply the correct augmentation for the environment. This is often donewith image processing (IP).

3. User interaction is also an important aspect as there are several additions you can haveto improve the AR experience further. For instance, eye-tracking, gesture, speech andtouch.

4. The fourth way is to improve the information infrastructure such as using cloud ser-vices to obtain data.

5. AR needs high computational power and a great communication infrastructure to bindeverything together.

One of the large factors of why AR is not being pursued that hard is the requirements thatare put on the environment that the AR should be running in. For example there is often asituation where brightness can make the AR experience worsen, because known features arenot found. The user that gets a bad AR experience will not return to the service [39].

AR can be used and implemented on smartphones. The processing power of a smart-phone (iPhone 6) is 120,000,000 times faster than the single best computer that sent people

8

3.1. Augmented reality

to the moon and back [44]. The Samsung Galaxy S7 which was used in this thesis has evenbetter processing power than the iPhone 6 [15], giving access to a powerful device with acamera that can be used for AR. The AR view is directly applied to the camera view, enhanc-ing the view. There are many AR applications available on the market where the replacementof facial features and showing a 3D model of an object are the most common. However, it ishard to distinguish what counts as AR which is why it is described below.

AR can be split into two cases, see-through AR and monitor based AR [36]. See-throughdisplays interact with the real environment directly with some extensions applied on theenvironment. The problem that see-through displays encounter is that they require accuratehead tracking with great calibration and an acceptable field of vision. An example of this isthe Microsoft HoloLens which is a head-mounted display that is see-through. The HoloLenscan be seen as a pair of bigger glasses that fits around the head.

The other case is a monitor based AR experience which is a display where the visualiza-tion of the augmented image is overlaid on a video that is either in real-time or a prerecordedone. This case have the most focus in this thesis as it involves the use of a cell phone with acamera that is directed at the sea to visualize the shallow.

In Figure 3.2 one can see the image of Milgram’s Reality-Virtuality Continuum [36]. Thefigure illustrates how AR and VR are related to each other by being some distance away inthe spectrum. The virtual environment is where VR is positioned and consists of an environ-ment that is based exclusively on virtual objects that are computer generated. This computergenerated environment can have either a realistic surrounding or an abstract environmentwhere physical laws no longer operate [39].

The real environment to the left and the virtual environment to the right are the oppositesthat are mentioned. The continuum displays the difference between AR and VR.

Figure 3.2: Reality-Virtuality Continuum

3.1.1 Complications of AR

There are some hindrances that is keeping AR to really break through. A list from Singh andSingh [51] shows that there are some technical aspects to evaluate and other obstacles thatexists which are needed to be overcome. To fully experience an AR view, a see-through headmounted display (HMD) is required. The integration of a HMD is hard and is illustrated withthe instance of Google glasses, introduced in 2013 [38]. This technology was thought of to bebefore its time in its features.

The obstacles that follows are not specifically directed at Google glasses. However theyare the most well-known HMD equipment that was meant for commercial use at its release.The obstacles that are blocking the breakthrough of AR are:

1. A stigma that it looks weird and nerdy to go around with a HMD.

2. The HMD is too expensive.

3. It is hard to have the HMD on at times.

4. Some people experience simulation sickness [28], often coupled with rapid movementsin the augmented image.

9

3. THEORY

5. "Changing behavior is much more challenging than changing technology" [10] is whata chief executive and founder of a design company commented, who is involved indesigning wearable devices.

Some of these hindrances will decrease faster than others over time as the expansion oftechnology still continues, which the majority of the population in U.S.A. believes it will[53]. Then time and continuous research will lead to smartphones and its components to beincreasingly sophisticated to deal with these hindrances. The hardest hindrance is to reducesimulation sickness which requires eye tracking and haptic feedback such as vibrations. Theeye tracking is there to help the eyes focus on a particular point and minor vibrations thatsimulate physical experiences.

3.2 Horizon-based orientation

This section involves the extraction of the horizon in the image and matching it to the horizonthat is generated from the 3D environment. This section can further be divided into two parts,namely image processing and matching.

3.2.1 Image processing

This part consists of the theory that regards the Image Processing (IP) techniques that wereused. In order to extract the horizon of the two images, there were some methods involved.

Preprocessing

Before one wants to use an image to process there are some preprocesses that yield a betterend result if they are done. The first that was done is translating the colored image to agrayscale image. The standard formula that is used to translate a colored pixel that consistsof three colors, red, green, and blue to a gray pixel is [33]:

gray = 0.299¨ red +0.587¨green +0.114¨blue

The other preprocess was to filter the image with an averaging kernel which is a matrixthat is shown in Table 3.1. The result of applying the kernel gives a blurring effect on animage [29]. This blurring effect will reduce the impact of noise that can be for instance a birdor a tree.

For each pixel in the image the kernel is placed and multiplied with the pixel’s adjacentpixels in order to obtain the targeted pixel’s averaged value. In other words, the pixel at(100,100) will have an averaged value that take into consideration all the pixels between (99-101, 99-101).

Table 3.1: Average kernel filter

19

19

19

19

19

19

19

19

19

Otsu’s method

After the normalization was done on the image, the next step was to segment the image intoblack and white. For this, Otsu’s threshold method was used [40]. Otsu’s method is aboutmaking a histogram which is a diagram of color value of 0 to L on the x axis and the numberof pixels that have the color value in the image on the y axis. L is often represented by 255

10

3.2. Horizon-based orientation

in Color32 format. Otsu’s method has the purpose of choosing a threshold value in betweentwo clusters of color values.

The theory behind Otsu is the following: Let the total number of pixels be N = n1 + n2 +... + ni, where ni is the number of pixels with value i. The histogram can be regarded as aprobability distribution calculated as:

pi =niN

(3.1)

The whole probability distribution of (3.1) is normalized by satisfying:

Lÿ

i=0

pi = 1 (3.2)

The threshold T that Otsu’s method generates will divide the pixels into two classes, C0and C1. C0 contains the pixels that have a gray-level in [0, T] while C1 contains the pixels thathave a gray-level in [T+1, L]. With these two classes, one can calculate the mean values ofthem both with:

u0 =T

ÿ

i=0

ipiw0

, u1 =L

ÿ

i=T+1

ipiw1

(3.3)

Where w0 =řT

i=0 pi and w1 =řL

i=T+1 pi.With these equations formulated, one can calculate the variance between the classes with:

σ2b = w0(u0 ´ uT)

2 + w1(u1 ´ uT)2 (3.4)

The best threshold T in [0, L] causes uT to be the one that maximize the variance (σ2b )

between the two classes [25].A histogram generated from the pixels values in the application can be seen in Figure 3.3,

which indicate that Otsu’s method would select a value around 150 as the threshold valuewith the highest variance. Whereas the lowest variance can be found around 0 and 255

Figure 3.3: Example of a histogram from Otsu’s threshold method

Extracting the curve

A kernel is also used to analyze patterns and is in this case a detection of a pattern in an image,however it has several uses in other areas [48]. To understand a pattern is to understandthe data’s relations, regularities and/or structure. This pattern is the foundation for doing

11

3. THEORY

predictions on where the pattern is supposed to be situated. An advantage of kernels isthat they do not require coordinates to function as they look at the adjacent pixels. Anotheradvantage of kernels is that their methods can be applied to a wide range of data. However,one should bear in mind that by using larger kernels, there will be an increase in computationcost [31]. The kernel in Table 3.2 can be used to generate a match if two pixels above and twopixels below the searched pixel have opposite values. A match would is generated if theoutcome of the kernel is zero.

Table 3.2: A 1x5 kernel

110-1-1

An edge detection kernel was used in this thesis which looked like the kernel in 3.3.

Table 3.3: Edge detection kernel

-1 -1 -1-1 8 -1-1 -1 -1

3.2.2 Matching with cross-correlation

The horizon curve that is extracted from the image is to be matched with the true horizon thatis generated from the geographical data. This is required in order to know the heading of thesmartphone. The matching of the two horizons are done with normalized cross-correlation(NCC). Cross-correlation provides a measure of similarity between two signals, and by nor-malizing the cross-correlation one gets the value of similarity to be in [-1, 1], where a value of1 corresponds to the exact same curve and -1 being the exact opposite. The exact opposite ofa curve is the same as the exact opposite of a line between two points. A line that increasesby one in height when it increases by one in width has the exact opposite line that decreasesby one in height when it increases by one in width.

The formula for obtaining the cross-correlation in 1D is altered from [32]:

řN´1i=0 xiyi

N is the length of the patterns x and y that is matched and a pattern’s value at point i isthe height of the pixel. This correlation calculates a similarity between two patterns, howeverthis similarity is illustrated in numbers and therefore hard to comprehend. To have a morestandardized value, one normalize the correlation with:

řN´1i=0 xiyi

(b

řN´1i=0 x2

i )(řN´1

i=0 y2i )

The final equation (3.5) that was used describes when pattern y is larger than pattern x.

řM´N+1j=0

řN´1i=0 xiyi+j

b

řM´N+1j=0

řN´1i=0 x2

iřN´1

i=0 y2i+j

(3.5)

M is the length of pattern y. M´ N + 1 is the difference plus one to go through the sumof j at least once.

12

3.3. Filter

NCC can also be used on a 2D image directly, where it measures the correlation of eachpixel in the image. However, to correlate a whole 2D image is computationally costly [24] andthe mesh data that the image is matched against is not a 360 video of the real environmentwith textures.

3.3 Filter

This thesis used two types of filters, the Kalman filter and a low-pass filter. These two filtersare illustrated and explained below.

3.3.1 Kalman filter

The Kalman filter was introduced in 1960 [26] and has since been one of the most used fil-ters to predict a forthcoming state. The Kalman filter algorithm constitutes of a loop thatsequentially predict the next state and after an update of the measurements are done, an out-put estimate is produced. The Kalman filter has the advantage of being able to be used livebecause of its small computational cost.

The Kalman filter handle uncertain noise with an approach similar to a Bayesian optimumsolution [61]. Kalman filters are often seen in navigation systems and in computer vision,making it a great candidate for this thesis. The goal of the Kalman filter is to produce anoutput signal that is as close to the ground truth as possible. The input to the Kalman filterin this thesis is the rotation in degrees of the heading that is one of three rotations from aquaternion which can be read further about in 3.2.3.

A general definition of the Kalman filter is explained together with Figure 3.4 where a readvalue is marked by a black cross. Each reading has an uncertainty, which is not displayed inthe Figure. The blue cross is the prediction of the next state in the Figure, where the nextprediction is based on the previous value. It is possible to predict several states in the future.However, the uncertainty would then increase in size considerably for each step.

After the prediction is made, the measurement is obtained and with these two values,a correction is made. The red line between the predicted and measured value is where thecorrected value will be selected. If the predicted value is close to the measurement, the un-certainty will decrease. The distance the corrected value is selected on the red line is decidedby the Kalman gain. The Kalman gain can be a constant or a variable that changes over timethat decides which value to trust the most between the measurement or the prediction. TheKalman gain is a number between 0 and 1 which is weighted by the distance between themeasured and predicted value.

The mathematical foundation for the Kalman filter can be read about further in AppendixA.

13

3. THEORY

Figure 3.4: How prediction and correction works

3.3.2 Low-pass filter

A low-pass filter allow signals that are of low-frequency to pass through easily without beingmodified much, while the higher frequencies are altered significantly [22].

This thesis focused on change over time, therefore the filter took into consideration theprevious value and then altered the new value based on the previous. The formula for thelow-pass filter is:

it = 0.75 ¨ it + 0.25 ¨ it´1

Where i is the input from the sensors and t is the time step.

3.3.3 Quaternions

The quaternion number system was often used throughout the thesis and was primarily im-portant to understand when there was a transformation around the y axis (pitch value) of theimage. If one wants to rotate 3D-objects, the rotation is done with quaternions. Furthermore,a quaternion is used in the filter when the data is inputted to the gyroscope, accelerometer,and magnetometer.

A quaternion is similar to a complex number except that a quaternion has three imaginaryterms. The first constant represent a scale (a in q) and the other three terms that containimaginary terms represent a vector (b, c, and d in q) [56].

q = a + bi + cj + dk

The vector is the one that was focused on in the filter as every gyroscope, accelerometer,and magnetometer uses quaternions for its position and rotation in the world.

There are two other popular ways to handle rotations in 3D. These are Euler angles [56]and 3x3 rotation matrix [56]. Euler angles have a problem with gimbal lock and use a lowmemory by the storage of 3 numbers. Gimbal lock arises when two rotation axes align witheach other, therefore the remaining rotational axis is locked to only rotate in its own axis andcannot always produce the true angle [19]. A 3x3 rotation matrix store 9 numbers but doesnot have the issue of gimbal lock. Lastly, a quaternion store 4 numbers and do not suffer fromgimbal lock, making it a balance between the two.

See Appendix A for a more mathematical explanation of quaternions.

14

3.4. Sensors

3.4 Sensors

A smartphone has four important sensors that can be used in AR: gyroscope, accelerometer,magnetometer, and geographic positioning system, where the three first belong to the micro-electromechanical system (MEMS) group [37]. A MEMS often consists of mechanical parts.However, every MEMS do not have moving parts. A MEMS could be a few millimeters insize but can be much smaller. The purpose of a MEMS is to observe something and later reactto it [37]. The reaction can for instance be set to update a value at a fixed frequency or makean operation if the observed value is over a specific limit.

3.4.1 Gyroscope

The gyroscope uses the force of Earth’s gravity to determine the orientation of the smart-phone. This orientation occurs fast but has a lot of drift. The gyroscope in smartphonesmeasure rotation around three axes x, y, and z. The gyroscope today is an optical gyroscopethat use lasers in an elipse of fiber optics [49].

3.4.2 Accelerometer

An accelerometer measures an object’s changes in velocity over time. Everyday life consist ofmany accelerations, therefore an accelerometer have a wide area of uses. This orientation isslow but is instead more precise compared to the gyroscope. The accelerometer fused with agyroscope is a good approach to measure the local orientation of the device.

The measurement of acceleration in an accelerometer can easier be understood by imagin-ing a weight attached to a string from the ceiling of an airplane [41]. This weight will measurethe acceleration depending on its position, meaning that if the weight is exposed to a nega-tive acceleration, it will move to the front of the airplane. With the position of the weight, anangle can be obtained from the offset from its resting position. The angle can then be used tocalculate the acceleration of the plane. Accelerometers in smartphones are designed to act onvibrations that occur with accelerations [18]. These vibrations are then read by microscopiccrystals which in turn induce a voltage to the measurements.

3.4.3 Magnetometer

A magnetometer measures the strength and direction of magnetic fields. Magnetometershave many daily uses that many are unaware of but a simple magnetometer is often knownas a compass that detects true north based on a magnetic field [55]. For a smartphone toobtain a local direction of magnetic north, it has three magnetometers, fixed perpendicular toeach other.

3.4.4 Global Positioning System

GPS have three components that forms it: the space segment (satellites), the user segment(receiver), and the control segment (control stations). The receivers obtains signal informationfrom the satellites and apply trilateration [27].

A signal contains three types of information: a psuedorandom code, ephemeris data, andalmanac data [46]. The psuedorandom code is the identity of the satellite that transmittedthe information. The ephemeris data contain information if it is healthy or unhealthy andits timestamp. The almanac data is the orbital information for the satellite and other satellitemembers in the system.

The receiver can then calculate the time it takes for a satellite’s signal to travel to the re-ceiver with the help of the signal information. This time times by the speed of light will

15

3. THEORY

retrieve the distance and with the help of two more satellites, trilateration can be done. Tri-lateration uses the signal from three satellites to identify the location of the phone [46]. How-ever, a minor error of one-thousandth of a second in the clock of the receiver will result in aposition error of 320 kilometres. This error can be fixed by adding another GPS signal whichwill result in a polygon and the center of this polygon is where the receiver is.

3.5 Usability

AR has a risk of showing to much information which can overwhelm the user and reducethe efficiency of AR [51]. This can become a problem if several augmentations are takingplace and the user cannot differentiate what is important and what is not. Furthermore, thepurpose of this application is to help persons at sea, therefore it needs some extra thoughtsregarding how to make it user-friendly. The main concern is how to easily transition to anexecuting state where the image processing is made without disrupting the user’s experience.

3.5.1 Requirements for mobile augmented reality

A user study that was done on AR in shopping centres [39] found crucial requirements onMAR devices. The relevant requirements from that study that can be used in this thesis isdisplayed below.

Privacy was a big concern and a specific restriction on how information was dealt withwas desired to be regulated with user control. The thought of having information stolen wasof a big concern as the information contains what the person has actually looked at. Notonly is the visual information important to secure but also the location data with all of itsmetadata. There is furthermore an unease regarding the increasing risk of physical safety forthe user when they are highly immersed in the AR environment. An instance of this would beto collide with nearby objects or fall overboard. Therefore, the service that is provided haveto be aware of and reactive towards the environment. This lead to an augmented image thatdo not contain too much information. There was an additional requirement that the contenthad to feel useful for the user which is similar to not displaying too much information.

The most important requirement for this thesis was reliability of the information. Peopledesired that the information was up-to-date and valid, e.g. how much it could be trusted. Animportant aspect is to have a trustworthy source that can assure the user of its credibility.

3.5.2 Usability attributes

The scenario that this thesis is focused on is when the user takes up the smartphone and looksaround the environment. The application has to be trusted by the user, which encourages anddemonstrates the benefits of a usability evaluation. Therefore there is a usability evaluationin chapter 5. The theory behind the evaluation is based on the article "Usability: CriticalAnalysis and Taxonomy" [2].

Usability is often characterized with imprecise and overly ambiguous terms leading to dif-ferent evaluation depending on the person [2]. Therefore if one mentions usability, a properstructure of the term is required. The article’s definition of usability is divided into six at-tributes: knowability, operability, efficiency, robustness, safety, and subjective satisfaction.These six attributes are the foundation that the application is evaluated for, where the lasttwo attributes were regarded as extra important in the implementation and therefore hasmore focus. The attributes and their subattributes are:

Knowability

Knowability is defined as how the user can understand, learn, and remember how to use theapplication. Its four subattributes are: clarity, consistency, memorability, and helpfulness.

16

3.5. Usability

Operability

Operability is defined as the system’s capacity to provide users with the necessary function-alities. Its four subattributes are: completeness, precision, universality, and flexibility.

Efficiency

Efficiency is defined as the system’s ability to generate results compared to the resources thatwas used. Its four subattributes are: efficiency in human effort, efficiency in task executiontime, efficiency in tied up resources, and efficiency in economic costs.

Robustness

Robustness is defined as the ability to resist error and unfavourable situations. Its four subat-tributes are: robustness to internal error, robustness to improper use, robustness to third partabuse, and robustness to environment problems.

Safety

Safety is defined as the system’s capacity to prevent risk and damage. Its three subattributesare: user safety, environment safety, and third party safety. Considering that the thesis focusextra on this attribute, this attribute is further described and every subattribute can be seenin Figure 3.5. It is drawn by the author but the design is from [2].

Figure 3.5: Safety classification and its subattributes

• User safety, the ability to avoid risk and damage to the user while the system is usedby a user. User safety can be divided further into: physical safety, legal safeguarding,confidentiality, and safety of user assets. These avoidance attributes are the same forthird party safety.

17

3. THEORY

• Environmental safety, the ability to avoid risk and damage to the environment whenused.

• Third party safety, the ability to avoid risk and damage to other people while the systemis used by the person.

Subjective satisfaction

Subjective satisfaction is the system’s ability to produce satisfaction in the user. Its two sub-attributes are: interest and aesthetics. A more extended illustration of all subattributes can beseen in Figure 3.6. It is drawn by the author but the design is from [2].

Figure 3.6: Subjective satisfaction classification and its subattributes

• Interest, the ability to catch and maintain user’s awareness and intellectual curiosity.

• Aesthetics, the ability to entertain the user in sensational terms and there are five sub-attributes that the sensation can be divided in.

3.5.3 Related work

There were not any work found in this area that had an approach similar to this thesis. How-ever, there have been several parts that have been picked from different articles.

There were many possible approaches to choose from to extract the horizon where mostof them were used in unmanned aerial vehicles (UAVs) in order to stabilize and estimate thealtitude of the UAV. An article detected the horizon with the use of psuedo spectra images(PSI) [58] with okay results, but their purpose was to illustrate that analyzing an image’s PSIworks to identify specific entities.

Another article from NASA with collaboration with Stanford University [8] used both theextraction of a horizon and matched it with 3D data. The article focused on navigating onother planets with the use of the horizon in extreme environments where information andsignals are restricted. The article showed great results but their is no indication of how longit took for the horizon to be detected. Therefore the difference in conditions further reducedthe tendency to use their method as they used a slow movement vehicle which did not haveany listed performance requirements.

The method that was picked was in an article named An Improved Algorithm for HorizonDetection Based on OTSU [25] which found a horizon with the same method that this thesisused. The method in the article was used as a foundation for an UAV’s altitude estimation

18

3.5. Usability

and stabilization and proved to be efficient. The difference is that this thesis also used sensorsto improve the method further which made it possible to rotate the image so the horizonalways was parallel to the image. Furthermore, the input from the sensors could with thehelp of the x rotation know if the horizon was to low and instead use the previous generatedpixels for the horizon curve. This was done to reduce glares caused by the sun reflecting onthe sea.

This thesis differ from other popular AR experiences, such as an AR experience that putout models that interact with the real world. This AR visualization is growing fast and a studymade by Eric Marchand et al. displays the underlying technique for pose estimation [34].The study shows how estimating the pose of the smartphone in relation to the environmentis done and how to keep track of where the object is positioned. ARCore uses this kind ofpose estimation to position objects locally. However, in this thesis ARCore is only used for itslocal orientation and not for placing out objects locally.

Furthermore, this thesis differ from articles that detects a straight horizon line when de-tecting a horizon which is done for instance in [20] which displays five different methods todetect such a straight line in a marine environment. This detection is often done in order fora unit that takes an image can stabilize and know an approximate of the altitude of itself.Instead, this article generates a curve to show the contours of the horizon in order to matchit against 3D data. A similar horizon line that can be obtained with the use of sensors whichwas done in this thesis.

There have also been studies of the Kalman filter previously and there are articles wherethe Kalman filter had been applied to estimate three rotations in [61]. The Kalman filter in thearticle only followed a step function and reacts similar to a low-pass filter, but still illustratesa useful scenario of using the Kalman filter. Another article showed that a Kalman filter canbe used to estimate altitude change during a fall [54]. The article used the Kalman filter ontwo different pressures and used the maximum of the difference to predict a fall or not. Theconclusion was that a Kalman filter was more responsive compared to a moving averagefilter.

The matching was done with normalized cross-correlation which is a common methodwhen checking if signals are similar and there is no related work that uses it in a different way.Furthermore, there have also been parts where no articles explained a process, for instancehow to make a panorama image from a 360 degree view.

19

4 Method

This chapter is divided into six sections: pre-study, data, design, implementation, usabilitytests, and measurable effects with the implementations. These six sections involves the sixphases that took place in this thesis. The sections are sorted in the same order that theyoccurred.

4.1 Pre-study

The pre-study comprised a literature study to gather the resources that were required. Thepart of gathering the resources were performed in a structured way to find as much relevantresources as possible. The pre-study was carried out at EBSCO’s [17] research database to-gether with Google Scholar. EBSCO was the main website for research and Google Scholarwas used to find popular references in the Augmented Reality area.

This pre-study also took into consideration the requirements in the background chapterwhen the literature study took place. The requirements can be simplified into high visualiza-tion accuracy on a smartphone comparable to Samsung S7.

Both the filtering of the input and the extraction of the horizon had the highest priority.The resulting approaches from the pre-study that was determined to be best suited for thisthesis were then further researched. When the research was ongoing, there were some arti-cles that was regularly referenced and these were further looked into. These articles mostlyinvolved definitions and new ideas that were tested for the first time. These articles wereused as a foundation for the AR research in this thesis.

4.2 Data

The geographical data from Lantmäteriet came in squares with UTM coordinates and had aside of 2.5km. Data came as LAS files of 300 Megabytes per each square, it was later convertedto .OBJ format in CloudCompare [13]. The file size was further reduced to 20 Megabytes bysubsampling the mesh to use vertices that had approximately a minimum space of seven me-ters between each other. The original data was very precise and if there were no subsampling,the mesh allocated 2 Gigabytes per square. Afterwards it was tested that in order for the meshto work in Unity with the correct transparent material, the limit was somewhere between 80

21

4. METHOD

Megabytes and 40 Megabytes. Furthermore, after the subsampling, there were some verticesthat stood out in the mesh which was removed in Meshlab [35].

The coordinate system was required to be altered in order for the center to be in the origoin Unity. Therefore a global shift was introduced in the mesh for it to be translated correctlyin Unity. The result was that the mesh was in the center and one unit length in Unity corre-sponded to one meter in the mesh. The center of the mesh had to be translated and be known.Therefore the UTM coordinates for the center of the mesh was transformed to latitude andlongitude for it to correspond to the GPS input on the smartphone.

4.3 Graphical design

The graphical design is about the prototype application that will be the end-product. Thedesign was developed based on the theory, pre-study and the requirements. The final resultin this thesis is a prototype, therefore references on how one should think of when designinga prototype has been taken from [9]. The article raises two important questions that havebeen taken into consideration when the implementation has taken place in this thesis.

• What is the essence of the existing user experience?

• What are essential factors that the design should preserve?

These questions are appropriate for a prototype that provide a proof of concept that re-alizes a theory into something practical. The first question was easy to work with as it isclear how there was a requirement for a better user experience where users could trust theapplication.

The second question is answered by that the smartphone is the only component that isrequired for the application. Furthermore, usability should be highly considered when de-veloping applications that uses AR. Therefore there are some principles that should be takeninto account which is presented in the theory chapter that regards the requirements for mo-bile augmented reality.

4.4 Implementation

The implementation can be divided into three smaller sections, one for each task that wasevaluated. The order of the following subsections are ordered so that the first approach isthe first that was started and estimated to take the longest time. Most implementations weredone in C# in Unity and if something was implemented anywhere else, it is declared in thetext.

4.4.1 Extracting and matching the horizon

This subsection regards the extraction and matching of the horizon. To easier understandthe steps below, every image can be represented by a matrix where every pixel in the imagecorresponds to a row and column in a matrix. This section answer the first research questionif horizon-based orientation can be realized so that the generated horizon can be matchedwith a correct heading.

A method chart of how the process works is illustrated in the list below:

1. First render an image of the screen without augmentations.

2. Rotate the image so the horizon is parallel to the the phone based on gyroscope andaccelerometer.

3. Observe the rotation in the x rotation for a cutoff height that the horizon can not beunder.

22

4.4. Implementation

4. Extract the horizon curve for the image.

5. Generate the panorama image and its horizon curve based on the coordinates that thesmartphone has.

6. Match the two curves.

Preprocessing

First, the image was transformed to gray-scale because it is faster to write and read a sin-gle pixel value, compared to three values that a color pixel have (red, green and blue) [29].Furthermore, the Otsu method that is presented below is easier to calculate in gray-scale.

After the conversion to gray-scale, an averaging filter was used, which has the effect ofblurring the image [29]. The idea of a mean filter is to take the average of the adjacent pixelsin order to reduce noise and have smoother transitions of the color values.

Otsu’s method

The gray-scaled image was segmented into either black or white depending if the pixel valuewas below or above the threshold value. When the decision to implement Otsu was taken,the direction to take was to use Bitmaps from the Bitmap Class in System.Drawing in theimplementation. That decision was based on that there were numerous examples online andit was a simple and effective way to extract pixel data as the image is seen as a matrix ofpixels. However, the Bitmap class was only available for Windows and not for Android soinstead Texture2D was used which is a class in the UnityEngine.

The thresholded value from Otsu’s method is calculated by searching the graph from bothsides until the largest cluster from the right and the largest cluster from the left is found. Themiddle of each cluster determine a mean point and the two mean points for both clusters areadded together and divided by two to find the highest variance.

Extracting the curve

The first step was was to retrieve two types of images. The first image to be fetched was of thereal image of the environment. This image was to be matched against a mesh that depictedthe environment. Therefore, it was necessary to receive a panorama image from the mesh.This was done by extracting a small rectangle in the middle of the 360 view to avoid scalingissues. The rectangle was then added onto a wider rectangle that represented the panoramaimage which can be seen in Figure 4.1. After each rectangle was extracted there was a rotationof the camera in the 360 view and another rectangle was added to the panorama image.

The panorama image was further extended by the width that the degree of the image inthe implementation represented. This is required so the cross correlation could also find amatch in that space. The panorama image was also set so the first pixel represented the truenorth direction in the map. Scaling was also an important aspect in the cross correlation,therefore much thought was put into the scaling for the image and the panorama.

The actual extraction of the curve was done with the help of a 1x5 kernel. There was a riskwith extracting something else, so the kernel was used to reduce the noise [31]. The increasein computation cost compared to a 1x5 kernel was negligible as the image is small.

Unfortunately there was not an actual test at sea for the application. However, the testwas instead simulated on the computer by using images that were taken just for this cause bysomeone that was on a boat. The images had location data from where the image was takenwhich made the simulation work the same. The latitude and longitude in the image was usedto place the camera on those coordinates which then depicted the mesh’s surroundings.

For the rotation and filter, an updating variable that included both the gyroscope and theaccelerometer rotation vector was retrieved for the usage of the y rotation. It was consideredfoolish not to use the other components of the rotation vector when extracting the horizon.

23

4. METHOD

Figure 4.1: Extraction process from 3D data to a panorama image

In Figure 4.2 there are four images which illustrates the rotation of the smartphone. The firstimage in the upper left is the starting image, and to the right of it, the z rotation has beenaltered. The lower left image is where the x rotation has been altered, and the lower rightimage displays an adjustment in the y rotation.

The x rotation was used as a check to see that the extracted horizon was not under acertain line that was extracted from the rotation vector. This was thought to be useful toreduce the impact in an image where the reflection of the sun was intense. If the generatedpixel were under the estimated line from the rotation vector, the previous pixel value in thearray was used instead. The z rotation was used to rotate every image so the horizon wouldapproximately be parallel and in the middle of the screen all the time. The rotation furtherhelps to better fit the panorama image.

There was a test on how the application would perform on buildings. In order to do this,a smaller block of 3D data was used to increase the accuracy of the data. Furthermore, theoption of how near the 3D mesh would be generated was altered from 100 to 5 meters.

The curve of the horizon on the panorama image was easy to generate as it was generatedby the application and the mesh had a different color value compared to the background.Therefore, it was only necessary to go through the image, column wise until it noticed achange in value. The pixel where the change in value occured was then allocated in an arraywhich defined the horizon for the panorama image. Both curves are stored in arrays whereeach cell has the height value of the curve.

Matching with normalized cross-correlation

When both of the curves was in an array each, matching was done between the arrays toidentify where the curve had the highest similarity. The matching value was ranging from -1to 1 after it was normalized. It measures the similarity between the curves. This value is usedin the context of usability, for instance to display a certainty of the outcome.

Edge detection

The edge detection kernel was was applied on the image right after Otsu’s method. Theresulting pixel would be black if the current pixel multiplied by eight, subtracted by the

24

4.4. Implementation

Figure 4.2: Rotations of the smartphone

adjacent pixels was higher than the Otsu’s threshold divided by 12. This edge detection wasdone as a test on how it would look like and display how the approach could be improved.The threshold that came from Otsu’s threshold had no scientific reasoning and was usedinstead of just using 0. Because when 0 was used it generated much more edges.

4.4.2 Filtering sensor values

The input that comes from the sensors on the smartphone was decided to be filtered. Thereare two types of filters, a simple low-pass filter and a Kalman filter that have a heuristic that ismodified according to this problem. This part of the method will answer the second researchquestion on how the orientation of a magnetometer compares to a filtered orientation.

Kalman filter

This section treats how the results from the gyroscope, accelerometer, and magnetometerwere filtered. This sensor data comes as a quaternion from Android Studio which is fusedand is updated on a frequency of 0.06s. The Kalman filter was required to take in a quaternionthat contained the readings of the magnetometer, gyroscope, and, accelerometer.

A more numerical concept of the Kalman filter can be seen in Figure 4.3 and is describedbelow.

The Kalman filter approach has an initial state in Next state where it obtains the readingsfrom the sensor. These readings are taken to the Prediction state where a prediction is madebased on the uncertainty of the reading.

Afterwards, the prediction for the model is applied. The prediction is based on the pre-vious state’s derivative, for instance a position can be predicted with the initial position,velocity and, time in the initial position. Depending on how confident the model is of thenext state, the covariance prediction will be smaller. The covariance matrix is updated withthe multiplication of the previous covariance matrix and the state-transition matrix.

Then there is the calculation of how the predicted value differed from the measurementwhich is the distance between the predicted center point subtracted with the actual mea-surement. The result is the prediction error and now the Kalman gain is what is left for thecorrection.

The Kalman gain decides where on the line between the measured and predicted pointthe corrected value will be. The Kalman gain K lies between 0 and 1, and decides which of

25

4. METHOD

Figure 4.3: Kalman filter

the two points is most believable. If the measurements are thought to be exact, we will havea small K and if we believe more in the prediction, the Kalman gain increases.

In the application, the filter observed the difference between the rotation in the quaternionthat used the magnetometer and the one who did not. If there was a difference between them,the Kalman gain would decrease.

The Kalman gain in Figure 4.3 is 0.66 as the output estimate is closer to the predictedvalue, i.e. we believe more in the prediction. The Kalman gain is used to correct the state andobtain the output estimate. Then the next iteration of the loop will start, however, the nextstate does not do any readings any more.

When the filter was implemented, there was one large factor that made the filter not per-form as well as intended. If there was a disturbance of the magnetometer, the Kalman filterwill still conform to the disturbance which will lead to a disturbance anyway even if it wasslightly smaller. The resulting effect was similar to a low-pass filter, but achieved in a morehazardous way.

There was one specific implementation to the Kalman filter that achieved the still effect,which was to alter the Kalman gain variable in another way. The Kalman gain was reduced to0 fast if the difference between readings did not match. If the difference between the previousand the actual reading of the rotation vector with no magnetometer was low, meaning thatthe smartphone was not meant to be moved. The real rotation vector should also mimic thisbehaviour, therefore a check if the difference is low can be made. If the check is true, theincoming rotation should use the previous rotation instead. However in order to reduce a bigdisplacement, linear interpolation between the previous and the current rotation can be donewhere the previous rotation is highly prioritized.

The first implementation of the Kalman filter was stand alone and acted only on the rota-tion that involves the magnetometer. Afterwards when the Kalman filter was integrated withthe rest of the application, the rotation quaternion that used the magnetometer was used untilthe horizon had been matched to the environment. Thereafter, the rotation with the Kalmanfilter was used to check if the rotation with only the gyroscope and accelerometer had a valueclose to its own.

The testing of the filter was divided into two smaller parts where the first was to checkhow the filter behaved when there were several orientations of the smartphone. The otherpart was to see how it handled disturbances, which in this case was a magnet that oscillatednear the smartphone. Under the progress of this thesis, the built-in self calibration on thesmartphone was done twice and was done to prevent the magnet disturbance test to confusethe sensors. The self calibration test can be arrived at by dialing *#0*# in the call keypad andthen navigate to Sensor and later Gyro Selftest and Selftest under Magnetic Sensor.

26

4.4. Implementation

Low-pass filter

The low-pass filter was applied directly after the readings were received on the smartphone.The low-pass filter was implemented with a cut off frequency of 0.3 (α) which is a commontactic while using the orientation sensors in smartphones [12]. It is done to reduce the noisethat the magnetometer cause. It was filtered directly as the current reading depends on theearlier and by doing it in Android Studio a better performance was expected.

4.4.3 Stabilizing the view with ARCore

ARCore is a platform for building AR applications on Android [3]. It tracks and understandsthe world with a local position and rotation based on images and sensors. ARCore was usedto compare how its orientation reinforced that a heading was correct and how much thedrift the sensors had. The drift of the accelerometer and gyroscope was compared againstthe ARCore in order to see if this could be a solution so that a new matching would not beneeded as often.

ARCore was used together with the rotation of the heading that the earlier positions pro-vided. This implementation was justified to decrease drift in the camera that is an issue ingyroscopes [16].

This drift is caused by the integration of the signal from the gyroscope. Gyroscopes givesreadings with a high frequency and outputs angular velocity. Therefore, in order to get theangle, you integrate. However, this integration causes noise to turn into drift from the gy-roscope. This can be further explained by integrating a cosine signal. The integration of acosine signal will be a sine but divided by the frequency which results in a smooth signal.

Figure 4.4 illustrates how the ARCore can be used to improve the experience. If the localorientation can be perceived in combination with the heading of the speed vector, there canbe another source that supply the orientation of the smartphone.

Figure 4.4: Understanding ARCore with its local perception

For the two coordinate systems to align there have to be some sort of trigger to do so. Thisthesis went with a button that should be used by the user with a push of the button when thesmartphone was directed in the same direction that the boat was heading. This was tested bycycling with the smartphone in hand to get the results in the result section.

4.4.4 Integration of approaches

The approaches were first implemented by themselves in order to test them individually andto satisfy a structured approach which included iterative steps with sprints. The final scenario

27

4. METHOD

was that the Kalman filtered quaternion was used as the orientation until an image was taken.After an image was taken the orientation quaternion without the magnetometer was used.

4.4.5 Difficulties

A problem that arise when we take the photo is that the application freezes while the cal-culations are made. This occurs because the thread is occupied in Unity. This problem wasattempted to be answered by threading parts of the calculations. Sadly most of the calcula-tions is forced to be done on the Unity main thread and therefore the problem could not beresolved.

The Bitmap class was tried out as a start for the image processing. However, Bitmaps donot have multi-platform compatibility and could therefore not be used. OpenCV, which isan open source computer vision and machine learning software library, was attractive to usewhen the thesis was started.

However, the setup time for OpenCV in Unity was also considered to be long becauseyou need to call C++ functions from C#. The process of making the exports was looked intoand the process of making a library file in C++ and export the functions from the library forthem to later be be called from C# was not a task for this thesis. A somewhat proof for thetediousness is that one can buy an OpenCV plugin in Unity Asset Store for 95$.

Another reason was that it was decided that this thesis would not compare different meth-ods for the horizon matching which made OpenCV impractical.

4.5 Measurable effects with the implementations

Three test were performed on three different smartphones, OnePlus 2, Samsung Galaxy S7,and Samsung Galaxy S8. The three tests can be divided into: performance, battery, and driftwhich were done to see how the implementations affected the performance of smartphones.

While the performance and battery test was done, the application placed out 20000 no-go rectangles randomly in a square of 6000 meters. Each rectangle that was placed out hada measure of 12x8 meters. This was done to simulate the effect of no-go areas where thewater would be more shallow than three meters. Furthermore, the whole mesh from thegeographical data was a square with a width of 5000 meters.

4.5.1 Performance

The performance was further split into two tests, the first test noticed the frames per secondthe application had, while the other noticed the time it took for the horizon detection to finish.The approach was to write a .txt file with the averaged value of the time it took for the horizonto be generated ten times. The average frames per second the application had over a periodof 16000 frames was also logged.

4.5.2 Battery

The application’s power consumption was measured by how much the battery was drainedby each of the three smartphones. The usage of the application was higher than normalbecause during the test the horizon was generated every 30th second. All three smartphoneswere measured at the same time in the hands of the author so that the rotation would be thesame for each smartphone. The battery status was then logged several times under sevenminutes. After the test was done, an average value was generated of how much the batterywas drained each minute.

28

4.6. Usability tests

4.5.3 Drift

The drift test was a test on how much the fused sensors actually drifted without conformingto a worldly heading such as a magnetometer. This was to see how long the matching of thehorizon could be held true before the heading was off and it was required to be done again.It was tested against ARCore in order to see if it could instead be a possible solution if thedrift happened too fast.

The drift was measured by logging the data when the smartphone was as still as possible.The environment for the testing was a smartphone holder and a solid glass window. Thesmartphone holder was of a plastic model that is often seen attached to the windshield of acar. The orientation was later written to a .txt file and could later be used as input to a graph.

4.6 Usability tests

Usability testing was both part of the evaluation of the result and the result itself as it an-swered the last research question on how the final application compared to a common ori-entation vector. But before the planned testing there were some important steps to preparebefore the actual meeting. These steps are inspired by how you plan for usability testing by[4].

• There should be clear test goals.

• How will the test occur?

• Determine the target group.

According to Barnum it is important to write a test plan to address the points above, whichwas done and can be read in Appendix B.

Before the usability tests started, it was only fair that the self calibration function was runon the smartphone. The testing started off with a brief introduction that explained the ap-plication and why the test was done. The participants could be divided into two subgroups,low technical knowledge and high technical knowledge. Every participant was between theage of 20-30.

The end purpose of the usability tests was to compare the two orientation vectors whichwas done by the participants to answer a questionnaire. The questions in the questionnairewas short and to the point. This was thought to welcome strong opinions and not be an-swered with the middle option: moderate, neither, or maybe. The questionnaire can also beread in Appendix B.

29

5 Results

This chapter presents the results of the thesis. The chapter is divided into six sections whichare pre-study, data, design, implementation, usability tests, and measurable effects with theimplementations.

5.1 Pre-study

The top findings in the pre-study were Otsu’s method and the Kalman filter. These twoapproaches were both chosen because of their robustness, speed, and matching attributestowards this thesis. Other viable findings were Canny Edge Detection, Hough transform andRandom Forest.

The Kalman filter was chosen because it could be done in real time at a low computationalcost and did not really have a competing method that had all of its strengths.

5.2 Data

The test map over a nearby test area can be seen in Figure 5.1 where it is rotated so that thenorth of the image corresponds to the correct direction in the Unity scene. One unit lengthin Unity corresponds to one meter in the real world. A thing that is required in the imple-mentation is that the map’s origin in UTM coordinates correspond to the origin in Unity’scoordinate system.

5.3 Design

The application’s view without randomized rectangles can be seen in Figure 5.2. The applica-tion has three buttons, one matches the horizon and another one adds the vessel’s heading ina local orientation vector to get a worldly orientation. The last one in the lower right is therefor debugging purposes to hide and show the mesh.

An implementation that was done to increase the trust towards the visualization is a gen-erated horizon that can be outputted on the screen. This feature allows the swipe of a fingerto alter the heading and can be tested against the real horizon to see if it is correct.

31

5. RESULTS

Figure 5.1: View in Unity

Figure 5.2: View from smartphone with a rendered horizon

There are many functions on the application that are associated with the buttons and themanual rotation of the horizon. These extra functionality may cause more of a distractionthan helping which was discussed during the usability tests.

5.4 Implementation

The results from the implementation is divided into three sections, extracting and matchingthe horizon, filtering incorrect values, and local orientations with vessel rotation.

5.4.1 Extracting and matching the horizon

The result from extracting the horizon is illustrated and described below, where the image inFigure 5.3 is the first image from the smartphone which has a resolution of 213x120.

The next image in Figure 5.4 is the resulting image after the image has been rotated, con-verted to gray-scale, averaged with a kernel, and segmented with Otsu’s method.

32

5.4. Implementation

Figure 5.3: Original image

Figure 5.4: Image after Otsu’s method

Figure 5.5 displays a found horizon after the matching kernel has found the first matchvertically starting from the top.

In Figure 5.6, the found horizon has been altered slightly depending on the rotation vectorand overlaid on top of the original image. One can see that the white house to the right of thetallest building matched below the horizon and have therefore obtained the previous valuefor the horizon curve.

Figure 5.7 depict the panorama that is generated in Unity from the geographical datafrom Lantmäteriet [30] and is what the found horizon is matched against. The camera on aSamsung Galaxy S7 has a field of view of approximately 60 degrees, therefore the panoramaimage has an aspect ratio that is six times larger in width.

33

5. RESULTS

Figure 5.5: Generated horizon curve

Figure 5.6: Overlaid horizon detection

Figure 5.7: Generated panorama from mesh

Another result from a rooftop can be seen in Figure 5.8 which is taken on a brighter dayshows more explicitly that white buildings results in a break in the horizon. However, thedecrease in the horizon curve is fixed by checking that the horizon can not be lower than anx rotation that comes from accelerometer and gyroscope.

The next images in Figure 5.9 and 5.10 show results from images taken from a boat at seawhich is the target environment.

34

5.4. Implementation

Figure 5.8: More results from rooftop

Figure 5.9: Results from boat images

Figure 5.10: Panorama where the boat images were taken

There were two images that stood out where the horizon generation did not work asintended, displayed in Figure 5.11 and 5.12. The horizon in Figure 5.11 was too high becausethere was an object in the way but to the right is how it could look like if the application tookinto consideration the max height of the generated panorama.

Figure 5.12 depicts the generated horizon to the left where the overall intensity of the im-age was similar. In the upper right is the expected horizon if the smartphone would havebeen there and could use the pitch value of the gyrometer and accelerometer that is imple-mented. The lower right is an image where the edges of the image has been generated by thekernel in Table 3.3.

There was an additional test that tried to see how the approach would do on buildings inthe 3D environment. The result can be seen below in Figure 5.13 and the panorama in 5.14.The image was taken near a building which can be seen to the left of the panorama image.

35

5. RESULTS

Figure 5.11: Image where the generated horizon were too high

Figure 5.12: Image where the generated horizon were too low

Figure 5.13: Overlaid horizon detection on buildings

36

5.4. Implementation

Figure 5.14: Panorama for house image 3

5.4.2 Filtering incorrect values

Both the two filters that was applied reduced the disturbance on the orientation of the mag-netometer.

Kalman filter

The Kalman filter reduced the measurement noise in the rotation vector in the smartphonewhich can be seen in Figure 5.15. If we look closer on how the filter takes care of disturbances,a magnet in this case, we can zoom in where it took place and see in Figure 5.16 that it handlesit well.

Figure 5.15: Kalman filter result

37

5. RESULTS

Figure 5.16: How the filter handles disturbances

Low-pass filter

The low-pass filter was applied directly to the rotation vector that was combined with themagnet orientation in Android Studio and improves the experience at the expense of reac-tiveness. This is displyed in Figure 5.17 where the magnet orientation is smoother.

Figure 5.17: Kalman filter on a low-pass filter input

5.4.3 Local orientations with vessel rotation

The result from using the vessel’s previous positions and local orientations.Figure 5.18 illustrates the value of ARCore and the fused gyrometer and accelerometer

with the vessel’s angle based on its previous positions.

38

5.5. Measurable effects with the implementations

Figure 5.18: World orientation with vessel rotation and local orientations

5.5 Measurable effects with the implementations

The results from the measurable effects of the implementations can be split into performance,battery, and drift. The test were done on three different smartphones: OnePlus 2, SamsungGalaxy S7, and Samsung Galaxy S8.

5.5.1 Performance

The performance test was further split into two tests, the first test noticed the frames persecond the application had, while the other noticed the time it took for the horizon detectionto finish.

The first test on the frames per second the application had on different smartphones canbe seen in Table 5.1. A small side is that if the mesh was in the scene, there was a decrease ofapproximately six frames per second.

Table 5.1: Frames per second for the application with different smartphones

Smartphone: OnePlus 2 Samsung Galaxy S7 Samsung Galaxy S8Number of rectangles:10000 41.54895 48.37805 59.7378320000 33.32494 36.87293 58.5155140000 24.09024 20.65146 39.8267

When the application with ARCore was tested on Samsung Galaxy S7 and SamsungGalaxy S8, the frame rate was 30Hz. However that was due to ARCore restrictions and onlyupdating textures and the device pose at 30Hz. There is an option to disable the MatchCam-eraFramerate which gives the application the target frame rate of 60 frames per second.

Results of the other test, on how long it took for the horizon detection with a SamsungGalaxy S7 on different resolutions to be utilized can be seen in Table 5.2. Table 5.3 belowdisplays how long it took for the horizon detection on different smartphones where the reso-lution were of a height of 120. The width was the height times the aspect of its screen.

The result is an average of 10 readings in seconds for the horizon to be generated.

39

5. RESULTS

Table 5.2: Allocated time for the horizon detection on Samsung Galaxy S7 with differentresolutions

Resolution (WxH) 213x120 355x200 710x400 1278x720Image time (s) 0.1669262 0.2737851 1.664396 3.64203Panorama time (s) 0.5491002 0.8571609 1.728626 3.147131Total time (s) 0.7160264 1.130946 3.393022 6.789161

Table 5.3: Allocated time for the horizon detection on the three different smartphones with aheight of 120

Smartphone: OnePlus 2 Samsung Galaxy S7 Samsung Galaxy S8Image time (s) 0.324309 0.1669262 0.1722252Panorama time (s) 0.466589 0.5491002 0.3996228Total time (s) 0.790898 0.7160264 0.571848

5.5.2 Battery

The average power consumption that each smartphone had, measured in drained percentageper minute can be seen in Table 5.4 below:

Table 5.4: Battery consumption for different smartphones

Smartphone: OnePlus 2 Samsung Galaxy S7 Samsung Galaxy S8Avg power consumption (%/min) 1.97 1.91 1,43

5.5.3 Drift

Figure 5.19 illustrates the drift of the Samsung Galaxy S7 when using ARCore and the orien-tation vector that uses the accelerometer and gyroscope. ARCore was thought to have a morestable view, which it had.

Figure 5.19: Drift in Android sensors

40

5.6. Usability tests

5.6 Usability tests

The result in Table 5.5 is the average response from six test persons on the application. Theanswers were graded from 1-5 where 1 imply Strongly disagree and 5 being Strongly agree.Some questions in Table 5.5 have been shortened to fit the width of the page and reformattedto positive statements, see Appendix B for the complete questions. There was one questionthat the test person had to answer in text which was: How would you describe the differencebetween the two different rotational feedback? Every answer differed some but they all meantthe same thing. Answers: the second one was less twitchy, slightly smoother, better, and morecorrect.

A question that two participants asked was that they wanted to know how the horizonwas generated and the methods involved that ensured why the button would display green.

Table 5.5: Average answer from the participants

Questions: Result:Using this application was easy 4.5The rotational feedback at the start was satisfying 2.83The rotational feedback after the switch was satisfying 4.08To match the horizon manually was easy 2.67I think that I could easier avoid the red areas with this visualization, compared to a 2D map 4,67The amount of visual information in the application is mild, (disturbing buttons and models) 5I can trust this application at sea when the horizon is auto-generated 3.83I would feel more secure with an indication on how much the heading could be trusted 4.5

From the results, one can see that the test group did not think that the application wasdifficult to use. The primary question got answered which was that the smartphone’s orien-tation was improved when the magnetometer was not used. A manual match with the use ofa swiping finger was not hard to do. They also validated that this kind of navigation is easierto use, compared to a 2D map that Thomas Porathe proves in his thesis. There is not toomuch visual information in the application. They would trust this application at sea even ifthe horizon was auto-generated, but they would trust it even more if there was a red, yellow,or green image that is linked with how much the heading could be trusted.

41

6 Discussion

This chapter analyzes and evaluates the method and the results from this thesis. This chapterfurthermore involves motivations on why some approaches were made. The chapter endswith a section that describes the work in a wider context.

6.1 Results

This section provides the discussion regarding each result that can be commented.

6.1.1 Data

The resulting quality of the mesh was acceptable in both when one, four, or sixteen squareswere used. The quality of the mesh when only one was used was a must when it buildingswas the target and it would be even better if it was reduced further. The case when 16 squareswere used was in an attempt to go out with a boat that was justified. The quality was quitehigh as the sea was just one single big rectangle with a low amount of vertices. It was noticedthat the data set is required to be updated regularly because trees grow and will otherwisedisturb the matching of horizons.

The result of the reduced meshes may have been better because the author had no priorknowledge on compressing 3D models and how to reduce them before this thesis. However,guidelines on the Internet was followed and different approaches were tested.

6.1.2 Extracting and matching the horizon

The matching of the horizon worked better than expected in this thesis. However, there existsa limitation on the horizon-based orientation as it can only be said to work if there are distin-guishable features on the horizon such as a high house or altitude changes. A completely flathorizon cannot be matched against because every angle will have a high matching score.

A large problem that was discussed in this thesis was if the horizon is matched incorrectly,what and how can it be displayed. If the horizon gives another heading to the user whichleads to that the visualization to be false, then lives are in danger.

Therefore it is important to underline that the application cannot be completely trusted.The application is thus displaying how much the heading can be trusted with the help of

43

6. DISCUSSION

an image that display green, yellow, or red depending on how the calculated horizon angletowards north is compared to the heading that comes from the rotation vector that involvesthe magnetometer. The results also proved that a world orientation could be achieved withthe press of a button in the direction the boat were going in. Therefore the indicating imagecan have two orientations that can strengthen that the generated angle from the horizon iscorrect.

Another aspect was to encourage the user with its own input on how the horizon actuallylies as the horizon from the mesh is outputted on the screen. This horizon in the AR experi-ence can also be interacted with the use of a sliding finger. The slider is a way for the user tocorrect the rotation with its own visual feedback from the horizon.

Instances where the horizon was mismatched, for example in Figure 5.12 can be consid-ered to be hard to notice even for a human. The other incorrect generation of the horizonin Figure 5.11 can be seen how it can be improved by knowing the maximum height of thegenerated panorama. When the maximum height is known, the search for a horizon can startat that height, saving both performance and reducing faults. The adjusted horizon is nowgenerated by a hard coded value and is only an instance of how it could have looked like.

By looking at the amount of time it took for the generation and matching of the horizonwith a resolution of 213x120. Therefore, if the picture is taken at roughly the same place, thepanorama image from earlier can be used. Then one can see that it is quite fast and could beused repeatedly to ensure that the direction is the correct one.

It was a big surprise that it could also work near buildings. However it was experiencedthat if the position was somehow off, the displacement of the mesh would cause scaling prob-lems and disturb the matching. The overall result from matching the horizon on buildingswould be better if there were different layers of details of the 3D data. The mesh would thenhave high quality near the smartphone’s position and lower quality when the distance fromthe position increased.

The issue with this method is that it does not work directly when the phone is brought upas the Kalman filter does. This method furthermore involves tasks to the user which can bedifficult in some situations.

Lastly, the most important point is that it works surprisingly well, for instance when theapplication was tested towards the high building in the result section (5.3), it is safe to say thatit worked 9 out of 10 times. However, what this thesis pointed out was that the 1 time that itdoes not work, there are many safety measures that would say that the heading is incorrect.

6.1.3 Filtering incorrect values

The final Kalman filter proved to be good at following the actual orientation response andnegating disturbances. The disturbances were huge and was caused by a magnet and it stillcorrected it well. The values in the result were measured under a short time period and thecase of a magnetic phone-case will still alter the heading.

When the low-pass filter was used, one can see that the signal was smoother and thereforeincreasing the AR experience slightly. The disadvantage is that the orientation reacts slowerand thus leaving a small delay. This small delay is acceptable and not so noticeable. However,if we increase the cut off frequency, the delay on orientations would be much more noticeablebut not showing much of a fluctuation on deviating values.

6.1.4 Measurable effect with the implementations

It was a pity that the OnePlus 2 was not in the Samsung Galaxy generation as it would havegiven great statistics on the increasing performance on smartphones. However, one can seethat the hardware specifications and the benchmarks on the OnePlus 2 is just slightly lessthan Samsung Galaxy S6 [60].

44

6.1. Results

The results displayed that the frame rate was slightly better on the OnePlus 2 comparedto the Samsung Galaxy S7. This is explained by that the camera usage can take more perfor-mance and that the OnePlus 2 is just better at drawing graphics. However, the biggest reasonis due to the difference in resolution. The Samsung Galaxy S7’s screen has a higher resolutionwhich will result in 1.78 more pixels to render each frame [14]. Therefore, if it is thought thata bigger resolution scales somewhat linearly, the Samsung Galaxy S7 is still faster than theOnePlus 2. Furthermore, the Samsung Galaxy S8 has en even larger resolution but only by16%, compared to the Samsung Galaxy S7.

If the resolution is taken into account and seen to drop the frame rate linearly, one can seethat there is approximately a 55% increase in frame rate from OnePlus 2 and the SamsungGalaxy S7 and an 85% increase in frame rate between the Samsung Galaxy S7 and SamsungGalaxy S8. This is a big and steady improvement in performance that answers the question ifsmartphones are still increasing in performance.

The test can be concluded with that the application can be run on all the smartphoneswithout much of a notice of a slow update of the frame. The number of rectangles is expectedto not exceed 20000 and considering that all smartphones had a frame rate above 30 for 20000rectangles it is deemed good. However, one still needs to take into consideration that thesmartphones did not have any other services or applications running.

The drained percentage did not differ much between the OnePlus 2 and the SamsungGalaxy S7. This can be explained by that the OnePlus 2 has 10% more capacity [14] com-pared to the other two smartphones. Furthermore, the resolution of the OnePlus 2 display issignificantly lesser. Now in retrospect, to measure the battery life of smartphones that havedifferent ages is not a good reference because the battery life of a battery decreases with age.This can maybe explain the difference in drainage between the S7 and the S8.

The battery results were dramatic because it showed that the application drained up to2% per minute which would drain the whole battery in only 50 minutes. As a result of thelow battery time, a power bank is preferred when the application should be used for a longerduration.

The drift test showed that the the fused sensors had drifted approximately 0.3 degreesafter 20 seconds, which shows that the matching of horizons can be done occasionally tocorrect the drift. However, instead of doing the matching again, ARCore can instead be usedwhich the results showed. The ARCore had significantly less drift which can be explained bythat it uses both a fused sensor similar to the one tested and Computer Vision technology.

6.1.5 Usability tests

The results showed a high knowability from the participants as the response was that theapplication was easy to use. This is explained by how the application is just an advancedcamera in some eyes and the visual response gives high clarity. The result further showedthat the rotational feedback was better without the magnetometer as it is not easily disturbed.The difference between the two subgroups were minimal, the only difference was that per-sons that had more technical knowledge was prone to know how the error of the headingwould work. The most bothering result from the usability tests is the high trust towards theapplication to auto-generate the horizon. How it could be so easily trusted is unknown and ismaybe an effect of a young test group. However, two participants still asked how the horizonwas generated and would trust the application more if its methods were described in a guide.

Safety is a usability attribute that the application focused on and the results have theimpression that the participants would feel safe to use this navigation application at sea. Theuser safety subattribute can be said to be met as this application has the intent of reducing thephysical safety of the user. The other aspects of this subattribute which involve saving dataon the application has not been taken into consideration in this thesis. The environmentalsafety would increase slightly as boat parts are not located on the seafloor which can also be

45

6. DISCUSSION

an increase in the next safety attribute. The third party safety is expected to rise if there wasreduction in accidents which can hurt others.

The other focused usability attribute was Subjective satisfaction, the application’s abilityto produce satisfaction in the user. The application captures the user’s attention when theysee what works and a hope to find some limitations. It further satisfies the user by reducingthe cognitive workload when out at sea as it is thought to be more easy to use.

The usability result added supporting arguments for two features. The first was that itwould be a great idea to have a button that will display red, yellow, or green depending onthe deviation from two other headings. There is additionally a supporting argument why arendered horizon line in the augmented environment is useful. However, if there was onlya manual match of the horizon it is expected to decrease the subjective satisfaction of theapplication.

6.2 Method

This section have five categories that this thesis have gone through that are discussed.

6.2.1 Pre-study

The pre-study decided the approaches that was taken and the two major approaches was thesame throughout the thesis. However, there were many different approaches to choose from.There were many methods available to extract the horizon, one being the Hough transform.

Hough transform was not viable as it can use either straight or curved edges dependingon if the mapping to Hough space is done with ellipses or rectangles [58]. Due to many curvesin the environment that the application is meant for, it was decided that it should not be used.

NCC can be used on a 2D image, so NCC could be done on the whole image, but it wouldnot yield better results as the panorama image does not include actual 360 degree video ofthe environment but uses a 3D mesh.

The best performing horizon-based orientation was thought to come from NASA withcollaboration with Stanford University [8]. The article was about how a planetary surfacenavigation was possible with horizon-based orientation. The article had an early impact onthis thesis, however it was noticed that the circumstances were different. Their vessel didnot move as fast and could take much more time for the calculation that the smartphonecould. The article used a random forest classifier for detecting the horizon which is a machinelearning algorithm that in this case classified either sky or ground. This method was deemedto take to much time and the point of the thesis was not to have the best extraction of thehorizon, it was to compare different methods that improved and assured the right orientation.

There was an additional paper that used horizon detection at sea that this paper is inspiredby, named Horizon Detection From Pseudo Spectra Images of Water Scenes [58]. This paperhas another approach on how to detect the horizon with the use of psuedo spectra images.The psuedo spectra images is a method that can eliminate distortions that are caused bynatural textures, such as grass, water, and skin. And could still be a viable pre-processingstep in this approach. However, the results that this thesis presents are satisfactory.

The last method was Canny’s edge detection [11] which was not used for the same reasonthat the Hough transform was not used. It was thought to be too many edges in the image,for instance clouds, mountains, islands, and other objects at sea. All these edges will be hardto distinguish for the actual horizon and will take much computational power to do. ThisCanny edge detection will have a result that is similar to the edge detection that was testedon the incorrect generated boat image in the result chapter.

Otsu’s method was therefore chosen, following An Improved Algorithm for Horizon De-tection Based on OTSU [25] because there is a clear distinction in the environment that theimage is expected to be taken. This can be seen with the sea and ground being of one colorintensity and the sky being in a different intensity.

46

6.2. Method

The other approach of filtering the orientation vector was filled with results pointing atthe Kalman filter. No other filter method had the small computing power and was provedto have good results. The orientation was important as the accuracy of the heading is crucialwhen displaying objects. This can be seen where only a single degree difference compared tothe true heading will result that a point 100 meters ahead will approximately be 1.65 metersoff.

The Kalman filter and Otsu’s metod were the best options to take as it suited the ap-plication’s need the best. The application required a robust and accurate image processingalgorithm. It should also be fast enough if there was a need to update the horizon repeatedlysometimes. Both the Kalman filter and Otsu’s method are old, but the focus in this thesis isif it is possible and how. If in the future there would come another horizon detection, Otsu’smethod could be replaced by it easily now that the structure and of the application is done.

6.2.2 Data

The data could be converted on various software but the ones that were chosen was be-cause they were good, based on internet searches. The conversion was also quickly tested inBlender instead of MeshLab, but with no different result. When the reduction was experi-mented with, it was noticed that it was better to reduce the mesh step wise by reducing thequality of the mesh in half in each step, compared to reducing the quality by for instance 95%in one single step.

When the mesh got larger and stretched over a bigger area, the application’s performancewas affected. It was quickly seen that the mesh did not depict the real environment well.This limitation is considered to be the largest as the area that is required to be covered at seais big. The amount of data that will be displayed is a huge problem and can only be solvedby decreasing the intensity of nodes in the mesh which will result in a poor quality meshthat is harder to match against. Another approach would be that the mesh had low qualityas standard and increase the level of detail towards the higher points of the mesh when thepanorama image was generated.

The red no-go area in this thesis was only rectangles at randomized positions. If it shouldbe done more realistically, the corners of the no-go area would be described with GeoJsoncoordinates. That would place the rectangles on specific latitude and longitude coordinates.

A bad thing with the red no-go area is that it does not take into consideration occlusion.However, if we use 3D terrain data visualized with the mesh it should be possible to hide theno-go area behind a lighthouse or mountain.

6.2.3 Measurable effects with the extra implementations

While measuring the effects it was important to have a fair competition, therefore all applica-tions were shut down and device maintenance was done. The device maintenance containedchecks that the internal memory was not used by something in the background before start-ing the tests. It was also noticed that the frame rate depended on how much the phone wasmoved. This was taken into consideration while the tests were done so the different smart-phones had similar movements on the tests.

6.2.4 Usability tests

The usability tests were motivated by the awareness that the implementations in this thesishad to do with how the user felt while having the AR experience. A goal regarding usabilityis that a good application makes the user do, not think. This application is a prototype thatcan solve a need for navigation in smartphones.

The usability tests could be improved and take place on an actual boat to have a moregenuine experience but unfortunately those kind of resources were not available. The testing

47

6. DISCUSSION

of the button that would use the boat’s previous positions with a local orientation could notbe evaluated. This function would have been great to test because the visualization will oftenbe used within a limited time when the boat is near areas that are shallow. To have a betterresult, the press of the button should be done before the boat is close to the no-go area. Thismakes the user obliged to plan ahead and therefore this function is not as useful.

The tests could furthermore be improved with more participants that stretched over abigger age span. This was noticed by how some results was thought to be different if olderpersons would have answered.

6.2.5 Source criticism

When choosing a source the mindset have been that a single piece of information shouldalways be evaluated and never trusted completely. The information displayed in the litera-ture has been accumulated by someone that may have another purpose or the informationcould have been interpreted incorrectly from the proper source. Or a mixture of both wherethe author may have interpreted it to fit the agenda that he or she had. When performancemeasures are done, biases are to be expected on how good the method actually performed.

To prevent these circumstances when reading a source, multiple sources are essential.This thesis used triangulation in order to verify that the information that was gathered hadcredibility. This is a process of using several sources to check the validity of the informa-tion that was transmitted. Secondary sources, such as websites were given more thought forevaluation of its trustworthiness.

6.3 The work in a wider context

In AR, most of the applications are still directed at proof of concept but there are predictionsfor the future that there will soon be a change in direction toward more complete applications.This can be seen in a graph in the Harvard Business Review [43] that illustrates that nearlyevery industry is investing in AR technology, and it is anticipated that each industry willdouble their investments in AR by 2020 [1]. From the review, one can understand that thebiggest marketing for AR is that it is enhancing human decision making.

AR has a lot of potential and while it continues to be experienced with, new solutions willbe discovered. The increase of usages will be seen as the years go by and it is not presump-tuous to say that the technology will be in contact lenses decades from now. But to bring thereasoning back to the present, it could be used on the streets to relive history with old imagesat the same location to see how the environment looked like years ago. It could also be usedto improve learning by doing visualizing and performing practical work.

There are many upsides with AR but what are the downsides?The largest disadvantage for AR is that it reduces the privacy of the wielder and everyone

around the person experiencing an AR. This is because the camera is always on when an ARexperience is taking place. The upside is that with a smartphone one has to use it in a waythat surrounding people can see. However, this can be even worse if a HMD is used as it ismore discrete when it is recording.

The huge amount of data that AR will generate is of big concern to keep secret. If the ARexperience is compromised in a way that it sends pictures throughout the session somewhere.The receiver of the images can know exactly what the person is looking at and when.

48

7 Conclusion

This thesis had a general purpose of understanding how good an AR experience was on asmartphone and how it could be improved without the use of external equipment. The textthat follows are answers to the four research questions.

• Can horizon-based orientation be realized so that the generated horizon matchesagainst a 3D mesh with the correct heading? If the heading from the matching is in-correct, how can the application and user know?

The horizon-based orientation was realized with good results when it was matchedagainst 3D data. The advantage of this approach was that the orientation vector that wasused after the matching could not be disturbed by magnetic forces. This thesis further illus-trated two ways to reinforce that a heading was correct in order to avoid accidents at sea.

• How does the heading of the smartphone’s magnetometer compare to a heading that isaltered with a correctly applied filter?

The Kalman filter accomplished a decrease in the outcome from an oscillating magnetwhich would otherwise disturb the signal. However a long-term magnetic disturbancewould still lose the correctness of the heading.

• How will the implementations affect the battery-life and the CPU usage of the smart-phone?

Furthermore, it was seen that the processor and battery power is somehow lacking fora longer period of visualization. However, there now exists newer smartphones with betterperformance and efficiency which can manage an AR experience even better.

• How will the application with all the approaches integrated compare against the com-mon orientation vector with magnetometer in Android Studio in usability?

The position and rotation of objects was argued to be better with this orientation vector inthe usability tests. The application proved to be easy to use and did not have any disturbingelements at all. It was also agreed by the participants of the usability tests that this navigation

49

7. CONCLUSION

would be better compared to a 2D map. This leads to the conclusion that the applicationincreased the safety and subjective satisfaction of users.

Google glasses was somehow a failure because they were a bit before their time but afuture where everyone is going around with a similar type of AR equipment may not be faraway. Before that, it is expected that the industry will use the technology first. The industry’sdriving force for AR is that the mastering of new jobs will become much easier, thereforereducing the cost of training an employee.

This thesis concludes that it is possible to improve the AR experience at sea by usingapproaches in this thesis. But watch out as the battery could be drained within 50 minutes.

7.1 Future work

The work that can be further developed in this thesis is to have another horizon detection,maybe get the horizon from a voting between Otsu’s method and the horizon that was gen-erated by the edge detection kernel. There could be more research in how the mesh can becompressed in certain ways.

A downside with the red no-go areas are that it does not take into consideration occlusion.This could maybe be solved with the use of the 3D mesh to hide no-go areas that are behinda lighthouse or mountain. Another approach could be to use data under the sea instead ofabove. By scanning the seafloor one could obtain the vessel’s position and orientation if itcould be matched against a data set of the seafloor.

50

Bibliography

[1] “A decade of digital Keeping pace with transformation Looking back to look forward”.In: Global Digital IQ R© Survey: 10th anniversary edition (2017). URL: https://www.pwc.com/sk/en/publikacie/assets/2017/pwc-digital-iq-report.pdf.

[2] D. Alonso-Ríos, A. Vázquez-García, E. Mosqueira-Rey, and V. Moret-Bonillo. “Us-ability: A Critical Analysis and a Taxonomy”. In: International Journal of Human-Computer Interaction 26.1 (Dec. 2009), pp. 53–74. ISSN: 1044-7318. DOI: 10 . 1080 /10447310903025552.

[3] ARCore Overview. 2018. URL: https://developers.google.com/ar/discover/(visited on 06/12/2018).

[4] Carol M. Barnum. Usability Testing Essentials: Ready, Set...Test! 1st. San Francisco, CA,USA: Morgan Kaufmann Publishers Inc., 2010. ISBN: 9780123785534.

[5] Båtfolket gikk oftere på grunn i 2017 | Redningsselskapet. 2018. URL: https://www.ntbinfo.no/pressemelding/batfolket-gikk-oftere-pa-grunn-i-2017?publisherId=89422%7B%5C&%7DreleaseId=16535255 (visited on 02/09/2018).

[6] Vegar Berntsen. Dødsulykker Fritidsfartøy. 2017. URL: https : / / www . sdir . no /globalassets/sjofartsdirektoratet/fartoy%5C-og-sjofolk%5C-%5C-%5C- dokumenter/fritidsbatkonferansen/2017/04%7B%5C_%7Dvegar-berntsen%7B%5C_%7Dulykkestall%5C-i%5C-aret%5C-som%5C-har%5C-gatt.pdf.

[7] Anton Biasizzo and Franc Novak. “Hardware Accelerated Compression of LIDAR DataUsing FPGA Devices”. In: Sensors 13.12 (May 2013), pp. 6405–6422. ISSN: 1424-8220.DOI: 10.3390/s130506405.

[8] X. Bouyssounouse, A. V. Nefian, A. Thomas, L. Edwards, M. Deans, and T. Fong. “Hori-zon based orientation estimation for planetary surface navigation”. In: 2016 IEEE Inter-national Conference on Image Processing (ICIP). IEEE, Sept. 2016, pp. 4368–4372. ISBN:978-1-4673-9961-6. DOI: 10.1109/ICIP.2016.7533185.

[9] Marion Buchenau and Jane Fulton Suri. “Experience prototyping”. In: Proceedings of theconference on Designing interactive systems processes, practices, methods, and techniques - DIS’00. New York, New York, USA: ACM Press, 2000, pp. 424–433. ISBN: 1581132190. DOI:10.1145/347642.347802.

51

BIBLIOGRAPHY

[10] Claire Cain Miller. Google Looks to Make Its Computer Glasses Stylish - The New YorkTimes. 2013. URL: http://www.nytimes.com/2013/02/21/technology/google-looks-to-make-its-computer-glasses-stylish.html (visitedon 01/29/2018).

[11] John Canny. “A computational approach to edge detection”. In: IEEE Transactions onPattern Analysis and Machine Intelligence (1986).

[12] Kenneth Christiansen and Alexander Shalamov. Motion Sensors Explainer. 2017. URL:https://www.w3.org/TR/2017/NOTE-motion-sensors-20170830/ (visitedon 05/20/2018).

[13] CloudCompare - Website. URL: http : / / www . danielgm . net / cc/ (visited on03/21/2018).

[14] Compare OnePlus 2 vs. Samsung Galaxy S7 vs. Samsung Galaxy S8 - GSMArena.com. URL:https://www.gsmarena.com/compare.php3?idPhone1=6902%7B%5C&%7DidPhone2=7821%7B%5C&%7DidPhone3=8161 (visited on 05/21/2018).

[15] Compare Samsung Galaxy S7 vs. Apple iPhone 6 - GSMArena.com. URL: https://www.gsmarena.com/compare.php3?idPhone1=7821%7B%5C&%7DidPhone2=6378(visited on 03/20/2018).

[16] Zhanlin Diao, Haiyang Quan, Lidong Lan, and Yifei Han. “Analysis and compensa-tion of MEMS gyroscope drift”. In: 2013 Seventh International Conference on Sensing Tech-nology (ICST). IEEE, Dec. 2013, pp. 592–596. ISBN: 978-1-4673-5222-2. DOI: 10.1109/ICSensT.2013.6727722.

[17] EBSCOhost Research Databases. 2018. URL: https : / / help . ebsco . com /interfaces/EBSCOhost (visited on 06/12/2018).

[18] A. Feldbusch, H. Sadegh-Azar, and P. Agne. “Vibration analysis using mobile devices(smartphones or tablets)”. In: Procedia Engineering 199 (2017). X International Confer-ence on Structural Dynamics, EURODYN 2017, pp. 2790–2795. ISSN: 1877-7058. DOI:https://doi.org/10.1016/j.proeng.2017.09.543.

[19] Yongxiang Gao, Andrew Kaan Balin, Roel P. A. Dullens, Julia M. Yeomans, and DirkG. A. L. Aarts. “Thermal analogue of gimbal lock in a colloidal ferromagnetic Janusrod”. In: (Nov. 2015). DOI: 10.1103/PhysRevLett.115.248301. arXiv: 1511.05517.

[20] Evgeny Gershikov, Evgeny Gershikov, Tzvika Libe, and Samuel Kosolapov. “HorizonLine Detection in Marine Images: Which Method to Choose?” In: International Journalon Advances in Intelligent Systems, vol 6 no 1 2 (2013). DOI: 10.1.1.682.4145.

[21] Michael Gervautz and Dieter Schmalstieg. “Anywhere Interfaces Using HandheldAugmented Reality”. In: Computer 45.7 (July 2012), pp. 26–31. ISSN: 0018-9162. DOI:10.1109/MC.2012.72.

[22] Torkel Glad and Lennart Ljung. Reglerteknik: grundläggande teori. Studentlitteratur AB,2006. ISBN: 9789144022758.

[23] Dan Grabham and Jones Robert. History of the iPhone 2007-2017: the journey to iPhone X| T3. 2018. URL: https://www.t3.com/features/a-brief-history-of-the-iphone (visited on 01/29/2018).

[24] A.J.H. Hii, C.E. Hann, J.G. Chase, and E.E.W. Van Houten. “Fast normalized cross cor-relation for motion tracking using basis functions”. In: Computer Methods and Programsin Biomedicine 82.2 (May 2006), pp. 144–156. ISSN: 0169-2607. DOI: 10.1016/J.CMPB.2006.02.007.

52

Bibliography

[25] Jiali Hou and Baokui Li. “An Improved Algorithm for Horizon Detection Based onOSTU”. In: 2015 7th International Conference on Intelligent Human-Machine Systems andCybernetics. IEEE, Aug. 2015, pp. 414–417. ISBN: 978-1-4799-8645-3. DOI: 10.1109/IHMSC.2015.116.

[26] R. E. Kalman. “A New Approach to Linear Filtering and Prediction Problems”. In: Jour-nal of Basic Engineering 82.1 (1960), p. 35. ISSN: 00219223. DOI: 10.1115/1.3662552.

[27] Elliott D. Kaplan and C. (Christopher J.) Hegarty. Understanding GPS : principles andapplications. Artech House, 2006, p. 703. ISBN: 9781580538947.

[28] Hannes Kaufmann and Andreas Dünser. “Summary of Usability Evaluations of an Ed-ucational Augmented Reality Application”. In: Proceedings of the 2Nd International Con-ference on Virtual Reality. ICVR’07. Beijing, China: Springer-Verlag, 2007, pp. 660–669.ISBN: 978-3-540-73334-8.

[29] Andrew P. King, Paul Aljabar, Andrew P. King, and Paul Aljabar. “Signal and ImageProcessing”. In: MATLAB Programming for Biomedical Engineers and Scientists. Elsevier,2017, pp. 221–253. ISBN: 9780128122037. DOI: 10.1016/B978-0-12-812203-7.00010-0.

[30] Lantmäteriet, Produktbeskrivning: Laserdata NH. 2018. URL: http : / / www .lantmateriet.se/globalassets/kartor-och-geografisk-information/hojddata/produktbeskrivningar/laserdat_nh.pdf.

[31] Haider O Lawend, Anuar M Muad, and Aini Hussain. “Robust Edge Detection Basedon Canny Algorithm for Noisy Images”. In: Journal of Theoretical and Applied InformationTechnology 1595 (2017). ISSN: 1817-3195.

[32] J. P. Lewis. “Fast Normalized Cross-Correlation”. In: Vision interface (1995). DOI: 10.1.1.21.6062.

[33] Maurizio Manera. “The Use of Texture Analysis in the Morpho-Functional Character-ization of Mast Cell Degranulation in Rainbow Trout (Onchorhynchus mykiss)”. In:Microscopy and Microanalysis 19.06 (Dec. 2013), pp. 1436–1444. ISSN: 1431-9276. DOI:10.1017/S1431927613013408.

[34] Eric Marchand, Hideaki Uchiyama, and Fabien Spindler. “Pose Estimation for Aug-mented Reality: A Hands-On Survey”. In: IEEE Transactions on Visualization and Com-puter Graphics 22.12 (Dec. 2016), pp. 2633–2651. ISSN: 1077-2626. DOI: 10.1109/TVCG.2015.2513408.

[35] MeshLab - Website. URL: http://www.meshlab.net/ (visited on 03/21/2018).

[36] Paul Milgram, Haruo Takemura, Akira Utsumi, and Fumio Kishino. “Augmented Re-ality: A Class of Displays on the Reality-Virtuality Continuum”. In: vol. 2351. 1994,pp. 282–292. DOI: doi:10.1117/12.197321.

[37] Elizabeth Mohn. “Microelectromechanical systems (MEMS)”. In: Salem Press Encyclope-dia of Science (2015).

[38] Jonathan Nakhla, Andrew Kobets, Rafeal De la Garza Ramos, Neil Haranhalli, YaroslavGelfand, Adam Ammar, Murray Echt, Aleka Scoco, Merritt Kinon, and Reza Yas-sari. “Use of Google Glass to Enhance Surgical Education of Neurosurgery Residents:“Proof-of-Concept” Study”. In: World Neurosurgery 98 (Feb. 2017), pp. 711–714. ISSN:18788750. DOI: 10.1016/j.wneu.2016.11.122.

[39] Thomas Olsson, Else Lagerstam, Tuula Kärkkäinen, and Kaisa Väänänen-Vainio-Mattila. “Expected user experience of mobile augmented reality services: a user studyin the context of shopping centres”. In: Personal and Ubiquitous Computing 17.2 (Feb.2013), pp. 287–304. ISSN: 1617-4909. DOI: 10.1007/s00779-011-0494-x.

53

BIBLIOGRAPHY

[40] Nobuyuki Otsu. “A Threshold Selection Method from Gray-Level Histograms”. In:IEEE Transactions on Systems, Man, and Cybernetics 9.1 (Jan. 1979), pp. 62–66. ISSN: 0018-9472. DOI: 10.1109/TSMC.1979.4310076.

[41] George R. Plitnik. “Accelerometers”. In: Salem Press Encyclopedia of Science (2013).

[42] Thomas Porathe. “3-D Nautical Charts and Safe Navigation”. In: Mälardalen UniversityPress Dissertations; 27 (Jan. 2006). ISSN: 1651-4238. URL: http://www.diva-portal.org/smash/get/diva2:120506/FULLTEXT01.pdfAn.

[43] Michael E. Porter and James E. Heppelmann. “A Manager’s Guide to Augmented Re-ality.” In: Harvard Business Review 95.6 (2017), pp. 45–57. ISSN: 00178012.

[44] Tibi Puiu. Smartphone is millions of times faster than NASA’s 1960s computers. 2017. URL:https : / / www . zmescience . com / research / technology / smartphone -power-compared-to-apollo-432/ (visited on 01/24/2018).

[45] Bjørn Rønningen .L. Hitra Frøya. 2016. URL: http : / / www . hitra - froya . no /nyheter/2016/07/30/Ny-grunnst%7B%5Co%7Dting-i-dag-13110445.ece(visited on 02/09/2018).

[46] David Royster. “Global positioning system (GPS)”. In: Salem Press Encyclopedia of Science(2013).

[47] Qian Shan, Thomas E Doyle, Reza Samavi, and Mona Al-Rei. “Augmented Real-ity Based Brain Tumor 3D Visualization”. In: Procedia Computer Science 113.00 (2017),pp. 400–407. DOI: 10.1016/j.procs.2017.08.356.

[48] John. Shawe-Taylor and Nello. Cristianini. Kernel methods for pattern analysis. CambridgeUniversity Press, 2004, p. 462. ISBN: 9780511210600.

[49] Richard Sheposh. Gyroscope. 2016. (Visited on 02/19/2018).

[50] Manjit Singh Sidhu, Singh Sidhu, and Jee Geak Ying. “Experiencing new Learning, In-teraction and Visualization Process using Augmented Reality Technology”. In: TEMJournal Journal 6.2 (2017), pp. 222–231. DOI: 10.18421/TEM62-05.

[51] Mona Singh and Munindar P. Singh. “Augmented Reality Interfaces”. In: IEEE InternetComputing 17.6 (Nov. 2013), pp. 66–70. ISSN: 1089-7801. DOI: 10.1109/MIC.2013.107.

[52] Adam Sinicki. Are smartphones still getting faster? Does Moore’s law still apply? | An-droidAuthority. 2017. URL: https://www.androidauthority.com/smartphone-speed-moores-law-801776/ (visited on 01/23/2018).

[53] Aaron Smith. U.S. Views of Technology and the Future. Tech. rep. Pew Research Center,2014. URL: http://www.pewinternet.org/2014/04/17/us- views- of-technology-and-the-future/2/.

[54] Michael C. Stevens, Wei Lu, Changhong Wang, Stephen J. Redmond, and Nigel H.Lovell. “A Kalman filter to estimate altitude change during a fall”. In: 2016 38th AnnualInternational Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).IEEE, Aug. 2016, pp. 5889–5892. ISBN: 978-1-4577-0220-4. DOI: 10.1109/EMBC.2016.7592068.

[55] Janine Ungvarsky. “Magnetometer”. In: Salem Press Encyclopedia of Science (2016).

[56] John (John A.) Vince. Quaternions for computer graphics. Springer, 2011, p. 140. ISBN:9780857297600.

[57] D. Wagner and D. Schmalstieg. “First steps towards handheld augmented reality”. In:Seventh IEEE International Symposium on Wearable Computers, 2003. Proceedings. IEEE,2003, pp. 127–135. ISBN: 0-7695-2034-0. DOI: 10.1109/ISWC.2003.1241402.

54

Bibliography

[58] Rahul Walia and Raymond A Jarvis. “Horizon detection from pseudo spectra images ofwater scenes”. In: 2010 IEEE Conference on Cybernetics and Intelligent Systems. IEEE, June2010, pp. 138–144. ISBN: 978-1-4244-6499-9. DOI: 10.1109/ICCIS.2010.5518566.

[59] What is a LAS dataset? - ArcGIS for Desktop. URL: http://desktop.arcgis.com/en/arcmap/10.3/manage-data/las-dataset/what-is-a-las-dataset-.htm(visited on 01/31/2018).

[60] Andrew Williams. OnePlus 2 vs Samsung Galaxy S6: Android face-off | Trusted Reviews.2015. URL: http://www.trustedreviews.com/opinion/oneplus- 2- vs-samsung-galaxy-s6-2928336 (visited on 05/18/2018).

[61] Seanglidet Yean, Bu Sung Lee, Chai Kiat Yeo, and Chan Hua Vun. “Algorithm for 3Dorientation estimation based on Kalman Filter and Gradient Descent”. In: 2016 IEEE 7thAnnual Information Technology, Electronics and Mobile Communication Conference (IEM-CON). IEEE, Oct. 2016, pp. 1–6. ISBN: 978-1-5090-0996-1. DOI: 10.1109/IEMCON.2016.7746263.

[62] Paul Zarchan, Howard Musoff, and American Institute of Aeronautics and Astronau-tics. Fundamentals of Kalman filtering : a practical approach, p. 876. ISBN: 9781624102769.

55

A Appendix A

A.1 Kalman filter

In more technical terms, a general Kalman filter starts at t = 0 and t ´ 1 = 0 in t|t´1 withan initial state X0|0 and an initial uncertainty which is expressed by a Gaussian covariancematrix P0|0 [62]:

X0|0 =

x0y0x0y0

, P0|0 =

L 0 0 00 L 0 00 0 L 00 0 0 L

This example has two values x and y where x0 is the starting value and x0 is the differencebetween the current value and the previous value which is often large for the first step. Thesame applies for y0 and y0. L is the uncertainty, known as the Gaussian sigma or the standarddeviation which is a number between 0 and 1. The L is spread diagonally because x and y arenot correlated.

Prediction

This step in the Kalman filter is to predict the value of the next state and the process noisewhere the point is expected to be in. This is done with the estimated point ((Ftxt´1) andthe process noise (Qt). The process noise is the deviation caused by that the value is neverperfect, i.e. there is noise about the estimate.

p(xt|xt´1) = N(Ftxt´1, Qt)

Where F is the state-transition matrix, a matrix whose product with the state vector x atan initial time t0 gives x at a later time t. N is the uncertainty and Q is the process noise whichis a 2D Gaussian.

In the prediction there is an update of the mean (xt|t´1) and covariance (Pt|t´1):

57

A. APPENDIX A

xt|t´1 = Ft xt´1|t´1

Pt|t´1 = FtPt´1|t´1FTt + Qt´1

One knows the current measure and the difference in the measure, compared to the previ-ous state, therefore one can see the rotation is updated as r(t) = r(t´ 1) + v ¨ p(t´ 1). Wherer stands for rotation and v velocity.

Correction

Before the correction is done, a noisy measurement zt has been received.

zt =[

xy

]

The corrected value has an uncertainty N(zt, Rt), where Rt is the measurement noise.Later, the predicted value and the measured value are observed to calculate the predictionerror. This prediction error is used to place the corrected value. The residual y and residualcovariance S is calculated by:

yt = zt ´ xt|t´1

St = Pt|t´1 + Rt

Afterwards is the correction of the prediction and for this the Kalman gain is used. AKalman gain decide how the bias is distributed between the prediction and the actual mea-surement. The Kalman gain can be a constant but with the update below it is possible tomake the filter adaptive, meaning that the Kalman gain will alter itself according to previousreadings.

Kt =Pt|t´1

St

The resulting Kalman gain will be low if the measurements have low measurement noiseR. This will lead to that the measurements are weighted more in the correction. In contrast, ifthe Kalman gain is high, the prediction is weighted more heavily.

The last sequence is to make the correction. The correction updates the mean and is cal-culated as:

xt|t = xt|t´1 + Ktyt

With the covariance of:

Pt|t = (1´ Kt)Pt|t´1

The xt|t is now the output to the rotation vector and the updated uncertainty is Pt|t.

58

A.2. Quaternions

A.2 Quaternions

Quaternions are an extension of the real numbers and can be easier defined with an exampleof two quaternions qa and qb:

qa = a + bi + cj + dk

qb = e + f i + gj + hk

And if you multiply these two quaternions with each other, one obtains:

qa ¨ qb = ae + a f i + agj + ahk

bei + b f i2 + bgij + bhik

cej + c f ji + cgj2 + chjk

dek + d f ki + dgkj + dhk2

With the multiplications of quaternions, one can fill in Table A.1 by following the rule ofthe imaginary units below. Always starting with ijk = ´1 one can derive Table A.1 and seethat multiplication with quaternions are noncummutative.

i2 = j2 = k2 = ijk = ´1 (A.1)

An example on how ij = k can be seen below.

ijk = ´1ô ijk2 = ´k (k2 = ´1)ô´ij = ´k ô ij = k

Table A.1: Quaternion multiplication

i j ki -1 k -jj -k -1 ik j -i -1

59

Test plan Test goals:

● Result of which approach has the better orientation and how much does it improve the overall feeling.

● Get valuable information regarding how the application is considered by outsiders. For instance, trust towards the application and if it would be too hard to learn.

How to test the product:

● The application should be tested by looking towards the horizon nearby and let the user experience the different rotational feeling of the application. (The orientation is based on the orientation vector that uses only the accelerometer and gyroscope, compared to the the orientation vector that also uses the magnetometer)

The groups that will be tested can be divided into two different subgroups: Low technical knowledge and high technical knowledge. The test should be done in the field. The scenario will compromise of a light introduction where the application is explained and what the test will boil down to. Then the test person will receive a smartphone where the application is running one of the orientation vectors. The test person shall under the test walk around and look with the phone in different directions and during the test the moderator (author) will also write down important statements. After the test, the test person will answer a post-test questionnaire. The end purpose of the test is to compare the two orientation vectors and later ask them to fill in a questionnaire regarding their experience.

B Appendix B

B.1 Test plan

Questionnaire: Using this application was difficult Strongly disagree Neither Strongly agree

O________________O________________O________________O________________O

How was the rotational feedback at the start?

Very bad Moderate Very good

O________________O________________O________________O________________O

How was the rotational feedback after the switch?

Very bad Moderate Very good

O________________O________________O________________O________________O

How would you describe the difference between the two different rotational feedback? ____________________________________________________________________ To match the horizon manually was hard Strongly disagree Neither Strongly agree

O________________O________________O________________O________________O

I think that I could easier avoid the red areas with this sort of visualization, compared to a traditional 2D map Strongly disagree Neither Strongly agree

O________________O________________O________________O________________O

I think that there are too much visual information in the application, for instance, disturbing buttons and models Strongly disagree Neither Strongly agree

O________________O________________O________________O________________O

B. APPENDIX B

B.2 Questionnaire

62

Would you feel secure, i.e. trust towards this application at sea where you press a button that captures an image that will be processed. The press of the button will generate a heading from the horizon in the image. No Maybe Yes

O________________O________________O________________O________________O

Would you feel more secure if there was a square that indicated how much the visual representation could be trusted in green, yellow, and red?

No Maybe Yes

O________________O________________O________________O________________O

B.2. Questionnaire

63