DLoc:Deep Learning based Wireless Localization for Indoor

Deep Learning based Wireless Localization for Indoor NavigationRoshan Ayyalasomayajula, Aditya Arun, Chenfeng Wu, Sanatan

Sharma, Abhishek Sethi, Deepak Vasisht and Dinesh Bharadia

1

Mobicom 2020

https://wcsng.ucsd.edu/dloc/

Outdoor Localization

2

Outdoor Localization

2

Navigation

https://www.kvhmobileworld.kvh.com/why-gps-spoofing-is-no-threat-autonomous-

navigation/

Autonomous Driving

https://play.google.com/

Delivery Bots

https://medium.com/@miccowang/delivery-robots-as-last-mile-solution-

aebebe557ad4

Autonomous Drones

Outdoor Localization

2

Navigation

https://www.kvhmobileworld.kvh.com/why-gps-spoofing-is-no-threat-autonomous-

navigation/

Autonomous Driving

https://play.google.com/

Delivery Bots

https://medium.com/@miccowang/delivery-robots-as-last-mile-solution-

aebebe557ad4

Autonomous Drones

Outdoor Localization

2

https://www.nasa.gov/sites/default/files/gps_constellation_0.jp

g

Navigation

https://www.kvhmobileworld.kvh.com/why-gps-spoofing-is-no-threat-autonomous-

navigation/

Autonomous Driving

https://play.google.com/

Delivery Bots

https://medium.com/@miccowang/delivery-robots-as-last-mile-solution-

aebebe557ad4

Autonomous Drones

Indoor Localization

3

Indoor Localization

3

https://www.clipartkey.com/upic/4785/

Indoor Localization

3

https://www.clipartkey.com/upic/4785/ https://medium.com/@brian.moran_91776/top-10-reasons-how-indoor-navigation-helps-shopping-malls-

be6376bf3d2

Indoor Localization

3

https://www.clipartkey.com/upic/4785/ https://medium.com/@brian.moran_91776/top-10-reasons-how-indoor-navigation-helps-shopping-malls-

be6376bf3d2

https://jonnegroni.com/2015/04/15/the-humans-of-wall-e-were-probably-better-off-without-him/

Lack of Context

4

Lack of Context

4

19, 13

X-axis (m)

Y-a

xis

(m

)

Lack of Context

4

19, 13

X-axis (m)

Y-a

xis

(m

)

Lack of Context

4

19, 13

X-axis (m)

Y-a

xis

(m

)

Lack of Context

4

19, 13

X-axis (m)

Y-a

xis

(m

)

WiFi based localization

5

WiFi based localization

5

MonoLocoMobiSys'18

Chronos NSDI'16

ToneTrackMobicom'15

SpotFiSigcomm'15

ArrayTrackNSDI'13

WiFi based localization

5

MonoLocoMobiSys'18

Chronos NSDI'16

ToneTrackMobicom'15

SpotFiSigcomm'15

ArrayTrackNSDI'13

Median: few decimeters

WiFi based localization

5

MonoLocoMobiSys'18

Chronos NSDI'16

ToneTrackMobicom'15

SpotFiSigcomm'15

ArrayTrackNSDI'13

Median: few decimeters >10% of cases: few meters

Deep Learning based Wireless Localization for Indoor

NavigationDLoc and MapFind

6

Deep Learning based Wireless Localization

7

Deep Learning based Wireless Localization

Localization: Novel learning based approach to solve for the environment dependent localization.

7

Deep Learning based Wireless Localization

Localization: Novel learning based approach to solve for the environment dependent localization.

Context: Bot that collects both Visual and WiFi data.

7

Table

TablesDesk

Deep Learning based Wireless Localization

Localization: Novel learning based approach to solve for the environment dependent localization.

Context: Bot that collects both Visual and WiFi data.

Dataset: Deployed it in 8 different in a Simple and Complex Environment

7

Table

TablesDesk

Deep Learning based Wireless Localization

Localization: Novel learning based approach to solve for the environment dependent localization.

Context: Bot that collects both Visual and WiFi data.

Dataset: Deployed it in 8 different in a Simple and Complex Environment

Results: Shown a 85% improvement compared to state of the art at 90th

percentile.

7

Table

TablesDesk

Challenge: Multipath, Non-Line of Sight

8

Challenge: Multipath, Non-Line of Sight

8

r

A

P

Smartphon

e

θ

Challenge: Multipath, Non-Line of Sight

8

r

A

P

Smartphon

e

Reflecto

r

θ

Challenge: Multipath, Non-Line of Sight

8

A

P

Smartphon

e

Reflecto

r

Obstacl

e

Challenge: Multipath, Non-Line of Sight

8

A

P

Smartphon

e

Reflecto

r

Obstacl

e

Need Knowledge of Environment

Requirements to design the neural network

9

Requirements to design the neural network

9

Input Representation

Requirements to design the neural network

9

Input Representation

Output/Target Representation

Requirements to design the neural network

9

Input Representation

NetworkOutput/Target Representation

Requirements to design the neural network

9

Input Representation

NetworkOutput/Target Representation

Objective/LossFunction

Requirements to design the neural network

9

Input Representation

NetworkOutput/Target Representation

Objective/LossFunctionGradient

Input Representation: Raw CSI data

10

Input Representation: Raw CSI data

10

Maximillian Arnold et. al., SCC 2019

Michal Nowicki et. al., ICA, 2017

Xuyu Wang, et al., IEEE Access 5, 2017

Xialong Zheng, et al., IEEE/ACM Transactions on Networking, 2017

Input Representation: Raw CSI data

10

Maximillian Arnold et. al., SCC 2019

Michal Nowicki et. al., ICA, 2017

Xuyu Wang, et al., IEEE Access 5, 2017

Xialong Zheng, et al., IEEE/ACM Transactions on Networking, 2017

Complex Channel Values and AWG noise

Input Representation: Raw CSI data

10

Can we represent them as images?

Maximillian Arnold et. al., SCC 2019

Michal Nowicki et. al., ICA, 2017

Xuyu Wang, et al., IEEE Access 5, 2017

Xialong Zheng, et al., IEEE/ACM Transactions on Networking, 2017

Complex Channel Values and AWG noise

Input Representation: AoA-ToF images

11

Input Representation: AoA-ToF images

11

Input Representation: AoA-ToF images

11

Does not have context of Space and AP locations

Input Representation: XY images

AoA-ToF (Polar) to XY (cartesian)

12

Angle

of A

rriv

al,

𝝷°

Time of

Flight(m)

Input Representation: XY images

AoA-ToF (Polar) to XY (cartesian)

12

Angle

of A

rriv

al,

𝝷°

X axis

(m)

Time of

Flight(m)

Y a

xis

(m)

Polar to Cartesian

Input Representation: XY images

AoA-ToF (Polar) to XY (cartesian)

12

Angle

of A

rriv

al,

𝝷°

X axis

(m)

Time of

Flight(m)

Y a

xis

(m)

Polar to Cartesian

Context embedded Input Representation

Location Decoder Output targets

13

Location Decoder Output targets

13

Location Decoder Output targets

13

Location Decoder Output targets

13

Image-to-Image translation problem

Network Architecture

14

Network Architecture

14

1. Conv2d

[7,7,1,3]

2. Instance norm

3. ReLU

1. Conv2d

[3,3,2,1]

2. Instance norm

3. ReLU

Resnet Block* 1. ConvTranspose2d

[3,3,2,1]

2. Instance norm

3. ReLU

1. Conv2d

[7,7,1,3]

2. Instance norm

3. Sigmoid

1. Conv2d

[7,7,1,3]

2. Instance norm

3. Tanh

6 Resnet Blocks3 Resnet

Blocks

Location

DecoderEncoder

4 64

128256 256

256

12

8 64 1

Input Images

Output Location

Network Architecture

14

1. Conv2d

[7,7,1,3]

2. Instance norm

3. ReLU

1. Conv2d

[3,3,2,1]

2. Instance norm

3. ReLU

Resnet Block* 1. ConvTranspose2d

[3,3,2,1]

2. Instance norm

3. ReLU

1. Conv2d

[7,7,1,3]

2. Instance norm

3. Sigmoid

1. Conv2d

[7,7,1,3]

2. Instance norm

3. Tanh

Llocation

6 Resnet Blocks3 Resnet

Blocks

Location

DecoderEncoder

4 64

128256 256

256

12

8 64 1

Input Images

Output Location

Location Loss

Closeness in MSE sense

𝐿𝑙𝑜𝑐𝑎𝑡𝑖𝑜𝑛 = 𝐿2[𝐷𝑙𝑜𝑐𝑎𝑡𝑖𝑜𝑛𝐸 𝐻 − 𝑇]

15

Location Loss

Closeness in MSE sense

𝐿𝑙𝑜𝑐𝑎𝑡𝑖𝑜𝑛 = 𝐿2[𝐷𝑙𝑜𝑐𝑎𝑡𝑖𝑜𝑛𝐸 𝐻 − 𝑇]

Penalize multiple peaks

15

Location Loss

Closeness in MSE sense

𝐿𝑙𝑜𝑐𝑎𝑡𝑖𝑜𝑛 = 𝐿2[𝐷𝑙𝑜𝑐𝑎𝑡𝑖𝑜𝑛𝐸 𝐻 − 𝑇]

Penalize multiple peaks

𝐿𝑙𝑜𝑐𝑎𝑡𝑖𝑜𝑛 = 𝐿2 𝐷𝑙𝑜𝑐𝑎𝑡𝑖𝑜𝑛𝐸 𝐻 − 𝑇 + λ 𝐿1[𝐷𝑙𝑜𝑐𝑎𝑡𝑖𝑜𝑛𝐸 𝐻 ]

15

High 90th percentile errors: Asynchronous Clocks

16

High 90th percentile errors: Asynchronous Clocks

16

r

AP

Smartphon

e

High 90th percentile errors: Asynchronous Clocks

16

r

Δr

AP

Smartphone

ToF offset

17

ToF offset compensation

18

DLoc: Network Architecture

19

6 Resnet Blocks3 Resnet

Blocks

Location DecoderEncoder

4 64

128256 256

256

12

8 64 1

Input Images

Output Location

Offset Corrected Images

DLoc: Network Architecture

19

6 Resnet Blocks3 Resnet

Blocks

Location DecoderEncoder

4 64

128256 256

256

12

8 64 1

Input Images

Output Location

6 Resnet Blocks

Consistency Decoder (𝐷𝑐𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑐𝑦)

256

12

8 64 4

Offset Corrected Images

Insight: Single source

20

DLoc: Network Architecture

21

6 Resnet Blocks3 Resnet

Blocks

Location DecoderEncoder

4 64

128256 256

256

12

8 64 1

Input Images

Output Location

𝐿𝑙𝑜𝑐𝑎𝑡𝑖𝑜𝑛+

6 Resnet Blocks

Consistency Decoder (𝐷𝑐𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑐𝑦)

256

12

8 64 4

Offset Corrected Images

DLoc: Network Architecture

21

6 Resnet Blocks3 Resnet

Blocks

Location DecoderEncoder

4 64

128256 256

256

12

8 64 1

Input Images

Output Location

𝐿𝑙𝑜𝑐𝑎𝑡𝑖𝑜𝑛

𝐿𝑐𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑐𝑦

+

6 Resnet Blocks

Consistency Decoder (𝐷𝑐𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑐𝑦)

256

12

8 64 4

Offset Corrected Images

Offset Compensation Loss

Defines consistency across images

𝐿𝑐𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑐𝑦 =1

𝑁𝐴𝑃

𝑖=1

𝑁𝐴𝑃

𝐿2[𝐷𝑐𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑐𝑦(𝐸(𝐻)) − 𝑇𝑐𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑐𝑦]𝑖

22

Offset Compensation Loss

Defines consistency across images

𝐿𝑐𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑐𝑦 =1

𝑁𝐴𝑃

𝑖=1

𝑁𝐴𝑃

𝐿2[𝐷𝑐𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑐𝑦(𝐸(𝐻)) − 𝑇𝑐𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑐𝑦]𝑖

22

Context: MapFind

23

Context: MapFind

23

WiFi Client

LiDAR

RGB-D

Server

Kobuki

Context: MapFind

23

Table

TablesDesk

Context: MapFind

23

How much data is needed?

Table

TablesDesk

Path Planning

24

Path Planning

Maximize coverage

Minimize traversal length

24

Path Planning

Maximize coverage

Minimize traversal length

24

Context Enabled Accurate Indoor Localization

Results

25

Setup

26

Setup

26

Complex High-multipath and

NLOS environment (1500 sq.

ft.)

AP

1

AP

3

AP

2

Plasma

Screens

Plasma

Screens

AP4 -

NLOS

Setup

26

Complex High-multipath and

NLOS environment (1500 sq.

ft.)

AP

1

AP

3

AP

2

Blocka

ge

Plasma

Screens

Plasma

Screens

AP4 -

NLOS

Setup

26

Complex High-multipath and

NLOS environment (1500 sq.

ft.)

Simple LOS based

environment (500 sq. ft.)

AP

3 AP

2

AP

1

Reflect

or

AP

1

AP

3

AP

2

Blocka

ge

Plasma

Screens

Plasma

Screens

AP4 -

NLOS

DLoc Result – Complex Environment

27

0 0.4 0.8 1.2 1.6 2 2.4 2.8 3.2 3.6 4 4.4 4.8Localization Error (m)

0.10.20.30.40.50.60.70.80.9

1

CD

F

DLoc Result – Complex Environment

27

0 0.4 0.8 1.2 1.6 2 2.4 2.8 3.2 3.6 4 4.4 4.8Localization Error (m)

0.10.20.30.40.50.60.70.80.9

1

CD

F

DLoc

SpotFi

Baseline DL Model

DLoc Result – Complex Environment

27

0 0.4 0.8 1.2 1.6 2 2.4 2.8 3.2 3.6 4 4.4 4.8Localization Error (m)

0.10.20.30.40.50.60.70.80.9

1

CD

F

DLoc

SpotFi

Baseline DL Model

DLoc Result – Complex Environment

27

0 0.4 0.8 1.2 1.6 2 2.4 2.8 3.2 3.6 4 4.4 4.8Localization Error (m)

0.10.20.30.40.50.60.70.80.9

1

CD

F

DLoc

SpotFi

Baseline DL Model

DLoc Result – Complex Environment

27

0 0.4 0.8 1.2 1.6 2 2.4 2.8 3.2 3.6 4 4.4 4.8Localization Error (m)

0.10.20.30.40.50.60.70.80.9

1

CD

F

DLoc

SpotFi

Baseline DL Model

DLoc Result – Simple Environment

28

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2Localization Error (m)

00.10.20.30.40.50.60.70.80.9

1

CD

F

DLoc Result – Simple Environment

28

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2Localization Error (m)

00.10.20.30.40.50.60.70.80.9

1

CD

F

DLoc

SpotFi

Baseline DL Model

DLoc Result – Simple Environment

28

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2Localization Error (m)

00.10.20.30.40.50.60.70.80.9

1

CD

F

DLoc

SpotFi

Baseline DL Model

DLoc Result – Simple Environment

28

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2Localization Error (m)

00.10.20.30.40.50.60.70.80.9

1

CD

F

DLoc

SpotFi

Baseline DL Model

Accurate Indoor Localization

Generalization across multiple setups

29

Generalization across multiple setups

29

Setup-1 Setup-2

Setup-3 Setup-4

Generalization across multiple setups

29

Setup-1 Setup-2

Setup-3 Setup-4

Trained on

Setup

Tested on

Setup

Median Error (cm)

90th Percentile Error (cm)

DLoc SpotFi DLoc SpotFi

1,3,4 2

1,2,4 3

1,2,3 4

Generalization across multiple setups

29

Setup-1 Setup-2

Setup-3 Setup-4

Trained on

Setup

Tested on

Setup

Median Error (cm)

90th Percentile Error (cm)

DLoc SpotFi DLoc SpotFi

1,3,4 2 198 420

1,2,4 3 154 380

1,2,3 4 161 455

Generalization across multiple setups

29

Setup-1 Setup-2

Setup-3 Setup-4

Trained on

Setup

Tested on

Setup

Median Error (cm)

90th Percentile Error (cm)

DLoc SpotFi DLoc SpotFi

1,3,4 2 71 198 171 420

1,2,4 3 82 154 252 380

1,2,3 4 105 161 277 455

Open-Sourced Dataset

30

Open-Sourced Dataset

30

• Enabling Baseline comparison for all algorithms

Open-Sourced Dataset

30

• Enabling Baseline comparison for all algorithms• Pushing Indoor Localization to realization

Open-Sourced Dataset

30

• Enabling Baseline comparison for all algorithms• Pushing Indoor Localization to realization• Pushing towards a competition similar to ImageNet program

Open-Sourced Dataset

Labelled WiFi CSI data (WILD-v1)

8 different setups

4 different days

108K datapoints

2 different environments

30

https://wcsng.ucsd.edu/wild/

• Enabling Baseline comparison for all algorithms• Pushing Indoor Localization to realization• Pushing towards a competition similar to ImageNet program

Open-Sourced Dataset

Labelled WiFi CSI data (WILD-v1)

8 different setups

4 different days

108K datapoints

2 different environments

30

https://wcsng.ucsd.edu/wild/

WILD-v2 Coming Soon

• 20 different setups

• 10 different days

• 1 million datapoints

• 8 different environments

• 20 different AP locations

• Enabling Baseline comparison for all algorithms• Pushing Indoor Localization to realization• Pushing towards a competition similar to ImageNet program

Conclusion and Future Work

• Novel Deep Learning based algorithm with 85% incremental performance

compared to state-of-the-art.

• MapFind we have collected over 108k datapoints (and expanding) that is open-

sourced.

• Enabling large scale and autonomous indoor navigation

31

https://wcsng.ucsd.edu/dloc/