Augmented Reality and Interactive Sensor Technology,

2015 Fall

Introduction of Augmented Reality and Interactive Sensor Devices

Min-Chun Hu

anita_hu@mail.ncku.edu.tw

MISLab, R65601, CSIE New Building

1

Outline

• EPSON Smart Glasses Moverio BT-200

• Kinect v2

• MYO

• Hololens

2

EPSON Smart Glasses Moverio BT-200

3

Introduction (1/2)

Epson Smart GlassesMoverio BT-2000Tiny projectors

Viewing area is transparent

88g

Front-facing camera640*480

Can insert user’s lens

Motion sensor

4

Introduction (2/2)

Epson Smart GlassesMoverio BT-2000Connect with glasses by cable

Can connect to earphone

Micro SD memory card

Wifi & Bluetooth

《MOVERIO Apps Market》No google app store

Motion sensor

5

Product specifications (1/4)

6

Product specifications (2/4)

7

Product specifications (3/4)

8

9

Product specifications (4/4)

10

Camera

Gyroscope / Accelerometer / Compass

By tapping here twice, you can activatethe mute knock function which tempo-rarily pauses play back ,hides the image, and mutes the audio. See the Referen-ceGuide for more details on enablingthe mute knock function.

Captures images and movies.The indicator flashes when t-he camera is operating.

Mute knock

Glasses

New motion sensors capture Headmovement.

By changing the width and position of thepads, you an adjust the height of the lensesto a suitable position.

Nose pads

Transparent display blends real and digitalworlds.

LCD

Viewing Tips

• Viewing TipsYou can change the size of the screen depending on the

distance from your point of view. If your of view is in the distance, it will feel as if the screen is in the distance and larger. If your point of view is close, it will feel as if the screen is close and smaller. The sense of size for the screen varies depending on the individual.

11

EPSON 3LCD Technology

• What is 3LCD Technology? Found in all Epson projectors, 3LCD is the world’s leading projection technology that

delivers unbelievably bright and natural color, amazing detail and proven reliability.

3LCD was pioneered by Epson in the 1980s and is marketed by an affiliated organization known by the same name: "3LCD ". The technology has been licensed for use by other manufacturers worldwide for the ultimate viewing experience in projectors.

12

OS

• LinuxKernel 3.0.21

• Android 4.0.4

IceCreamSandwich

13

SDK - wikitude

14

Android

iOS

Smart Glasses

Full Screen Mode

SDK - wikitude

15

SDK - Camera control function

16

SDK - Sensor switching function

17

SDK - 3D Content Display

18

SDK

• P-GPS, WiFi Assist, Location/GPS APIObtain position data through GPS location service.

Google positioning service that is only available from Google certified device.

Moverio uses P-GPS data from WiFi network.

19

Dev Environments (IDE)

20

Based on Eclipse

Dev Environments (IDE)

21

Gaming Application for Smart glass • Camera

Real-world item (Augmented Reality)

• InteractionPlay with partner

• Surrounding Environment360° Playing

22

Microsoft Kinect V2

23

Introduction (1/3)

• Kinect是由微軟開發，應用於Xbox360和Xbox One主機的周邊設備，可以利用語音指令或手勢來操作系統介面，也能捕捉全身上下的動作，是微軟研究院的研究成果之一。

• 「Kinect」為 kinetics（動力學）加上 connection（連接）兩字所自創的新詞彙。

24

Introduction (2/3)

25

Introduction (3/3)

26

Kinect一次可擷取三種東西，分別是彩色影像、3D深度影像、以及聲音訊號。首先是Kinect機身上有3顆鏡頭，中間的鏡頭是一般常見的RGB彩色攝影機，左右兩邊鏡頭則分別為紅外線發射器和紅外線CMOS攝影機所構成的3D深度感應器，Kinect主要就是靠3D深度感應器偵測玩家的動作。

SDK

• 非官方

• OpenNI 2 (Open Natural Interaction)

• 微軟官方

• Kinect for Windows SDK 2.0 (Date Published: 10/21/2014)

27

SDK

• 微軟於在 2011 年 6 月推出Kinect forWindows SDK Beta

• 可以使用 C# 與.NETFramework4.0 來進行開發。

• 內含驅動程式、豐富的原始感測數據串流程式開發介面、自然用戶介面、安裝文件以及參考範例。

• Kinect for Windows SDK 可讓使用 C++、C#或 Visual Basic

28

SDK Browser

• 官方範例程式

29

Kinect Fusion Basics-D2D Body Basics-WPF Face Tracking Basics-WPF

SDK - System Requirement

• 作業系統：Windows 8、Windows 8.1、Embedded 8 Standard

• 硬體：64bit、雙核心 3.1Ghz 以上的處理器

4GB 以上的系統記憶體

可支援的 USB 3.0 控制器（Intel 或 Renesas 晶片）

支援 DirectX 11 的顯示卡

• 開發環境：Visual Studio 2012、Visual Studio 2013

30

Kinect 4 Unreal

31

SDK - Structure

32

Dev Environments - Hardware

• 1.電腦

• 作業系統：Windows 8、Windows 8.1、Embedded 8 Standard

• 硬體：64bit、雙核心 3.1Ghz 以上的處理器4GB 以上的系統記憶體可支援的 USB 3.0 控制器（Intel 或 Renesas 晶片）支援 DirectX 11 的顯示卡USB 3.0孔 x 1

• 2. Kinect v2

• 3. USB3.0線

33

Dev Environments - Software

• Visual Studio 2012

• Kinect for Windows SDK 2.0

• OpenCV 3.0

• Openframeworks 0.8.1

34

Application

35

Face TrackingSkeleton Tracking

3D Detection

• Light coding技術理論是利用連續光對空間進行編碼，經感應器讀取編碼的光線，交由晶片運算解碼後產生具深度的圖像。

• 其關鍵是laser speckle雷射光散斑，散斑據高度隨機性，相當於將整個空間加上標記，light coding發出雷射光對測量空間進行編碼，就是指產生散斑。

• 最後透過紅外線攝影機記錄空間中的每個散斑，擷取資料後再透過晶片計算3D深度圖像。

36

Skeleton Detection

• 透過light coding技術只能獲得基本影像資料，重點還是要便是影像並轉為指令。

• 微軟會將偵測的3D圖像，轉換到骨架追蹤系統；它可以同時偵測6人，包含同時辨識2人的動作；每個人可紀錄20組細節，並利用機器學習技術來建立龐大的圖像資料庫以看動使用者的動作。

37

MYO

38

Introduction (1/4)

39

Introduction (2/4)

40

Introduction (3/4)

41

Introduction (4/4)

• Mouse mode

• Keyboard mode

• Presentation mode

• Menu

42

SDK

43

Development (1/4)

• C++

• Build environment : include / lib / dll

44

Development (2/4)

• onOrientationData() -> 4 values (quaternion)

45

Development (3/4)

• onEmgData() -> 8 values

• onGyroscopeData() -> 3 values

46

Development (4/4)

• onAccelerometerData() –> float * 3(in x, y, z)

• onPose()

• onArmSync()

• onUnlock()

• etc

47

Hololens

48

Introduction (1/5)

• 多攝像頭運算技術取得景深，立體視覺 / Stereo Vision 技術來獲取深度（depth）

49

Introduction (2/5)

• 深度感應器結構光－Kinect 一代，

TOF，time-of-flight－Kinect One

雙眼攝影鏡頭，Google Tango 以及 Leap Motion，前者配有四個攝影鏡頭，後者有兩個。

50

Introduction (3/5)

• HOLOLENS的深度感應可能HoloLens 擁有四台攝影鏡頭

從多個攝影鏡頭的圖像數據中得到真實場景中的物體離攝影鏡頭的距離。

51

Introduction (4/5)

• 精準的即時定位與地圖構建 SLAM（Simultaneous Localization And Mapping），即同步定位與建圖系統。

52

53

Introduction (5/5)

• 立體抬頭顯示技術相機鏡頭看到的圖像，即當前場景

沉浸式的真全像渲染畫面

更立體、更即時也更清楚的3D畫面呈現

54

55

Hardware of Hololens (1/3)

• Comfort

56

Hardware of Hololens (2/3)

• Sensor fusion

• Advanced optics

57

Hardware of Hololens (3/3)

• Custom holographic processing unit

• Built-in speakers

58

Application (1/4)

• Game

• Teaching

59

Application (2/4)

• Design

• Smart House

60

Application (3/4)

• Instruction

• Visualizing

61

Application (4/4)

• Space

62