Exploration Robot with Stereovision

Vladislav Richterrichterv2@fit.cvut.czMiroslav Skrbek FIT, CTU in Prague

skrbek@fit.cvut.cz

Vladislav Richter, Miroslav Skrbek PESW2014 2

Outline

• Introduction

• Robot's hardware

• Control Application

• Sensor visualization

• Kinect

Vladislav Richter, Miroslav Skrbek PESW2014 3

Exploration Robot

Android

Android phone or tablet

Wi-Fi

3D image(anaglyph)

Commandsforwardbackwardrotateset_wheel_speedsayget_eye_image…

Vladislav Richter, Miroslav Skrbek PESW2014 4

Robot's Hardware

Cameras (stereo pair)

WiFi antena

Roboard RB110(1GHz PC,Linux)

Speaker(speech synthesis)

80 cm distance IR

sensor

Accelerometer, gyroscope, battery

voltage, battery current and temperature sensors inside

Audio codec withan amplifier

EyeMovement(up/down)

Vladislav Richter, Miroslav Skrbek PESW2014 5

Remote Control Application

• Directed for tablets and mobile phones

• Android based application– Earlier version Android 2.3– Latest version Android 4.4

• Motor control

• Sensor visualization

• Sound localization

• Voice output

Vladislav Richter, Miroslav Skrbek PESW2014 6

Sensor Visualization

• Accelerometer• Gyroscope• Motor speed• System temperature• Battery

– Voltage– Current

• IR distance sensors• Sound source

location

Vladislav Richter, Miroslav Skrbek PESW2014 7

Accelerometer Data Visualizations

Accelerometer data

The model follows the movement of the robot

Terrain

Vladislav Richter, Miroslav Skrbek PESW2014 8

Sound Source Localization

Bar with four microphones (two stereo pairs)

Cross correlation I

Cross correlation II

Localization

Δt1

Δt2

Vladislav Richter, Miroslav Skrbek PESW2014 9

Camera Images

3D View

Anaglyph

Cameras:

640x480, 30 fps, YUYV

Vladislav Richter, Miroslav Skrbek PESW2014 10

Nearest object detection

Left image

Right image

OpenCV based stereo correspondence

calculation

Depth map

Nearest objects are

white

Vladislav Richter, Miroslav Skrbek PESW2014 11

Motor Control

Virtual Joystick

Movementkeypad

Vladislav Richter, Miroslav Skrbek PESW2014 12

Advanced Feature – GUI Adaptation

Android

get_xml_info

XML file

Description of sensors

and commands

available on the robot

GUI adaptatio

n

Vladislav Richter, Miroslav Skrbek PESW2014 13

Kinect on the Robot

Wi-Fi

Server

UDP multi-cast streams

- color image - depth map - infrared image - skeleton tracking data - accelerometer - audio

UDP stream

- motor control - stream control

15 fps

Vladislav Richter, Miroslav Skrbek PESW2014 14

Kinect - Depth Map

Image depth is 12 bits. To view the full range, a color scheme is used.

Depth map

Vladislav Richter, Miroslav Skrbek PESW2014 15

Kinect - Color Image Head-up Display

Vladislav Richter, Miroslav Skrbek PESW2014 16

Kinect – Infrared Image

Vladislav Richter, Miroslav Skrbek PESW2014 17

Kinect - Skeleton Data

Vladislav Richter, Miroslav Skrbek PESW2014 18

Conclusion

• Four-wheeled robot hardware• Stereo camera pair for distance measurement• Reach set of sensors• Sound source location• Voice output• Android based control application

– 3D view by anaglyph– Sensor data visualization– Motor control

Vladislav Richter, Miroslav Skrbek PESW2014 19

Thank you for your attention