Kinecting the Dots: 3-Dimensional Mapping

with a Scalable Drone System

by

Nikhil Devanathan

10th grade, Kennewick High School, Kennewick, WA 99336

Nikhil Devanathan

1

Acknowledgments

I wish to acknowledge my parents, Ram Devanathan and Subha Narayanan, for helping me by

providing materials and equipment, reviewing my design, providing guidance, answering

questions, offering pointers, instructing me on data analysis, taking photographs, and

proofreading my written material.

I am indebted to Dr. David Atkinson at Pacific Northwest National Laboratory for taking the

time to serve as my project’s mentor once more and listening to my progress reports on this

project on numerous Sundays. The advice, suggestions, and review from Dr. Atkinson was

invaluable to my project and prompted me to develop many of the capabilities and systems on

the drone.

2

Contents 1. Introduction ............................................................................................................................................. 3

2. Engineering Goals and Design Specifications ...................................................................................... 5

3. Design, Development, Calibration and Testing of Drones .................................................................. 6

3.1. Hardware Selection ............................................................................................................................ 6

3.2. Initial Drone Assembly ...................................................................................................................... 8

3.3. Drone Stabilization ............................................................................................................................ 9

3.4. USBIP Integration ............................................................................................................................ 12

3.5. Kinect Integration ............................................................................................................................ 13

3.6. Division of Computation .................................................................................................................. 13

3.7. Mapping Results .............................................................................................................................. 15

.3.8. Innovation ....................................................................................................................................... 16

4. Conclusion and Outlook ....................................................................................................................... 17

5. Bibliography .......................................................................................................................................... 18

3

Abstract

The leading approach to generate 3-dimensional maps with drones using photogrammetry

requires multiple images of a singular object and significant computational power. While

photogrammetry can create 3-dimensional models with great accuracy, it is too slow to be

conducted in real-time and lacks value in augmenting autonomous drone navigation. The goal of

this engineering project is to design and develop a scalable drone system for real-time 3-

dimensional map generation and environment exploration. These goals were realized through the

use of a depth-of-field sensor and the utilization of remote processing. The drone utilized a

Raspberry Pi 4 microcomputer connected to a Microsoft Kinect sensor. The Kinect used an

infrared (IR) light projector and IR camera to generate a depth scan of its field of view. The

Raspberry Pi relayed the sensor data from the Kinect along with position and orientation data

from the drone’s Pixhawk flight controller to a remote Jetson Nano microcomputer via a 5 GHz

wireless network. The Jetson Nano is an embedded computer developed for graphics-processing-

intensive workloads. It was used to reconcile depth scans to develop a real-time 3-dimensional

map and orient connected drones. This drone system serves as a proof of concept, which can be

expanded upon. More drones can be connected to single base stations, as the only limits on the

system are the computational power of the base station and the data transfer rate of the wireless

network. Using this technology, drones could be utilized to navigate and explore hazardous

environments without exposing people to risk.

4

1. Introduction

Small autonomous unmanned aerial vehicles, also known as drones, have potential

applications in a number of civilian technologies, such as search and rescue, disaster response,

environmental monitoring, factory and structural inspections, and warehouse management

(Floreano 2015). These drones can have fixed wings, rotors, or a bio-inspired design based on

insects or birds. To perform the desired function, the drones must sense their environment

reliably and safely maneuver in both indoor and outdoor environments. The outdoors can be

challenging, because Global Positioning System (GPS) signals may not be available, the

environment may be highly unpredictable, and intermittent forces such as the wind can act on the

drone. There is growing interest in taking advantage of emerging advances in optical flow

technologies for stabilization and powerful graphical processing units to handle the

computational challenge of autonomous drone flight.

As stated above, the ability to observe the environment, analyze signals from multiple

sources, and make decisions rapidly is key to stable and autonomous drone flight. There is an

inherent tradeoff that needs to be made between analysis time, accuracy of the analysis, and

energy consumption. The computational hardware needed to support autonomous flight can be

heavy and generate considerable heat, which will be detrimental to the stability, flight time, and

performance of the drone. At the same time sending data and control signals to and from a

centralized cloud is not feasible for real-time drone control. Edge computing or decentralized

computing is rapidly gaining ground for autonomous drone operations (Callegaro 2020). There is

also the need for efficient computer vision algorithms (McGuire 2017) for obstacle avoidance in

autonomous flight. Collision and obstacle avoidance along with efficient communication is

needed to take the next step in this technology, which is autonomous drone swarms (Vásárhelyi

2018). Machine learning is also being utilized to prevent collisions in drone swarms (Schilling

2018).

Autonomous drones or swarms of such drones are being used to create detailed maps of

terrain based on images obtained from a camera. This technique, called photogrammetry,

involves fusing many images or video frames to develop a 3D map. The availability of GPUs has

made it possible to develop such a map without waiting for post-processing. Photogrammetry

5

may require multiple images of the same object from different angles and is still constrained by

available computing power.

The opportunities and challenges addressed above served as motivation for this project.

This engineering project aims to design, program, test and develop a scalable drone system for

real-time 3D map generation and environment exploration. This ambitious goal was made

feasible by the availability of the Jetson Nano to consumers in March 2019, which is a very

recent development. The Jetson Nano is a powerful embedded computer developed by NVIDIA

for power-efficient, graphics-processing-intensive workloads. It is ideally suited for computer

vision applications. It can be used to reconcile depth scans with position and orientation data to

develop a real-time 3D map and orient connected drones. The innovative idea behind this project

is to build a drone with a Raspberry Pi 4 microcomputer connected to a Microsoft Kinect sensor

that can generate a depth scan of its field of view. The Raspberry Pi can relay the sensor data

from the Kinect along with position and orientation data from the drone’s Pixhawk flight

controller to a remote Jetson Nano microcomputer via a 5 GHz wireless network. It is expected

that this project will demonstrate a proof of concept of autonomous drones, that could be used in

swarms, for the exploration of unstructured and hazardous environments.

2. Engineering Goals and Design Specifications The goals of this project were to:

● develop a drone capable of using a depth-of-field camera, GPS, and optical flow to

collect 3D map data of varying environments,

● configure a base station to remotely receive the map data and assemble a 3D map in real-

time and

● utilize drivers and background processes to create an system for scalable drone mapping.

The long-term goal was to build multiple drones that can work cooperatively in a swarm.

This is a stretch goal because it requires demonstration of non-interference between drones,

efficient mapping, and optimization of mapping from multiple simultaneous feeds.

The design specifications established the boundaries for this project. The first specification

was that the drone should be smaller than 800 mm in diameter to limit the cost, complexity, and

6

hazard. The second specification was that the drone should be equipped with an onboard

computer, depth camera, GPS module, and a downward-facing optical flow sensor. The third

specification was the need to have at least 20 minutes of flight time using an onboard battery in

the interest of practicality. The fourth specification was that the onboard computer must be able

to send depth data to a remote computer via a network and the remote computer must run

mapping and visualization software. The next two specifications were stretch goals for the

project given the available time and budget of under $500. The fifth specification was that the

drone must operate in almost any lighting condition and the sixth was the need for collision

avoidance capability.

3. Design, Development, Calibration, and Testing of Drones

3.1. Hardware Selection

The design process began with choosing the drone platform. Based on the available

literature, the combination of a drone and a powerful base station was chosen as the most cost-

effective and computationally efficient approach for this task. The Raspberry Pi was chosen as

the onboard computer for the drone, because of its small and lightweight form factor, ease of

programmability, and my prior experience with the Raspberry Pi. Based on a survey of available

drone platforms, the following components were chosen for the drone shown in Figure 1.

● 550mm glass fiber frame for low cost, strength, and low weight

● 4-cell, 6500 mAh battery to provide 20 minutes of flight time based on estimated mass

● Raspberry Pi 4B as the onboard computer

● Pixhawk Flight controller

● 4S 2212 high-torque brushless motors for low speed, high efficiency, and drone stability

● High-flexibility nylon propellers were chosen as they were safer than the stiffer carbon

fiber propellers but offered sufficient crash resistance. They could also be easily replaced.

● 360° LIDAR and Kinect for map data

● Optical flow and GPS for position data

7

Figure 1. Photograph of the drone with Kinect sensor in the front, LIDAR, and GPS.

The base station shown in Figure 2, had a small Jetson Nano computer with a Wi-Fi module to

act as the center of the network to which multiple drones could connect and share LIDAR and

Kinect devices with the base station. This station was designed to remain stationary and use data

from the shared devices along with position data from each drone to build a map. The Jetson

Nano is designed to easily handle a small number of drones but could lose responsiveness with

increased feeds. Despite having 128 graphics cores, the Jetson Nano only has 4 ARM 54 CPU

cores. Network transfer speeds can be reduced as the distance between drones and base station,

the number of drones, and the number of obstacles in the environment increase. The network

must handle a steady stream of data from one depth-of-field camera and one 360° LIDAR

module per drone.

8

Figure 2. The Jetson Nano with a Wi-Fi module. The USB ports show the scale of the device.

3.2. Initial Drone Assembly

A custom battery bay was designed and dremelled out of a polycarbonate sheet to

accommodate the battery specifically chosen for this drone. The largest possible battery for this

drone was determined to be 4S 6500mAh based on calculations of lift and mass. The battery

needs to accommodate power drain from motors, Raspberry Pi, flight controller, sensor systems,

depth-of-field, and optical flow cameras, and power lost as heat. With these power requirements,

the battery was required to power the drone for a flight time of 20 minutes with the drone

carrying all onboard systems and maneuvering. The components used to assemble the drone are

shown in Figure 3.

9

Figure 3. The components used to assemble the drone.

3.3. Drone Stabilization

To stabilize the drone, a GPS module was used to provide reliable position data for the

drone in outdoor environments. An optical flow module was used alongside a downward-facing

rangefinder to provide additional position data. This setup is shown in Figure 4. The Inertia

Measurement Units (IMUs) in the Pixhawk flight controller estimated the drone’s displacement

based on its acceleration. An Extended Kalman Filter (EKF) was used on the Pixhawk to

reconcile sensor readings while filtering out sensor noise or extreme readings to determine the

necessary correction. Using an EKF-assisted flight mode, the drone was able to move accurately

and hold position when not receiving instructions.

10

Figure 4. Downward-facing LIDAR (left), optical flow-camera (middle), and SONAR (right).

Proportional, integral derivative (PID) loops, shown in Figure 5, were needed to stabilize

the drone. In the PID loop, sensors were used to determine the deviation (error) from an ideal

state. The rate of change of error (derivative) and the sum of accumulated errors (integral) were

then determined. These three values are multiplied by individual constants and added together,

and the result was used as a correction factor to stabilize the altitude of the drone. Both the

gyroscope and the accelerometer were integrated with the flight controller. The gyroscope

measured the rotational attitude and rotational velocity of the drone, while the accelerometer

measured the rotational acceleration of the drone. The flight controller polled the values from the

sensors in real-time at +100Hz. The values were used to stabilize the drone through the flight

controller without prompt or command from the Raspberry Pi. My flight controller was initially

stabilized using a remote controller. To stabilize my flight controller, I had to determine PID

values to use in the accelerometer and gyroscope-based correction. I used a potentiometer on my

remote controller to slowly increase the correction factor applied to my drone to determine a

UVO, or unique value of oscillation which is the correction factor that causes the drone to begin

overcorrecting, which is visible as oscillation. Using a tool called Optune, I was able to

determine PID constants for use with my drone.

11

Figure 5. Illustration of a PID loop.

The optical flow module was crucial to the stabilization of the drone. The IMU in the

flight controller and the gyroscope agreed perfectly as shown in Figure 6. The optical flow also

agreed well with both the IMU and gyroscope readings. As a result of this accurate optical flow,

the drone held its position and stabilized itself.

Figure 6. Graphs showing good agreement between IMU (green), optical flow sensor gyroscope

(blue), and optical flow camera (red) estimates of drone position during a flight test.

12

3.4. USBIP Integration

In order for the Jetson Nano to develop a map of the environment explored by the drone,

the Jetson Nano needed access to the data from the Kinect. Initially, a Python script was tested

for transferring Kinect data from the Raspberry Pi onboard the drone to which the Kinect was

wired to the Jetson Nano, but due to the high volume of data produced by the Kinect, the script

lagged severely. To reliably transfer Kinect data, a more computationally efficient approach had

to be considered.

The Linux USB over IP (USBIP) kernel module allows for the sharing of a USB device

over a network connection. The computer physically connected to the USB device can run a

USBIP server. The server will allow computers running a USBIP client over the same network to

connect to the USB device as if it was physically connected. This approach was advantageous for

numerous reasons. Foremost, because USBIP is a kernel module, it runs efficiently as a

subprocess, allowing for the sharing of devices like the Kinect which produce large volumes of

data. USBIP also supports Linux natively, so a Linux subsystem or virtualized operating system

was unnecessary to use the module.

Unfortunately, the operating system published by NVidia for the Jetson Nano did not

come with USBIP built-in. In order to use USBIP, the kernel sources had to be recompiled with

USBIP included for the Jetson Nano. Once USBIP was installed on both the Raspberry Pi and

the Jetson Nano, operating system services were configured to utilize the module to share the

Kinect USB device. Two USBIP host daemons were set up on the Raspberry Pi to continuously

share the Kinect camera and audio device over the network. Two USBIP client services were

configured on the Jetson Nano to connect to the Raspberry Pi and share the Kinect devices.

Once USBIP was configured, it proved to be an optimal solution. The module allowed for

a reliable stream of Kinect data to be streamed from the Raspberry Pi to the Jetson Nano at

speeds approaching 24 Hz. The Jetson Nano recognized the Kinect as if it was connected

physically, and the Jetson Nano was able to successfully capture Kinect depth scans over USBIP.

13

3.5. Kinect Integration

The Kinect is an inexpensive ($40) depth of field camera. It has significant online support

with open source drivers like freenect and OpenNI. It is also natively supported by 3D mapping

platforms like rtbmap natively. The Kinect module was extracted from its original casing to

reduce size and weight as shown in Figure 7. The module was attached to the drone using a

custom 3D-printed part. This part was designed with TinkerCAD (free and online) and printed

using a 3D printer at home. The Kinect module consists of a camera, an infrared (IR) camera,

and an IR projector. It can take normal pictures and videos with its camera, and it uses the IR

projector and camera duo to determine the depth of its field of view.

Figure 7. The Kinect device (left) and internals mounted on the drone (right).

3.6. Division of Computation

The computational tasks were divided between the Raspberry Pi on the drone and Jetson

Nano off the drone (in the base station). Any computation the Raspberry Pi does drains the drone

battery and reduces flight time. Thus, it makes sense to move some of the computation off the

drone. The Jetson Nano has more memory and stronger graphics processing than the Raspberry

Pi 4. Due to these differences, it was ideal to perform minimal computation on the Raspberry Pi

and instead use it to relay and manage data from the flight controller, onboard sensors, and

Jetson Nano. The Jetson Nano served as a stationary base station to process map data from all

14

drones. In the future, as the number of drones increases, the Raspberry Pi can remain unchanged,

and only the base station would need an upgrade. This is an systemic advantage because data is

not held on the drone. Even if the drone crashed, data will be preserved. The Raspberry Pi that

may be damaged in a crash is also significantly less expensive than the Jetson Nano making the

drone more expendable. The remote processing set up is shown in Figure 8.

Figure 8. Schematic diagram illustrating the remote processing set up.

A system of background processes handled the transmission of Kinect and LIDAR data

from the Raspberry Pi to the Jetson Nano. A USB-over-IP (USBIP) server was run on each

Raspberry Pi connected to the Jetson Nano while USBIP clients ran on the Jetson Nano. The

Kinect and LIDAR were shared at the USB device level, which minimized computational

demand on the Raspberry Pi. The operating system level background processes ran more

efficiently than scripts. Though background processes were used to share devices between the

Raspberry Pi and Jetson Nano, position and orientation data were sent via Python scripts. The

Python socket interface was used to stream position and orientation data from the Raspberry Pi

to the Jetson Nano. The network scripts were parallelized to increase efficiency and reliability.

The Jetson Nano interfaced with the Kinect using the freenect and OpenNI Linux drivers.

Mapping was done using the rtbmap module which utilized odometer data from the Kinect along

with the depth images to generate a map of the drone’s environment. These modules were built

on top of the Robot Operating System (ROS), a system of modules and workspaces which

provide many low-level functionalities for polling data from and controlling robots.

15

3.7. Mapping Results

The Robot Operating System (ROS), a software framework for robotics development,

was used to assemble the maps onboard the Jetson Nano. The OpenNI Kinect driver was built in

a ROS workspace and used in conjunction with rtabmap to provide data which was visualized

using the ROS-visualization tool (rviz). Initially, the Kinect was tested indoors in a well-lit room.

Because the Kinect consists of both a normal camera and an IR depth camera, it was able to

create a true-color 3D depth scan of the room.

Figure 9. A real-time 3D map capture taken in a well-lit room. The capture shows a window

behind a TV to the left and an adjacent wall and part of a couch to the right.

The Kinect also performed extraordinarily well in a dark outside setting. It was able to

produce a 3D scan with negligible difficulty. This can be attributed to the IR project on the

Kinect which provides the IR depth scanner all of the light it needs. The normal camera on the

Kinect lacked the necessary light to accurately capture the environment, so the scan could not be

16

rendered in true color.

Figure 10. A 3D scan of an outdoor setting in dark, nighttime lighting conditions. To the bottom

left is a picnic bench, and at the top is a line of shrubs.

Ultimately, while the Kinect scans were somewhat lacking in resolution, they were

rendered in real-time with great computational efficiency. The depth images provide the base

station with all of the information it needs to functionally map the environment and navigate a

drone through it.

.3.8. Innovation

There are several innovative aspects of this project. First, the computational power, data,

and costly components reside outside the drone in the base station. This design reduces the

computational requirements and power needed for the drone. It also ensures that data is protected

if the drone is lost. In addition, it makes the drone inexpensive and easily replaceable. This

feature is attractive for setting up drone swarms, where there is a risk of collision and loss of

drones. The hub-and-spoke network structure facilitates the efficient addition of new drones.

Second, the use of a depth-of-field camera enables the collection of 3D data of an object without

17

multiple passes. Depth images can be reconciled into a 3D map with significantly less

computational power than alternative methods (photogrammetry).

4. Conclusion and Outlook

In this project, I designed, prototyped, programmed and tested multiple drones that used a

Raspberry Pi4 computer, a Kinect, LIDAR and GPS for autonomous stabilization and 3D

mapping in indoor and outdoor environments. The drone demonstrated autonomous position

holding and stability both indoors in a garage and outdoors under windy conditions. The

computational workload on the drone and consequently power consumption was reduced by

transferring much of the computation to a powerful Jetson Nano computer in a base station. The

base station collected data by wireless communication with drones. This work shows that

although depth-of-field scans are of lower resolution than photogrammetry scans, they are

significantly more computationally efficient and faster. The innovation in removing the core

computer of the system from the drones reduced the cost of each drone and removed crucial map

data from the drones. It was also evident that efficient background processes and services can

serve as an alternative to computationally costly scripts in some cases

There are several possible future enhancements for my drone system. Multiple drones can

be added to the system to test the limits of the Jetson Nano as the base station. Further software

and hardware enhancements will help achieve collision avoidance and obstacle avoidance with

multiple drones in flight in a small area like a garage. Efficient drone-to-drone communication

will enable the drones to divide and conquer tasks such as mapping, search and rescue or

inspection of hazardous environments. The loss of a drone in that scenario will not be

catastrophic as the data and much of the computational power will reside in the base station.

18

5. Bibliography

Callegaro, Davide, Sabur Baidya, and Marco Levorato. "Dynamic distributed computing for

infrastructure-assisted autonomous UAVs." In IEEE International Conference on

Communications. IEEE ICC. 2020.

Floreano, Dario, and Robert J. Wood. "Science, technology and the future of small autonomous

drones." Nature 521, no. 7553 (2015): 460-466.

Hambling, David. “Stitching a drone’s view of the world into 3D maps as it flies.” New Scientist

July 20, (2016).

McGuire, Kimberly, Guido De Croon, Christophe De Wagter, Karl Tuyls, and Hilbert Kappen.

"Efficient optical flow and stereo vision for velocity estimation and obstacle avoidance

on an autonomous pocket drone." IEEE Robotics and Automation Letters 2, no. 2 (2017):

1070-1076.

Schilling, Fabian, Julien Lecoeur, Fabrizio Schiano, and Dario Floreano. "Learning vision-based

cohesive flight in drone swarms." arXiv preprint arXiv:1809.00543 (2018).

Vásárhelyi, Gábor, Csaba Virágh, Gergő Somorjai, Tamás Nepusz, Agoston E. Eiben, and

Tamás Vicsek. "Optimized flocking of autonomous drones in confined environments."

Science Robotics 3, no. 20 (2018): eaat3536.