Fog-Based Visual Gesture Control and ExternalStabilization for Micro-UAVs

Anandarup Mukherjee1⇤, Sudip Misra1, Nilanjan Daw2, and Debapriya Paul21Department of Computer Science and Engineering, Indian Institute of Technology Kharagpur, India

2 Department of Computer Science and Engineering, Institute of Engineering and Management, Kolkata, India

Abstract—We propose a method for a single groundcamera-based visual gesture control of a quadrotor mUAVplatform, making its �ight more responsive and adap-tive to its human controller as compared to a humancontroller using keypads or joysticks for controls. Theproposed camera-based gesture control scheme providesan average accuracy of 100% gestures detected, as com-pared to accuracies obtained using expensive Kinect-based hardware, or processing intensive CNN-based poseestimation techniques with 97.5% and 83.3% averageaccuracies, respectively. A fog-based stabilization mech-anism is additionally employed, which allows for �ight-time stabilization of the mUAV, even in the presenceof unbalanced payloads or unbalancing of the mUAVdue to minor structural damages. This allows the useof the same mUAV without the need for frequent weightreadjustments or mUAV calibration. This approach hasbeen tested in real-time, both indoors as well as outdoors.

Index Terms—mUAV controls, Gesture-based control,UAV stabilization, Fog-based stabilization.

I. I�����������An Unmanned Aerial Vehicle (UAV) is a pilot-less

�ying vehicle or platform, which depends on remotelystationed pilots or autonomous algorithms for navigationand guidance. In this paper, we restrict our work toquadrotor micro UAVs (mUAVs), which have a twin-pairof �xed pitch, counter-clockwise rotors, located at fourdistal ends of the aircraft. The use of autonomous/semi-autonomous control makes it easy to handle the multi-rotors by a wide variety of users ranging from noviceto experts. The use of autonomous platforms has appli-cations in search and rescue operations, communication,surveillance, as well as military operations. However, incertain challenging scenarios – survivor localization indamaged buildings, checking the extent of damage, thepresence of hazardous fumes in buildings or caverns –the most optimal solution is manual control. Manualcontrol can easily account for the unknown nature of the�ying environment in the path of a mUAV, in which au-tonomous algorithms fail, as these algorithms are mostlytailored for the general environment. A more robustalgorithm for mUAVs requires the inclusion of moresensors to the mUAV system. This not only increases themUAV’s payload, thereby a�ecting its maneuverability,but also reduces its �ight time due to the additionalpower requirements. The mUAVs deployed in such sce-narios are generally minuscule, supporting only bare-basic sensors to allow for their restricted payload andeasy navigation within con�ned spaces. In this paper,we present a line-of-sight based control method formUAVs using visually recorded human-re�exive gestures.

It is shown that gesture-based re�exive controls [1]provide more agility and better control as compared tojoystick-based controllers [2]. The additional inclusionof fog-based gesture detection and control generationfor the mUAV further enhances the UAV’s capabilities.The command and control method of these mUAVs play

USER IMAGING DEVICE

COMPUTING DEVICE RADIO mUAV

3 4

67

GESTURE WIRED WIRELESS

INTERFACES:

12

5

Fig. 1: A graphical work-�ow for the proposed methodan important role in various mission-speci�c tasks. Op-eration of these quadrotors in GPS denied situations –indoors, and cloudy outdoor conditions – visually guidednavigation is the only precise approach. However, inscenarios where a fast and e�cient deployment of thesemUAVs is required, dependence on external camera-guided navigation is not feasible. A visual line-of-sight(LOS) based manual control approach provides robust andprecise control in previously uncharted and hazardousterrains without proving heavy on the mUAV processingand energy. Therefore, in our work, we keep the gesture-detection interface as simple as possible by choosing avideo-based re�exive gesture detection method, whichhas a simple interface and avoids errors due to suddenor jerky movements, as is true with motion sensor basedgesture detection units.

A. System DescriptionIn our work, a quadrotor mUAV is tethered to a fog

node (here, a computer) over a 2.4GHz wireless radiolink, allowing the computer to send commands to themUAV and the mUAV sensor values to be received by thefog node. The computer is set-up with a camera whichallows a human-controller to manipulate the mUAV’s�ight based on re�exive hand-gestures to the camera.These simple visual gestures are translated to mUAVcontrol sequences after being processed by a simpleimage-based gesture detection algorithm on the fog node.The functional steps used in our work are enumeratedconcerning Fig. 1 as follows:1) A human user wearing a colored glove (Blue),

gestures in front of the imaging device. For this

work, the gestures are restricted to a few directionalmovements of the gloved hand only – Up, Down,Left, and Right.

2) The imaging device is connected to a fog processingunit, which detects human-hand re�exive gesturesusing color-tracking.

3) Control signals for the mUAV are generated basedon the detected gestures and are forwarded to atransmitter or radio module.

4) The radio module scans the presence of the mUAV,connects to it, and then forwards the control signalsto it.

5) The mUAV interprets the control signals and actu-ates its motors accordingly.

6) The mUAV’s onboard sensors periodically poll thesystem �ight parameters ( , ✓, �) and status. Theseare then forwarded to the radio.

7) The radio returns the forwarded status signals tothe processing unit, which interprets it and checksagainst the next set of control signals. Steps 3 to 7form a closed loop control system, which stabilizesthe mUAV and balances it. This protects the mUAVagainst external disturbances, as well as sudden andfalse over-actuation.

B. ContributionsThe following are the contributions of our work:1) A method for visual re�exive gesture-based con-

trol for enhancement of visual-range operationalmUAV control has been developed, which worksboth indoors (under supervised lighting conditions)as well as outdoors (under unsupervised lightingconditions).

2) A network-based stabilization method has been ap-plied, which detects and adjusts excessive imbalancedue to �ight maneuvers, or external factors likewind, or inter- nal factors like sensor noise, orstructural damage.

II. R������W����Ongoing works in the �eld of manual or semi-

autonomous UAV control include gesture-based controls.Several gesture-based UAV control approaches use thepopularly worked upon Kinect-based 3D-gesture recog-nition system for interaction with UAVs [3]. Lementecet al. [4] propose a gesture-based control system forUAVs using multiple orientation sensors. They imple-ment it in a laboratory environment. In the previouslymentioned works dependent on gesture recognition forremote control of a system, the task of utmost criticalityis the correct detection of gestures and its translation tocontrol signals. Valiallah et al. [5] implement a computer-vision based gesture recognition system to control UAVsin a multi-robot system. However, these systems eitherrequire an elaborate and costly setup or are con�ned tooperations within a very close range.

The generation of control signals for remote oper-ation is followed by stabilization of the UAV, whichmay be either sensory stabilization or stabilization ofthe system as a whole. UAV stabilization concerningits position and attitude have been extensively studied

[6]. Previous works on UAV stabilization have mostlyfocused on hardware-based stabilization. Ho�mann et al.[7] implemented a �ight control system for a quadrotorhelicopter test-bed. They used a feed-forward compensa-tion to serve aerodynamic �ight motions for the e�ectiverejection of large systematic disturbances. Their feed-forward compensation through the control loops of aSTARMAC quadrotor protects the platform from anyuncertainties in the model. In the control system of asmall helicopter, Roy et al. [8] used an inner (positioncontrol)– external (altitude control) looped structure thatstabilizes the hover �ight of a UAV in the presence ofwind-gusts. The outer loop is designed for robust back-stepping to control translational trajectory, while theinner loop employs a PID-controller for the stabilizationof the UAV altitude. Similar works by Vinaykumar [9]report better results with Kalman �lter on UAV controlsystems as compared to Alpha-Beta �lters.

Synthesis: Vis-a-vis the works mentioned above andmany more, our system provides a computationally ef-fective, low-cost, easily deployable system for gesture-based control of mUAVs which achieves excellent results— both indoors as well as outdoors. Additionally, ourapproach works equally well on stabilized mUAVs as wellas destabilized ones. Our approach does away with theneed for mUAV recalibration or load readjustment each,and every time the UAV su�ers minor damages.

III. UAV D�������

A. Flight Dynamics

Quadrotor mUAVs are generally four rotor aerial ve-hicles, each of which provides angular velocities !

i

tothe mUAV for its motion. The four rotors are groupeddiagonally into pairs of two. One pair rotates in theclockwise direction and the other pair in the anti-clockwise direction. This con�guration cancels out theoverall rotating e�ect of the rotors over the mUAV,thereby stabilizing it. The �ight of mUAVs are generallygoverned by four parameters – roll, pitch, yaw andthrust. For a linear velocity v, the drag force F

D

onthe mUAV is calculated as 1

2 ⇢CD

Av2, where ⇢ is theenvironmental mean �uid density, A is the sweep-areaof the mUAV’s rotor blades, and C

D

is a dimensionlessconstant. For an angular velocity !, rotor-blade radiusR and proportionality constant b, the torque due to thefrictional drag ⌧

D

is calculated as b!2. Assuming a pointsource for force, the torque ⌧

z

required along the z-axisis calculated as b!2 + I

M

!, where IM

is the moment ofinertia, ! is the angular acceleration of the rotor, and b isthe drag coe�cient. Simplifying and assuming a steady�ight, the total torque along yaw-axis (⌧ ) is given byb(!2

i

� !2j

+ !2k

� !2l

).Similarly, assuming any two motors i and j lying

along the roll axis, the roll torque (⌧�) is given asLk(!2

i

� !2j

), and the corresponding pitch torque ⌧✓ iscalculated as Lk(!2

k

� !2l

). In these equations L is thedistance between the center of gravity of the quadrotormUAV and any of the four rotor centers. The anti-clockwise and clockwise motions require equal thrust onthe four motors, numbered from 1 to 4. The motion

along the z-axis is achieved by the change in propellerpair velocities. The resultant angular velocity for thecorresponding motor, k = 1, 2, 3, 4, is denoted by !

k

.In case of actual motion along the xy-plane, during leftor right motion, the two of the opposite motors shouldhave the same !

k

|k 2 [1, 3], [2, 4]. However, the adjacentmotors should have varying !

k

such that the followingcondition is satis�ed:

!k

|k 2 [1, 3] , !k

|k 2 [2, 4] (1)B. Sensor Stabilization

During an mUAV’s �ight the on-sensors such as thegyroscopes providing the mUAV’s orientation have atendency to drift away from their initial calibration andneed a constant stabilizing algorithm to counter thisdrifting e�ect. The on-board mUAV sensors are stabilizedusing a derived form of Kalman �lter. This is used toestimate the parameters of interest, in our case, the on-board derived sensor values – , ✓, �. The Kalman �ltersneed only the previous and current state values to predictthe correction parameters for future states. Consideringthe yaw axis, the state estimate for yaw ( ) is denotedby (k |k). Thus, (k + 1|k) is the estimation parameterfor the immediate future step (k + 1) such that

(k + 1|k) = F(k) (k |k) + G(k)z(k) (2)where (k |k) is the historical prediction value. F(k)and G(k) are experimentally determined constants. Thisestimation correction parameter is then used to predictthe change in measurements needed, and is representedas z(k + 1|k) = H(k) (k + 1|k). The state estimate isagain updated during a second iteration, and is denotedby (k+1|k+1), which equals (k+1|k)+w(k+1)v(k+1),where v(k + 1) is the measurement residue. For anexpectation matrix H(k), the Kalman �lter gain is givenby w(k + 1) and is calculated as

v(k + 1) = z(k + 1) � z(k + 1|k) (3)W(k + 1) = P(k + 1)P(k + 1|k)H(k + 1)0S(k + 1)�1 (4)

The error covariance of the state estimates for the el-ements are denoted by P(k |k) for z(k), z(k � 1), andP(k + 1|k) for z(k + 1), z(k). Additionally, the error esti-mation parameter is denoted by S(k + 1). The predictionestimation P(k + 1|k) and error estimation S(k + 1) arecalculated as F(k)P(k |k)F(k)0 +Q(k), and H(k + 1)P(k +1|k)H(k +1)0 +R(k +1), respectively. Thus, the estimatedparameter value (k+1|k+1) is a measure of the amountof compensation that is mixed with the yaw readings tomake them reliable. Similarly, the estimated parametervalues for pitch (✓) and roll (�) are given by ✓(k+1|k+1)and �(k + 1|k + 1), respectively.C. UAV Stabilization

The mUAV is stabilized by using the Proportional-Integral-Derivative (PID) controller scheme. The threecorrecting terms, viz., the proportional term, the integralterm, and the derivative term are summed up to formu-late the output as

u(t) = Kp

e(t) + Ki

πt

0e(⌧)d⌧ + K

d

de(t)dt

(5)

Here, Kp

,Ki

& Kd

are constants of proportionality, ande(·) is a time-varying parameter, such that e(·) 7�! f (t).

The Laplace transform (L(s)) of the above function isKp

+ K

i

⌘ + Kd

⌘, such that ⌘ is the complex numberfrequency. The proportionality constants – K

p

,Ki

& Kd

–are tuned manually until the optimum values are reached,which stabilizes the mUAV �ight.

IV. V����� G������ D��������Gesture detection by visual means initially involves

the acquisition of a three-channel Red-Green-Blue (RGB)color-image (I 2 (m ⇥ n ⇥ 3)). A color-mask basedbackground subtraction operation is performed, and themodi�ed image is converted to a single channel gray-scale image (I 2 (m ⇥ n ⇥ 1)). Subsequently, white noiseis removed using blurring and morphological transfor-mation on the gray-scale image. A threshold operationis then performed on the gray-scale image to obtain abinary image such that I(m ⇥ n)|m, n 2 (1, 0). Subse-quently, a blue-color tracker is implemented for inferringthe direction of motion of the blue-gloved hand. Sincethe RGB color space is highly susceptible to lightingchanges, we port the image into the Hue-Saturation-Value (HSV) color space, which is much more stablethan the RGB color space. A color-mask is prepared inthe blue spectrum, which masks everything other thanblue objects. For each of the pixels in the image I(t), itsvalue is obtained and is then subtracted with the pixelscorresponding to the ones in the mask range value atthe same position, and is denoted by I( f ) = I(t) � I(m),where I( f ) is the �nal image obtained after applying thecolor mask.

A. Threshold and BinarizationThe 3-channel HSV image is converted to gray-scale

intensity image (I( f )m⇥n⇥p), which discards the hue and

saturation values. The gray-scale image is then binarizedusing an unsupervised method, which considers the 0thand the 1st order cumulative moments of the gray-scaledimage histogram. This converts the image pixels from8-bits to 1-bit binary values, optimizing both spatialand temporal requirements for future calculations. Thethreshold range for image binarization is calculated byminimizing the weighted within-class deviation (�2

w

(t)),where t is the threshold value calculated experimentally.The weighted within-class deviation is calculated as

�2w

(t) = q1(t)�21 (t) + q2(t)�2

2 (t) (6)For the probability of background class occurrence, de-noted by P

i

, q1(t) and q2(t) are given byÕ

t

i=1 P(i),and

ÕI

i=t+1 P(i), respectively. We consider the variablesµ1(t),�2

1 (t) and µ2(t),�22 (t) such that

µ1(t) =t’

i=1

iP(i)q1(t), �2

1 (t) =t’

i=1[i � µ1(t)]2

P(i)q1(t)

(7)

µ2(t) =I’

i=t+1

iP(i)q2(t), �2

2 (t) =I’

i=t+1[i � µ2(t)]2

P(i)q2(t)

(8)

The threshold, thus obtained, is applied on the imageI( f )

m⇥n⇥p , which converts it into a single-channel imageI( f )

m⇥n. This operation is mathematically represented asI( f )

m⇤n, which is similar to I( f )m⇥n⇥p ⇥ �2

w

(t). In therest of the paper, I( f )

m⇥n is denoted as I for ease ofrepresentation.

B. Image Noise Filtering

A blurring operation followed by a morphologicaltransformation of the blurred image is performed torule out the e�ect of noise in detecting gestures. Thegray-scale image I is passed through the �rst round of�ltering, which is the blurring phase. The image pixelsare convolved with a kernel ( f (x, y)), which is describedby a Gaussian Function in two-dimensions. This resultsin smoothing of edges and removal of pixel-level aber-rations. For cells (n) in the �attened image matrix, theconvolution function is mathematically represented as

f (x, y) ⇤ I[n] =1’

m=�1f [m].I[n � m] (9)

The Gaussian kernel f (x, y) is generated asI.exp(�( (x�x02)

2�2x

+(y�y02)

2�2y

)), where the amplitude isdenoted by the pixel intensity, the center as (x0, y0), andthe standard deviation is denoted as (�

x

,�y

) in the xand y axes, respectively.

The morphological transformation uses mathematicalset theory for the analysis of images. The blurred imageI( f ) is convolved with a kernel B, such that the minimalpixel value overlapped by B is assigned as the new valueI B to that anchor point such as {z 2 E |B

z

✓ I}, whereBz

is the vector z’s translation of kernel B and equals {b+z |b 2 B}, 8z 2 E . This entire transformational operationis often termed as opening of the image. Opening resultsin the removal of multi-pixel aberrations or specks causeddue to Gaussian noise. The image obtained at the end ofthis pipeline is mostly free of commonly a�icting noisein digital images.

C. Gesture Detection

Eventually, gesture detection and its subsequent recog-nition, involves the identi�cation of the target gestureobject. The gesture is computed using an oriented gra-dient signal, denoted by G(x, y, ✓

a

), for a binary imageI. The algorithm executes by anchoring a circular discat the x, y coordinates, which is divided into two semi-circular discs inclined at an angle ✓

a

to each other. Thegradient G is de�ned by �2 and is given as

�2(g, h) = 12

1’i

(g(i) � h(i))2g(i) + h(i) (10)

The cell coordinates are denoted by g(i) and h(i), where iis the distance of the center of the circular disc to a pointof interest in the image matrix. The result obtained is apossibility matrix for the position of the target object.A 2nd order Savitzky-Golay �lter is applied to enhanceand smoothen the detection zenith’s orthogonal to theangle ✓

a

. This operation increases the signal-to-noiseratio. This �ltering is adapted here to remove noise inthe probability distribution without distorting the matrixelements beyond permissible limits. This operation ismathematically abstracted as

Yj

=

+(m�1)

2’i=� (m�1)

2

Ii

yj+i,

(m + 1)2 j n � (m � 1)

2 (11)

=135 (�3y

j�2 + 12yj�1 + 17y

j

+ 12yj+1 � 3y

j+2) (12)

where I(x, y) is a set of n tuples in the gradient matrix,y is a dependent variable governed by x, and is treatedas a set of convolution coe�cients. This matrix is thencompared with the previously available gradient matrixto determine the course of movement, and eventuallydetect the directional gesture being executed by the user.

V. M����������The camera being used for detecting visual gestures

records images at a rate of 30 frames per second (fps).This rate is too fast for generating command signals,which for our case, should be at a maximum rate of10Hz, which corresponds to the mUAV’s sensor pollingrate. An approximation function takes the mode of 10consecutive frames and outputs the detected gesture asa signal to the mUAV at a rate of 3Hz, which forour implementation is su�cient to control the mUAVwithout over-whelming its sensors. The proposed methodis applied to the open-source micro-UAV platform –Crazy�ie. A regular o�-the-shelf webcam is used forgesture detection. The experimental �ight tests are done,both indoors, under controlled lighting conditions andenvironment, as well as outdoors, under natural day-timelighting and environmental e�ects – winds and externalEM noise. To gauge the e�ectiveness of our approach,we used an unbalanced mUAV, with an additional loadon the front right quadrant of the mUAV.

A. mUAV Fog-based StabilizationReal-time UAV �ights pose some serious challenges

as, if the UAV is unbalanced, the gyroscopes tend todrift away, resulting in the UAV turning turtle. To avoidsuch occurrences, a secondary fog-based stabilizationmechanism is implemented, besides the regular hardwarestabilizers. The telemetry data generated on-board themUAV is fed-back into the gesture recognition systemon the fog node. These telemetry values are used forgenerating new PID values, which are sent to the mUAV.It is a multi-step process, which begins with the gen-eration of an estimation of the PID values required to�y the mUAV according to the user’s visual gestures.The current values are used for generating an errorestimate �. For an estimated value E, and feedback valueF , this error estimate is the correction needed in the�ight stabilizers, and is given by � ,✓,� , which equals(E ,✓,� � F ,✓,�)(E ,✓,�)�1. The � obtained is used asthe correction parameter to generate the new pitch, rolland yaw values ✓

correct

= ✓f eedback

+ �✓ , �correct =�f eedback

+ �� , and correct

= f eedback

+ � . Finally,these compensated telemetry values are fed back into themUAV for a stable �ight.

B. mUAV- Gesture Control LoopThe mUAV stabilization system is forked into two main

branches – 1) stabilisation of the mUAV as whole, and2) stabilization of the on-board sensors. The sequenceof events during over-the-network stabilization of themUAV are represented in Fig. 2, and enumerated sequen-tially in the following stages:Stage-0: The mUAV initializes with its original PID con-

stant values (Kp

,Kd

,Ki

). The initialization starts the

mUAV PROCESS

KALMAN FILTER

ADAPTIVE CONTROL

GENERATOR

ERROR ESTIMATOR

GESTURE CONTROL

GENERATORRADIO LINK

RADIO LINK

1

2 3

456

0

Fig. 2: The control system loop for our proposed method

on-board data-logging process, which starts pollingall the sensors and actuators connected to the sys-tem.

Stage-1: The process generates the t

, ✓t

, �t

values,which is the initial value at time t = 0, and generally,quite unstable.

Stage-2: A Kalman �lter based hardware stabilization isapplied which generates stabilized sensor outputs.The stabilized outputs derived from the parameterscorresponding to the orientation sensor – Gyroscope– at instant k is given as , ✓, �, for the instant k+1.These state estimates are transmitted to the remotecontrol-station over a radio-link.

Stage-3: The received state estimates from the mUAVare used to estimate the error between the stateestimated value and the feedback value from themUAV. The generated error � ,✓,� is forwarded tothe next processing block.

Stage-4: A gesture-control generator generates control-signals for the mUAV in the form of , ✓, � values.These generated values from the gesture controlgenerator are referred to as

f

, ✓f

, �f

, respectively.Stage-5: The

f

, ✓f

, �f

values, along with the errorestimate � ,✓,� , are used for computing the correctedcommand signals, which are sent to the mUAV overthe radio link in the form of

c

, ✓c

, �c

.Stage-6: For t > 0, the updated

t

, ✓t

, �t

control themUAV and the whole cycle repeats from Stages 1to 6.

VI. E����������� R������We divide this section into two parts for adjudging the

– 1) e�ectiveness of gesture detection, and 2) e�ective-ness of the fog-based stabilization on the mUAV. It is tobe noted that only a part of the temporal telemetry valuesof the mUAV’s �ight parameters , ✓, � are representedfor ease of interpretation.

(a) Indoors (b) Outdoors

Fig. 3: Performance of various hand gesture detectionwithout gesture approximation.

A. E�ectiveness of Gesture Detection

Fig. 3 shows the performance of visual detection ofgestures under various environments. Speci�cally, Figs.3(a) and 3(b) depict the performance of gesture detectionalgorithms in indoor and outdoor environments, respec-tively. Fig. 3(a) shows the comparison of correct andincorrect detection instances of various gestures in anindoor environment, with stable lighting conditions. Theincorrect detections appear for a fraction of a secondand are attributed to little involuntary variations of thehuman hand concerning a comparatively faster imagecapturing device. Similar to the indoor environment, thegesture-detection method, when tested outdoors, pro-vides the statistics shown in Fig. 3(b). Unlike the indoorenvironment with controlled lighting conditions, the out-door gesture-detection proves to be more challenging.However, the �xed color-tracking of the gloved handproves bene�cial for outdoor gesture detection. Same asbefore, the approximation algorithm reduces the chanceof the erroneously detected gestures to be translated tomUAV commands.

Fig. 4: Comparison of minimum gesture detection accu-racies of the proposed scheme with other approaches.

Fig. 4 compares the e�ectiveness of the proposedgesture approximation method with the usual approach.Two additional methods are also compared with ourapproach — an expensive Kinect-based gesture detectionscheme, termed Many Parameters Restriction Algorithm(MPRA), and a computationally expensive CNN-basedpose estimation method. It is seen that the CNN-basedpose estimation [10], and the MPRA-based UAV controlscheme [11] give minimum gesture detection accuraciesof 83.3% and 97.5%, respectively. Vis-a-vis, the proposedgesture detection approach without gesture approxima-tion has a minimum correct detection accuracy of 77.4%,and with the gesture approximation scheme, which takesthe mode of the gestures detected, achieves an accuracyof 100% due to exclusion of outliers generated as a resultof chance misdetections.

(a) Uncompensated Indoors (b) Compensated Indoors

Fig. 5: E�ect of network feedback on gesture-based con-trol of mUAV under controlled indoor lighting conditions.

B. E�ectiveness of Stabilizing algorithm

Fig. 5 shows the mUAV �ight parameters duringgesture-based control in an indoor environment. The�ight tests are carried out for two gestures – left andright. Figs. 5(a) and 5(b) represent the mUAV �ight pa-rameters during left gesture as command for unstabilizedand network stabilized methods, respectively. In Fig. 5(a),it is seen that, upon detection of the control signal tomove left, which primarily changes the yaw ( ) of themUAV, the sensor values change and stay in negativeyaw for some time (20ms approx.) before returning tothe positive yaw axis. The pitch (✓) and roll (�) valueschange because of the unbalancing load attached to themUAV. However, in Fig. 5(b) it is seen that changesgradually, allowing for much smoother control of themUAV, as compared to the un-stabilized approach. Theother associated parameters, ✓ and �, show transgressionfrom their stable value, but recovers quickly due to ourimplemented network-based stabilization. It is observedin our test �ight that the mUAV sensors tend to overloadand freeze beyond a certain value (�, ✓, = ±180�).Providing control signals to the mUAV and forcing it tomove in the same direction repeatedly, in an un-stabilizedmUAV, causes it to lose control and veer-out uncontrol-lably. These sensor overloading and freezing problemsare also avoided using our network-based stabilizationapproach.

(a) Uncompensated Outdoors (b) Compensated Outdoors

Fig. 6: E�ect of network feedback on gesture-based con-trol of mUAV during uncontrolled lighting conditions inan outdoor �ying environment.

Fig. 6 shows the mUAV �ight parameters duringgesture-based control in natural lighting during outdoor�ight. Figs. 6(a) and 6(b) represent the mUAV �ightparameters during left gesture as command for the un-stabilized and fog stabilized methods, respectively. InFig. 6(a), upon receiving the control signal for turningleft, for a larger duration of time, the sensory overloadat t = 125ms. This overloading a�ects the � values,causing it to behave abruptly, which eventually results ina crash. Crossing the ±180� causes the mUAV to becomeuncontrollable, and eventually crash. It is noteworthy tomention that the mUAV sensors tend to overload, whichis attributed to the e�ect of wind on it. This causesthe mUAV to be already unbalanced, even before gettingcontrol signals from the user. However, using the fog-based stabilization, as shown in Fig. 6(b), even the e�ectsof wind do not destabilize the mUAV as is evident fromthe � and ✓ values of the mUAV’s �ight. The e�ects ofwind can be seen from the hump and valleys formedbetween t = 90 � 150ms in the � and ✓ values of Fig.6(b).

VII. C���������The work presented in this paper uses a low-cost

approach, requiring minimal set-up time to establish agesture-based control of a mUAV, which can be usedin speci�c challenging and restrictive scenarios. Besidesthe low-cost gesture detection approach, which dependson re�exive gesture-based controls to attain impeccablecommand over mUAVs, our approach of employing a fog-based stabilization mechanism, in addition to the mUAV’sonboard stabilization system, allows the mUAV to operateeven when it is structurally unbalanced or unequallyloaded. This network-based stabilization mechanism pre-vents the mUAV from �ipping-over and crashing byrestricting the user-generated commands to stay withinlimits of the normal sensor operating range to avoidoverloading of the mUAV’s sensors.

In the future, we plan on increasing the functionalitiesof the mUAVs by increasing the number of commandinggestures and employing this approach for the control ofmultiple mUAVs or swarms. Additionally, the deploymentof more advanced hardware-based on-board mUAV sta-bilizing mechanisms is being worked upon.

R���������[1] R. R. Yan, K. P. K. P. Tee, Y. Y. Chua, H. H. Li, and H. H.

Tang, “Gesture Recognition Based on Localist Attractor Networkswith Application to Robot Control [Application Notes],” IEEEComputational Intelligence Magazine, vol. 7, no. 1, pp. 64–74, Feb2012.

[2] R. K. Megalingam, V. Rangan, S. Krishnan, and A. B. E. Alinkeezhil,“IR Sensor-Based Gesture Control Wheelchair for Stroke and SCIPatients,” IEEE Sensors Journal, vol. 16, no. 17, pp. 6755–6765, Sept2016.

[3] K. Pfeil, S. L. Koh, and J. LaViola, “Exploring 3D gesture metaphorsfor interaction with unmanned aerial vehicles,” in Proceedingsof the 2013 international conference on Intelligent user interfaces.ACM, 2013, pp. 257–266.

[4] J.-C. Lementec and P. Bajcsy, “Recognition of arm gestures usingmultiple orientation sensors: gesture classi�cation,” in The 7thInternational IEEE Conference on Intelligent Transportation Systems,2004. Proceedings. IEEE, 2004, pp. 965–970.

[5] V. M. Monajjemi, J. Wawerla, R. Vaughan, and G. Mori, “HRI inthe sky: Creating and commanding teams of uavs with a vision-mediated gestural interface,” in Intelligent Robots and Systems(IROS), 2013 IEEE/RSJ International Conference on. IEEE, 2013,pp. 617–623.

[6] A. Mohamed, R. Clothier, S. Watkins, R. Sabatini, and M. Ab-dulrahim, “Fixed-wing MAV attitude stability in atmospheric tur-bulence, part 1: Suitability of conventional sensors,” Progress inAerospace Sciences, vol. 70, pp. 69–82, 2014.

[7] G. M. Ho�mann, H. Huang, S. L. Waslander, and C. J. Tomlin,“Precision �ight control for a multi-vehicle quadrotor helicoptertestbed,” Control Engineering Practice, vol. 19, no. 9, pp. 1023–1036,2011.

[8] T. K. Roy, M. Garratt, H. Pota, and H. Teimoori, “Hover �ightcontrol of a small helicopter using robust backstepping and PID,”in 10

th

World Congress on Intelligent Control and Automation(WCICA), 2012. IEEE, 2012, pp. 1688–1693.

[9] M. Vinaykumar and R. K. Jatoth, “Performance evaluation ofAlpha-Beta and Kalman �lter for object tracking,” in InternationalConference on Advanced Communication Control and ComputingTechnologies (ICACCCT). IEEE, 2014, pp. 1369–1373.

[10] J. Bolin, C. Crawford, W. Macke, J. Ho�man, S. Beckmann, andS. Sen, “Gesture-Based Control of Autonomous UAVs,” in Proceed-ings of the 16th Conference on Autonomous Agents and MultiAgentSystems. International Foundation for Autonomous Agents andMultiagent Systems, 2017, pp. 1484–1486.

[11] M. Dong, L. Cao, D.-M. Zhang, and R. Guo, “UAV �ight controllingbased on Kinect for Windows v2,” in International Congress on Im-age and Signal Processing, BioMedical Engineering and Informatics(CISP-BMEI),. IEEE, 2016, pp. 735–739.