Control Engineering Practice 10 (2002) 1309–1313

Haptic interfaces for the remote control of mobile robots

Otto J. R .oscha, Klaus Schillinga,*, Hubert Rothb

aUniversity of Applied Sciences FH Ravensburg-Weingarten, Postfach 1261, Steinbeis Transferzentrum ARS, Doggenriedstrasse,

D-88241 Weingarten, GermanybUniversity of Siegen, Holderlinstr. 3, D-57068 Siegen, Germany

Received 14 January 2002; accepted 22 May 2002

Abstract

Teleoperation of mobile robots benefits from the extension of the sensor capabilities by force feedback. Usually telerobotic

applications provide to the remote user images and navigation sensor data. With additional force feedback, not only visual but also

haptic interfaces provide inputs to the teleoperator in order to improve the remote control performance. This paper describes the

hard- and software implementation for controlling a mobile robot over the Internet by the use of a force feedback joystick.

r 2002 Elsevier Science Ltd. All rights reserved.

Keywords: Remote control; Haptic interfaces; Mobile robots; Force feedback; Sensors for mobile robots

1. Introduction

Modern telepresence systems address in addition tovisual, also auditory and haptic senses to provideinformation to a human teleoperator (Buss & Schmidt,1999). This active area of research in remote controlanalyzes the integration of multimodal sensor inputs,the compensation of time delay effects, and thesynchronization of teleoperations/autonomous controlreactions. In such multimodal teleoperation systems thehaptic interface comprises kinesthetic and tactile,possibly even temperature feed back (Lederman Susan,2001). Interesting applications of haptic interfaces havebeen realized for teleoperated robotic arms (Niemeyer &Slotine, 2002; Preusche et al., 2001).These results motivated the transfer of similar

techniques to the remote control of mobile robots,being a key research area at the University of AppliedSciences FH Ravensburg-Weingarten (Schilling, Roth,& R .osch, 2000, 2002; Schilling & Meng, 2002; Rosli,Loureiro & Caldwell, 2000). These rovers participate inseveral international remote laboratories (Schilling,

Adami, & Irwin, 2002; Schilling et al., 2000), wherestudents from worldwide partner universities can per-form experiments with hardware via Internet. Hapticinterfaces for the teleoperations of remote rovers wouldoffer interesting industrial application potential forassembly/deassembly tasks in dangerous environments,as well as in space exploration with planetary rovers(Schilling, Richter, Bernasconi, Jungius, & Garcia-Marirrodriga, 1997).

2. The MERLIN vehicle design

The Mobile Experiment Robot for Locomotion andIntelligent Navigation (MERLIN) vehicles (Schilling &Meng, 2002) have been designed and implemented foroutdoor experiments and teleoperations via Internet(Fig. 1). This wheeled vehicle is 40 cm long and ischaracterized by an electrical architecture, which sup-ports integration of a broad variety of sensors fornavigation and for characterization of the workingenvironment. In the front of this car a force sensor islocated, which measures the force, when the car pushesagainst an obstacle.In this section the hardware design is summarized (for

further details cf. Schilling & Meng, 2002), while in thefollowing section the software architecture will beaddressed.

*Corresponding author. Tel.: +49-751-551-146; fax: +49-751-48-

523.

E-mail addresses: roesch@ars.fh-weingarten.de (O. J. R .osch),

schi@ieee.org, schi@ars.fh-weingarten.de (K. Schilling), roth@rst.

e-technik.uni-siegen.de (H. Roth).

0967-0661/02/$ - see front matter r 2002 Elsevier Science Ltd. All rights reserved.

PII: S 0 9 6 7 - 0 6 6 1 ( 0 2 ) 0 0 1 5 3 - 3

2.1. The chassis

The MERLIN chassis has to accommodate a micro-controller board, sensors, batteries and the radio link.The chassis for a cross country model car, based on adifferential drive system and four wheels, was selected asthe most versatile mobility platform. The wheels have adiameter of 10 cm, providing sufficient clearance tohandle typical terrain roughness.The chassis is made of ABS-plastic, being lightweight,

hard to break, and very stiff. Thus it can carry a payloadof more than 1 kg. The stretch in front of the suspensioncan be varied, or different springs can be built into thesuspension of the car, allowing even more load to becarried. The major mass contribution is from thebatteries in order to allow driving tests in the order ofhours.

2.2. Electrical architecture and sensors

The MERLIN hosts a 80C167 microprocessormotherboard with a CAN and serial interface, support-ing interfaces to a broad spectrum of sensor configura-tions. The micro-controller also provides a pulse widthmodulated output, which is used for the drive controland steering.The sensors on-board the rover have to provide the

information to the teleoperator to characterize therover’s working environment (cf. Everett, 1995 fortypical sensor systems). The standard sensor systemfor MERLIN includes bumper, wheel encoders (hallsensors see Fig. 2), four ultrasonic ranging sensors, gyro,3D-compass and GPS.Odometry is derived from Hall sensors by detecting

the passage of magnetic markers placed on the wheel’srim. As MERLIN has a rear wheel drive, the sensors areattached to the front wheels. Two hall sensors are usedon each wheel to measure also the rotation direction ofthe wheel.

2.3. The radio link

For teleoperated mobile robots the telemetry andtelecommand data are to be transferred in a short-distance link to a nearby fixed station, which thenprovides the access to long-distance telecommunicationsystems. As this short-distance link is critical for theoverall performance, several alternative link implemen-tations on basis of Bluetooth, DECT, and wireless LANhave been analyzed (cf. Mayer, Schilling, Halme, &Harmo, 2002), before a radio package controller (RPC)was selected for this task.This RPC is an intelligent transceiver module, which

enables a radio network/link to be implemented betweendigital devices. The selected Radiometrix RPC-433-A isa low-energy-demanding device. It combines a UHFradio transceiver and a 40 kbit/s packet transfer. Areliable range indoors is around 30m, and outdoorsabout 120m. The module provides all the RF circuitsand processor functions for intensive low-level packetformatting and packet recovery, required to intercon-nect several micro-controllers in a radio network.

2.4. The components of the haptic control system

Based on measurement of forces, detected in front ofthe vehicle, the teleoperator’s haptic joystick generatesproportional forces. Thus, when the vehicle is pushingan object, the teleoperator feels the related friction forceon his joystick. In order to be able to correlate theseforces with good precision for the joystick, an Immer-sion Impulse Engine 2000 was used. Two motors arebuilt into this device to apply a torque in X and Y

directions. A PCI board connects the joystick to thecomputer.The force sensor is a two-way full bridge strain gauge

fixed in the car’s front centre (cf. Fig. 1), which ends in abumper extending over the vehicle’s width. It measuresthe bending of a 0.8mm thick steel beam. The bendinglength is approximately 47mm. The output voltageof the strain gauge is between 1 and 10mV. An

Fig. 2. Wheel with the Hall sensors.Fig. 1. The mobile robot MERLIN with the strain gauges in front

acting as force sensor.

O.J. R .osch et al. / Control Engineering Practice 10 (2002) 1309–13131310

instrumentation amplifier is used to amplify the signal toa range of 0–5V.The RPC interferes with the very small sensor signal,

therefore shielding techniques are to be applied to thewires, and the amplifier location was optimized, in orderto reduce the noise of the sensor signal.

3. Remote control of mobile robots including haptic

interfaces

The remote control of mini-rovers via the Internet isdetermined by the network performance composed of

* the radio link between the rover and the local PC and* the Internet link between the local server computerand the client computer.

Crucial issues in that context are related to

* bandwidth for transferring sensor data from therover to the teleoperator and

* latency in passing the control commands from theteleoperator to the rover.

Therefore a robust software architecture optimizingthese aspects had to be investigated.In the following, first the telecommand direction of

this distributed system is described, then the aspects ofthe sensor data transfer from the rover to theteleoperator is analyzed.

3.1. The teleoperator workstation

The joystick is the main input device for theteleoperator commands in the telematic system config-uration sketched in Fig. 3. It is connected by a PCIinterface board to the client’s computer. This enables ahigh data rate between the program on the clientcomputer (here a workstation with Red Hat Linuxoperating system) and the joystick. In order to providefast and easy access to the board, an interface in theobject-oriented programming language C++ was im-plemented.

3.2. The data link between client/server

Data transfer between the client and the servercomputer over the Internet has been realized by acommunication based on sockets, using the Internet userdatagram protocol (UDP). In contrast to the TCP/IP-Protocol, used as Internet standard, no acknowledge-ment of received data packages is provided. Thus in caseof lost packages also no retransmission is initiated. Thedata transfer reaches with UDP better real-timeperformance at the cost of a potential undetected lossof data.

Two threads were created in the main program forsending and receiving data packages independently overthe Internet. On both computers the same methods areused to send and receive data packages. In thisapplication, platform independence is not realizable,because the joystick as well as the robot are specialdevices, which can only be used with dedicated hardware(the appropriate PCI board, respectively, the MERLINvehicle).

3.3. The server computer

On the server (here a workstation with Solarisoperating system) the radio link (cf. 2.3) to the mini-rover is installed. A micro-controller preprocesses thedata received by the radio modem and writes them to aRS232 interface (serial port) of the server computer. Themaximum data transfer rate is around 8 kbit/s.A cycle to transfer one data package from the server

to the car and back takes approximately 35ms. If theserver and the client are in the same sub-net of the LAN,then nearly no time delay is noticeable, when the robotis controlled with the joystick. Typical delay periodsmeasured for the Internet data transfer to locations ondifferent continents ranged between 80 and 110ms.

3.4. The MERLIN rover on-board data processing

All sensors and actuators on the car are connected tothe analog and digital inputs of the micro-controller.

Fig. 3. The telematic system.

O.J. R .osch et al. / Control Engineering Practice 10 (2002) 1309–1313 1311

For the micro-controller on-board the rover, the ANSIC programming language provides sufficient real-timeperformance and was therefore preferred to Assemblercoding. A cross-compiler creates the hex-files for themicro-controller C167.Interrupt service routines are used for receiving and

sending data packages over the radio link and also toderive the speed of the car from the position calculationbased on hall-sensor measurements. The X- and Y-coordinate values from the joystick are transmitteddirectly to the car and these inputs are then transformedinto control commands for the drive and the steeringmotor. They are assigned to the pulse width modulatedoutput of the micro-controller, used to control the driveand steering angle.

3.5. The force feedback loop

As soon as the car hits an obstacle while drivingforward, the force sensor output is to be applied to themotors in the joystick system. The amount of force atthe joystick corresponds to the force measured in frontof the car. If the car is moving downhill withoutapplying current to the drive, a force should also be fedback, such that the operator recognizes the movement ofthe car. The force value, current of the motor, velocityof the car and driving direction are sent back to theserver.The force for the joystick in the X- and Y-directions

has to be calculated from the sensor value of the straingauge force sensor, the velocity of the car and the motorcurrent from the drive. For this task a rule-based controlis used. These rules include also environment effectsrelated to car movement without a driving motor force,or car slipping on a wet surface. In addition theteleoperator receives force feedback related to airresistance and friction, in particular at higher speeds.To measure slipping wheels, the velocity of the front

wheels has to be determined and compared to thecurrent applied to the drive. With all these measuredsensor data as inputs, a sensor data fusion algorithmcalculates the force, which will be applied to the joystick.

4. Calibration and test results

The feedback force on the joystick, presents themovement of the car to the operator, taking intoaccount information from force sensor, velocity of thecar and the current applied to the motor. The relationbetween the measured force on the car, compared to theapplied force on the joystick is displayed in Fig. 4 andthe calibration complies well with the ideally lineardependence.The correlation between the velocity of the car and

the force applied on the joystick is displayed in Fig. 5.

For moving the joystick backward, the output values tothe joystick vary between 127 (no torque applied) and255 (max. torque backward).A typical test application is to push objects to target

locations. In the scenario depicted in Fig. 6, the robotwas placed in contact with the box and began from asituation at rest. The force increases until the box beginsto move; from that moment the force starts to decrease.When the car stops, the force becomes zero.This additional sensor information is now integrated

into the context of remote experiments in engineering

Fig. 4. Force–torque relation between the car and the joystick.

Fig. 5. Velocity of the car in relation to the joystick output force.

Fig. 6. Force/velocity profile for pushing an object.

O.J. R .osch et al. / Control Engineering Practice 10 (2002) 1309–13131312

teleeducation (Overstreet & Tzes, 1999; Schilling et al.,2000, 2002). Established experiments related to pathplanning and obstacle avoidance for mobile robotsreceive interesting extensions, when the rover has tomove also an object to the given target location. Inaddition to the navigation task, strategies are to bedeveloped to keep contact with the object. In particularin combination with the inherent latencies in Internet-based teleoperations, interesting and challenging tasksrelated to remote control and combinations of auton-omy/teleoperations tasks are to be investigated.In the area of multiple cooperating rovers (cf. Fig. 7),

interesting research tasks are to be addressed, when therovers are equipped with complementary specializedsensor systems: some rovers might have sensor equip-ment with emphasis on force detection, while othersare specialized with respect to navigation/rangingdevices. Both sensor configuration types are hard toaccommodate in parallel on one robot, thus cooperationbetween the different rover types is needed in a naturalway.

5. Conclusions

Haptic interfaces complement traditional sensorinformation for telepresence systems related to theremote control of mobile robots. This paper addressedimplementation details for the hard- and softwareaspects of a haptic interface for rovers, teleoperatedvia the Internet. A robust system has been realized,providing the basis for research on effects of latencies inforce feedback control, as well as on strategies formultiple cooperating rovers.

Acknowledgements

This work has been supported within the project‘‘Verbund Virtuelles Labor’’ from the ‘‘Virtual Uni-versity’’-program of the state Baden-W .urttemberg. Theauthors also want to thank the European Union forsupporting the work in the projects IECAT (EU/UScooperation in higher education) and TEAM (EU/Canada cooperation in higher education).

References

Buss, M., & Schmidt, G. (1999). Control problems in multi-modal

telepresence systems. In P. Frank (Ed.), Advances in control (pp.

65–101). New York: Springer.

Everett, H. R. (1995). Sensors for mobile robots—theory and

application. AK Peters Ltd.

Lederman Susan, J. (2001). Designing haptic and multimodal

interfaces. A cognitive scientist’s perspective. In G. F.arber, & J.

Hoogen (Eds.), Advances in interactive multimodal telepresence

systems (pp. 71–80). Munich: Institute of Automatic Control

Engineering.

Mayer, H., Schilling, K., Halme, A., & Harmo, P. (2002). Mobile

robot control via bluetooth technology. Proceedings MSy 02

embedded systems in mechatronics, Winterthur.

Niemeyer, G., & Slotine, J.-J. E. (2002). Toward bilateral Internet

teleoperation. In K. Goldberg, & R. Siegwart (Eds.), Beyond

Webcams (pp. 193–213). Cambridge, MA: MIT Press.

Overstreet, J. M., & Tzes, A. (1999). An Internet-based real-time control

engineering laboratory. IEEE Control Systems Magazine, October

(pp. 19–34).

Preusche, C., Koeppe, R., Albu-Sch.affer, A., H.ahnle, M., Sporer, N.,

& Hirzinger, G. (2001). Design and haptic control of a 6 DoF

force-6. In G. F.arber, & J. Hoogen (Eds.), Advances in interactive

multimodal telepresence systems (pp. 99–110). Munich: Institute of

Automatic Control Engineering.

Rosli, S., Loureiro, R., & Caldwell, D. G. (2000). Haptic feedback for

VR based minimally invasive surgical (MIS) training, Vol. 2.

Fourth session, EUREL robotics conference 2000, Manchester,

UK.

Schilling, K., Adami, T. M., & Irwin, R. D. (2002). A virtual

laboratory for space systems engineering experiments. Proceedings

of the 15th IFAC symposium on automatic control in aerospace

(pp. 326–331).

Schilling, K., & Meng, Q. (2002). The MERLIN vehicles for outdoor

applications. In G. R. Gerhart, C. M. Shoemaker, & D. W. Gage

(Eds.), Unmanned ground vehicle technology. Proceedings of SPIE,

Vol. 4715, SPIE Aerosense, Bellingham.

Schilling, K., Richter, L., Bernasconi, M., Jungius, C., & Garcia-

Marirrodriga, C. (1997). Operations and control of the mobile

instrument deployment device on the surface of Mars. Control

Engineering Practice, 5, 837–844.

Schilling, K., Roth, H., & R .osch, O. J. (2000). Mechatronik-

experimente in virtuellen labors. KI-K .unstliche Intelligenz, 2, 41–46.

Schilling, K., Roth, H., & R .osch, O. (2002). Mobile mini-robots for

engineering education. Global Journal of Engineering Education, 6,

79–84.

Fig. 7. Cooperating MERLIN robots dealing with the task of pushing

the box into the corner.

O.J. R .osch et al. / Control Engineering Practice 10 (2002) 1309–1313 1313