CHAPTER 9

Understanding haptics by evolvingmechatronic systems

Gerald E. Loeb{,*, George A. Tsianos{, Jeremy A. Fishel{,Nicholas Wettels{ and Stefan Schaal{,{

{ Department of Biomedical Engineering, University of Southern California, Los Angeles, California, USA{ Department of Computer Science, University of Southern California, Los Angeles, California, USA

Abstract: Haptics can be defined as the characterization and identification of objects by voluntaryexploration and somatosensory feedback. It requires multimodal sensing, motor dexterity, and highlevels of cognitive integration with prior experience and fundamental concepts of self versus externalworld. Humans have unique haptic capabilities that enable tool use. Experimental animals have muchpoorer capabilities that are difficult to train and even more difficult to study because they involverapid, subtle, and variable movements. Robots can now be constructed with biomimetic sensing anddexterity, so they may provide a suitable platform on which to test theories of haptics. Robots willneed to embody such theories if they are ever going to realize the long-standing dream of workingalongside humans using the same tools and objects.

Keywords: tactile sensing; haptics; robots; perception; cognition; learning; sensorimotor control;dexterity.

Introduction

The term “haptics” has been used variously bypsychologists to mean the science of the sense oftouch, by computer technologists for tactile feed-back from an electronic device, and by aestheticphilosophers to denote affective responses tomanual exploration. We shall use it here to

denote the intersection of these usages as “theidentification of properties of objects via voluntaryexploration and somatosensory feedback.” Theentity doing the identification can be human ormachine, the properties of the object can be physi-cal or aesthetic, somatosensory includes both tac-tile and proprioceptive modalities from any bodypart (or their machine equivalents), and the inter-action must involve active movement.

Natural scientists usually seek to understand aphenomenon by reducing it to its componentparts. Psychologists have catalogued the

*Corresponding author.Tel.: þ213-821-5311E-mail: gloeb@usc.edu

A. M. Green, C. E. Chapman, J. F. Kalaska and F. Lepore (Eds.)Progress in Brain Research, Vol. 192ISSN: 0079-6123Copyright ! 2011 Elsevier B.V. All rights reserved.

129DOI: 10.1016/B978-0-444-53355-5.00009-9

Author's personal copy

exploratory behaviors made by human adults andinfants when confronted by a novel object andneuroscientists have identified and characterizedthe receptors activated by such behaviors (Jonesand Lederman, 2006). Nevertheless, we arelacking what David Marr called a “theory of com-putation” (Marr, 1982). In recent years, fairlystrong theories of computation have been devel-oped for many aspects of perceptual and even cog-nitive behavior. These usually derive fromsystematic recordings of neural activity from vari-ous brain regions of highly trained animalsperforming tasks and correlations of such activitywith various parameters of the performance. Thisstrategy is difficult to apply to haptics because fewanimals have anything like the manual dexterityof humans. For haptic behaviors that are feasiblefor animals, the movements and forces betweenthe digits and objects are difficult to capture andthe steps in the complex sequences tend to be vari-able and uncontrollable by the experimentalist.The difficulties of a reductionist approach tohaptics can be appreciated by comparison withlocomotion, another class of sensorimotorbehaviors. The mechanics and control of walkingin animals are now fairly well understood becausereproducible behavior is easily generated andmeasured in both intact and reduced preparations(McCrea and Rybak, 2008). Some of the principleshave been successfully incorporated intorobots that perform reasonably well (Buchli andIjspeert, 2004).

The process of building machines that incorpo-rate principles of operation of living organisms iscalled biomimetic design. We propose that a the-ory of haptics can be developed and tested bystarting with biomimetic robots and attemptingto emulate the behaviors and capabilities ofhuman subjects. Recent advances in mechatronics(engineered systems combining mechanical com-ponents with electronic control) have made it pos-sible to emulate the mechanical behavior ofbiological hands and limbs (Delcomyn, 2007).Sensors built into the actuators and mechanicallinkages provide the equivalent of proprioceptive

information. Impedance control of the actuatorscan be used to emulate the natural complianceof biological limbs (Hogan, 1984; Pratt et al.,1996). Multimodal tactile information can beprovided by a new sensor array that mimics themechanical properties and robustness of a humanfingertip (Wettels et al., 2008). It may be possibleand perhaps even necessary to emulate the hier-archical structure of the biological nervous sys-tem, in which the brain formulates high-levelstrategies and tactics and the spinal cordcoordinates the action of muscles and regulatesthe interactions with external objects (Raphaelet al., 2010). If the controller of a mechatronicequivalent of a biological system could use a the-ory of computation to achieve humanlike hapticperformance, this would be suggestive that thebrain may be using a similar theory of computa-tion. It should also be possible to apply suchknowledge to the design of neural prosthetic sys-tems to restore dexterity to patients with paralysisor amputation of their arms (Fig. 1).

Components of haptic behavior

The motor strategies that humans use to explore,characterize, and identify objects have beencatalogued by psychologists (Jones andLederman, 2006; Lederman and Klatzky, 1987).These strategies seem reasonable in view ofthe various somatosensory modalities that havebeen identified neurophysiologically, but they fallfar short of a theory of computation for hapticbehavior and they provide no insight into howthey developed in the first place.

Development of internal representationsof objects

We shall start with the assumption that thedeveloping nervous system has little or nopreconceived information about itself or the otherentities that comprise its world and must instead

130

Author's personal copy

self-organize its internal representations bydetecting patterns of coherence in the incomingsensory information and its own outputs. This isthe starting assumption for the unsupervised neu-ral networks that were modeled in the early daysof artificial intelligence (Haykin, 1999; Hebb,1949). It is now clear that a certain amount of sig-nal processing is genetically hardwired into thephysical structure of primary sensory receptorsand their associated local circuitry, which presum-ably predisposes the perceptual nervous system tostart with specific salient features. Nevertheless,

the development of internal representations isknown to depend on actual experience, oftenassociated with “critical periods” of neonataldevelopment and plasticity (Crair et al., 1998;Hubel and Wiesel, 1970). We shall here assumethat the objects to be represented centrally havean identity that is static, but this identity must bebuilt up from highly dynamic interactions and itincludes information required to predict dynamicbehavior of the object.

Identification of self

The first thing that a self-organizing brain is likelyto recognize is that some, but not all, of the sen-sory information coming back is contingent onmotor signals that the brain sends out, initiallyrandomly and eventually purposefully. The con-tingent sensory data are associated with the exis-tence of a self; sensory data that are constant orinconsistently modified by efferent signals con-stitutes evidence of external entities, which aredefined later. The representation of self is essen-tially a mapping between efferent signals and pro-prioceptive and visual afferent signals thatrepresents the set of movements that the organ-ism can make.

Efferent and afferent feedback In order to recog-nize the correlations between efferent and affer-ent signals, both must be available as inputs to aperceptual center. This explains why efferentprojections are accompanied by recurrentprojections that are organized similarly to affer-ent sources.

Internal reference frames Much has been writtenabout the coordinate frames used by various partsof the brain to represent the relationships of theself to the external world. Some of them appearto be inherent in the structure of sense organs(e.g., retinotopic and cutaneous maps), othersare likely to arise as the CNS detects robust cor-relations among the senses (e.g., extrapersonal

Prostheticlimb

NMES

BIONic spindles Neurogram

Paralyzedlimb

coordinationalgorithms

Motorcommandinterface

Dexterous neural prosthetic systems

Sensoryperceptioninterface

Controlsystem

Patient

Signalprocessing

ActuatorsMechanical

linkage

Tactilesensors

Fig. 1. Restoration of haptic function in patients with paralysisor amputation of the upper extremity requires bothbidirectional interfaces with sensory and motor signals and acontrol system that can integrate those signals with commandsignals from the patient to provide coordination and rapidadjustments. Neuromuscular electrical stimulation (NMES)interfaces in the paralyzed limbs could utilize injectable,wireless microdevices called BIONs (Loeb et al., 2001; Sachsand Loeb, 2007).

131

Author's personal copy

space as fusion of visual information with gazedirection and proprioceptive feedback of pos-ture), and some are purely mathematicalcreations designed to account for psychophysicaldata (e.g., shoulder-centered kinesthesia;Soechting and Flanders, 1989).

One likely outcome of adaptive neuralnetworks is that they can extract more-or-lessorthogonal representations of whatever principalcomponents are present in their input data, butthis begs the question of what the extractedcoordinates might represent. If a given datasetcan be adequately represented by a particularorthogonal coordinate frame (an eigenvector), itcan also be equally well represented by othercoordinate frames that are rotations of the firstin eigenspace. At the early stages of sensoryinformation processing, the coordinate frames ofthe neural representations may bear a strongresemblance to the physical structure of the senseorgans from which the input signals are derived,but they will become increasingly abstract athigher levels where inputs are multimodal. Themotor cortex integrates highly abstractedrepresentations from many areas of associationcortex plus relatively low-level direct input fromproprioceptors. Thus, it is not surprising that sim-ple attempts to correlate its output signals or itstopography to simple physical coordinates basedon either sensory input (extrapersonal space) ormotor output (muscles) have been frustrating(Churchland and Shenoy, 2007).

The notion of first identifying “self” suggeststhat the internal coordinates of many, if not most,cortical areas will be different from each otherbut will reflect a combination of both afferentand efferent information. Such hybrid coordinateframes do not have simple physical or mathemat-ical analogs and may vary from subject to subject(see below), so we have no guesses to correlatewith neural activity. If they exist, we must firstidentify the neural processes and hierarchies thatlead to their emergence.

Definition of surfaces

Once the organism has a repertoire of movementsin extrapersonal space, it is in a position to recog-nize visual signals that do not change as a result ofsuch movement (except occlusions by self) butthat give rise to somatosensory and perhaps audi-tory signals when the movement reaches the placeof those visual signals. Thus, the first externalityto be represented in the brain seems likely to bethe notion of surfaces that obstruct otherwise freelimb movements.

Definition of rigid objects

Once the organism knows how to make contactwith surfaces in extrapersonal space, it is in aposition to recognize that contact between itselfand a subset of those surfaces can result inchanges in subsequent visual appearance andsomatosensory feedback from those surfaces. Thisis the definition of a movable object as opposed toa fixed surface. By systematically reorienting itsview of a rigid object, the brain can associate allthe different patterns of sensory feedback thatcan be produced by that object, forming a fusedpercept. By systematically grasping and exploringthe object, the brain can include in this perceptinformation about weight, inertia, friction, ther-mal properties, etc.

Definition of deformable objects

Once the organism can recognize and manipulaterigid objects, it can appreciate that certain typesof object have an even richer set of visual andsomatosensory feedback depending on which ofa limited set of states they occupy. Those states,in turn, may depend on the history of interactionswith the object (e.g., hingelike motion) or the for-ces being applied (e.g., elastic deformation).

132

Author's personal copy

Definition of tools

Once the organism can recognize and manipulateany object, it is in a position to cause interactionsbetween objects and surfaces, for example, usingone object to hit and cause noise and/or move-ment of another object or surface. This is the def-inition of rudimentary tool use.

Definition of materials

An organism that can recognize objects that havea closed set of possible appearances can then rec-ognize interactions that cause an object to changeits appearance permanently. This is the definitionof breaking or otherwise permanently changingan object, which renders it a material that canbe used to create other objects (e.g., flaking stonetools, molding clay pots). Infants commonly han-dle known objects aggressively, as if they are fas-cinated by whether or not they fall into thecategory of things that can be broken.

Integrating exteroceptive senses

In the developmental sequence hypothesizedabove, visual information is only one of manytypes of sensory and efference copy informationfrom which the structure of the self and the exter-nal world must be defined. By contrast, mostroboticists start with sophisticated machine visionand use that as the basis for motor planning andsensory fusion. Infants are relatively slow todevelop visual acuity, relying instead on tactileinformation from both fingers and mouth (Gib-son, 1988). Adults who are blind from birthdevelop sophisticated representations of objectsand dexterity in manipulating and characterizingthem (Marks, 1983; Smitsman andSchellingerhout, 2000). What they do lack is asense of how the visual appearance of objectschanges with point of view (Heller et al., 1996).This is actually a problem rather than an attribute

of vision, and much of machine vision and pre-sumably human vision is concerned with over-coming this problem so as to maintain averidically fused representation of individualobjects. Thus, it seems both easier and moreappropriate for these high-level visual capabilitiesto emerge spontaneously based on the perceivedutility of visual information rather than as a pre-ordained organizing principle for the brain.

The sounds made by objects and theirinteractions are rarely considered in robotics butthey appear to be of great interest to infants.Sound contains valuable information that maybe difficult to extract from other sensorymodalities and it tends to be integrated centrallywith other sensory modalities including touch(Bresciani and Ernst, 2007). Most obviously,sound provides precise temporal information,which can be useful for understanding kineticinteractions during touch. It also provides a valu-able hint about the mutability of objects—articu-lated objects often change shape silently or withrepeatable sounds whereas breaking an object isoften accompanied by a nonrecurring sound.Attributes such as hollowness may be impossibleto perceive in any other way. The vibrotactilespectrum (up to "800 Hz) substantially overlapsthe acoustic spectrum (20–20,000 Hz). Both mayuse temporospatial cross-correlation to extractuseful information (Johansson and Flanagan,2009; Loeb et al., 1983), but neither biologicalmechanism is well understood. Nevertheless,acoustic information is easy to acquire electroni-cally and to preprocess in a way that conveys atleast some of its biological saliency.

Testing new entities against internalrepresentations

Once the organism has defined all major classesof entities in its world and has a reasonablelibrary of sensorimotor patterns associated withpreviously encountered entities in each class, thenits daily existence and continued developmentdepend on two complementary capabilities:

133

Author's personal copy

# Recognizing when a newly encountered entity issufficiently similar to a previously known entityto treat them as identical.

# Recognizing when a newly encountered entity issufficiently dissimilar to all previously knownentities to warrant creation of a new item inthe library.

These decisions can be seen as forced-choiceoutcomes of Bayesian decision making, in whichthe probability of picking one or the other usuallydepends on fragmentary data and a complex setof prior information about the probability of aknown entity being present, the importance ofnot making a mistake, and the cost of obtainingadditional information. This cost has at least twodimensions, reflecting the energetic cost of theexploratory movement and its execution time.For many of the haptic discrimination tasks facinga hunter-gatherer, execution time will be criti-cal—foraging time is limited and prey whoseown motor behaviors cannot be identified and

countered tends to escape. Bayes’ theorem is use-ful for decision making once the new data are inhand (Kording, 2007), but it must be extendedto account for the decision about which data topursue based on the relative costs and theexpected discriminative value of the new data.This depends on having an accurate and immedi-ately accessible representation of all the possibleassociations between exploratory movements andthe sensory data that they are likely to yield (Fig. 2).This suggests that the internal representation of theproperties of objects in the brain is not in the usualcanonical coordinates that we define in physics(e.g., mass, rotational inertia, hardness, friction,etc.), but rather in the associational coordinates oflearned exploratory movements and the criticalraw sensory signals associated with them. Forexample, the representation of mass of a givenobject is the association between the parametersof a hefting exploratory movement and the propri-oceptive and tactile data resulting from hefting thatobject. In fact, it is possible that the goal of hefting

Fig. 2. Haptic exploration to identify an object can be conceptualized as an interactive sequence of hypothesis testing designed tocollect specifically those data Di that are most likely to shift the prior probabilities P(A), P(B) sufficiently to conclude that the objectmatches an object that produces similar data when subjected to those exploratory strategies.

134

Author's personal copy

is to obtain similar sensory feedback from allobjects, in which case the discriminative informa-tion is actually in the parameters of the heftingmovement itself (e.g., amount of motor effortrequired to produce a given acceleration of theobject). Similarly, the shape of an object obtainedfrom contour following may be reflected in thesequence of exploratory movements required tokeep the fingertip following a feature and therebygenerating constant tactile feedback.The choice of exploratory movement also

involves the decision to stop exploration and set-tle on whatever is the currently most likely identi-fication. There are many circumstances whenerroneous “lumping” with previous experience is“good enough,” which accounts for a wide rangeof illusions, beloved of psychophysicists,magicians, and trial lawyers. When the braininsists that all is not familiar even after exhaustiveexploration, the problem becomes one of creatinga new percept that is associated with and basedupon the closest known match. This leads to botha parsimonious extension of the coordinates ofthe existing internal representations andassociations that facilitate access to previouslydeveloped strategies for handling similar but dif-ferent entities. It is important to recognize thatthis incremental learning about the world is likelyto give rise to internal representations of the sameentity that are quite different among individuals,because they depend on the sequence of all otherentities to which each individual was exposedduring its prior experience.

Hierarchical system of sensorimotor control

The nervous system may start out “tabula rasa”but it is not free from genetically determinedstructures that evolved because they promoteand accelerate the functionality that must developontogenetically. Each part of the nervous systemtends to start with certain types of computationalelements (neurons) whose various properties are

specified. The initial interconnections amongthem appear to follow certain general rules andthe rules for modifying that connectivity as aresult of experience appear to be quite specific.In sensory systems, the sequence of embryologi-cal development gives rise to topographicgradients that are related to physical coherencein the signals that will later be experienced (e.g.,retinotopic, cochleotopic, and somatotopic maps),so it has been natural (but not necessarily useful;see below) to interpret the higher levels of per-ceptual processing in those same coordinateframes. Organizing the motor system fromperiphery upward is appealing because theperipheral elements effectively define the compu-tational problem that must be solved at higherlevels of motor planning, but the major informa-tion flow is necessarily in the opposite directionfrom sensory systems. This discussion considerswhat is known about the structure andrelationships from the bottom up but from theperspective of motor function, which is top-down.

Design of the spinal regulator

Industrial robots use hierarchical control that isusually divided into a movement planning leveland a servocontrol level. For the last 50 years,sensorimotor neurophysiologists have tried tointerpret the brain and spinal cord as analogousto those respective engineering subsystems, withlittle success. As details of the anatomical cir-cuitry and physiological signals have beenelucidated, it has become clear that they are notconsistent with such an interpretation. The cir-cuitry in the spinal cord is much more complexand integrative than an engineering servocontroland the signals in the brain correlate only looselywith many different aspects of the output behav-ior. The spinal cord may function more likeanother, more sophisticated engineered system—a programmable regulator (He et al., 1991; Loebet al., 1990). Recently, models of

135

Author's personal copy

neuromusculoskeletal systems based on theknown circuitry of the spinal cord have beenshown to have highly useful emergent propertiesthat were completely unexpected (Raphaelet al., 2010). The biological regulator consists ofcomponent circuits with specific combinations ofinput and output signals that have evolved andremained stable over many millions of years ofvertebrate evolution. The brain can “program”the gains of these elements via relatively simpledescending command signals so that the highlyphasic and coordinated sequences of muscleactivations required to perform each task are gen-erated by the nonlinear combinations of thesecommands and ongoing sensory feedback.Despite the large number of gains that must beset, simple gradient descent algorithms convergerapidly and inevitably to stable solutions that per-form similarly to human subjects doing the sametasks. The set of interneurons defines a highdimensional space that is rich in “good enough”local minima. When started from random gains,the controller quickly “discovers” a nearby localminimum defined by the cost function used fortraining. Because the output is generated bycircuits that include rich combinations of sensoryfeedback, the solutions automatically handlewhatever types of noise, perturbations, or com-plex loads were included in the training set.

Since the pioneering work of Nicolas Bernsteinin the 1930s, both motor psychologists and robot-icists have been concerned with resolving the prob-lem of “redundancy,” in which there are morearticulations or actuators than necessary to per-form a given task (Bernstein, 1967). The program-mable regulator turns this problem around,greatly expanding the redundancy at the interneu-ronal level to ensure that a single system can gener-ate a virtually unlimited repertoire of desirablebehaviors depending on the goals at hand. Evolu-tionary success is defined by rapidly finding goodenough solutions rather thanmeticulously comput-ing globally optimal solutions. We intend to dem-onstrate that robots controlled by similarprogrammable regulators can achieve similarly

robust functionality. We can compare theirstrengths and weaknesses with state-of-the-artcompliant controllers based on more traditionaltrajectory planning and servocontrol.

The problemwith applying a regulator to a robotis that we do not have any blueprints. The pro-perties of most mechatronic systems are so differ-ent from biological musculoskeletal systems thatthe patterns of interneurons known in the spinalcord cannot be applied to the robot. Sowewill haveto recapitulate the evolution of the spinal regulatorby using a genetic algorithm (Ijspeert, 2001).Genetic algorithms create and evolve structuresincrementally, keeping components that improveperformance and rejecting those that do not. In thiscase, the performance criterion is the speed andsecurity with which a simple gradient descent con-troller can learn to perform a repertoire of diversetasks using each new generation of regulator.Because the mechanical dynamics and sensors ofthe robot are actually substantially simpler tomodel than their biological counterparts, it shouldbe feasible to evolve a fairly sophisticated regulatorusing a fast PC.

Learning to perform tasks

The redundancy problem is not really a problem forthe nervous system because, unlike most roboticcontrollers, it does not try to compute analytical sol-utions to new problems. Rather the brain startswith one of the motor programs that it alreadyknows (or a random output if an infant) and gradu-ally refines that program until it gets what it wants.Given a reasonable repertoire of motor skills, mostof the problem posed by a new task is the percep-tual one of recognizing similarities to somethingalready in the repertoire. The process of refine-ment by trial-and-error is made efficient and reli-able by the many, good enough local minima ofthe regulator. This is distinctly different fromservocontrollers, whose gains and input commandsaremore critical and are better set by analytical sol-utions for optimal control (Todorov and Jordan,

136

Author's personal copy

2002).Given a suitable regulator and a suitable costfunction defining a task, it should be straightfor-ward to train a model of cortical motor control. Toachieve performance similar to a human, the costfunction should probably include both kinematicterms related to accuracy and energetic termsrelated to effort. Interestingly, the spinal-like regu-lator seems predisposed to identifying motorstrategies that result in minimal coactivation ofactuators (Raphael et al., 2010), similar to thepatterns seen in well-learned tasks performed bybiological subjects.

Cortical consolidation

Surprisingly, after adequate performance hasbeen achieved in a motor task, the controller inthe brain continues to introduce large, apparentlyrandom fluctuations in its behavior (Churchlandet al., 2010). We can surmise that the next levelof controller upward in the hierarchy (e.g., pre-motor cortex) is performing its own trial-and-error learning to minimize some cost function.What might that cost be? From the well-describedplasticity of cortical representations, we can sur-mise that the computational resources of the cor-tex are finite and perhaps a limiting factor in therepertoire and resolution of motor tasks that canbe learned. When good enough performance isfirst achieved, the number of cortical mot-oneurons contributing to the net gains in the spi-nal regulator is likely to be much larger thannecessary. This is because those net gains arethe result of diverse excitatory and inhibitoryfunctions controlled directly by each corticospinalmotoneuron as well as indirectly via extrapyrami-dal subsystems that are also ultimately under thecontrol of the cerebral cortex. By graduallyadjusting the commands to the motor cortex, thepremotor cortex should be able to reduce over-lapping and conflicting activity. Shrinking the cor-tical neural representation of a given task frees upcomputational machinery to learn another task.The robustness of the spinal regulator, with its

many good enough local minima, makes it rela-tively easy and safe to employ trial-and-errorlearning in all stages.

Cortical representations

Because there are many good enough solutions tocommon motor tasks, the repertoire of solutionsthat a given brain has at its disposal is likely tobe rather different from another brain that haslearned to achieve similar nominal performance.If a previously learned task is complicated bynew loads, perturbations, or accuracyrequirements, those different starting strategiesmay have different utilities and consequences.Psychophysicists studying motor learning usuallycreate learning curves by averaging the data frommany different subjects with similar starting skills,but these smooth, averaged curves do not reflectthe apparently random, stepwise processes thatactually occur in the individual subjects (Gallistelet al., 2004). They also fail to account for sportcoaching practices that are required to forceathletes out of idiosyncratic, well learned but ulti-mately suboptimal habits.

By utilizing trial-and-error learning of goodenoughbehaviors insteadof analytical optimal con-trol, biological systems have accepted a Faustianbargain that roboticists have traditionally rejected:biological systems can rapidly achieve acceptableand stable performance with noisy componentsbut are unlikely ever to achieve globally optimalperformance. One consequence of that bargain isthat the signals that can be recorded from behavingbiological systems can never be fully understood bycorrelating them with the predictions of analyticalengineering tools (e.g., Churchland and Shenoy,2007). The deeper one dives from the unavoidablerequirements of physics into the more arbitrarydetails of neural representations, themore the databecome colored by the unique and essentially ran-domexperiences of the individual subject. This fun-damental limit to experimental reductionismprovides the rationale for our alternative approach

137

Author's personal copy

based on simulation and synthesis. We choosehaptics over other motor behaviors that havealready been studied reductionistically becausehaptics requires rich sensorimotor integration andit has little reductionist baggage to discard.

Development and testing of Haptic robots

Mechatronic platform

A great variety of robotic arms and hands forboth research and industrial applications havebeen developed over many years (Table 1).The more anthropomorphic systems of late tendto be more expensive and more fragile, bothbecause of their complexity and their limited pro-duction for research. It is not clear what level ofverisimilitude is necessary to generalize lessonslearned from a haptic robot to a haptic humanor vice versa.

Biomimetic tactile sensing

A variety of technologies have been used in tac-tile sensors (Table 2), but commercially availabletactile sensors tend to be limited to relativelycoarse arrays of normal-force sensors based oncompression of elastic materials. In fact, most ofthe commercially available hands listed in Table 1are not supplied with any tactile sensing. Manytechnologies are difficult to apply to the curved,deformable “skin” that facilitates grip and feware able to resist damage in the electromechani-cally hostile environments in which hands areoften used (moisture, grit, sharp edges, etc.).One promising new candidate is the BioTacÒ, abiomimetically designed, multimodal array thatprovides most of the dynamic range of human tac-tile sensing for location, magnitude, and vectordirection of contact forces, microvibrationsassociated with slip and textures, and thermal fluxresulting from contact with objects that differ in

thermal effusivity (Lin et al., 2009; Raphaelet al., 2010; Wettels et al., 2008, 2009) (Fig. 3).

Compliant control algorithms

Many of the exploratory movements that underliehuman haptics involve force or impedance controlrather than position control, meaning that the tra-jectory of the limb results from a dynamic interac-tion between the robot and objects that itencounters. Most robots use highly geared DCand stepping motors that generate whatevertorque is required to produce precise movements.When external forces are applied to them, theybehave in a stiff rather than a compliant manner.In order to behave compliantly, these motorsmust be fitted with torque or force sensors thatactively modify the position commands to whichthey are responding. If these control loops andmotors are sufficiently fast, the system canachieve a reasonable approximation of the com-pliant control that arises naturally from biologicalmuscles, which generate forces that depend onposition and velocity of movement. Recently,such control principles have been organized intoDynamic Movement Primitives (DMP; Schaal,2006), a system of nonlinear differential equationswhose parameters correspond to the speed andaccuracy criteria that are typically appliedto biological affordances (Pastor et al., 2009;Schaal, 2007).

Visual targeting

Much haptic behavior in humans tends to startwith visual information about an object of interestand its location in extrapersonal space. If theobject is familiar, this information alone may besufficient to identify the object and its expectedhandling properties. If not, it may provide conve-nient, albeit occasionally misleading, startingassumptions (e.g., large objects tend to be heavy,shiny surfaces tend to be slippery, etc.). Location,

138

Author's personal copy

Table 1. Summary of mechatronic hands

Research group Tactile sensing technology DOF Comments

Utah/MIT hand Capacitive normal force 16USC/Belgrade hand Normal-force-sensitive resistors 20Honda hand Normal-force-sensitive resistors 2 þ part of ASIMO robotHirzinger hand Normal-force-sensitive resistors 12 þNASA robonaut 2 6-DOF load cells in fingertip 24 þGiFu III hand Pressure-sensitive conductive ink 16 þSouthampton hand Piezoelectric polymer film 4 Prosthetic prototypeStanford/JPL hand 6-axis strain gauge 9UB III hand 6-axis IT sensors 16 þSmart hand Triaxial MEMS array 16 Follow-on from cyber handDist hand Triaxial force, slip sensors 16 þDLR II hand Conductive polymer grid 13 þShadowhand Quantum tunneling composite 24 *Yokoi III Force sensitive resistors 13 þ Tsukasa Kiko engineeringiLIMB pulse None * for prosthetic useLMS hand Unknown 16 þBUAA hand Unknown 16 þZurich/Tokyo hand Force sensitive resistors 13 þTorino hand Unknown UnknownRCH-I hand Unknown UnknownMA-I hand None 16SARAH hand None 10 *RTR II hand None 9 þ Prosthetic prototypeVanderbilt hand None 9 þ Pneumatic (peroxide)ACT hand None 24 þBarrett hand None 4 *Vecna HG2 None 6 * Hydraulic, hand cameraHeidelberg fluid hand None Unknown þLMS hand None 16Anybots Monty hand None 18 þTuat/Karlsruhe hand None 20 Prosthetic prototypeUltralight None 13 þElumotion–Sheffield hand None 20 þOrebro University Sweden None 12Manus Colobi None 3TBM hand None Unknown þ Prosthetic handOtto Bock Michelangelo 2 None 2 * for prosthetic use

* denotes currently available commercially; þ denotes currently under development by research teams.http://www.davidbuckley.net/RS/HandResearch.htmhttp://asimo.honda.com/asimo-specs/http://robonaut.jsc.nasa.gov/default.asphttp://www-arts.sssup.it/newCyberhand/smarthand/index.htmhttp://www.dlr.de/rm/en/desktopdefault.aspx/tabid-3802/6102_read-8923/http://www-lar.deis.unibo.it/activities/dexterousmanipulation/UBHandIII/index.htmlhttp://robot.gmc.ulaval.ca/en/research/theme304.htmlhttp://www.touchbionics.com/Pulsehttp://www.vanderbilt.edu/exploration/stories/bionicarm.htmlhttp://www.vecnarobotics.com/solutions/end_effectors/hg2.shtmlhttp://haptic.buaa.edu.cn/English_dexteroushand.htmhttp://neurobotics.cs.washington.edu/projects.htmlhttp://www.barrett.com/robot/products-hand.htmhttp://www.elumotion.com/shefarm.htmlhttp://www.ottobock.com/cps/rde/xchg/ob_com_en/hs.xsl/32149.html?openteaser¼1

139

Author's personal copy

http://www.davidbuckley.net/RS/HandResearch.htmhttp://asimo.honda.com/asimo-specs/http://robonaut.jsc.nasa.gov/default.asphttp://www-arts.sssup.it/newCyberhand/smarthand/index.htmhttp://www.dlr.de/rm/en/desktopdefault.aspx/tabid-3802/6102_read-8923/http://www-lar.deis.unibo.it/activities/dexterousmanipulation/UBHandIII/index.htmlhttp://robot.gmc.ulaval.ca/en/research/theme304.htmlhttp://www.touchbionics.com/Pulsehttp://www.vanderbilt.edu/exploration/stories/bionicarm.htmlhttp://www.vecnarobotics.com/solutions/end_effectors/hg2.shtmlhttp://haptic.buaa.edu.cn/English_dexteroushand.htmhttp://neurobotics.cs.washington.edu/projects.htmlhttp://www.barrett.com/robot/products-hand.htmhttp://www.elumotion.com/shefarm.htmlhttp://www.ottobock.com/cps/rde/xchg/ob_com_en/hs.xsl/32149.html?openteaser=1http://www.ottobock.com/cps/rde/xchg/ob_com_en/hs.xsl/32149.html?openteaser=1

Table 2. Summary of tactile sensors

Transductionmethod

No.ofaxis Multimodal Advantages Disadvantages Example

Capacitive 1 No Small, very high resolutiontaxels, can be flexible, widedynamic range, sensitive

Hysteresis, noise Pressure profile systemsrobotouch:http://www.pressureprofile.com/technology-capacitive.php

Inductive 1 No High sensitivity, repeatability Complex, vulnerableconstruction, lowspatial resolution

Futai et al. (2003)

Resistive:deformablecontact area

1 No Flexible, thin Hysteresis Inaba: Inastomer http://www.inaba-rubber.co.jp/en/b_products/inastomer/index.html

Resistive:conductivefabric

1 No Flexible, robust, simple Unable to resolvemore than onecontact point

Pan and Zhu (2005)

Resistive:quantumtunnelingcomposite

1 No Sensitive, wide dynamicrange

Hysteresis, gasabsorption

QTC: http://www.peratech.com/

Resistive: straingauge

6 No Same as above Bulky, expensive ATI: Nano 17 load cell: http://www.ati-ia.com/products/ft/sensors.aspx

Resistive:Piezoresistiveconductivepolymer

1 No Thin, low cost, simple Hysteresis, stiff Tekscan Flexiforce: http://www.tekscan.com/flexiforce.html

Resistive:Piezo-MEMS

6 Yes Small, multielement Large number ofwires in workspace

Oddo et al. (2009), Beccaiet al. (2005)

Polymer-MEMs(multimodal)

6 Yes 6-DOF force, temperatureand heat flow, roughness

Large number ofwires in workspace,wiring complexity

Engel et al. (2006)

Piezoelectric 1 Yes Detects dynamics for slip andtexture

Only detectsdynamic events,thermal sensitivity

Dario et al. (1984), Howe andCutkosky (1993)

Optical: videoprocessing

3 No Very high resolution,sensitive

Computationallyintensive, sensitiveto ambient light

Hristu et al. (2000), Ohka et al.(2004)

Optical:resistive

1 No Flexible, low hysteresis Complex fabrication http://www.skilsens.com/index.html

Magneto-elastic 1 No Very sensitive, low hysteresis Sensitive to externalmagnetic fields

Mitchell et al. (1986)

Magneto-resistive

6 No Robust, sensitive, lowhysteresis

Noisy Hackwood et al. (1983)

Ultrasound 1 Yes Static and dynamic High voltage,complex electronics

Brashford and Hutchins(1996), Grahn and Astle(1986)

140

Author's personal copy

http://www.pressureprofile.com/technology-capacitive.phphttp://www.pressureprofile.com/technology-capacitive.phphttp://www.inaba-rubber.co.jp/en/b_products/inastomer/index.htmlhttp://www.inaba-rubber.co.jp/en/b_products/inastomer/index.htmlhttp://www.inaba-rubber.co.jp/en/b_products/inastomer/index.htmlhttp://www.inaba-rubber.co.jp/en/b_products/inastomer/index.htmlhttp://www.peratech.com/http://www.peratech.com/http://www.ati-ia.com/products/ft/sensors.aspxhttp://www.ati-ia.com/products/ft/sensors.aspxhttp://www.ati-ia.com/products/ft/sensors.aspxhttp://www.tekscan.com/flexiforce.htmlhttp://www.tekscan.com/flexiforce.htmlhttp://www.tekscan.com/flexiforce.htmlhttp://www.skilsens.com/index.htmlhttp://www.skilsens.com/index.html

size, and regions of interest for grasp or contourfollowing can be extracted readily from ste-reovision. Machine vision has developed fairlysophisticated algorithms for all of these functions.

Exploratory algorithms

As discussed above, the decision about whichexploratory movement to employ at any giventime depends on whatever prior information iscurrently available about the probable identityof an object and the property of the object thatis most likely to distinguish it from other possibleobjects. After selecting and initiating a specificexploratory strategy, the details of the movement

may need to be driven by received sensory data(e.g., contour following) or adjusted iterativelyto fine-tune discrimination (e.g., repeated heftingto assess weight or stroking to assess texture).

Internal representations

In order to extract a typical canonical physicalproperty of an object (e.g., weight, texture, etc.),any sensory data received during an exploratorymovement would have to be deconvolved withthe parameters of the exploratory movement inorder to yield information specific to the object.In order to use canonical physical properties toinform the dexterous manipulation of that object,

Rigid core

External texturefingerprints

Impedanceelectrodes

Conductivefluid

Elastic skin

Pressuresensor

Thermistor

Fig. 3. A biomimetic, multimodal tactile sensor with the physical form and mechanical properties of a fused middle and distalphalanx consisting of a deformable skin inflated over a rigid core by a conductive fluid. Location, magnitude, and direction ofcontact forces can be extracted from changes in the pattern of electrical impedances measured through the conductive fluid byelectrodes distributed over the surface of the core (visible through transparent skin in photo below). A pressure sensorconnected to the fluid functions like a hydrophone to detect acoustic vibrations associated with slip or sliding over texturedsurfaces, which are enhanced by fingerprint-like ridges molded into the skin. The thermal material properties of objectscontacting the finger can be assessed by a thermistor that measures heat flow from the heated core (Lin et al., 2009; Wettelset al., 2009). Sources: Upasani et al. (1999), Biagiotti et al. (2002), Puig et al. (2008).

141

Author's personal copy

the anticipated dynamic effects of those pro-perties on the proposed manipulation would haveto be computed, something that has beenattributed to “internal dynamic models”(Imamizu et al., 2000; Kawato, 2008). Alterna-tively, the brain could represent objects as a setof learned associations in which each associationincludes both the output parameters of theexploratory movement (available from the manycopies of efferent signals that project recurrentlyin the central nervous system) and the sensorydata associated with the object being manipulatedin that way (e.g., tactile, proprioceptive, visual,auditory; Pastor et al., 2009). This would facilitatethe Bayesian strategy selection described aboveas well as the recall of appropriate motor strategiesfor dexterous manipulation of the object.

Conclusions

Hapticbehaviorsdonot lend themselves to the typesof reductionist studies in animals that have beeneffective in revealing the neural computationalalgorithms that underlie other perceptualcapabilities such as vision and hearing. Instead, itmay be possible to develop and test theories of hap-tic computation by applying them to robotic pla-tforms whose sensory and motor capabilities areincreasingly biomimetic. For this to be effective,however, it may be necessary to recapitulate at leastsome of the early cognitive stages during whichintelligent organisms develop representations ofthemselves and the external world. Thoserepresentations and strategiesmaybe somewhat idi-osyncratic, further emphasizing the importance ofunderstanding the processes rather than the results.

References

Beccai, L., Roccella, S., Arena, A., Valvo, F., Valdastri, P.,Menciassi, A., et al. (2005). Design and fabrication of ahybrid silicon three-axial force sensor for biomechanicalapplications. Elsevier, Sensors and Actuators, 120, 370–382.

Bernstein, N. (1967). The coordination and regulation of move-ment. New York: Pergamon.

Biagiotti, L., Lotti, F., Melchiorri, C., & Vassura, G. (2008).How far is the human hand? A review on anthropomorphicrobotic end-effectors. Bologna, Italy: University of Bologna.

Brashford, A. G., & Hutchins, D. A. (1996). Characteristics ofultrasonic micromachined capacitance transducers in water.IEEE Ultrasonics, Ferroelectrics, and Frequency ControlSociety, 2, 955–958.

Bresciani, J.-P., & Ernst, M. O. (2007). Signal reliabilitymodulates auditory-tactile integration for event counting.Neuroreport, 18, 1157–1161.

Buchli, J., & Ijspeert, A. J. (2004). Distributed central patterngenerator model for robotics application based on phasesensitivity analysis. Biologically Inspired Approaches toAdvanced Information Technology, 3141, 333–349.

Churchland, M. M., Afshar, A., & Shenoy, K. V. (2010). Acentral source of movement variability. Neuron, 52(6),1085–1096.

Churchland, M. M., & Shenoy, K. V. (2007). Temporal com-plexity and heterogeneity of single-neuron activity in pre-motor and motor cortex. Journal of Neurophysiology, 97,4235–4257.

Crair, M. C., Gillespie, D. C., & Stryker, M. P. (1998). The roleof visual experience in the development of columns in catvisual cortex. Science, 279, 566–570.

Dario, P., De Rossi, D., Domenici, C., & Francesconi, R.(1984). Ferroelectric polymer tactile sensors with anthropo-morphic features. IEEE International Conference on Robot-ics and Automation (1984), pp. 332–340.

Delcomyn, F. (2007). Biologically inspired robots. AdvancedRobotic Systems International and I-Tech, (in press).

Engel, J., Chen, N., Tucker, C., Liu, C., Kim, S.-H., &Jones, D. (2006). Flexible multimodal tactile sensing systemfor object identification. 5th IEEE conference on sensors,pp. 563–566.

Futai, N., Matsumoto, K., & Shimoyama, I. (2003). Simulation,fabrication and evaluation of microinductor-based artificialtactile mechanoreceptor embedded in PDMS. In IEEEMEMS, pp. 206–209.

Gallistel, C. R., Fairhurst, S., & Balsam, P. (2004). Thelearning curve: Implications of a quantitative analysis. Pro-ceedings of the National Academy of Sciences of the UnitedStates of America, 101(39), 13124–13131.

Gibson, E. J. (1988). Exploratory behavior in the developmentof perceiving, acting, and the acquiring of knowledge.Annual Review of Psychology, 39, 1–41.

Grahn, A. R., & Astle, L. (1986). Robotic ultrasonic force sen-sor arrays. Robot Sensors, 2, 297–315.

Hackwood, G. B., Hornal, L. A., Wolfe, R., & Nelson, T. J.(1983). A torque sensitive tactile array for robotics. Interna-tional Journal of Robotics Research, 2, 46–50.

142

Author's personal copy

Haykin, S. (1999). Neural networks: A comprehensive founda-tion. In (2nd ed.). Upper Saddle River, NJ: Prentice Hall.

He, J., Levine, W. S., & Loeb, G. E. (1991). Feedback gainsfor correcting small perturbations to standing posture. IEEETransactions on Automatic Control, 36(3), 322–332.

Hebb, D. O. (1949). The organization of behavior. New York:Wiley.

Heller, M. A., Calcaterra, J. A., Tyler, L. A., & Burson, L. L.(1996). Production and interpretation of perspectivedrawings by blind and sighted people. Perception, 25(3),321–334.

Hogan, N. (1984). An organising principle for a class of volun-tary movements. Journal of Neuroscience, 4(11), 2745–2754.

Howe, R. D., & Cutkosky, M. R. (1993). Dynamic tactile sens-ing: Perception of fine surface features with stress rate sens-ing. IEEE Transactions on Robotics and Animation, 9, 2.

Hristu, D., Ferrier, N., & Brockett, R. W. (2000). The perfor-mance of a deformable-membrane tactile sensor: Basicresults on geometrically-defined tasks. IEEE InternationalConf. on Robotics and Automation 1, pp. 508–513.

Hubel, D. H., & Wiesel, J. N. (1970). The period of susceptibil-ity to the physiological effects of unilateral eye closure inkittens. The Journal of Physiology, 206, 419–436.

Ijspeert, A. J. (2001). A connectionist central pattern genera-tor for the aquatic and terrestrial gaits of a simulated sala-mander. Biological Cybernetics, 84(5), 331–348.

Imamizu, H., Miyauchi, S., Tamada, T., Sasaki, Y., Takino, R.,Putz, B., et al. (2000). Human cerebellar activity reflectingan acquired internal model of a new tool. Nature (London),403(6766), 192–195.

Johansson, R. S., & Flanagan, J. R. (2009). Coding and use oftactile signals from the fingertips in object manipulationtasks. Nature Reviews. Neuroscience, 10, 345–359.

Jones, L. A., & Lederman, S. J. (2006). Human hand function.New York: Oxford University Press.

Kawato, M. (2008). From “Understanding the Brain by Creat-ing the Brain” towards manipulative neuroscience. Philo-sophical Transactions of the Royal Society of London.Series B: Biological Sciences, 363, 2201–2214.

Kording, K. (2007). Decision theory: What “should” the ner-vous system do? Science, 318, 606–610.

Lederman, S. J., & Klatzky, R. L. (1987). Hand movements: Awindow into haptic object recognition. Cognitive Psychol-ogy, 19, 342–368.

Lin, C.-H., Erickson, T. W., Fishel, J. A., Wettels, N., &Loeb, G. E. (2009). Signal processing and fabrication of abiomimetic tactile sensor array with thermal, force andmicrovibration modalities. ROBIO 2009 IEEE internationalconference on robotics and biomimetics.

Loeb, G. E., Levine, W. S., & He, J. (1990). Understandingsensorimotor feedback through optimal control. Cold SpringHarbor Symposia on Quantitative Biology, 55, 791–803.

Loeb, G. E., Peck, R. A., Moore, W. H., & Hood, K. (2001).BION system for distributed neural prosthetic interfaces.Medical Engineering & Physics, 23(1), 9–18.

Loeb, G. E., White, M. W., & Merzenich, M. M. (1983).Spatial cross-correlation. A proposed mechanism foracoustic pitch perception. Biological Cybernetics, 47(3),149–163.

Marks, L. E. (1983). Similarities and differences among thesenses. The International Journal of Neuroscience, 19(1–4),1–11.

Marr, D. (1982). Vision. New York: W.H. Freeman & Co.McCrea, D. A., & Rybak, I. A. (2008). Organization of

mammalian locomotor rhythm and pattern generation.Brain Research Reviews, 57(1), 134–146.

Mitchell, E. E., DeMoyer, R., & Vranish, J. (1986). A newmetglas sensor. IEEE Transactions on Industrial Electronics,IE-33, 166–170.

Oddo, C. M., Beccai, L., Felder, M., Giovacchini, F., &Carrozza, M. C. (2009). Artificial roughness encoding witha bio-inspired MEMS-based tactile sensor array. Sensors,9, 3161–3183.

Ohka, M., Mitsuya, Y., & Takeuchi, S. (2004). Sensing cha-racteristics of an optical three-axis tactile sensor under com-bined loading. Robotica, 22, 213–221.

Pan, Z., & Zhu, Z. (2005). Flexible full-body tactile sensor oflow cost and minimal output connections for service robot.Industrial Robot: An International Journal, 32, 485–491.

Pastor, P., Hoffmann, H., Asfour, T., & Schaal, S. (2009).Learning and generalization of motor skills by learning fromdemonstration. IEEE International Conference on Roboticsand Automation.

Pratt, J., Torres, A., Dilworth, P., & Pratt, G. (1996). Virtualactuator control. IEEE/RSJ international conference onintelligent robots and systems.

Puig, J. E. P., Nestor, E. N., Rodriguez, N., & Ceccarelli, M.(2008). A methodology for the design of robotic hands withmultiple fingers. International Journal of Advanced RoboticSystems, 5(2), 177–184.

Raphael, G., Tsianos, G., & Loeb, G. E. (2010). Spinal-likeregulator facilitates control of a two-degree-of-freedomwrist. The Journal of Neuroscience, 30(28), 9431–9444.

Sachs, N. A., & Loeb, G. E. (2007). Development of a BIONicmuscle spindle for prosthetic proprioception. IEEE Trans-actions on Biomedical Engineering, 54(6), 1031–1041.

Schaal, S. (2006). Dynamic movement primitives—A frame-work for motor control in humans and humanoid robotics.Adaptive Motion of Animals and Machines, 261–280 Part 6.

Schaal, S. (2007). The new robotics—Towards human-cen-tered machines. HFSP Journal, 1, 115–126.

Smitsman, A. W., & Schellingerhout, R. (2000). Exploratorybehavior in blind infants: How to improve touch? InfantBehavior & Development, 23, 485–511.

Soechting, J. F., & Flanders, M. (1989). Sensorimotorrepresentations for pointing to targets in three-dimensionalspace. Journal of Neurophysiology, 62(2), 582–594.

Todorov, E., & Jordan, M. I. (2002). Optimal feedback controlas a theory of motor coordination. Nature Neuroscience, 5(11), 1226–1235.

143

Author's personal copy

Upasani, A. V., Kapoor, C., & Tesar, D. (1999). Survey ofavailable sensor technology for robotic hands. Proceedingsof the DETC 99, ASME design engineering technicalconferences.

Wettels, N., Fishel, J., Su, Z., Lin, C.-H., & Loeb, G. E. (2009).Multi-modal synergistic tactile sensing. Tactile sensing in

humanoids—Tactile sensors and beyond workshop, 9thIEEE-RAS international conference on humanoid robots.

Wettels, N., Santos, V. J., Johansson, R. S., & Loeb, G. E.(2008). Biomimetic tactile sensor array. Advanced Robotics,22(8), 829–849.

144

Author's personal copy

Understanding haptics by evolving mechatronic systemsIntroductionComponents of haptic behaviorDevelopment of internal representations of objectsIdentification of selfEfferent and afferent feedbackInternal reference frames

Definition of surfacesDefinition of rigid objectsDefinition of deformable objectsDefinition of toolsDefinition of materials

Integrating exteroceptive sensesTesting new entities against internal representations

Hierarchical system of sensorimotor controlDesign of the spinal regulatorLearning to perform tasksCortical consolidationCortical representations

Development and testing of Haptic robotsMechatronic platformBiomimetic tactile sensingCompliant control algorithmsVisual targetingExploratory algorithmsInternal representations

ConclusionsReferences