robotics.sciencemag.org/cgi/content/full/4/27/eaau9924/DC1

Supplementary Materials for

Restoring tactile sensations via neural interfaces for real-time force-and-slippage

closed-loop control of bionic hands

Loredana Zollo*, Giovanni Di Pino, Anna L. Ciancio, Federico Ranieri, Francesca Cordella, Cosimo Gentile, Emiliano Noce, Rocco A. Romeo, Alberto Dellacasa Bellingegni, Gianluca Vadalà, Sandra Miccinilli, Alessandro Mioli,

Lorenzo Diaz-Balzani, Marco Bravi, Klaus-P. Hoffmann, Andreas Schneider, Luca Denaro, Angelo Davalli, Emanuele Gruppioni, Rinaldo Sacchetti, Simona Castellano, Vincenzo Di Lazzaro, Silvia Sterzi,

Vincenzo Denaro, Eugenio Guglielmelli

*Corresponding author. Email: l.zollo@unicampus.it

Published 20 February 2019, Sci. Robot. 4, eaau9924 (2019) DOI: 10.1126/scirobotics.aau9924

The PDF file includes:

Materials and Methods Fig. S1. Median and ulnar nerve. Fig. S2. Intraneural electrode sutured to epineurium. Fig. S3. Cuff electrode. Fig. S4. Percutaneous cables. Fig. S5. Threshold charge over 11 weeks in the thumb, index, and middle fingers. Fig. S6. Classification performance of the EMG pattern recognition algorithm. Fig. S7. Real-time force-and-slippage closed-loop control of a power grasp. Table S1. Percept qualities evoked by electrical stimulation of the cuff electrode on median nerve before T0. Table S2. Percept qualities evoked by electrical stimulation of the cuff electrode on ulnar nerve before T0. Table S3. Percept qualities evoked by electrical stimulation of the ds-FILE intraneural electrode on median nerve before T0. Table S4. Percept qualities evoked by electrical stimulation of the ds-FILE intraneural electrode on median nerve after T0.

Other Supplementary Material for this manuscript includes the following: (available at robotics.sciencemag.org/cgi/content/full/4/27/eaau9924/DC1)

Movie S1 (.mp4 format). Restoring tactile sensations.

Supplementary Material

Materials and Methods

Surgery

Surgery was conducted under general anesthesia, in supine decubitus using an arm table.

The overall procedure lasted about four hours, while the net surgical implant took almost two hours.

Antibiotic prophylaxis was administered following the hospital protocol for infection prevention. A

medial approach to the neurovascular bundle was performed, following the medial edge of the

biceps muscle for about 12 cm distally, to obtain sufficient space to introduce the electrodes and

avoid possible conflicting interactions. After careful smooth dissection ulnar and median nerve

were isolated and exposed for about 5 centimeters (Fig. S1). A microsurgical epineural dissection

was performed using a surgical microscope (Opmi Vario/NC33, Zeiss). Two intraneural electrodes

(ds-FILEs) were introduced in each of the two main motor nerves for hand and finger flexion

(medial and ulnar nerve trunks), achieving a total of 64 channels opened toward the nerves. The

intraneural electrodes were introduced using a dedicated needle at 45 degrees to the nerve axis, and

pulled inside the nerve, in order to reach the widest contact with nerve fascicles as possible. Then,

the electrodes were sutured onto the epineurium through dedicated slots, using a 8.0 nylon suture

(Fig. S2). In order to minimize the mechanical stress on the electrodes, the ceramic connector was

anchored to contiguous fascial tissues. Furthermore, two epineural electrodes (“cuff electrodes”)

were wrapped around each nerve and sutured using a 8.0 nylon suture (Fig. S3) proximally

compared to the intraneural electrodes. By tunneling the subcutaneous tissue, the cables connected

to the electrodes passed the skin through 4 different holes on the anterior aspect of the arm,

avoiding the skin grafting. Moreover, to avoid any accidental traction to the cables, two loops were

arranged along the cable, one inside and one outside the skin surface, then anchored to the skin (Fig.

S4).

Eleven weeks after the implantation, the same procedure was carried out and all the

electrodes were removed.

Mapping of elicited sensations

In the sensory mapping the stimulation waveform was a train of cathodic rectangular biphasic

pulses with a fixed frequency of 50 Hz. The pulse amplitude and the pulse width were set to fixed

values, progressively modified in order to identify all the sensations elicited in the subject by the

electrical stimulation.

The participant was instructed to report quality, location and intensity of the perceived sensation on

a custom-developed platform with a computer interface. This has been ad-hoc developed to help the

subject take note about the reported sensation. The quality was assessed using the following

options: touch/pressure, vibration, tingling, pinch, pain, cold, hot, finger extension, finger flexion,

wrist extension and wrist flexion (Tables S1-S4). The location of the sensation was indicated by the

patient using two picture boxes representing the frontal and dorsal side of the hand. Moreover, the

intensity and/or the pain of the perceived sensation were reported in a scale from 0 to 10.

The minimum threshold to elicit perceived sensations on the hand was monitored during the eleven

weeks of the experiment and was identified by slowly increasing the intensity of the stimulation of

intraneural and cuff electrodes. The minimum stimulation charge on intraneural electrodes ranged

from 7 nC to 86 nC in eleven weeks, significantly lower than threshold recorded for cuff electrodes

in the same period. Cuff on median and ulnar nerves varied their charges from 60 nC to 240 nC and

from 120 nC to 150 respectively. The injected charges adopted in this study for intrafascicular and

cuff electrodes were consistent with the charges used in previous studies and involving the same

types of electrodes (7) (12) (13).

Force and slippage sensations were provided to contacts number 10, 12, 16 of the intraneural

electrode in the median nerve that the subject referred to map on the thumb, index and middle

fingers. Minimum threshold tracking of these channels over the eleven weeks of experimental study

is shown in Fig. S5. The sensory stimulation threshold of channel 16 slightly increased during time

from 7 nC to 36 nC. For channel 12, the minimum threshold changed from 19 nC to 86 nC, while

the minimum injected charge of channel 10 increased with days ranging from 19 nC to 41 nC.

Validation of the multifingered stick-slip model

Ten healthy subjects (seven males and three females, mean age (±s.d.): 36±4 years) volunteered to

participate in this study and provided written informed consent. All the participants received

detailed instructions and familiarized with tasks before starting the acquisitions. They were

comfortably seated at a desk with the arm sustained by a support.

The blindfolded and acoustically shielded subjects were asked to grasp an object placed close to the

fingers and lift it. Ten repetitions of a power grasp and ten repetitions of a tridigital grasp per

subject were performed. For the power grasp an object with a parallelepipedal shape and a weight

of 0.25 kg was adopted; for the tridigital grasp, an object with a parallelepipedal shape and a weight

of 0.050 kg was used. The objects were equipped with force-sensing resistors for recording the

applied normal forces and reflective markers for monitoring the object displacement by means of

BTS Smart-D optoelectronic system. One marker was placed at the upper extremity and two

markers were at the lower extremity of the object. Moreover, a magneto-inertial unit (MTw-

38A70420 Xsens Technologies B.V.) was located at the top of the object to record object

acceleration and orientation.

In order to generate a repeatable perturbation (i.e. Fs in the model), an additional mass was linked to

the object and released when the object was lifted by the subject. For the power grasp, the

additional mass was of the same weight of the object, while for the tridigital grasp the mass was

twice the object weight. The weight of the additional masses was empirically chosen to be sure to

induce slippage and obtain an observable object displacement.

The same experimental setup and the same experimental conditions were reproduced with the

amputee participant for the experiment of force-and-slippage closed-loop control with neural

feedback.

Grasp assessment

The weighted success is a normalized measure of the task success rate and is expressed as the task success modulated byby the number of occurred slippage events and normalized over the maximum

number of slip events detected with the same feedback condition. It is computed as

𝑠𝑢𝑐𝑐𝑒𝑠𝑠𝑤𝑒𝑖𝑔ℎ𝑡 = 𝑠𝑢𝑐𝑐𝑒𝑠𝑠 (1 −𝑠𝑙𝑖𝑝𝑝𝑎𝑔𝑒

𝑠𝑙𝑖𝑝𝑝𝑎𝑔𝑒𝑚𝑎𝑥)

where:

𝑠𝑢𝑐𝑐𝑒𝑠𝑠 is a binary value that can assume value 1 when the trial is successfully performed, and value 0 when the trial fails;

𝑠𝑙𝑖𝑝𝑝𝑎𝑔𝑒

𝑠𝑙𝑖𝑝𝑝𝑎𝑔𝑒𝑚𝑎𝑥 is the number of slip events detected during the trial; it is normalized by the

maximum number of slip events detected in all the trials and performed in the same sensory

feedback condition (i.e. with neural feedback, or without feedback).

Therefore, the 𝑠𝑢𝑐𝑐𝑒𝑠𝑠𝑤𝑒𝑖𝑔ℎ𝑡 ranges in the interval [0, 1], where 0 is obtained when the trial fails,

while 1 is obtained when the trial is successfully performed with no slippages. In between, the

success index is weighted with slippage and decreases as the number of slippages increases.

The force index, expressed in Newton, measures the total force applied by the fingers involved in the grasping or manipulation task. The force signal is segmented between the time instant where

the force exceeded a threshold of 2% of the peak force and the time instant where the force dropped

below the same threshold. Force index 𝐹𝑖 is evaluated as

𝐹𝑖 = ∑ 𝐹𝑚𝑒𝑎𝑛𝑖𝑘𝑖=1

where 𝐹𝑖 is the mean value of the force for finger 𝑖, and 𝑘 can vary from 2 to 5, depending on

the number of fingers involved in the task

The execution time is the time employed for performing the task, elapsed from the trial onset and termination triggered by the experimenters.

Data analysis and statistics

A statistical analysis based on Friedman non-parametric tests with Wilcoxon post-hoc test and

Bonferroni correction (p< 0.016) was applied for multiple comparisons of the weighted success at

T0, T1 and T2. A Wilcoxon Signed-Rank test was used to compare grasp performance in the two

conditions of neural feedback and no feedback, with significance threshold set to 0.05.

Neurophysiological assessment

Assessment of nerve motor fiber excitability

Motor fiber activation after stimulation by the implanted electrodes was assessed by recording

electromyographic (EMG) responses from muscles of the median and ulnar innervation territories:

anterior forearm muscles (wrist flexors) and flexor carpi ulnaris muscle were chosen to test median

and ulnar nerve excitability, respectively. Increasing stimulation intensities up to 400 uA (stimulus

duration: 80 us) were used with contacts of the median dsFILE electrodes and up to 1000 uA

(stimulus duration: 200 us) with contacts of median and ulnar cuff electrodes.

Assessment of motor cortical excitability

The excitability of the primary motor cortex (M1) was assessed by single pulse transcranial

magnetic stimulation (TMS), delivered through a Magstim 2002 magnetic stimulator (The Magstim

Company Ltd, Whitland, Carmarthenshire, UK) generating a monophasic magnetic pulse. The

stimulator was connected to a figure-of-eight coil with an external diameter of 9 cm, held over the

right motor cortex at the optimum scalp position to elicit EMG responses in the contralateral arm

and forearm muscles. The induced current flowed in a posterior-to-anterior direction across the

central sulcus.

Recording of muscle evoked responses

Muscle evoked potentials, by either TMS or nerve stimulation, were recorded using 9 mm diameter

Ag-AgCl surface EMG electrodes, with the active electrode over the motor point of the muscle and

the reference placed distally on the surface of the ulnar bone (for forearm muscles) or over the

tendon of the biceps brachialis muscle at the elbow. The signal was amplified and filtered (gain:

1000; bandwidth: 3-3000 Hz) by a Digitimer D360 amplifier (Digitimer, Welwyn Garden City, UK)

and stored on a computer with a sampling rate of 5 KHz using a CED 1401 A/D converter

(Cambridge Electronic Design, Cambridge, UK).

Supplementary Figures and Tables

Surgery

Fig. S1. Median and ulnar nerve.

Fig. S2. Intraneural electrode sutured to epineurium.

Fig. S3. Cuff electrode.

Fig. S4. Percutaneous cables.

Mapping of elicited sensations

Fig. S5. Threshold charge over 11 weeks in the thumb, index, and middle fingers. The thresholds

corresponded to the minimal sensations reported by the subject stimulating the channels of ds-FILE

intraneural electrode implanted in median nerve.

Table S1. Percept qualities evoked by electrical stimulation of the cuff electrode on median nerve

before T0. The number of trials evoking intensity level, quality and perceived area is shown. An intensity

level “0” indicates a stimulus not perceived by the patient.

Table S2. Percept qualities evoked by electrical stimulation of the cuff electrode on ulnar nerve before

T0. The number of trials evoking intensity level, quality and perceived area is shown. An intensity level “0”

indicates a stimulus not perceived by the patient.

Table S3. Percept qualities evoked by electrical stimulation of the ds-FILE intraneural electrode on

median nerve before T0. The number of trials evoking intensity level, quality and perceived area is shown.

An intensity level “0” indicates a stimulus not perceived by the patient.

Table S4. Percept qualities evoked by electrical stimulation of the ds-FILE intraneural electrode on

median nerve after T0. The number of trials evoking intensity level, quality and perceived area is shown.

An intensity level “0” indicates a stimulus not perceived by the patient.

Myoelectric control

Fig. S6. Classification performance of the EMG pattern recognition algorithm. (A) Confusion matrix

indicating 99.3% mean accuracy for selected gestures. Main diagonal shows accuracy for each class (rest,

power, pinch, open and lateral). (B) Classification performance expressed through the F1Score for each

class, mean F1Score and mean accuracy

Real-time force-and-slippage closed-loop control

Fig. S7. Real-time force-and-slippage closed-loop control of a power grasp. (A) With neural feedback.

The participant performed a power grasp: the power gesture was selected by the EMG classifier and all the

fingers stared moving. Once the object was touched, force feedback was provided. The first slippage event

was felt by the participant, who actively tuned the level of force by producing a variation in the EMG signal.

After an additional correction of the force, due to a latter slippage event, grasp stability was reached up to the

end of the trial. Hence, the open hand gesture was classified and the hand re-opened. (B) Without feedback.

The participant performed a power grasp: the power gesture was selected by the EMG classifier and all the

fingers stared moving. Once the object was touched, the applied force was measured and slippage was

detected by the sensors. There was no stimulation. The patient was not able to feel the detected slippage

event, however the grasp was stable and the object does not fall. At the end of the trial, the open hand gesture

was classified and the hand re-opened. All the traces are normalized with respect to the maximum time

duration.