Adaptive Tensegrity Locomotion: XX(X):1–20 The Author(s ...kb572/pubs/reinf_learning_tensegrity_locomotion.pdf6-bar tensegrity rovers (Caluwaerts et al.2014;Sabelhaus et al.2015)

Adaptive Tensegrity Locomotion:Controlling a Compliant Icosahedron withSymmetry-Reduced Reinforcement Learning

Journal TitleXX(X):1–20c©The Author(s) 0000

Reprints and permission:sagepub.co.uk/journalsPermissions.navDOI: 10.1177/ToBeAssignedwww.sagepub.com/

David Surovik1, Kun Wang1, Massimo Vespignani2, Jonathan Bruce2, and Kostas E. Bekris1

AbstractTensegrity robots, which are prototypical examples of hybrid soft-rigid robots, exhibit dynamical properties that provideruggedness and adaptability. They also bring about, however, major challenges for locomotion control. Due to highdimensionality and the complex evolution of contact states, data-driven approaches are appropriate for producing viablefeedback policies for tensegrities. Guided Policy Search (GPS), a sample-efficient hybrid framework for optimizationand reinforcement learning, has previously been applied to generate periodic, axis-constrained locomotion by anicosahedral tensegrity on flat ground. Varying environments and tasks, however, create a need for more adaptiveand general locomotion control that actively utilizes an expanded space of robot states. This implies significantly higherneeds in terms of sample data and setup effort. This work mitigates such requirements by proposing a new GPS-based reinforcement learning pipeline, which exploits the vehicle’s high degree of symmetry and appropriately learnscontextual behaviors that are sustainable without periodicity. Newly achieved capabilities include axially-unconstrainedrolling, rough terrain traversal, and rough incline ascent. These tasks are evaluated for a small variety of key modelparameters in simulation and tested on the NASA hardware prototype, SUPERball. Results confirm the utility ofsymmetry exploitation and the adaptability of the vehicle. They also shed light on numerous strengths and limitationsof the GPS framework for policy design and transfer to real hybrid soft-rigid robots.

KeywordsAdaptive Control, Learning Techniques in Soft Robotics, Locomotion, Mobile Robotics, Tensegrity

1 Introduction

As modern robotic applications target increasingly unstruc-tured environments and variable objectives, the systemsdeployed must be made more adaptive and robust in orderto succeed. Soft robots meet these needs by providing manydegrees of freedom in combination with significant structuralcompliance. These traits can be induced either by utilizingsoft materials or through the extensive use of elastic jointsbetween hard components (Lipson 2014), as in the case oftensegrity robotics.

Tensegrity structures suspend isolated rigid elementswithin a network of nonrigid elements, which are keptin pure tension to maintain structural integrity. Dueto the lack of rigid joints, forcing upon the structurecauses compliant reconfiguration that distributes stressthroughout the network. Because this scheme avoidslocalized accumulation of stress at failure points, suchstructures can withstand large forces in proportion totheir own weight. Given an ability to actuate thelengths of individual elements—most commonly the tensilemembers—tensegrities can furthermore be made to controltheir own shapes. This combination of passive complianceand active deformation provides the beneficial robustnessand adaptiveness of soft robotics.

A wide variety of architectures have been consideredfor mobile tensegrity robots, with the number of rigidbars ranging from as few as 3 in a simple prism (Paulet al. 2006) to dozens within a spine (Mirletz et al.2015) or quadruped (Hustig-Schultz et al. 2016). The most

Figure 1. Left: hardware prototype of SUPERball version 2(SBv2) at NASA Ames. Right: Simulation of SBv2 adaptivelylocomoting across rough terrain.

extensively studied arrangement is the 6-bar icosahedron,which is close enough to spherical in shape to enablelocomotion that resembles a rolling ball. In combinationwith the favorable traits of tensegrities in general, thisscheme strikes a balance between versatility and simplicitythat lends it merit as a platform for mobility on theunstructured surface environments of other planets or moons.

1Rutgers University, New Brunswick, NJ, USA2NASA Ames Research Center, Moffett Field, CA, USA

Corresponding author(s):David Surovik, Department of Computer Science, Rutgers University,110 Frelinghuysen Road, Piscataway, NJ 08854, USAEmail: [email protected]

Prepared using sagej.cls [Version: 2016/06/24 v1.10]

2 Journal Title XX(X)

Additional logistical benefits include low manufacturingcost, the ability to pack efficiently into a small volume ofa rocket payload, and the potential to dissipate large forcesduring landing. Consequently, NASA Ames Research Centerhas developed a series of hardware prototypes of self-landing6-bar tensegrity rovers (Caluwaerts et al. 2014; Sabelhauset al. 2015). The most recent of these, SUPERball version 2or SBv2, is shown in Fig. 1 (Vespignani et al. 2018b).

1.1 Tensegrity LocomotionThe system complexities that give mobile tensegritiesappealing properties also bring about deep challenges interms of control and motion planning. These systems havemany degrees of freedom and are subject to subtle energeticeffects through elasticity and static instability. Furthermore,during locomotion, numerous diverse contact modes areexperienced in rapid succession. Such factors prevent astraightforward process for deriving a high-quality controlpolicy directly from a system model.

The most basic approach for producing planar locomotionis hand-tuning of control inputs to one or two actuators, whilethe rest remain in neutral configuration (Hirai and Imuta2012; Sabelhaus et al. 2015). This can also be applied tomore challenging tasks such as incline ascent (Chen et al.2017) but cannot be expected to generalize broadly or toleverage the full capabilities of the vehicle.

Simplified models that reason about static stability only,ignoring dynamics and environmental perturbations, can beused to discover open-loop locomotion policies that utilizelarge numbers of actuators. This may be accomplishedby Monte Carlo search of the input space to a non-differentiable model that accounts for elasticity (Kim et al.2015), or via gradient descent in a further simplified butdifferentiable model (Littlefield et al. 2017). With offlineprecomputation of a large number of circumstances, similargeometric reasoning can be leveraged online for quasi-staticlocomotion on terrain (Zhao et al. 2017); however, thisapproach requires position and contact information that maynot be reliably available onboard.

Alternatively, higher fidelity dynamics can be accountedfor via data-driven approaches to control policy design.Open-loop periodic locomotion has been generated forthe 3-bar using an evolutionary algorithm (Paul et al.2006) and for the 6-bar using central pattern generatorsand Bayesian optimization (Rennie and Bekris 2018).Using similar combinations of few-parameter oscillator-based architectures with various search and learningmethods, periodic locomotion with tension feedback hasbeen achieved on the spine (Mirletz et al. 2015) and the6-bar (Caluwaerts et al. 2013). Evolutionary methods havealso been applied to produce smooth rolling of the 6-barin discrete choices of directions on hilly terrain, providedwith contact information (Iscen et al. 2014). Superviseddeep learning has been used to train a contextual policyfor directed rolling on flat ground, when provided withan extensive training set generated by model predictivecontrol (Cera and Agogino 2018).

Another recent strategy is the hybrid deep reinforcementlearning method of Guided Policy Search (GPS), whichseeks to provide improved sample efficiency without anapriori dynamics model (Levine and Abbeel 2014). This

technique has recently been applied to produce a policy forfixed-axis rolling of SUPERball version 1 on flat ground,with successful execution on hardware (Zhang et al. 2017).A similar result has also been achieved in simulation for afully actuated 6-bar on mild terrain (Luo et al. 2018).

1.2 Outline and ContributionsThis investigation seeks to expand the capability of tensegritylocomotion controllers to adapt to diverse contexts in termsof the environment and objective. A primary implication isthat the policy must actively utilize broad regions of thevehicle’s state space during sustained nonperiodic behavior.This stands in contrast to previous results on flat or nearlyflat ground that appear to closely track a periodic motion, i.e.a 1D manifold of the control space.

In part, this investigation has been prompted bythe exceptional deformational capabilities of the newestprototype of SUPERball (Vespignani et al. 2018b).Alternately, an exploration of the possibilities of more highlydynamic motion is also desired. This could come aboutvia either an increase in hardware actuation speed or adecrease in gravitational acceleration, as is characteristicof many target environments for deployment. This moreenergetic motion could also shed light on the extensibilityor limitations of current algorithmic approaches.

In Sec. 2, broader context will be provided for the study oforientation-free mobile platforms and the control of adaptivesystems with high dimensionality and compliance. Detailsof the structure and locomotive possibilities of SUPERballversion 2 are then described in Sec. 3, while notingdifferences with previous 6-bar platforms. The remainder ofthe paper is next outlined in terms of research contributions.

Contribution 1: comprehensive symmetry reduction.This contribution, which was shown in preliminary format a conference venue (Surovik and Bekris 2018), targetsa drastic reduction in the data requirements for producingan effective policy. While some prior works exploited theinterchangeability of structural element IDs (Iscen et al.2014; Vespignani et al. 2018a), the symmetry reductionpresented here in Sec. 4 additionally handles frame-dependent quantities such as node positions and bar angularvelocities. Consequentially, it can be leveraged duringGPS when fitting surrogate models and when computingcontextual feedback, such as for fine-grained steering.

Contribution 2: altering GPS for sustained nonperiodicbehaviors. Past applications of GPS focused upon eithertransient manipulation or periodic locomotion and utilizedrestrictive setups that ensured an apriori decompositionof sample data into local regions. Sec. 5 first describesthe original GPS pipeline before introducing changesthat eliminate such setup requirements, which would beprohibitive to implement for broadly adaptive and contextualpolicies. These changes are facilitated by the symmetryreduction technique. Relative to an early conference versionof this work (Surovik et al. 2018), the scheme has beenstreamlined and is presented in much greater detail.

Contribution 3: demonstration on multiple tasksand setups. The modified GPS implementation, termedGPS-SyNC, is applied to generate adaptive motion undera small variety of system parameters and environmentsdetailed in Sec. 6. These test cases alter both the capability

Prepared using sagej.cls

Adaptive Tensegrity Locomotion 3

of the vehicle and the challenge of the environment, rangingfrom flat ground to a rough 25-degree incline. Sec. 7provides analyses of performance and adaptive behavioralcharacteristics. Preliminary deployment on hardware isdetailed in Sec. 8, including challenges faced in transferringa policy developed in simulation. All of these aspectsare vastly extended relative to the preceding conferencearticle (Surovik et al. 2018).

Finally, discussion in Sec. 9 considers the nature ofthe successes and limitations, which may inform relatedcontroller design efforts in soft robotics and other high-dimensional adaptive systems.

2 Related Work

2.1 Orientation-Free VehiclesSpherical robots and rolling locomotion have been studiedas an orientation-free paradigm for mobility. Some effortsconsidered a rigid sphere on a flat plane, with rotationinduced via internal reaction wheels (Bhattacharya andAgrawal 2000). Another scheme using internal massdisplacement and data-driven controller design achievedrolling on smooth hills in simulation (Der et al. 2006). Earlywork in self-deforming round robots utilized radial actuatorsto pull on a circular or spherical shell (Sugiyama et al. 2005).

Notable recent self-deforming prototypes include avehicle composed of 3 perpendicular, rigidly articulatedloops, retaining 6 degrees of freedom for shape control (Weiet al. 2019). Another platform features 32 radially-orientedtelescopic legs for highly expressive command over itscontact interface (Nozaki et al. 2018). While these designshave only seen preliminary trials using model-based controlon level ground, they hold the potential to adapt their contactinterfaces effectively on irregular terrain; however, they lackthe advantageous property of compliance.

2.2 Compliance and TensegritiesPhysical compliance helps to reconcile the usage of adaptiveconfigurations with the influence of dynamic properties. Thishas been extensively leveraged in the field of legged robotsfor benefits such as speed and energy efficiency (Zhou andBi 2012). Furthermore, robots that employ soft materials orlarge numbers of compliant linkages exhibit a heightenedpotential for adaptiveness and can unlock novel modes oflocomotion (Calisti et al. 2017). Examples range from leggedvehicles that squeeze through confined spaces (Jayaram andFull 2016), caterpillars that can both crawl and roll (Lin et al.2011), deforming-wheel vehicles (Lee et al. 2013) and bat-like fliers with high maneuverability (Ramezani et al. 2017).

Through a phenomenon known as morphological com-putation, the passive dynamics induced by compliance cangreatly benefit the formulation of control solutions (Hauseret al. 2011). This principle, which can be considered asphysical reservoir computing on a mechanical system, hasbeen recognized as another benefit of tensegrity morpholo-gies (Rieffel et al. 2009; Caluwaerts et al. 2013; Fujita et al.2018), which achieve a high degree of compliance via manyelastic linkages. Along with the corresponding ruggedness,this trait provides significant advantages to the icosahedral

6-bar tensegrity as an adaptive and orientation-free mobileplatform.

Alternate forms of orientation-free tensegrities includethose with curved bars (Kaufhold et al. 2017) or largernumbers of straight bars (Kim et al. 2017). Another NASAconcept considers the integration of an 18-bar rover with adeformable heatshield to aid control during descent throughthe atmosphere of Venus (Schroeder et al. 2018). Otheractuation paradigms include active displacement of massesattached to the bars (Kaufhold et al. 2017) or of a suspendedcentral payload (Caluwaerts et al. 2014). Propulsive payloadssuch as thrusters (Kim et al. 2016) or propellers (Mintchevet al. 2018) have been considered as well. Motion can alsobe induced by vibrating different rigid elements at differentfrequencies (Rieffel and Mouret 2018).

2.3 Controlling Adaptive SystemsFor certain manipulation tasks, compliant and soft-bodiedsystems can be successfully controlled by tailoring model-based approaches (Best et al. 2016; Santina et al.2018) or even by relying entirely on morphologicalcomputation (Deimel and Brock 2015). Many soft orhighly articulated locomotors are amenable to simple controlschemes using hand-tuned or learned open-loop input at lowdimension, with high-dimensional effects ensured by passivedynamics or imposed couplings (Shepherd et al. 2011; Onaland Rus 2013; Ijspeert 2008). Relatedly, soft locomotion caneven be optimized by first specifying the coupled controlscheme and subsequently evolving the morphology of therobot (Cheney et al. 2013).

Adaptation to the most challenging contextual variations,however, may require control policies with high-dimensionaloutput, which becomes possible with large-scale data-drivenmethods such as reinforcement learning. Such capabilitiesare only recently emerging for adaptation to object shapesin dextrous manipulation (Rajeswaran et al. 2018), to terrainvariation in 2D locomotion (Peng et al. 2016), and to self-reconfiguration in 3D locomotion (Christensen et al. 2013).

The explosion of sample complexity that occurs instep with broader contexts and capabilities calls fordevelopment of various strategies for its reduction.Possibilities include the evaluation of an experiencedsample trajectory with a different objective than wasconsidered at runtime (Andrychowicz et al. 2017) anddeliberation about the retainment and replay of various suchexperiences (De Bruin et al. 2018). Other approaches mayseek a reduction of the problem description, such as thelearning of a lower-dimensional model (Zhang et al. 2018).

3 SystemThis section and the next will introduce notation describingthe system and control problem. Table 1 summarizes valuesthat are referenced extensively in the remainder of the paper.

3.1 Structure and PropertiesSUPERball version 2, denoted SBv2 and pictured in Fig. 1,is a large-scale 6-bar icosahedral tensegrity with bar lengthb = 1.94m. These compressive elements are isolated fromeach other within a network of 24 elastic cables. The “nodes”



x,y state and observation vectorsu control input (targeted rest lengths of cables)φ nonlinear system dynamics modelπ, θ nonlinear control policy and its parameterscx, cy contextual params of environment and objectiveb length of compressive bars (constant)

∆,Λ “closed” and “open” triangular facet typesµ ID of bottom triangle (unique facet below CoM)

v, ν CoM movement direction as vector or angleH, j symmetry remapping operator and its index

Table 1. Key notation used broadly throughout the paper.

at which the cables and bars are connected contain motorizedspools to independently control each cable’s rest length,which differs from its actual length due to stretching.

Although many aspects of construction differ fromversion 1 of the prototype (SBv1), the most significantconsiderations for control design are the changes tothe tensile elements. The elasticity of these cables issubstantially increased in service of the goal of withstandinglarge impacts during landing. More fundamentally, SBv1could actuate only 12 cables, arranged in a pattern thatimposed a specific axis for rolling motion (Zhang et al.2017). SBv2 can actuate all 24 cables, as has also beenpossible for some smaller-scale 6-bar prototypes.

A significant difference relative both to SBv1 and otherpast prototypes is SBv2’s very large range actuator range ofmotion of [0, 1]b, which is sufficient to demonstrate packinginto a flat disk or a thin cylinder. In the current investigation,artificial bounds of [0.25, 0.75]b is applied in order to limitthe space of explored shapes to a manageable volume.Given that deformation effects accumulate across a set of24 cables, possibilities are drastically increased relative toSBv1’s range of motion of [0.35, 0.65]b.

Figure 2 gives the connectivity scheme for the physicalelements of the vehicle while also depicting conceptuallyuseful “virtual” elements. The edges of the convex hullconsist of all 24 tensile elements and 6 additional virtualedges where no physical element is present. Triangular facesof two types tile the hull. “Closed” faces are defined bythree cables and denoted with the symbol ∆. The 8 facesof this category never share a common edge. The remaining12 “open” faces, denoted Λ, occur in pairs that share a virtualedge.

∆ Λ

Figure 2. Topology of a 6-bar icosahedral tensegrity shown in3D and 2D, using a cutting section shown in gray. Solid blacklines indicate cables; dotted lines are virtual edges. ∆ typetriangles are shaded blue, Λ unshaded. In 2D, left and right grayboundaries are identified with each other; top and bottomboundaries are self-identified points in the center of the top andbottom ∆’s (which appear rectangular). Bars, shown as thickcolored lines, span through the interior and appear curved in 2D.

3.2 State and DynamicsThe vehicle state x consists of the 6-DoF rigid body stateof each bar along with the rest length of each cable. Whileactual cable lengths are dependent upon the bar states, theirrest lengths differ due to stretching, which stores significantelastic energy. The control input u is the vector of desiredcable rest lengths, where the correspondence between spoolactuation and resulting rest lengths is handled by a separatelayer of PID controllers. Sensor observations y (x) includethe actual cable rest lengths, available from the spoolencoders, and the angular velocities of the bars, obtainedfrom gyroscopes.

Noting that the above definition of x expresses theintrinsic state of the vehicle, let cx denote the environmentalcontext, e.g., the contact interface. Furthermore let cy givethe objective context, which in this work will indicate thereal-valued desired direction of motion and is assumed to beavailable independent of y via higher layers of autonomy.Under a contextual feedback policy π with design parametersθ, the discrete-time nonlinear system dynamics are then

x′ = φ (x,u; cx) (1)u = πθ (y (x) ; cy) (2)

This formulation indicates that the evolution of xvaries substantially throughout the environment and is nottrivially related to x alone. Nonetheless, a relationshipmust somehow be inferred through y (x) in order todetermine a control response. The contextual nature ofthe policy, meanwhile, indicates that the the responseshould incorporate considerations that are independent ofthe environment. These subtle inferences will be addressedthrough data-driven controller design, with the simulationtestbed representing φ consisting of the NASA TensegrityRobotics Toolkit, NTRT (Caluwaerts et al. 2014).

3.3 OrientationA fundamental quantity for expressing the orientation of thevehicle (which is not explicit in y) is the “bottom” triangleof the vehicle, µ. This is the facet of the convex hull that liesdirectly below the center-of-mass (CoM), regardless of thecontact states of the nodes that define that facet. Notably,this definition is independent of the concept of a supportpolygon and uniquely identifies a single triangle. The bottomtriangle will be used for defining and illustrating numerousquantities, including those related to symmetry reduction.

∆ Λ

h

hv

v

α

β

γ

α

γ

β

Figure 3. Top-down views of the two types of bottom triangle µalong with orientation quantities and the direction of motion v.The handedness of the ∆ triangle depends on the direction inwhich the connected bars extend.



(a) (b) (c)

Figure 4. Patterns of mass-displacement or “rolling” based locomotion of an icosahedral tensegrity, illustrated as paths of thecenter-of-mass shadow across the convex hull. Paths may wrap across the projection’s left and right boundaries or move instantlyalong the top or bottom boundaries (self-identified points). (a) Step-wise motion with a small set of discrete directional options. (b)Smooth fixed-axis motion, with gray cables inactive on SBv1. (c) Any-axis motion with a continuous set of directional options.

Figure 3 gives general illustrations for the two possiblecases of bottom triangle type. The positions of the feet aredenoted qi with i ∈ {α, β, γ}. These indices are alwaysordered such that the face normal (qβ − qα)× (qγ − qβ)is topologically consistent between all faces (i.e., all facenormals point outward from the vehicle). On Λ faces, α isalways the node opposite from the virtual edge.

Two further useful quantities are also expressed in Fig. 3:the center-of-mass movement unit vector v and the bottomtriangle orientation unit vector h. The latter quantity isderived using the vector (qβ + qγ) /2− qα, shown as adashed arrow. For ∆ cases, h lies along this vector; for Λcases, a 90◦ offset is applied. This manner of definition wasfound to be more stable than the use of an edge normal,as it incorporates information from all three of the bottomtriangle’s nodes. These values can also be used to definea transformation A from the world frame into the vehicleframe,

A(h, v

)h = [1, 0, 0]

T (3)

Finally, it will occasionally be convenient to express v as anangle, the track ν = atan2 (vz, vx).

3.4 Rolling PatternsDifferent patterns of rolling locomotion can be illustratedby combining the 2D topology introduced in Fig. 2 withthe definition of the bottom triangle. Curved paths in Fig. 4represent the evolution of the point at which the downwardprojection of the CoM intersects the vehicle’s convex hull.Pattern availability relates to the precision with which theobjective context cy can be satisfied. Variability of cx, whichaffects the ability to actually execute these patterns, is notillustrated here.

Step-wise motion is depicted as a geometrically consistentopen-loop motion primitive for transitioning from any ∆onto a neighboring Λ face and potentially continuing ontothe next ∆ face before coming to a rest. This categoryencompasses many past results (Hirai and Imuta 2012; Chenet al. 2017; Kim et al. 2015; Vespignani et al. 2018a).The possibility of Λ→ Λ transfers, colored blue, is onlyconsidered by (Zhao et al. 2017; Littlefield et al. 2017).

Next, fixed-axis motion is shown as a smooth periodic paththat winds around the vehicle by alternating between ∆ andΛ faces, always crossing edges at a consistent position andangle. This motion uses feedback to reject small disturbancesin the neighborhood of a 1D manifold of the intrinsic statespace, and is achieved by (Zhang et al. 2017; Luo et al.

2018). Not depicted, (Iscen et al. 2014) achieves similarcontinuous motion over any sequence of alternating ∆ andΛ faces, permitting abrupt turns.

The new rolling paradigm targeted by this work andpreliminarily presented in (Surovik et al. 2018) is illustratedin Fig. 4(c). This smooth “any-axis” motion utilizesunconventional transition geometries that cross edges atarbitrary positions and angles, includes Λ→ Λ transitions,and may even cross directly over nodes. The CoM canthus be moved in an arbitrary direction prescribed by cy ,and its feasible paths densely cover the illustrated domain.This corresponds to usage of a nominal 2D manifoldof the intrinsic state space, subject to perturbations andenvironmental context.

4 Symmetry Reduction

4.1 Dynamically Invariant RemappingThe symmetry properties of a multibody system can beexploited by combining two types of operations:

1. Relabeling: Permutation of the assignment of uniqueIDs to different elements of the same type. Thisrepresents how elements may play the same role atdifferent times, e.g., either a biped’s “left” foot or its“right” foot could be the “forward” foot.

2. Frame Transformation: Expressing the associated vec-tor quantities in a reference frame. E.g., consideringan element “tilting left” to be dynamically identical to“tilting right” after an appropriate reflection.

The combination of these two operations can be used tomatch the actual state of the system (e.g., stepping with leftfoot) to a reference case (stepping with right foot) for whichan appropriate control response is known. Sensor readings ofleft-side elements must be swapped with right-side ones, andframe-dependent values (such as lateral rotation rate of anankle) must also be reflected on the left/right plane.

Denoting the relabeling operator as L and the frametransformation matrix as A, the combined remappingoperation is expressed, with some abuse of notation, as H =AL. If a robot state x consists of vector values qi, then

x =[qT1 ,q

T2 , . . . ,q

TN

]T(4)

Hx =[AqTL(1), Aq

TL(2), . . . , Aq

TL(N)

]T(5)

When the state contains vector quantities associated withnumerous types of elements (e.g., feet and legs), the map



L(i) = i′ will furthermore depend on the element type.For simplicity, this will not be explicitly expressed in thenotation.

The crucial property that lends validity to the use ofremapping operations is invariance of the system’s intrinsicdynamics. Following directly from the invertibility of frametransformations and ID permutations, this is expressed as:

φ (x) = H−1φ (Hx) (6)

This fact simply states that the evolution of the system doesnot fundamentally depend on the choices of the frame orlabels.

With j ∈ J as the relabeling index, there are #J validoperators Hj , equal to the robot’s order of symmetry. Fig. 5sketches the remapping of a trajectory using a constantchoice of j.

x1

xnx(t) ∈ X ⊂ Rn

x′(t) ∈ H1X = X

Figure 5. After a trajectory x(t) is remapped by Hj with aspecific choice of j = 1, it remains valid under the intrinsicdynamics φ and maintains its fundamental characteristics.

4.2 State Volume Reduction via RemappingConsider that the space depicted in Fig. 5 is the space inwhich a symmetric robot operates. Nominally, learning aneffective policy would require experience or demonstrationdata that cover the entire operational space. Selective use ofremapping, however, can decrease the required coverage bya factor equal to the order of symmetry.

This involves the definition of a selection function j (x)that determines which remapping operation will map a givenpoint onto the reduced space XR

Hj(x)x ∈ XR ∀ x ∈ X (7)Vol (XR) = Vol (X ) /#J (8)

with equivalent consequences for the observation space Yand its reduction YR. Figure 6 illustrates this selectiveremapping for reduction of three trajectories. It is then onlynecessary for the data driven process and resulting policy πto cover the smaller regions XR and YR. Control outputsof the policy can simply be inversely mapped via H−1j forsuccessful application on the actual state of the robot, asdepicted by Fig. 7.

4.3 SUPERball State Volume ReductionIn its neutral configuration, where all cable lengths areequivalent, SUPERball is a pseudo-icosahedron. This shapeexhibits 24th-order symmetry, such that there exist 24 uniquespatial transformations that map it back onto itself (Surovikand Bekris 2018). Under general deformations, SUPERballis no longer physically symmetric, and the transformationsdo not preserve shape. They correspond, however, to

(a) (b)

Figure 6. (a) Trajectories shown in raw observation space. (b)Remapping operations chosen with the selection function j (x)map all trajectories into a condensed volume.

y′ ∈ YRHj(y)

πH−1j(y)

φ (x,u)

u′ ∈ UR

y (x) ∈ Y

u ∈ U

(y′)

Figure 7. Block diagram of the use of remapping operations,governed by the selection function, to process feedback within areduced space and apply the output on the original system.

#J = 24 relabeling operations Lj such that relabeling anarbitrary deformed state results in another valid state.

As described in Sec. 4.2, a selection function j (x) isdesired such that Vol (XR) = Vol (X ) /24. The first step inconstructing the selection function is the designation of oneparticular exterior triangle of each type to serve as the bottomtriangle in the reduced space. In other words, Lj must relabelthe actual bottom triangle with ID ∆R or ΛR, dependingupon its type. Rather than uniquely determining j, however,this rule produces a set Jµ of valid options, with cardinality3 when µ is a ∆, and 2 when it is a Λ. This arises from thefact that each of the 8 ∆ triangles has 3rd-order symmetryitself (preserving handedness), while each of the 12 Λ’s has2nd-order symmetry.

Jµ (x) ={j∣∣ µ (Ljx) ∈ {∆R,ΛR}

}(9)

∆

Λ

α

β

γ

β

γ

α

γ

α

β

α

γ

β

α

β

γ

Figure 8. Different choices for relabeling the nodes of thebottom triangle correspond to different world-frame orientationsof the relabeled system.

As shown in Fig. 8, these options correspond to alternateorderings of the nodes that define the bottom triangle,nominally defined in Fig. 3. The specific choice thus affectsthe value of the heading vector h that is computed fromthe remapped data. Numerous definitions of the secondaryreduction rule can be considered. For example, it could



simply be enforced that the γ node of the true trianglemaps onto the γ node of the reference triangle. The lackof a meaningful and consistent relationship between nodeorderings and system behavior, however, means that this rulewould not satisfy Eq. 8.

An amenable definition is most directly motivated by firstconsidering the frame in which the reduced data is to behandled. As specified by Eq. 3, the transformation A definesthe vehicle frame such that h becomes the x-axis. Noting thatthe relabeling option selected from j ∈ Jµ does not affectthe world-frame value of the CoM movement direction v, theminimization of the magnitude of its vehicle-frame angularvalue νR can be applied as the final criterion:

j (x) = argminj∈Jµ(x)

|ν (Hjx) | (10)

Defining the selection function via Eqs. 9 and 10 ensuresthat the desired property of Eq. 8 is satisfied. Figure 9 depictsthe resultant frame for each triangle type, with bounds on νRof ±60◦ for ∆ faces and ±90◦ for Λ. Note that relative toFig. 3, the geometry of the ∆ example has undergone a purerotation while the Λ has undergone rotation and reflection.Yet, the node orderings satisfy the original conventions asthey correspond to entries of Fig. 8 that were generated byapplying valid relabeling operations.

∆ Λ

β

γ

α

α

βγ

v

v

Figure 9. Bottom triangle directional quantities expressed afterapplying a remapping determined by the track-based selectionfunction. Forward and left directions match the frame’s basisvectors. Green arcs indicate the possible range of theremapped track vector.

Ultimately, the frame transformationA is simply a rotationabout the vertical followed, when appropriate, by a lateralreflection. For the case of Λ bottom triangles, the use ofreflection is optional based upon Eq. 10. For the ∆ case,reflection is used if and only if the handedness is oppositethat of ∆R. Not depicted, this would appear as a change inthe bar connection directions indicated by the thick beigelines.

4.4 Illustrative DataTo demonstrate the utility of the symmetry reduction scheme,a dataset corresponding to any-axis rolling in simulation isvisualized. Fig. 10 shows the CoM-relative node positionsbefore and after symmetry reduction, with a different colorfor each bar. The ground plane is not illustrated as its locationrelative to the CoM is variable.

Because the dataset involves the vehicle rotating aboutmany different axes, no meaningful structure is apparentwithin the raw-frame data and all nodes visit all regions

Figure 10. CoM-relative node positions are plotted duringany-axis rolling. Color varies by bar; marker style (visiblezoomed in) distinguishes two nodes of the same bar Rowscorrespond to top, side, and front views. The left column showsraw data, while the center and right columns show remappeddata organized by bottom triangle type ∆ or Λ, respectively.

of the space. After applying remappings Hj(x), each nodeis relegated to a specific neighborhood of the frame. Thesenodes can now be conceived of as playing dedicated roles inthe vehicle’s locomotion, such as “front-left foot” or “backknee”. The corresponding reduction to the operational state-space volume will be leveraged in the following section.

5 Policy Learning

Using the combined state/action vector r =[xT uT

]T,

a trajectory τ =[r0, r1, . . . , rT−1

]is the length-T

sequence generated by φ and πθ for an initial state x0. Let therunning cost l (r) be the performance metric for any giventime step of controlled dynamics. The term Policy Searchrefers to a class of algorithms that calibrate a parameterizedcontrol policy by searching the space of parameter values θ.Under a dataset consisting of N trajectories of length T , thiscorresponds to the objective

θ∗ = argminΣN−1i=0 ΣT−1t=0 l (τi (t)) (11)

5.1 Guided Policy SearchAs the complexity of the control policy architectureincreases, evolutionary algorithms and other black-boxoptimization approaches to discover θ∗ require burdensomeamounts of sample data. Guided Policy Search (GPS) isa technique designed to train an artificial neural networkrepresentation of πθ with only moderate sample complexityyet without requiring a provided differentiable dynamicsmodel (Levine and Abbeel 2014). The method combinesclassical linearization-based control optimization (the C-step) with supervised learning (the S-step). A simple diagramof the approach is shown in Fig. 11.

Key to the sample efficiency of GPS is its exploitationof system gradient information within the C-step, whichcorresponds to the gray region of Fig. 11. In the absence ofa differentiable model, gradients of φ and πθ with respect



Sample Data

Dynamics Fit

ImprovedActions

SupervisedLearning

Optimization

θprev θObservations

States,Actions LOCAL p∗1 p∗2

p∗3

πθ

x1

xn

Figure 11. Left: the flow of data during one iteration of GPS.Right: A depiction of GPS within state space; a single globalpolicy is trained to approximate a set of locally optimizedpolicies, producing similar closed-loop dynamical flow.

to the state-action r are approximated by fitting linear time-varying surrogate models f and p to the sample data:

x′ = f (t, r) = F (t)r + f(t) (12)u = p (t,x) = P (t)x + p(t) (13)

Letting s =[xT uT x′T

]T, fitting a linear dynamics

model for time step t involves first collecting the distributionof s that occurred at sample time t. Using the mean s, theempirical covariance Σ = (s− s) (s− s)

T/N is computed

and the model is obtained by solving for the matrix Fand vector f from the relations Σr,s = FΣr,r and x′ =F r + f , with subscripts of Σ indicating block indexing ofcorresponding dimensions. A similar process is followed forthe policy model using the appropriate dimensions.

With the assumption that l (r) is analytic, it is thenpossible to compute an improved local policy p∗(t,x) viathe iterative Linear Quadratic Gaussian (iLQG) (Tassa et al.2012) using differentiable l(t, r) and f(t, r). Applying thispolicy to the original state sequence produces variationallyimproved actions u∗.

Because both φ and πθ are nonlinear, their surrogatemodels have only limited local validity. Several of thesemodels mi = (fi, pi) may then be necessary for improvingfeedback actions for a large region of the state space.The S-step consists of using a globally accumulated set ofobservations and corresponding locally improved actions,D = {y,u∗}, for supervised training of the policy, i.e.,learning θ. Fig. 11 illustrates this step as the matching ofthe global policy field πθ (x) to a set of local policies p∗i thatfunnel nearby states along a targeted path.

Algorithm 1 outlines the high-level procedure for oneiteration of GPS. For each x0 of a set of initial conditionsX0, the previous policy θprev is executed N times togenerate sample trajectories τk ∈ T . The time varying localmodel m is then fit and used to conduct a backwardpass, updating each u to u∗. These are paired withcorresponding observations in the dataset D = {y,u∗} tosupervise learning of θ.

For cases in which the gradient fit process is a databottleneck, an additional step is added to increase theinformation utility per data point. A global Gaussian MixtureModel (GMM) of the dynamics is fit using at most Nmaxsamples to limit the exponential compute time, assigningeach sample point s a component index. Linear models arefit for each component of the mixture and collected in the setM . Later, this global model is used as a prior when fittingeach local model within the time-indexed set m.

Algorithm 1: GPS Iteration

1 D ← ∅2 foreach x0,i ∈ X0 do3 Ti ← RunSamples(x0, θprev, N )

4 M ← FitGlobalGMM (∪iTi;Nmax)5 foreach x0,i ∈ X0 do6 m← FitLocalModel(Ti, M )7 T ∗ ← LQGBackwardPass(Ti,m, l (r))8 D ← D ∪ GetObservationActionPairs(T ∗)9 θ ← SupervisedLearning(D; θprev)

Convergence is aided by augmenting the cost functionwith a KL-divergence term, l′ (r, p) = l (r) + κdKL (p∗||p).With weight κ, this penalizes the difference between thepolicy p∗, which improves the cost, and the surrogate modelp of the policy that generated the original trajectory.

5.2 Data Decomposition ConsiderationsThe standard GPS pipeline of Alg. 1 relies upon an apriorilabeling of local regions and time indices, implying that:

1. The user should provide a set X0 of initial conditionsthat will each be used numerous times.

2. The horizon T of each sample should be appropriatefor the LQG backward pass.

3. The X0 and T combination should provide sufficientlygeneral experiential coverage of the operationalregime for controller deployment.

These points introduce some limitations that have beenaddressed to varying degrees in subsequent work.

Reset-Free Guided Policy Search (RFGPS) seeks toremove the requirement for repeated use of specific initialconditions (Montgomery et al. 2017). This was motivatedboth to eliminate the manual effort required for precise resetswhen running GPS on physical hardware, as well as forthe potential to improve generalization via stochastic X0.To this end, RFGPS automatically identifies time-varyinglocal regions via an expectation-maximization algorithm thatconsiders the likelihood of each trajectory belonging to eachlocal region. Results indicated that the method improved thegeneralization of policies for manipulation tasks.

Conversely, the original application of GPS to a 6-bartensegrity targeted a periodic behavior, the traversal of 12exterior faces as depicted in Fig. 4(b) (Zhang et al. 2017).This goal corresponds to a duration much too long forvalidity of a linear time-varying model, prompting a schemefor manually decomposing the targeted operational manifold.This was done by defining initial states on each of the 6∆ faces of the pattern, providing the first 6 local regions,and adding an additional local region beginning at eachcorresponding set of stochastic end states. This schemeimplies an output neural net policy that differs from LQRprimarily in terms of the input, a temporally independentobservation. The same scheme was applied in (Luo et al.2018) with a 24-actuated system and the usage of pathintegrals for local surrogate modeling.



As discussed in Sec. 3.4, any-axis tensegrity locomotionimplies the need to track a higher dimensional manifold ofthe state space, and to alter the route along this manifoldbased on the objective context cy . Similarly, deployment inrough terrain corresponds to large influential variations ofthe environmental context cx, meaning that a much broaderneighborhood of the manifold must be accommodated. Thesefactors nominally imply a drastic increase in the amount ofsample data needed and in the difficulty of constructing a setX0 that efficiently provides good experiential coverage.

5.3 GPS For Contextual Nonperiodic MotionA combination of simple changes to the GPS pipeline aredetailed in this section in order to provide the coverageneeded for the newly sought behavioral properties without anexcessive growth in sample requirements or setup effort. Thealtered pipeline, presented in Algorithm 2, will be referredto as GPS-SyNC to indicate the exploitation of Symmetryand the Nonperiodic, Contextual nature of the behavior. Mostcentrally, this implementation employs a global dynamicsGMM as the final set of local dynamics models.

Algorithm 2: GPS-SyNC Iteration

1 M,Dc ← ∅2 T ← RunSamples(X0, θprev)3 T ← SymmetryReduction(T )4 foreach ξ ∈ Ξ do5 Sξ ← FilterPoints(T , ξ)6 M ←M ∪ FitClassGMM(Sξ)

7 foreach τ ∈ GetSubTrajectories(T ) do8 m← PatchModelSeries(τ ,M )9 τ∗ ← LQGBackwardPass(τ , m, l (r))

10 Dc ← Dc ∪ GetObservationActionPairs(τ∗)

11 Dc ← Dc ∪ SymmetryAugmentation(Dc)12 θ ← SupervisedLearning(Dc; θprev)

By relying solely upon the GMM fit, any correspondencebetween the time index of a sample point and itsassociation with a local model is discarded. Without furtheralterations, this choice would imply either diminishedmodel accuracy under reasonable Nmax, or excessivecomputational requirements for the model fit. The symmetryreduction of Sec. 4, incorporated on Line 3 of the algorithm,greatly reduces the volume of the state-action operationalregion that must be covered by the set of local models. Twoadditional changes help further alleviate the nominal issuesof this choice.

A degree of apriori local decomposition can be achievedby exploiting the definition of the symmetry-reduced frames,which induces discontinuities into trajectories. Two criteraare assessed for each sampled dynamics step s: whether thebottom triangle of x is of type ∆ or Λ, and whether or notthe frame reduction operation of x′ is the same as that ofx. These criteria are illustrated in Fig. 12 and are combinedto define four classes ξ, collected in the set Ξ. Classes thatinclude frame discontinuities implicitly account for framechanges within their surrogate models. Note that this occursnot only for a change of bottom triangle (circular emptyand filled markers in Fig. 12) but also when a change in

velocity direction alters the remapping index selection ofEq. 10 (square markers).

∆ Λ ∆ Λ

Figure 12. Left: A discrete trajectory crossing over both typesof triangles. Right: Fully-reduced trajectories with directionconstrained by Eq. 10 to meet the angular bounds shown ingreen. Arrows representing s are colored red if they encounterframe discontinuities, such as when the frame is changed inorder to stay within the angular bounds.

To further alleviate potential issues, a lower-dimensionalsubspace is defined for the dynamics, decreasing thedimension that the models must cover in complement to thevolume reduction. A simple linear mapB : x→ x is appliedwithin FitLocalModels such that m expresses x = f (t, r)

and u = p (t, x), where r =[xT uT

]T. The purpose is

to capture only the primary dynamical influences, so thatthe variational relationship between the control input andthe resulting cost is coarsely but robustly approximated.Selection of B must permit the evaluation l (x), i.e.,observability of the cost. This dimesion reduction decreasesthe number of sample points required per local model, andthe computational burden of the GMM fit.

Local optimization requires dynamics gradients, providedfrom the set M of surrogate models, and cost gradients,obtainable analytically for each trajectory. The backwardpass horizon Tsub is moderated by dividing each trajectoryinto sub-trajectories τ = τ (tstart ≤ t < tend) in GetSub-Trajectories of Alg. 2 such that tend − tstart = Tsub. Thisprocess keeps the horizon short enough to avoid excessiveaccumulation of linearization error, but long enough tosmooth out behavior across transitions. Unlike prior work,each sampled data point is updated by a unique backwardpass with its own cost expansion, rather than one pass beingused for all points contained within a local region.

Within the S-step, the training dataset becomes Dc ={y,u∗; cy} due to the contextual nature of the policy. Afinal consideration addresses the fact that during deployment,selection of the appropriate symmetry map index jdepends upon limited and noisy sensor data. This warrantsexploitation of the fact that any training point can be triviallyduplicated to an alternate frame via:

Hj′H−1j (y,u∗; cy) (14)

which fills expanded bounds of the subvolume represented inFig. 6(b) with redundant data, rather than corresponding toany loss of data density. One option is to duplicate all pointsinto all frames; however, this could require a large increasein the size of the network and the related training effort.Instead, it is desired to determine a limited selection of {j′}per sample point to provide a smaller expansion motivatedby the most likely errors.

Recalling that the two steps of the nominal selectionj (x) depend upon µ and v, two duplication strategies aredefined. By using j′ ∈ Jµ \ j, i.e., removing the criterion



of Eq. 10, SameTri Duplication accounts for the fact thatprecise detection of the footprint-relative v from sensor datais not feasible. To furthermore account for error in detectionof µ, NeighborTri Duplication applies j′ ∈ Jµ′ where µ′ isthe triangle neighboring µ whose face center has the shortestground-plane distance to the CoM.

6 Setup and Preliminaries

6.1 ObjectiveThe running cost used for optimization within Sec. 5 isa simple quadratic tracking error for the center-of-massvelocity vCoM :

l (r) = (vCoM (x)− v∗g)TW (vCoM (x)− v∗g) (15)

with g representing the intended direction of motion andv∗ the desired speed. The weight matrix W penalizes bothground-plane components of velocity error equally in orderto discourage excessive lateral movement. Vertical velocityerror is de-weighted so as not to overly constrain the mannerof forward motion, especially in cases with uneven terrain.

As implied by Sec. 5.3, the cost must be observablefrom the dimensionally reduced state x. Furthermore, sincesurrogate models are fit within the symmetry-reduced frame,the direction g is transformed into that frame via Hj .This direction corresponds to the objective context, cy =atan2 (gz, gx). The training cost of one iteration of GPS issimply the running cost l averaged across all samples andtime steps, i.e., the argument minimized in the definition ofpolicy search in Eq. 11.

For better management of the influence of environmentand objective context during training and testing, thedirection g is governed by a waypoint system. A series ofwaypoints are defined with locations q∗k, such that g(k) =q∗k − xCoM is their vehicle-relative displacement. With ‖g‖as the distance to the waypoint, g = g/‖g‖ and satisfactionof the criterion ‖g‖ < b/4 is a “hit” of the waypoint, utilizingbar length b. If the timeout duration δt∗ elapses since themost recent waypoint hit, the current waypoint is marked“missed”. In either case, the next waypoint k + 1 is activated.

Waypoint sequences {q∗k} are generated as randommeandering paths with distance b/2 between the centersof neighboring waypoints. This serves as an approximatepath-following criterion in which deviation is penalized onlyimplicitly through l. During training, the formulation ensuresexposure to a large variety of (cx, cy), e.g., traversing inmany directions over each of many terrain features. Interms of testing, it demonstrates what could be achievedin combination with a higher level planner acting to avoidobstacles or harsher terrain than is present on the chosenpath.

6.2 Test CasesController performance is evaluated on a variety of systemparameterizations listed in Table 2. A key factor in systemcontrollability and dynamism is the ratio of active forcesto passive ones. This is investigated by altering both themaximum rate of cable length actuation and the accelerationdue to gravity. The Hardware case denotes the currentavailable physical testbed, with Earth gravity and a cable

Gravity Cable Rate TestbedHardware 9.8 m/s2 8 cm/s SBv2 Prototype

Normal 9.8 m/s2 8 cm/s NTRT SimulatorFast 9.8 m/s2 30 cm/s NTRT Simulator

LowGrav 2.0 m/s2 8 cm/s NTRT SimulatorLGFast† 2.0 m/s2 30 cm/s NTRT Simulator

Table 2. Named system parameterizations. †used only intraining.

actuation speed of roughly 8 cm/s under tension. Theremaining variants are executed in simulation, beginningwith Normal which seeks to match the hardware parameters.In contrast, Fast is a more quickly actuated system thatrepresents what might be achieved with mechanical designparameters considered in earlier studies. Finally, LowGravrepresents current hardware deployed on a generic spaceexploration destination with a gravitational value less thanthat of Mars and greater than those of Earth’s Moon andthe Saturnian moon Titan. This also increases the relativeinfluence of elastic effects.

The baseline Flat testing environment is depicted inFig. 13 along with a random waypoint path. In this simpleenvironment, cx varies only with the vertical positioning ofthe vehicle; the primary intent is to assess basic mobilitydirected by cy .

Figure 13. The Flat environment, in which a basic variety ofcontact states are possible. White cylinders, which do not havecollision properties, indicate a random path of waypoint regions.

Of great interest to tensegrity deployment, and in contrastto most prior work, very rough terrain is considered in theRough environment of Fig. 14. Computing the maximalslope on a per-facet basis, the mean across all facets is 44◦

with standard deviation 16◦. Hills and troughs exist on lengthscales similar to the size of SBv2, with frequent heightchanges on the order of b/2 and as high as b in extreme cases.As a result, extreme variation of cx is encountered duringpolicy search.

Finally, uphill locomotion is tested in the Ascentenvironment, which features a substantial incline from theset {20◦, 25◦, 30◦} and significant roughness at a scalesmaller than the size of SBv2. This environment shown inFig. 15 using a slope of 25◦. While the roughness impartssome additional challenge due to irregularity, it also permitssome benefits in terms of traction. Although the physicalprototype of SBv2 features compliant “softball” elementsat the ends of its bars, which may provide good grip, thesimulation testbed of NTRT does not feature compliantcontact modeling. This causes numeric instability in contactdetection when multiple nodes are in single-point contact,permitting slippage regardless of the friction coefficient.Thus, ascent of smooth slopes in simulation may face unduelimitations that are not related to the policy search method.



Figure 14. The Rough environment, which provides a vast increase in the set of possible contact states. Left: a random waypointpath. Right: ground-level view revealing height variations comparable to one bar length.

Figure 15. The Ascent environment, which requires largedeformations in order to achieve uphill motion based on CoMdisplacement. The pictured instance utilizes a mean slope of25◦.

In all cases, initial conditions X0 feature randomizationof their deformation within the operational actuation boundsas well as of their orientation and location within theenvironment. For Ascent, initial locations are restricted to alinear set at the bottom of the slope. When random waypointpaths are used, they begin at the vehicle’s initial location.

6.3 HyperparametersThe GPS method involves a large number of hyperparame-ters, including many that are not explicitly detailed in Sec. 5.This section will briefly discuss several hyperparameterselections which, in the interest of scope, are not extensivelyanalyzed within this work.

Given the alterations of GPS-SyNC, sample complexityis primarily driven by the total count of s per iteration,equal to NT . The Flat scenario utilizes N = 100, T =250 while Rough uses N = 150, T = 1000. Values ofbackward pass length Tsub above 5 were found to producegood performance without much variability; Tsub = 20 wasselected. Discretization occurs at 10hz, the control frequencythat was feasible on SBv1. Total runtimes on a desktopworkstation, using parallelization of sampling operationsacross 16 cores, were about 20 minutes for Flat and 3 hoursfor Rough. No individual step of Alg. 2 dominated thecomputational cost.

Smooth control noise with amplitude of about 0.08b wasadded to sample rollouts during training in order to induce

exploration; values above 0.15b or below 0.05b degradedperformance. A random scaling between 0 and 1 was appliedto this value on a per-sample basis in order to strike a balanceof exploration and exploitation. The target speed was setas v∗ = 80cm/s — so long as this value was higher thanachievable by the system, varying it had little effect on theoutcome. Very low values caused poor performance of thepolicy search, suggesting difficulty in discovering behaviorsthat arrest forward momentum.

Observations y include the bottom triangle type alongwith symmetry-reduced values of the cable lengths, barangular velocities, and the angular position of the nextwaypoint as cy . The architecture of πθ is a fully connectednetwork of 4 hidden layers with 128 neurons each,using ReLU activation functions and trained with Adamoptimization. Training with the use of SameTri Duplicationslightly improved convergence, despite the fact that theobservation regions containing the duplicated data fromEq. 14 are not witnessed in simulation (where j is uniquelydetermined). The effect could not be replicated by simplyincreasing the amount of training done on non-duplicateddata alone, suggesting that the alternate views of a givenfeedback example may have eased the learning of theunderlying principle.

6.4 State Dimension SelectionThe state dimension reduction map B described in Sec. 5.3was chosen from a small set of options that simply extracta subset of state values. Each option included the vCoM toensure compatibility with the gradient of the cost of Eq. 15during the backward pass. Remaining contents were drawnfrom the following categories:

1. CoM-relative positions of the three nodes of µ, givingbasic static stability and contact information

2. CoM-relative positions of the lower six nodes,increasing such information

3. CoM-relative positions of all 12 nodes, additionallydetermining the vehicle shape

4. Cable rest lengths, which express tension if the shapeis known

5. Velocities of all 12 nodes

A baseline case utilizing the full state (node positions, nodevelocities, and cable rest lengths) resulted in memory issuesdue to excessive dimensionality.



Figure 16 plots average sample cost versus iteration countfor several subspace definitions evaluated in the Roughenvironment. When modeling the cost state alone, vCoM ,GPS-SyNC could merely improve the initial pure-noisepolicy to a motionless policy. The inclusion of node posi-tions (corresponding to “nodes” in the graph) providedstrong performance, converging to low value indicatingeffective locomotion. Additionally including their velocities(“nodes+vels”) marginally increased cost, signaling dimin-ishing returns in the trade off of additional dimensionalityincurred for additional information. When instead incor-porating the rest lengths of the cables (“nodes+cables”)performance dropped significantly, suggesting that tensionaleffects are highly sensitive and would require much narrowerlocal regions.

Figure 16. Average sample cost versus policy search iterationfor different choices of surrogate model subspace, averagedover 3 experiments each.

Not shown, omitting the upper six node positions onlyslightly increased the cost, highlighting the importanceof interface geometry over that of the general shape.Furthermore, in a separate evaluation on Flat, retaining justthe bottom three node positions allowed for only moderatelydegraded performance to be obtained at a low sample count.

6.5 Sample ComplexityDue to fundamental differences between periodic andnonperiodic controllers, as well as variations betweenparticular implementations of 6-bar tensegrities, directquantitative comparison to past work is not possible. As acoarse comparison point with prior use of GPS (Zhang et al.2017; Luo et al. 2018), an attempt was made to produce any-axis CoM movement on the Flat environment while omittingsymmetry reduction and using the classic decompositionbased upon consistent initial-conditions and timing as inAlgorithm 1.

Beginning at rest upon a particular ∆ triangle with afixed orientation, the commanded direction of motion wasdiscretized into J = 36 values, corresponding to the setof local surrogate models mj . For each direction, N = 50samples of equal length T = 16 are collected. After I = 10GPS iterations, the controller was able to smoothly move theCoM in commanded directions relative to the initial bottomtriangle for one or two transitions — a result of limited scope

for a tuned sampling cost of IJNT = 288, 000 total timesteps.

This already compares unfavorably to the samplecomplexity INT = 125, 000 required to produce good any-axis rolling via GPS-SyNC. Furthermore, an increase bya factor of 20 (the total number of triangles) would benecessary for the standard GPS pipeline to cover all transitioncases. Such issues would only be compounded when trainingfor rough terrain, requiring another discretization dimensionfor K types of terrain features to cover the space ofpossible cx, with accompanying setup effort. These findingsmotivate the modifications for nonperiodicity implementedin GPS-SyNC, which complement the use of symmetryreduction while more naturally and efficiently covering thestate space with long sample trajectories.

6.6 Baseline ControllerAnother appropriate comparison is the difference in behaviorachievable in the any-axis rolling paradigm relative tothe previous paradigms described in Sec. 3.4. In earlierwork (Surovik et al. 2018), an attempt to train a neuralnetwork policy with a model-based supervisor yielded anoteworthy result: an undertrained network produced aconstant output, i.e., πθ(·) = u. This policy represents asingle shape that is a broad average of all shapes experiencedby the locomoting model-based controller. Embodying thisshape causes a transition between support faces, triggeringa change in the raw control u = H−1j u via the selectionfunction j (x).

The resulting sustained motion, seen in Fig. 17, resemblesthe ∆→ Λ→ ∆ step-wise paradigm and has good dynamicproperties when executed continuously. As such, it willbe referred to as the QuickStep controller and usedas a strong representative of conventional non-adaptivelocomotion paradigms. Comparison will be limited to theFlat environment as the controller fails elsewhere.

7 Simulation ResultsThis section first assesses the performance of the controlpolicies in terms of waypoint path following and smoothnessof motion. Further analyses then provide closer insights intothe adaptive nature of the policies’ behavior.

7.1 Performance on Flat GroundAverage performance metrics on Flat for various combina-tions of controller type, training system, and testing systemare provided in Table 3. Fig. 18 then provides typical CoMground-tracks for selected controllers as they attempt tofollow a path of waypoints. Video of these four trials isavailable in Extension 1.

It is immediately apparent that the QuickStep controlleris insufficient for following waypoints of the selected size.In Fig. 18(a), chatter occurs at multiple locations where thedirection of motion required for closely following the pathis not available on the controller as it does not include aΛ→ Λ option. After the waypoint times out, the updatedtarget direction may permit motion to proceed — potentiallycrossing the missed waypoint after deviating from the path.

The GPS-SyNC controller, by contrast, is able to achievea much higher degree of precision. Fig. 18(b) shows



Figure 17. A sequence of states as the QuickStep control policy locomotes SBv2 to the right on the Flat environment.

Type Train Sys Test Sys Hits Misses Forward Speed Lateral Speed Λ→ Λ Fig. 18QuickStep Fast Normal 14.0 ± 3.4 5.5 ± 1.5 14.4 ± 18.0 0 ± 25.3 0.0%GPS-SyNC Fast Normal 28.7 ± 2.8 0.5 ± 0.7 23.8 ± 17.6 0 ± 20.1 4.1%GPS-SyNC Normal Normal 25.1 ± 3.6 1.8 ± 1.2 21.0 ± 18.9 0 ± 24.0 0.0% (b)QuickStep Fast Fast 21.9 ± 3.8 4.8 ± 1.1 20.2 ± 38.6 0 ± 51.6 0.0% (a)GPS-SyNC Fast Fast 64.7 ± 5.2 0.1 ± 0.8 53.1 ± 25.8 0 ± 20.4 7.0% (c)GPS-SyNC LowGrav LowGrav 18.8 ± 3.8 2.6 ± 1.8 16.9 ± 14.1 0 ± 18.1 6.3% (d)GPS-SyNC LGFast LowGrav 19.8 ± 3.5 2.1 ± 1.4 17.5 ± 14.8 0 ± 18.3 7.6%GPS-SyNC Fast LowGrav 21.1 ± 2.7 1.1 ± 1.2 18.1 ± 14.2 0 ± 16.2 11.4%

Table 3. Performance in Flat environment averaged over 64 trials of 100s duration, with 1σ uncertainty. Speeds are given in cm/s.By enabling any-axis motion (Fig. 4(c)) via use of GPS-SyNC, higher speed and waypoint counts are achievable than with smoothstep-wise motion (Fig. 4(a) and (b) combined within QuickStep). Actuation speed and gravity strength both correlate withlocomotion speed. Performance is slightly increased when the training system is more capable than the testing system.

(a)

(b)

(c)

(d)

Figure 18. Ground track of the CoM while tracing ameandering sequence of waypoints for 100s. Empty circlesindicate unmet or timed-out waypoints. The limited directionaloptions of QuickStep impede its progress. GPS-SyNC policiesachieve accurate path tracing on all systems. Fast continues farout of frame (see Table 3), while LowGrav is slowed by thereduced pace of both actuation and unstable motion.

reasonable success with Normal system parameters eventhough the controller did not discover the possibility of Λ→

Λ transfers, as reported in Table 3. The added versatility inchoosing how to cross the same edge options is sufficient fora substantial performance improvement. Performance on theFast system shows a smooth and precise path that is tracedquickly. Meanwhile, a controller trained on Fast was the bestperformer on Normal, likely due to the fact that Λ→ Λwas discovered. In addition to the benefits of a faster rateof exploration, this may imply that the momentum attainableby Fast effectively inflated the set of commands that achievesuch transitions; after improving such behaviors, they maybe robust enough to succeed even with reduced momentum.

Finally, the LowGrav system proved the most challengingto control due to the increase in the role of elastic energyrelative to that of gravity. Motion was much slower due toboth actuation and gravitation being weak, and the distancetraveled was comparatively short. Nonetheless, consistentprogress was achieved with reasonable precision. Trainingon more capable systems appeared to provide only amarginal improvement in performance, suggesting that thelimits here arise mainly due to vehicle capability rather thanfrom the role of exploration.

Figure 19. Snapshots of GPS-SyNC controller on Normalsystem traversing Rough environment to the right. A moderateamount of rotation in place occurred on the steepest featuredue to slippage.



Figure 20. Example groundtracks for GPS-SyNC controller on Rough environment. Left, center, and right plots correspond toNormal, Fast, and LowGrav systems. Performance above, at, below, and far below average is indicated by line colors green, blue,magenta, and red.

Type Train Sys Test Sys Hits Misses Forward Speed Lateral Speed Λ→ Λ Fig. 20GPS-SyNC Normal Normal 9.5 ± 3.8 1.1 ± 1.0 6.9 ± 15.4 0 ± 18.0 0.3%GPS-SyNC Fast Normal 10.5 ± 3.9 0.8 ± 0.9 8.8 ± 16.2 0 ± 17.8 0.3% (a)GPS-SyNC Fast Fast 25.8 ± 8.8 0.5 ± 0.9 20.0 ± 29.1 0 ± 32.1 0.3% (b)GPS-SyNC LowGrav LowGrav 7.1 ± 3.2 1.2 ± 1.0 6.1 ± 14.7 0 ± 15.2 3.3% (c)GPS-SyNC LGFast LowGrav 8.4 ± 3.5 1.0 ± 1.0 7.1 ± 13.7 0 ± 13.2 5.6%GPS-SyNC Fast LowGrav 8.9 ± 3.5 1.0 ± 0.9 7.1 ± 13.9 0 ± 16.2 0.9%

Table 4. Performance in Rough environment averaged over 64 trials of 100s duration, with 1σ uncertainty. Speeds are in cm/s. TheQuickStep controller cannot locomote in this environment and is omitted. Trends for GPS-SyNC remain unchanged: actuationspeed and gravity strength increase performance, and training on a more capable system than the test system is slightly beneficial.

7.2 Performance on Rough TerrainSuccessful locomotion across a challenging feature ofthe Rough environment is depicted in Fig. 19. Twodemonstrative path-following trials can be viewed inExtension 1. Aggregate performance statistics are providedin Table 4. The QuickStep controller is omitted as it is notcapable of traversing significantly uneven ground.

In comparison to Flat, performance on Rough isdiminished by a factor of about 2.5 due to the increasedchallenge of the terrain. Relative variations between systems,however, tell a similar story. Fast remains the most capableand LowGrav the least, and controllers trained on morecapable systems than they were tested on again show a smallperformance advantage. Instances of Λ→ Λ transitions wererare other than in the LowGrav case. This may largely bedue to the fact that the virtual edges of the vehicle are usuallyrelatively short, and are easily further reduced when their twonodes are inside a concave terrain feature.

Figure 20 gives groundtracks for multiple trials witheach of the three simulated systems. These examples areselected to represent the full range of performance for eachsystem, including some of the least successful trials. Allthree systems successfully track the desired paths in mostcases, with overall speed and smoothness varying dependingupon the challenge of the terrain feature relative to thevehicle’s orientation. Occasional significant deviations fromthe path reveal limitations in the controllers’ capability toarrest unwanted momentum, especially at steep drops.

In some cases, such as the red trajectories for Normal andLowGrav or the magenta trajectory for Fast, a protractedamount of time will be spent at a challenging uphill feature

before it is eventually overcome. These instances generallyinclude a combination of two phenomena: slippage, in whichthe vehicle rotates in place due to insufficient traction, andstuck states, where the controller is locked in an outputthat does not produce motion in the current environmentcontext cx. The former case may spontaneously resolve whenan advantageous conact configuration eventually arises. Thelatter case may resolve due to a combination of low-levelactuator noise and elastic effects perturbing the state, oralternately when waypoint expiration perturbs the objectivecontext cy .

7.3 Performance on Slope AscentFinally, performance on Ascent is examined. Controllerstrained directly on this environment did not learn successfuluphill locomotion, likely due to the high degree of riskassociated with exploration and the relatively narrow scopeof behavioral options that achieve progress. In contrast, theRough environment contains features of widely varyingdifficulty and has more limited consequences to unstablemotion in the wrong direction. As such, controllers trained onRough were found to be the most viable in testing on Ascent.This implies that in a sense Rough functioned similarlyto curriculum training. It also ensured that a broad varietyof experiences were maintained throughout the trainingprogress rather than a narrowing of scope around the nominalbehavior. Snapshots of a successful trial on Ascent are shownin Fig. 21, with video footage available in Extension 1.

Aggregate performance is analyzed using a sequence of 6waypoints that advance directly up the slope, with randomoffsets in the lateral direction. Mean inclinations of 20◦, 25◦,



Figure 21. Snapshots of GPS-SyNC controller on Normalsystem traversing the rough 25-degree slope of the Ascentenvironment to the right.

and 30◦ were tested. Fig. 22 provides the occurrence ratesof the final waypoint count over 64 trials, showing that allsystems perform well on the 20◦ incline. At 25◦, the Fastsystem successfully ascends the slope just above 80% of thetime, while the less capable systems succeed less than half ofthe time. Fast was also tested on a 30◦ incline, reaching thehalfway point on 75% of attempts and succeeding on only20%.

Figure 22. Occurrence rate of total waypoint counts for 64 trialson Ascent. Left: on a mean incline of 20◦, all systems achievefull ascent on most trials. Right: on a 25◦ incline, the Fastsystem retains a high success rate while the less capablesystems more commonly get stuck or tumble backward.

7.4 Examining Adaptive TraitsSeveral analyses are available for revealing the adaptivenature of the GPS-SyNC control policies applied above.On the Flat environment, it was seen that the restrictiveoptions for transition geometries in the QuickStep controllerlimited the precision of its directional control. An exampleof highly variable transitions achieved with GPS-SyNC isplotted in Fig. 23, which shows the geometry of the bottomtriangle in addition to the CoM path. For many transitions,the CoM shadow passes close or almost directly over aground-contacting node and at variable angles relative to thecrossed edge, indicating the any-axis locomotion paradigmdepicted in Fig. 4(c).

A larger-scale analysis of this trait is given in Fig. 24,which also provides definitions for two geometric measuresof a transition. The modest distributions of the QuickStepcontroller indicate variation due to the interaction of passivedynamics with the non steady-state nature of the motion

Figure 23. Top-down view of the path of the center-of-massand the positions of the nodes of the bottom triangle, generatedby the GPS-SyNC controller on the Fast system and Flatenvironment. Black lines and orange lines indicate cables andvirtual edges, respectively.

(a) (b)

(c) (d)

Figure 24. Bottom triangle transition measurements collectedby GPS-SyNC controller on Fast system in Flat environment.(a) Definition of position and angle values as the center of masspasses over an edge of the convex hull. Occurrences of thesevalues are plotted for each transition type: (b) Λ → Λ, which isnot available to QuickStep, (c) Λ → ∆, and (d) ∆ → Λ.

that occurs due to turning. Distributions of these values forthe GPS-SyNC controller are far broader, suggesting theactive use of a wide swath of state space for adapting to theobjective context cy .

Adaptiveness to environmental variation cx can bevisualized by plotting the locations of the bottom trianglenodes during locomotion as in Fig. 26. In the baselinecase of motion on Flat, the CoM experiences only mildvertical movement relative to the bottom nodes. The Roughenvironment shows not only vertical variation of the entirevehicle along the path, but major changes in bottom nodeheight relative to the CoM on laterally varying terrainfeatures. Similar traits are seen on Ascent, with longer nodeposition paths corresponding to slippage seen primarily inthe latter half of the trajectory.

A final illustration of adaptiveness, shown in Fig. 25,assesses variation of the vehicle shape. Because these dataare symmetry-reduced, each cable ID corresponds to a



Figure 25. Heat maps of commanded cable rest lengths, as a fraction of bar length b, after symmetry reduction. Left:QuickStep-Flat, using a relatively small region of the control space. Center: GPS-SyNC-Flat using a greatly expanded region.Right: GPS-SyNC-Rough using the broadest range of shapes.

Figure 26. Heights of the CoM and of the three nodes of thebottom triangle as the Normal system locomotes with theGPS-SyNC controllers. Top: Flat with little variation in bottomnode height relative to the CoM. Middle: Rough with substantialvariation. Bottom: Ascent with indications of slippage in the righthalf of the dataset.

specific role, and no effect is introduced by the fact that agiven cable may play several roles as the vehicle reorientsitself. Within the QuickStep plot, the consistent instructionsper role are apparent as hot spots on the spectrum of possiblecable lengths. Moderate variation occurs on some entriesas locomotion reallocates roles among cables. Conversely,during any-axis motion by the GPS-SyNC controller, eachrole exercises a much broader set of possibilities. Thisis exaggerated even further during locomotion on roughterrain.

8 Hardware Results

This section describes the deployment of selectedGPS-SyNC policies, trained in simulation, on the hardwareprototype of the SBv2 robot (Vespignani et al. 2018b).Hardware trials were executed at the NASA AmesRoverscape facility on two types of environment: flatground, and a relatively smooth 15◦ slope of packed dirt.Although a tether was utilized for convenience, the vehicleis capable of untethered operation. Analysis was conductedprimarily via video footage.

8.1 Policy TransferAs described in Sec. 3, the policy output u is a set ofdesired cable rest lengths, which motorized spools attemptto satisfy via PID control. A linear first-order model wasused to approximate the relationship between spool rotationand cable rest length, which was assumed to be identical forall cables. This model was tuned by an expert user in orderto account for possible differences in cable elasticity andattachment geometry relative to the simulated system. Themodel does not account for the fact that the spooling ratevaries as layers of cable accumulate on the spool.

Detection of the bottom triangle µ could not be directlycomputed due to the lack of positional sensing data. Instead,it was identified using a k-nearest-neighbors classifier trainedon the data from accelerometers contained within the nodes.This led to occasional discrepancies between simulation andhardware values of µ, which initially resulted in controlchatter and stuck states. The NeighborTri Duplicationtraining strategy was developed to resolve this and succeededin simulation using synthetic error on µ; however, it failed onhardware, likely due to interaction with other “reality gap”effects. Instead, the output of the µ detector was not accepteduntil the same value was reported consistently for a few timesteps in a row, resolving the chatter phenomenon.

The present hardware evaluation setup can extractpositional data from the external overhead video recordings,but these data are not available in real time and are notincorporated into the feedback process. Consequently, therobot lacks a direct way to sense its direction of motion ν andcannot apply the final step of the selection function, Eq. 10.By training with the SameTri Duplication strategy, anyremapping index from Jµ will result in an equivalent actionon hardware and ν is not needed. The relative target directiong, however, is also unavailable. To avoid significant issues, aconstant value of g was used rather than a waypoint system.The vehicle-frame value was approximated by assuming thatthe vehicle orientation did not change via deformation andthat transitions imparted a fixed-value change. This valuewas based on rigid rotation of the neutral vehicle shape, withthe rotation edge inferred from the history of µ.

8.2 ResultsDuring execution of trials for actuator calibration, thevariable types of transitions generated by the GPS-SyNCpolicy were apparent. One such example in Fig. 27 showsmovement of the CoM directly over a lone contacting node,an any-axis locomotive behavior (see Fig. 4(c) and Fig. 23).This action was observed in multiple trials.



Figure 27. The hardware prototype SBv2 using GPS-SyNCtrained on Flat transitions over a single node, much like theorange “any-axis” trajectory of Fig. 4(c) and the last threeevents indicated in Fig. 23.

After calibration, five longer trials of the Flat policywere executed on flat ground and recorded from an elevatedperspective. Each trial lasted roughly 60 to 75 seconds andtraveled about 8 meters using around 20 transitions. Thebest-case trial matched the average simualted speed of 20cm/s, and long sub-sequences of other trials also met orexceeded this value. The overall average speed, however,was reduced to 15 cm/s by occasional instances of delayedtransitions. These cases suggested that the shape embodiedon hardware did not displace the center-of-mass as muchas the same command issued in simulation, suggesting thatpolicy transfer may benefit from further calibration of thesoftware model and hardware actuation.

Figure 28 shows a top-down perspective of the progressionof ground-contacting node positions for one of the hardwaretrials, whose raw footage is also available in Extension 1.Positions are extracted based on a 5-meter square grid onthe ground, which also aids image transformation. The CoMpath is omitted as it cannot be reliably determined via thisimage processing. Moderate drift in the direction of motionoccurs due to the approximated tracking of the forwarddirection. In several occasions, additional nodes contact theground other than those of the bottom triangle. The exhibitedtransitions mostly follow the standard ∆→ Λ→ ∆ despitesignificant occurrence of non-standard transitions in shorttrials. This can likely be attributed to the necessary use ofa constant commanded direction; similar trials in simulationtypically settled into this pattern as well.

Lastly, a policy trained on the Rough environment wasexecuted on hardware to ascend a 15◦ slope. Footage of onesuch trial can be viewed in Extension 1, while snapshotsof another are shown as a composite image in Fig. 29.These reveal that the robot successfully ascends the slopeand frequently experiences highly deformed configurationsin doing so, along with occasional slippage of one or twonodes of the bottom triangle. This is the first demonstrationof incline ascent on SUPERball hardware and, to the authors’knowledge, the second demonstration on any 6-bar hardwareprototype after (Chen et al. 2017).

Figure 28. An elevated perspective of a hardware trial isspatially transformed into an overhead perspective. Coloredpoints indicate the detected locations of nodes that touch theground.

9 DiscussionThis research effort has produced highly adaptive feedbacklocomotion policies for a six-bar tensegrity rover via severalmodifications to the Guided Policy Search method. Incontrast to the specialized transient or periodic behaviorsof past work, contextual and sustained nonperiodic behaviorwas learned.

These outcomes were enabled by dropping the standardreliance upon an apriori decomposition of data into localregions. Computational feasibility of that change dependedon two reductions of the state description. Symmetryreduction greatly decreased the state volume without lossof information, while dimensionality reduction droppedless fundamental state components such that surrogatemodels were coarse but more broadly valid. The GaussianMixture Model decomposed data while accounting forcontextual influences, such as contact modes, which areleast amenable to gradient descriptions. Although theimplemented reductions were specific to the vehicle,similar steps could be taken for other high-dimensional,underactuated soft robots, which may also exhibit significantsymmetry and dominant dynamical modes.

9.1 Further ConsiderationsEvaluations on a variety of environments and systemparameterizations revealed the overall success of the methodwhile also indicating some limitations. Adaptive behaviors,such as rolling on an arbitrary axis, traversing roughterrain, and ascending a steep slope had strong performancewhen granted fast actuation. Although slower locomotionis inevitable on the more slowly actuated system, its pathsmoothness suffered even on the flat environment. There,unstable motion cannot be outpaced by actuation but mightbe avoided by, e.g., lowering the forward node beforeshifting weight onto it. Such possibilities might requireunrealistic effort and accuracy to discover via gradientupdates. Reasoning about a deliberate change of contactmodes within this paradigm, furthermore, would only bepossible through painstaking reward shaping.

The final recourse for uncovering subtle behavioralopportunities is increased exploration via control noise or, assuggested in the analysis, by exaggerating the capabilities ofthe vehicle. Due to the high dimensionality of the control,this remains demanding in terms of data, especially invery unstable regimes, such as the neighborhood of Λ→



Figure 29. A composite of three snapshots as the hardware prototype SBv2 locomotes up a 15-degree slope using theGPS-SyNC controller trained on the Rough simulation environment.

Λ transitions. The incorporated state reduction changes donot eliminate this issue as it is still desired to fully exploitthe available control space. Control reduction via imposedcouplings, such as Central Pattern Generators, could lessendemands but would require careful design to minimize lossof useful capabilities. Lastly, the incorporation of newlydiscovered behaviors into the updated policy would benefitfrom explicit reasoning about which data should be includedin the training set, i.e., experience selection (De Bruin et al.2018).

9.2 Reality GapHardware trials indicated a good but imperfect transfer ofbehavior from simulation to reality after moderate calibrationeffort. The reality gap could likely be further narrowed withadditional effort upon physical parameter calibration for thesimulated system, a topic of ongoing investigation (Zhuet al. 2018), as well as more detailed model formulationsfor compliant contact and cable spooling. One appealingpossibility would be to execute the last few iterations ofthe policy search using samples taken from hardware trials.This approach, however, would require additional externalsensing to provide adequate data for fitting surrogate modelsin the policy search C-Step, as well as a few hours worth ofhardware operation.

Other shortcomings, such as the need for the vehicle-relative target direction as a policy input, could only beappropriately solved by integration with additional onboardsensors. Environmental localization of this sort would likelybe provided by the integrated scientific payload on anactual deployed mission. The intermittent incorporation ofpositional data from such sources could also be used toimprove the quality of other state information available tothe policy.

Some aspects of modeling and hardware design, suchas the wear and tear induced upon cables, would requirefurther improvement before deploying hardware on veryrough terrain. Notably, the simulation environment utilizeddoes not account for interactions between cables and convexterrain features. In reality, this would increase the tensileforce between connected nodes beyond what would beexpected based on their linear separation. Occurrence of suchcases might have poor observability through limited sensing

data. This phenomenon might be overcome through purerobustness of the locomotive strategy, or otherwise throughthe incorporation of additional sensors to directly reporttensile and contact forces.

FundingThis work was supported by a NASA Early CareerFaculty grant awarded to Kostas E. Bekris (grant numberNNX15AU47G). Hardware development and operation byMassimo Vespignani and Jonathan Bruce was supported byNASA Space Technology Research Fellowship (NSTRF),NASA Ames Center Innovation Fund (CIF), and UniversitiesSpace Research Association Award (collectively undercontract number NNA16BD14C).

References

Andrychowicz M, Wolski F, Ray A, Schneider J, Fong R,Welinder P, McGrew B, Tobin J, Abbeel P and ZarembaW (2017) Hindsight Experience Replay. In: Advancesin Neural Information Processing Systems 30 (NeurIPS).pp. 5048–5058.

Best CM, Gillespie MT, Hyatt P, Rupert L, Sherrod V andKillpack MD (2016) A New Soft Robot Control Method:Using Model Predictive Control for a PneumaticallyActuated Humanoid. IEEE Robotics AutomationMagazine 23(3): 75–84.

Bhattacharya S and Agrawal SK (2000) Spherical rollingrobot: a design and motion planning studies. IEEETransactions on Robotics and Automation 16(6): 835–839.

Calisti M, Picardi G and Laschi C (2017) Fundamentals ofsoft robot locomotion. Journal of The Royal SocietyInterface 14(130): 20170101.

Caluwaerts K, Despraz J, Iscen A, Sabelhaus AP, Bruce J,Schrauwen B and SunSpiral V (2014) Design and controlof compliant tensegrity robots through simulation andhardware validation. Journal of the Royal SocietyInterface 11(98): 20140520.

Caluwaerts K, D’Haene M, Verstraeten D and Schrauwen B(2013) Locomotion without a brain: physical reservoircomputing in tensegrity structures. Artificial life 19(1):35–66.



Cera B and Agogino AM (2018) Multi-cable rollinglocomotion with spherical tensegrities using modelpredictive control and deep learning. In: IEEE/RSJInternational Conference on Intelligent Robots andSystems (IROS). pp. 1–9.

Chen LH, Cera B, Zhu EL, Edmunds R, Rice F, BronarsA, Tang E, Malekshahi SR, Romero O, Agogino AKand Agogino AM (2017) Inclined surface locomotionstrategies for spherical tensegrity robots. In: IEEE/RSJInternational Conference on Intelligent Robots andSystems (IROS). pp. 4976–4981.

Cheney N, MacCurdy R, Clune J and Lipson H (2013)Unshackling Evolution: Evolving Soft Robots with Mul-tiple Materials and a Powerful Generative Encoding. In:Conference on Genetic and Evolutionary Computation(GECCO). pp. 167–174.

Christensen DJ, Schultz UP and Stoy K (2013) Adistributed and morphology-independent strategy foradaptive locomotion in self-reconfigurable modularrobots. Robotics and Autonomous Systems 61(9): 1021–1035.

De Bruin T, Kober J, Tuyls K and Babuska R (2018)Experience selection in deep reinforcement learning forcontrol. The Journal of Machine Learning Research19(1): 347–402.

Deimel R and Brock O (2015) Soft Hands for ReliableGrasping Strategies. In: Verl A, Albu-Schaffer A, BrockO and Raatz A (eds.) Soft Robotics. Springer, pp. 211–221.

Der R, Martius G and Hesse F (2006) Let it roll — emergingsensorimotor coordination in a spherical robot. In:Artificial Life X. pp. 192–198.

Fujita K, Yonekura S, Nishikawa S, Niiyama R andKuniyoshi Y (2018) Environmental and structural effectson physical reservoir computing with tensegrity. Journalof the Institute of Industrial Applications Engineers 6(2):92.

Hauser H, Ijspeert AJ, Fchslin RM, Pfeifer R andMaass W (2011) Towards a theoretical foundationfor morphological computation with compliant bodies.Biological Cybernetics 105(5): 355–370.

Hirai S and Imuta R (2012) Dynamic simulation of six-strut tensegrity robot rolling. In: IEEE InternationalConference on Robotics and Biomimetics (ROBIO). pp.198–204.

Hustig-Schultz D, SunSpiral V and Teodorescu M (2016)Morphological design for controlled tensegrityquadruped locomotion. In: IEEE/RSJ InternationalConference on Intelligent Robots and Systems (IROS).pp. 4714–4719.

Ijspeert AJ (2008) Central pattern generators for locomotioncontrol in animals and robots: A review. NeuralNetworks 21(4): 642–653.

Iscen A, Agogino A, SunSpiral V and Tumer K (2014) Flopand roll: Learning robust goal-directed locomotion for atensegrity robot. In: IEEE/RSJ International Conferenceon Intelligent Robots and Systems (IROS). pp. 2236–2243.

Jayaram K and Full RJ (2016) Cockroaches traverse crevices,crawl rapidly in confined spaces, and inspire a soft,

legged robot. Proceedings of the National Academy ofSciences 113(8): E950–E957.

Kaufhold T, Schale F, Bhm V and Zimmermann K(2017) Indoor locomotion experiments of a sphericalmobile robot based on a tensegrity structure withcurved compressed members. In: IEEE InternationalConference on Advanced Intelligent Mechatronics(AIM). pp. 523–528.

Kim K, Agogino AK, Toghyan A, Moon D, Taneja Land Agogino AM (2015) Robust learning of tensegrityrobot control for locomotion through form-finding.In: IEEE/RSJ International Conference on IntelligentRobots and Systems (IROS). pp. 5824–5831.

Kim K, Chen LH, Cera B, Daly M, Zhu E, Despois J,Agogino AK, SunSpiral V and Agogino AM (2016)Hopping and rolling locomotion with spherical tenseg-rity robots. In: IEEE/RSJ International Conference onIntelligent Robots and Systems (IROS). Daejeon, SouthKorea, pp. 4369–4376.

Kim K, Moon D, Bin JY and Agogino AM (2017) Designof a spherical tensegrity robot for dynamic locomotion.In: IEEE/RSJ International Conference on IntelligentRobots and Systems (IROS). pp. 450–455.

Lee DY, Koh JS, Kim JS, Kim SW and Cho KJ(2013) Deformable-wheel robot based on soft material.International journal of precision engineering andmanufacturing 14(8): 1439–1445.

Levine S and Abbeel P (2014) Learning Neural NetworkPolicies with Guided Policy Search under UnknownDynamics. In: Advances in Neural InformationProcessing Systems 27 (NeurIPS). pp. 1071–1079.

Lin HT, Leisk CG and Trimmer B (2011) GoQBot: aCaterpillar Soft-bodied Rolling Robot. Bioinspirationand Biomimetics 6(2): 026007.

Lipson H (2014) Challenges and opportunities for design,simulation, and fabrication of soft robots. Soft Robotics1(1): 21–27.

Littlefield Z, Surovik D, Wang W and Bekris KE (2017)From quasi-static to kinodynamic planning for sphericaltensegrity locomotion. In: International Symosium onRobotics Research (ISRR). Puerto Varas, Chile.

Luo J, Edmunds RF, Rice F and Agogino AM (2018)Tensegrity robot locomotion under limited sensoryinputs via deep reinforcement learning. In: IEEEInternational Conference on Robotics and Automation(ICRA). pp. 6260–6267.

Mintchev S, Zappetti D, Willemin J and Floreano D(2018) A soft robot for random exploration of terrestrialenvironments. In: IEEE International Conference onRobotics and Automation (ICRA). pp. 7492–7497.

Mirletz B, Bhandal P, Adams RD, Agogino AK, Quinn RDand SunSpiral V (2015) Goal directed CPG based controlfor high DOF tensegrity spines traversing irregularterrain. Soft Robotics 2(4): 165–176.

Montgomery W, Ajay A, Finn C, Abbeel P and Levine S(2017) Reset-free guided policy search: Efficient deepreinforcement learning with stochastic initial states.In: IEEE International Conference on Robotics andAutomation (ICRA). pp. 3373–3380.



Nozaki H, Kujirai Y, Niiyama R, Kawahara Y, YonezawaT and Nakazawa J (2018) Continuous shape changinglocomotion of 32-legged spherical robot. In: IEEE/RSJInternational Conference on Intelligent Robots andSystems (IROS). pp. 2721–2726.

Onal CD and Rus D (2013) Autonomous undulatoryserpentine locomotion utilizing body dynamics of afluidic soft robot. Bioinspiration & Biomimetics 8(2):026003.

Paul C, Valero-Cuevas FJ and Lipson H (2006) Designand control of tensegrity robots for locomotion. IEEETransactions on Robotics 22(5): 944–957.

Peng XB, Berseth G and van de Panne M (2016) Terrain-adaptive Locomotion Skills Using Deep ReinforcementLearning. ACM Transactions on Graphics 35(4): 81:1–81:12.

Rajeswaran A, Kumar V, Gupta A, Vezzani G, SchulmanJ, Todorov E and Levine S (2018) Learning ComplexDexterous Manipulation with Deep ReinforcementLearning and Demonstrations. In: Robotics: Science andSystems XIV.

Ramezani A, Chung SJ and Hutchinson S (2017) Abiomimetic robotic platform to study flight specializa-tions of bats. Science Robotics 2(3).

Rennie C and Bekris KE (2018) Discovering a libraryof rhythmic gaits for spherical tensegrity locomotion.In: IEEE International Conference on Robotics andAutomation (ICRA). Brisbane, Australia, pp. 2290–2295.

Rieffel J and Mouret JB (2018) Adaptive and Resilient SoftTensegrity Robots. Soft Robotics 5(3): 318–329.

Rieffel J, Valero-Cuevas F and Lipson H (2009) Morpho-logical communication: exploiting coupled dynamics ina complex mechanical structure to achieve locomotion.Journal of the Royal Society Interface 7: 613–621.

Sabelhaus AP, Bruce J, Caluwaerts K, Manovi P, Firoozi RF,Dobi S et al. (2015) System design and locomotion ofSUPERball, an untethered tensegrity robot. In: IEEEInternational Conference on Robotics and Automation(ICRA). pp. 2867–2873.

Santina CD, Katzschmann RK, Biechi A and Rus D (2018)Dynamic control of soft robots interacting with theenvironment. In: IEEE International Conference on SoftRobotics (RoboSoft). pp. 46–53.

Schroeder K, Samareh J and Bayandor J (2018) Tandem:Tension adjustable network for deploying entry mem-brane. Journal of Spacecraft and Rockets : 1379–1392.

Shepherd RF, Ilievski F, Choi W, Morin SA, Stokes AA,Mazzeo AD, Chen X, Wang M and Whitesides GM(2011) Multigait soft robot. Proceedings of the NationalAcademy of Sciences 108(51): 20400–20403.

Sugiyama Y, Shiotsu A, Yamanaka M and Hirai S (2005)Circular/Spherical Robots for Crawling and Jumping.In: IEEE International Conference on Robotics andAutomation. pp. 3595–3600.

Surovik D, Bruce J, Wang K, Vespignani M and BekrisKE (2018) Any-axis tensegrity rolling via symmetry-reduced reinforcement learning. In: InternationalSymposium on Experimental Robotics (ISER).

Surovik DA and Bekris KE (2018) Symmetric reductionof tensegrity rover dynamics for efficient data-driven

control. In: ASCE Earth and Space Conference.Tassa Y, Erez T and Todorov E (2012) Synthesis and

stabilization of complex behaviors through onlinetrajectory optimization. In: IEEE/RSJ InternationalConference on Intelligent Robots and Systems (IROS).pp. 4906–4913.

Vespignani M, Ercolani C, Friesen JM and BruceJ (2018a) Steerable locomotion controller for six-strut icosahedral tensegrity robots. In: IEEE/RSJInternational Conference on Intelligent Robots andSystems (IROS). pp. 2886–2892.

Vespignani M, Friesen JM, SunSpiral V and Bruce J (2018b)Design of superball v2, a compliant tensegrity robot forabsorbing large impacts. In: IEEE/RSJ InternationalConference on Intelligent Robots and Systems (IROS).pp. 2865–2871.

Wei X, Tian Y and Wen S (2019) Design and locomotionanalysis of a novel modular rolling robot. Mechanismand Machine Theory 133: 23–43.

Zhang M, Geng X, Bruce J, Caluwaerts K, VespignaniM, SunSpiral V, Abbeel P and Levine S (2017) Deepreinforcement learning for tensegrity robot locomotion.In: IEEE International Conference on Robotics andAutomation (ICRA). pp. 634–641.

Zhang M, Vikram S, Smith L, Abbeel P, Johnson MJand Levine S (2018) Solar: Deep structured latentrepresentations for model-based reinforcement learning.arXiv preprint arXiv:1808.09105 .

Zhao Y, Zhou S, Lin C and Li D (2017) An efficientlocomotion strategy for six-strut tensegrity robots.In: IEEE International Conference on Control &Automation (ICCA). pp. 413–418.

Zhou X and Bi S (2012) A survey of bio-inspired compliantlegged robot designs. Bioinspiration & Biomimetics7(4): 041001.

Zhu S, Surovik D, Bekris KE and Boularias A (2018)Efficient model identification for tensegrity locomotion.In: IEEE/RSJ International Conference on IntelligentRobots and Systems (IROS). pp. 2985–2990.


Documents

Adaptive Tensegrity Locomotion: XX(X):1–20 The Author(s ...kb572/pubs/reinf_learning_tensegrity_locomotion.pdf6-bar tensegrity rovers (Caluwaerts et al.2014;Sabelhaus et al.2015)