E L S E V I E R Robotics and Autonomous Systems 21 (1997) 139-147

Robotics and

Autonomous Systems

The importance of fast vision in winning the First Micro-Robot World Cup Soccer Tournament

R a n d y Sa rgen t *, B i l l B a i l e y , Car l Wi t ty , A n n e W r i g h t Newton Research Labs. 14813 NE 13th St., Bellevue, WA 98007, USA

Abstract

This paper describes the Newton Labs entry in the 1996 MIROSOT Micro-Robot World Cup Soccer Tournament (http://www.mirosot.org), which won first place. Control of the three robot players is centralized; this requires relatively high bandwidth connections to the robots, as well as fast spatial information at the central controller. Spatial sensing is ac- complished by use of the Cognachrome Vision Tracking System, which is capable of tracking the positions and orientations of several objects at the full NTSC frame rate of 60 Hz.

Keywords," MIROSOT; Robot soccer; Robot football; Machine vision; Fast visual tracking

I. Introduction

This paper describes the Newton Labs entry in the 1996 MIROSOT Micro-Robot World Cup Soccer Tournament, which won first place (the Hanminjok cup).

The MIROSOT game is played on a surface that measures 130 cm x 90 crn. Each team is composed of three robots, each fitting within a cube 7.5 cm on a side. Robots are marked with team colors to aid in visual identification. The soccer ball is a standard orange golf ball (see Figs. 1 and 2).

Although multiple independent, interacting robots is a very interesting field of research, we chose to use centralized control for the three robots in our MIROSOT entry. We chose this approach partly for efficiency, and partly because it is easier to implement.

We found fast spatial sensing to be very important for the MIROSOT contest; the faster the spatial sens-

* Corresponding author. E-mail: vision@newtonlabs.com.

0921-8890/97/$17.00 1997 Published by Elsevier Science B.V. PI! S0921-8890(97)00022-5

Fig. 1. Newton Labs' MIROSOT team members: force, mass, and acceleration (mass is the goalie).

ing, the faster the feedback loops for controlling robot positions, and, thus, the faster the robots can travel. An ideal sensor for fast spatial sensing is a video camera coupled with a vision system. We found putting vision

140 R. Sargent et al./Robotics and Autonomous Systems 21 (1997) 139-147

mnmmlmmmmmm mmmmm mmmm mmmmmmmmmmmmmm[

mmmmmmmmmmmmmmmmmm

Fig. 2. The MIROSOT playing field.

systems on each robot to be impractical due to size constraints (although we are thinking about attempting this in the future). Therefore our only vision system was offboard, looking down at the playing field from above. This view is probably more useful for control- ling the robots anyway; views from the robots could be used for more accurate servoing relative to objects that are nearby, such as the soccer ball, but are other- wise less useful than the view from above.

In order to simplify communications, we used a sim- ple, one-way link from the central controller, rather than a two-way link where robots could send data back to the controller. This meant the central control strategy had no access to sensing that could be local to each robot, such as bump sensors or IR proxim- ity sensors. We hoped that the data from the over- head vision system alone would be adequate for our needs.

2. Spatial sensing

Fast and accurate spatial sensing is critical to the success of cooperating mobile robots. In the case of multiple-robot games, such as MIROSOT, it is paramount to maximize speed and minimize latency of spatial sensing, as these parameters ultimately limit how quickly robots can move in a controlled fashion, and thus limit how effectively robots can compete.

For our entry into the MIROSOT '96 contest, we made use of our own commercial vision-based

tracking system, the Cognachrome Vision System. ] The Cognachrome Vision System is cheap (retail US$2450), fast, and consumes little power. It is ca- pable of tracking several objects in the field of view at a full 60 Hz frame rate, with performance dipping to 30 Hz when a large number of objects are tracked. Objects are recognized by the system on the basis of colored tags. The system can be trained to recognize up to three simultaneous colors, and there may be many objects of each color present in the field of view. For each object, various statistics are automati- cally calculated, such as centroid, area, and direction of major and minor axes when desired.

The Cognachrome Vision System consists of a 68332-based CPU board, and a special hardware- accelerated vision subsystem for fast, high-resolution tracking. The system is also capable of a lower- resolution frame grab into on-board memory, useful for algorithms such as edge-based obstacle detection. The system is capable of running with built-in soft- ware, or software written by the user. The built-in software allows the user to specify tracking criteria, and then outputs tracking data in a user-defined format over one of the board's two asynchronous serial ports. For user-written software, the ARC C development system 2 can be used. All the preprogrammed vision algorithms are accessible, and the system has some spare digital I/O, as well as a bus interface requiring no glue logic (the 68 332 provides programmable chip selects).

At 3 . 2 c m x 6 . 4 c m x l 5 . 9 c m (1.25 in x 2.5 in x 6.25 in), our system is typically small enough to be placed on mobile robots. Unfortunately, our system will not fit into the 7.5cm x 7.5cm x 7.5cm size limits of the MIROSOT robot. Therefore, we con- tent ourselves with a single view of the MIROSOT playing field from above.

All robots in the MIROSOT contest must be marked with a colored beacon (one color per team) and the bali is bright orange. The Cognachrome Vision System is trained on the three colors and ex- tracts the following information about the state of the game: • location of the three players, • angle of orientation of the three players,

I http://www'newt°nlabs'c°m/c°gnachr°rne/ 2 http://www.newtonlabs.com/arc.

R. Sargent et al./Robotics and Autonomous Systems 21 (1997) 139-147 141

f Colored tag

Robot body (top view)

Dlfei~lon of

torward motion

Fig. 3. Top view of robot.

• location of the three opponents, • location of the ball.

Our players are marked with a colored stripe to fa- cilitate determination of angle of orientation, as shown in Fig. 3. Since our tag has 180 ° rotational symmetry, we must maintain the state of forward and backward directions, for a given robot.

3. Hardware architecture

Our hardware is composed of the robots, a central controlling computer, a one-way radio link from the controller to the robots, and the Cognachrome Vision System (Figs. 4 and 5).

3. I. Robot hardware

The robot is composed of two basic assemblies. The lower assembly contains the wheels, batteries, and

Fig. 4. The prototype soccer robot, without jersey.

Fig. 5. The final soccer robot, without jersey.

motors. Having the heavy parts in the lower assembly keeps the center of gravity low. The upper assembly contains three printed circuit boards in a stack: the processor board, the radio receiver board, and the in- terface board. The interface board provides a mount- ing for the other boards and points of attachment to the lower assembly.

The robot is powered by two 9 V batteries wired in series. Like most of the robots in the MIROSOT contest, we employed a standard "wheelchair" drive with two driving wheels and two casters. Because of the symmetry of the robot, the "front" and "back" sides of the robot can be used to push the ball equally well.

To drive the two wheels, we used a pair of Canon EN20-R5 motors with attached RA20 gearheads, which we then hand-modified to have a gear ratio of approximately 20 to 1. The plastic gears, drive wheels, shafts that carry the drive wheels, and the shaft bearing blocks are LEGO components. We used LEGO extensively for our prototype soccer robot, and found it was so effective that most of the LEGO assembly was kept for the final robot, The robot's wheels are LEGO wheels, with standard rubber O- rings replacing the tires (the original LEGO tires did not produce sufficient friction with the surface of the contest table).

For the main processing board, we used an off- the-shelf Miniboard 2.1, 3 which sports a 68HC 11 E2

3 http://fredm.www.media.mit.edu/people/fredm/papers/ rob/.

t42 R. Sargent et al./Robotics and Autonomous Systems 21 (1997) 139-147

8-bit microprocessor and two L293D motor drivers. We found it necessary to trim the Miniboard to fit within our 7.5 cm × 7.5 cm size constraint - we re- moved the portion of the board with the large serial connectors.

A pulse-width-modulation drive is used to control power applied to the motors. Since the robot is respon- sible for controlling the speed of each wheel, it must be able to measure the speed of each wheel. Instead of using encoders, we measured motor speed by mea- suring the back EMF across the motor during the off time of the PWM cycle.

3.2. Central controller

The communication system has a limitation which is fairly easily worked around. The DC component of the transmitted signal must be kept fairly close to zero - that is, a fairly even number of ones and zeros must be sent in close proximity. Empirically, we found that transmitting bytes with 3, 4, or 5 "1" bits (out of 8) resulted in reliable reception. This meant that instead of the 256 values which could usually be sent in a byte, we were limited to 182. (Surprisingly, this represents a loss of only 6% of the bandwidth).

Despite our success, we were more impressed with teams sporting the much smaller Radiometrix TXM/RXM FM Transmitter/Receiver pair, 4 and may pursue using these modules in the future.

Our host computer, which is responsible for the high-level planning and middle-level control of the robots, is a desktop PC-compatible machine. Perfor- mance was important, as we recalculate all plans 60 times a second to take advantage of the 60 Hz vision data, so we chose a model with good performance (based on a 180 MHz Intel P6 microprocessor).

3.4. Vision system

Our vision system hardware was previously de- scribed in Section 2.

4. Software architecture

3.3. Radio link

Our one-way radio link from the host computer to the robots consists primarily of an off-the-shelf audio radio system. After some experimentation, we found that the Radio Shack Optimus 900 MHz wire- less headset system (catalog number 33-1145) could be adapted for 9600 baud serial communication. The receiver board, designed to fit in the left earpiece of the headset, needed to be trimmed only slightly to make it fit the 7.5 cm × 7.5 cm dimension necessary to fit on our robot.

Connection to the transmitter was quite simple: we used a resistor divider to reduce the voltage from a standard RS-232 signal by approximately a factor of 10. This attenuated signal was connected to both the left and fight audio input jacks on the transmitter.

Connection to the receiver was slightly more com- plicated. After removing the board from the headset and trimming it, we turned up the "volume" control to its maximum setting. We ran the audio signal originally indended for one of the two speakers to a comparator, and we manually adjusted the comparator threshold voltage until reception worked.

Our software is divided into three levels. The "low- level" software runs on the robot itself, and is responsi- ble for controlling the speeds of the two drive motors. The "high level" takes current position information about the robots and the ball, and decides short-term position goals for each of our robots. The "middle level" connects everything: it receives and filters po- sition information from the vision system, and sends speed commands to the "low-level" controller on each robot to servo the robot to its current positional goal, which is given from the "high level".

4.1. Low-level control

The lowest level of the control software runs on the robot's 8-bit microprocessor. This program takes left and fight speed commands from the radio link and attempts to servo the robot's left and right motors to match the target speeds. Feedback on the motor's current speed is given by the back EMF across the motor. The duty cycle of the motor's drive is chosen as a function of the measured speed and desired speed.

4 http://www.radiometrix.co.uk/.

R. Sargent et al./Robotics and Autonomous Systems 21 (1997) 139-147 143

4.2. Middle-level control

The middle-level software is responsible for con- trolling the paths of the soccer robots. It receives current position information from the vision system, filters the data, and predicts the current state of the world (attempting to account for the small latency of the vision data). The post-processed position and ori- entation information is provided to the high-level soft- ware, and is used by the middle level for controlling the paths of the robots.

As described before, the only commands that can be sent to the robot are "set left wheel speed" and "set right wheel speed". The robot does not know its current position or orientation. It is the responsibility of the middle level to servo each robot's direction and speed to most closely match the high-level software's request.

We thought about and experimented with several ways for the high level to express the desired robot path to the middle level.

We rejected the idea of the high level commanding the desired position and orientation for the next frame of video, because this would require the middle level and high level to operate in lock-step. In order to run the middle and high levels asynchronously, the high level had to specify commands in a way that the com- mand still made sense after the frame during which it was calculated.

(As it turns out, our final system was in fact able to run both the middle-level and high-level software at the full frame rate of 60 Hz. However, we still rejected running the two in lock-step because the latency in- troduced by the 1/60th of a second processing time of the high level was unacceptable to add to the middle- level servo loop. Instead, the middle-level uses the last received command from the high level, resulting in using commands that are typically a frame old in our system - once again, requiring commands that con- tinue to be useful after the frame during which they were calculated.)

We eventually settled on a command format that encoded a "virtual target" and speed for each robot. The virtual target is an actual two-dimensional posi- tion on the playing surface. The middle level's job is to keep each robot pointed towards its virtual target, and running at its commanded speed. The high level commands virtual targets that are always a fair dis-

tance away from the robot, so that the robot will not reach the target before a new command.

At first glance, this command format might not seem to be any better than sending simply direction and speed - two parameters, rather than three. Indeed, if virtual targets are always very far from the robot, there is no benefit to having a two-dimensional target posi- tion instead of simply a direction. It turns out that the target's distance from the robot encodes how much the middle level should correct the direction of the robot when it "strays" from the expected path. When the tar- get is close, a sideways' positional error results in a larger correction than when the target is further away.

On the surface, it probably seems unimportant to give the high level the ability to specify the side-to- side accuracy with which the middle level follows its paths. However, we found that it gave us the ability to improve our performance by quite a bit. In cases where less side-to-side accuracy was required, we found that the middle level could run the robots more quickly. This is primarily a side-effect of our robot's two- wheeled differential drive - in order to correct a side- to-side error, the robot must change its direction and thus lose forward motion.

We found that we could trade side-to-side accuracy to gain speed both in the case of the goalie (whose forward-backwards motion mattered much more than side-to-side), and in the case where a robot was at- tempting to kick the ball, but was still far from it.

4.3. High-level control

The high-level strategy takes position data from the middle level at 60 Hz, and generates position and speed goals for each of the robots. While we were able to have the high-level finish plans within 1/60 of a sec- ond, in fact our architecture does not require that the middle level and high levels work in lock step, so a slower high level could be accommodated.

In each frame, the high-level strategy simulates the ball's path over the next two or so seconds in the future. In this simulation, the ball bounces off the sides of the playing fields. Interactions with robots are ignored.

Once the ball path is predicted, the high level creates plans for the three robots. The "goalie" robot has a special-purpose, fairy simple strategy. The other two robots use a different, more complex strategy.

144 R. Sargent et aI./Robotics and Autonomous Systems 21 (1997) t39 t47

4.3.1. Attacker's strategy We call the two robots other than the goalie attack-

ers. In our team, the attackers do most of the work. Their primary objective is to kick the ball into the op- ponent's goal. Secondarily, when a such a kick is not possible, the attacker will attempt to improve its po- sition in the playing field.

4.3.1.1. Kicking a goal. In the first phase of the at- tacker's strategy, the high-level software decides if either attacker can attempt a kick to place the ball directly into the opponent's goal. (In fact, we have no actual kicking mechanism. Rather, both the front and back sides of our robot are flat, and we "kick" by running into the bali at speeds up to 60 cm/s. The bali's resulting bounce usually exceeds the speed of the robot by a fair margin.) Kicking the ball is a little trickier than it sounds, primarily because the ball is moving.

In order to perform the kick behavior, we have a fairly simple algorithm that gives a path from the present location to a future desired location plus de- sired orientation. It is important that we control both the future location and orientation precisely, since these two parameters determine whether we intercept the ball, and if so, what direction we kick it in.

Given the path algorithm, and a possibly complex predicted path of the ball (which can include bounces off walls), we were not able to find a closed-form solution for what time in the future we would be able to kick the ball. Rather, we search times in the future for possible kicks as follows.

First, the high-level software performs a sparse search across the next two seconds, asking the follow- ing question: If the robot attempts to kick the ball at this position in the future, how long would it take to get there? When the time it would take for the robot to get to the position is close to the time it would take for the ball to get to the position, the system performs a finer-grained search to try to nail down at what point in the future the robot can most accurately kick the ball.

This search is rather expensive, so to speed it up, we constructed a two-dimensional lookup table that gives the estimated time for the robot to intercept a given relative location at a given orientation. This way, each individual test for future intersection is fairly quick.

If neither robot can kick the ball directly into the opponent's goal, the system then checks if either robot can perform a "bank s h o t " - knocking the ball off a wall into the goal. We found this strategy to be some- what effective in initial tests, but not nearly as effective at making goals at the actual MIROSOT contest, be- cause the walls of the actual MIROSOT playing field damped bounces more than did the walls of our test playing field. However, we decided against removing this behavior, because even a failed kick off the side of the wall moves the bali closer to the opponent's goal.

Assuming the system finds a future kick for one of the attackers, it sets that attacker in motion for the kick. Only one robot can kick at a time; if two robots both have kicks available, direct kicks are preferred, and secondarily, the robot with the soonest kick is preferred.

4.3.1.2. Improving position. Depending on the re- sults of the "kick a goal" behavior, at least one, and possibly both of the attackers will not currently be kicking the bail. When an attacker is not currently kicking the ball, it seeks to improve its position on the field.

In order to improve robot positions, the system must be able to measure the desirability of different posi- tions. Robots then treat the desirability as a potential field, and essentially move downhill in this field.

A position's desirability computed primarily using negative criteria (its "undesirability" is computed): (I) An attacker is repelled from the area between the

ball and the opponent's goal. This is a counter- productive place to be, as it might block a shot from the other attacker.

(2) An attacker is attracted to the half of the field on "our" side of the ball. This attraction is less strong than (l), so the robot wilt first move out from behind the ball, and then back to our side of the field. An attacker is strongly repelled from the im- mediate vicinity of an opponent's robot. This is intended to prevent collisions while optimizing position. An attacker is strongly repelled from the immedi- ate vicinity of the wall of the playing field. This is intended to prevent collisions with the wall while optimizing position.

(3)

(4)

R. Sargent et al./Robotics and Autonomous Systems 21 (1997) 139-147 145

(5) An attacker is weakly repelled from other mem- bers of our team. This causes robots to "play po- sition" and not huddle around the ball. Unlike the repelling from opponents, this works even at large distances. However, the "force" with which the at- tacker is repelled from his kindred is very weak, and is generally overpowered by the other forces.

4.3.2. Goalie's strategy The goalie's strategy is quite simple: stay between

the ball and the goal. While a human goalie might face away from the goal, our goalie faces to the left or right so that it can move back and forth to block the ball with its side. This simple strategy prevents the goalie from turning to "kick" the ball outside the goal area, but we did not find this to be a major problem in practice.

Under normal circumstances, the goalie stays even with the ball (see Fig. 6). However, when the system predicts that the ball will enter the area close to the goal in the next two or so seconds, the goalie will move to the expected point of entry. The goalie's back and forth motion is limited so that the goalie's position always overlaps the goal area.

4.4. Software results

Fig. 7. The SCAMP underwater vehicle positions itself relative to the target.

at times surprised, by the results. Although not pro- grammed explicitly, the ball "passes" between robots which emerged frequently from the combination of rules and chance - simply, when one robot kicked the ball, the ball often moved to a location where the other robot would be closer to intercept it. For video of the soccer players in action, please refer to ht tp: / /www.newtonlabs.com/soccer/(Fig. 7).

Although we feel that our system can be im- proved in many ways, we were very pleased, and

mmmm mm mnnnn

mml

5. Other applications of fast vision tracking

Making a winning entry for MIROSOT '96 was made much easier by having a vision system capable of quickly tracking targets of interest. The authors be- lieve that fast vision tracking has the potential to help many other applications as well. Included here are an assortment of projects for which the Cognachrome Vision System is currently used.

mnmml i N N mmll

Fig. 6. Goalie position under normal circumstances (left), and when ball is predicted to enter area close to goal (right).

5.1. Catching balls and paper airplanes [ 1,2]

Two Cognachrome Vision Systems were integrated in the new version of the adaptive robot catching project led by Prof. Jean-Jacques Slotine of MIT. The project uses an advanced manipulator and fast-eye gimbals developed under Dr. Kenneth Salisbury of the MIT AI lab.

Using two-dimensional stereo data from a pair of Cognachrome Vision Systems, they predict the three-dimensional trajectory of an object in flight, and

146 R. Sargent et al./Robotics and Autonomous Systems 21 (19.97) t39-147

control their fast robot ann (the Whole Arm Manipula- tor, or WAM) to intercept and grasp the object. (Please see http: / /www.ai.mit .edu/projects /wam/index .h tml# $2.2 for more information, and animations of the arm catching various objects.)

5,2. Cleaning Up Tennis Courts

Two mobile robots tied for first place at the "Clean Up the Tennis Court" contest, held at AAAI '96. One team was fielded by Newton Research Labs. The other was led by Sebastian Thrun of Carnegie Mellon University.

Both winning robots used the Cognachrome Vision System. The system provided fast tracking informa- tion on the positions of tennis balls, as well as the positions of the randomly moving, self-powered "squiggle" balls.

5.3. Group behavior and social interaction

of robots

Maja Mataric, Barry Werger, Dani Goldberg, and Francois Michaud at the Volen Center for Complex Systems at Brandeis University 5 study group behav- ior and social interaction of robots. Along with other robots, they use Pioneer mobile robots outfitted with Cognachrome Vision Systems.

In conjunction with shorter range or less specific sensors, such as sonar, the Pioneers use color-based tracking to help recognize other robots, obstacles, and goals.

Barry Werger says:

"I have combined these two [vision-based long-range obstacle avoidance and vision-based following of in- termittently blocked objects] to address some of the problems we have in our mixed robot environment... that is, the Pioneers are faster and bigger than our other, more fragile robots; the long range avoidance allows them to keep a safe distance from other robots, even in fairly dynamic environments, when follow- ing a dynamic target. The vision allows us to make

5please see http://www.cs.brandeis.edu/~agents/projects. html and http://www.cs.brandeis.edu/--.barry/research.html for more information.

these distinctions very easily, which the sonar does not".

5.4. Autonomous docking of spacecraft

The University of Maryland Space Systems Lab- oratory and the Kiss Institute for Practical Robotics have simulated autonomous spacecraft docking in a neutral buoyancy tank for inclusion on the UMD's Ranger space vehicle. Using a composite target of three brightly colored objects designed by Miller (see [3]), the spacecraft knows its distance and orientation, and can servo to arbitrary positions around the tar- get. (See http://www.kipr.org/robots/scamp.html for more information and pictures.)

6. Summary

We are very happy that our hardware and strategy gained us first place at the first annual MIROSOT con- test. Our primary strength appeared to be the speed of our vision system; several teams had more sophis- ticated approaches to strategy, but were unfortunately limited by sensing speed.

We feel that robot soccer is a very interesting ap- plication for robot technology, and that such contests will drive the development of robot sensing and con- trol strategies. We gratefully acknowledge Prof. Jong- Hwan Kim and his group at KAIST for developing and organizing MIROSOT. We look forward to future contests !

References

[11 W.J. Hong, Robotic catching and manipulation using active vision, M.S. Thesis, Department of Mechanical Engineering, Massachusetts Institute of Technology, September 1995.

[2] W.J. Hong and J.J.E. Slotine, Experiment in hand-eye coordination using active vision, in: Proc. 4th Int. Syrup. on Experimental Robotics, ISER '95, Stanford, CA (1995).

[3] D.P. Miller and A. Wright, Autonomous spacecraft docking using multi-color targets, in: Proc. 6th Topical Meeting on Robotics, Monterey, CA (1995).

14] M.K. Sahota, A.K. Mackworth, R.A. Barman and S.J. Kingdon, Real-time control of soccer-playing robots using off-board vision: the dynamite testbed, in: IEEE Int. Conf. on Systems, Man, and Cybernetics (t995).

[5] A.Wrighl, A high-speed low-latency portable visual sensing system, in: Proc. the SPIE (1993).

:.zi~!

R. Sargent et al./Robotics and Autonomous Systems 21 (1997) 139-147 147

Randy Sargent holds the title of De- sign Engineer at Newton Research Labs in Bellevue, WA. He received the B.S. in Computer Science at MIT, and the M.S. in Media Arts and Sci- ence from the MIT Media Laboratory. Formerly holding titles of Lecturer and Research Specialist at MIT, he is one of the founders of the MIT LEGO Robot Contest (a.k.a. 6.270), now in its seventh year.

old movies.

Carl Witty holds the title of Research Scientist at Newton Research Labs in Bellevue, WA. He grew up in Adrian, Oregon, received the B.S. in Computer Science from Stanford University, and the M.S. in Electrical Engineering and Computer Science from MIT. A member of the winning team in the 1991 ACM Programming Contest, his interests include robots, science fic- tion and fantasy, mathematics, formal methods for software engineering, and

Bill Bailey holds the title of Design Engineer at Newton Research Labs in Bellevue, WA. He specializes in analog and digital electronics, software, and mechanical design. When not working on robotics, he also works at a small software firm in Redmond, WA.

Anne Wright is the President of Newton Research Labs in Bellevue, WA. The original developer of the Cognachrome Vision System, she re- ceived the B.S. and M.Eng. degrees in Computer Science from MIT. She also helped lead and develop technology for the MIT LEGO Robot Contest from 1992 to 1994.