View
445
Download
0
Category
Preview:
Citation preview
Robotprogrammatie: enkele lessen uit de praktijk, trends en uitdagingen
Eric DemeesterFaculteit Industriële Ingenieurswetenschappen
Technologiecampus Diepenbeek
Onderzoeksgroep ACRO
CEVORA IT event, 25 oktober 2015
Robotprogrammatie: enkele lessen uit de praktijk
Outline
2
• ACRO activities: robots with vision and a plan
o Mobile, assistive robots with vision and a plan
o Agricultural robots with vision and a plan
o Industrial robots with vision and a plan
• Programming robots: learnt lessons
o The RADHAR project as an example
o Requirements
o Solutions
• Trends and challenges for the future
Robotprogrammatie: enkele lessen uit de praktijk
Overview ACRO activities
3
• Profibus & profinet, PLC (UCLL)
o Training for operators in industry
o Service and consultancy regarding industrial networks
• Vision and robotics (KU Leuven)
o Vision: choice of cameras, optics, lighting, image processing
o State estimation and machine learning, e.g.:
• Object recognition/classification and 6D pose estimation
• Mobile robot localisation and mapping
• Estimation of human intentions by robots
o Sensor-based decision making and planning, e.g.:
• Collision-free trajectory planning for mobile robots and industrial manipulators
• Shared human-machine control
Remark: ACRO is part of Faculty of Engineering Technology => research closer to industry (higher TRL levels)
Robotprogrammatie: enkele lessen uit de praktijk
Outline
4
• ACRO activities: robots with vision and a plan
o Mobile, assistive robots with vision and a plan
o Agricultural robots with vision and a plan
o Industrial robots with vision and a plan
• Programming robots: learnt lessons
o The RADHAR project as an example
o Requirements
o Solutions
• Trends and challenges for the future
Robotprogrammatie: enkele lessen uit de praktijk
Introduction
5
• Semi-autonomous robotic wheelchairs
o Motivation:
• Many elderly and disabled people suffer from a reduced mobility
• Electric wheelchairs enhance this mobility, but manoeuvring an
electric wheelchair is often difficult and time-consuming
• Consequences: accidents, frustration, dependence on others,
reduced social contact, reduced self-esteem, lower quality of life
o Solution? Equip electric wheelchairs with sensors and
computing power
=> Combine the strengths of both human (global planning) and
computer (fine motion control)
Collaborations/cases with: Permobil, Invacare, HMCI, Ottobock, BlueBotics, National MS Centre
Belgium, Windekind + various research institutes
Robotprogrammatie: enkele lessen uit de praktijk
Research questions
6
• Semi-autonomous robots should be able to answer
three questions:
1. Where am I?
Robotprogrammatie: enkele lessen uit de praktijk
Research questions
7
2. Where am I going?
o Key assumption: user should not be required to
communicate explicitly the desired navigation assistance
o Consequence: user (navigation) plans or intentions are
hidden and should be estimated from uncertain user signals
and sensor signals
o This is the problem of plan (intention) recognition:
Plan recognition is the problem of inferring the goal of
an actor and his plan to achieve this goal, based on a
sequence of actions performed by the actor.
Robotprogrammatie: enkele lessen uit de praktijk
Research questions
8
3. How should I get there?
o Even if it is known where the user would like to drive to, it
remains unclear how the manoeuvre should be executed
jointly by human and robot
o This is the problem of shared control:
The situation in which the control of a device is
shared between one or more users and one or more
robotic controllers.
Robotprogrammatie: enkele lessen uit de praktijk
Research questions
9
o Results with different interfaces: standard joystick, switch
interface, brain-computer interface, haptick joystick
Robotprogrammatie: enkele lessen uit de praktijk
Outline
10
• ACRO activities: robots with vision and a plan
o Mobile, assistive robots with vision and a plan
o Agricultural robots with vision and a plan
o Industrial robots with vision and a plan
• Programming robots: learnt lessons
o The RADHAR project as an example
o Requirements
o Solutions
• Trends and challenges for the future
Robotprogrammatie: enkele lessen uit de praktijk
Autonomous apple picking
11
• Introduction:
o It is hard to find personnel for fruit picking
o High wages, low prices for fruit on the global market
o Orchard management: selective spraying/fertilization
• Research questions:
o Autonomously determine the fruit’s position
o Autonomously pick the fruit without damaging the fruit nor the
tree, keeping the stem on the apple
o Feasibility study: is it possible to perform this operation with
vision and a robot arm?
o Can it be performed fast enough?
o And many other ... (logistics, wheather conditions, fruit
selection on the spot, etc.)
Robotprogrammatie: enkele lessen uit de praktijk
Autonomous fruit picking
12
• Current state:
This image cannot currently be displayed.
Robotprogrammatie: enkele lessen uit de praktijk
Outline
13
• ACRO activities: robots with vision and a plan
o Mobile, assistive robots with vision and a plan
o Agricultural robots with vision and a plan
o Industrial robots with vision and a plan
• Programming robots: learnt lessons
o The RADHAR project as an example
o Requirements
o Solutions
• Trends and challenges for the future
Robotprogrammatie: enkele lessen uit de praktijk
Random bin picking (IWT Tetra RaPiDo)
14
• Introduction:• Nowadays, objects are typically fed to robots in a well-defined,
mechanised manner, which takes time and money to setup
• Alternative: random bin picking:
• Make a 3D scan of a bin with randomly positioned objects
• Recognise and estimate pose of objects
• Compute a collision-free trajectory taking size of robot, gripper, objects, environment into account
• Research goals:
o Build an open demonstrator
o Evaluate existing open source code
o Focus on integration of vision and robotics
o Speed up of object detection and path planning
o Evaluate potential of novel sensors and algorithms
Robotprogrammatie: enkele lessen uit de praktijk
Random bin picking (IWT Tetra RaPiDo)
15
• Current state:
o Bin picking setup with sheet-of-light working
o Analysis of randomised path generation techniques
Collaborations/cases with: ABB, KUKA, Sick, Materialise, Ceratec, Egemin, cards PLM
solutions, Clock-O-Matic, Dewilde Engineering, Vision++, Exmore Benelux, Flanders Food,
Intrion, Optidrive, PEC, Intermodalics, Rabbit, Robberechts, Robosoft, Robomotive, Sedac
Meral, SoftKinetic, Beltech, Borit, Meditech, Sirris, Phaer,Gibas, Octinion, Dana, Van Hool
Robotprogrammatie: enkele lessen uit de praktijk
Flexible robot welding (IWT VIS SmartFactory)
16
• Introduction:
o Current trends:
• globalisation, individualised products, small lot sizes =>
production industry should become highly flexible
• High wages, lack of skilled labour, global competion => production
industry should become highly automated
• Factory of the future: combine flexibility and automation by using
novel technologies: machine vision, path planning software, force
control, intelligent transport
• Research goals: zero ramp-up & auto-programming• Recognise and determine the pose of objects to be welded using
2D or 3D vision,
• perform quality checks before and after welding
• Computation of optimal (time, energy) trajectories for robots
taking into account: robot geometry, kinematics and dynamics,
singularities, components mounted on the robot, environment
Robotprogrammatie: enkele lessen uit de praktijk
Flexible robot welding (IWT VIS SmartFactory)
17
• Current state:
Robotprogrammatie: enkele lessen uit de praktijk
Vision-based wood waste sorting
18
• Introduction:
o SME innovation project for NV Gielen
o Business model: grind wood waste to wood chips and sell
o Problem: percentage MDF: 15% to 50% => price of wood
chips very low
o Solution: separate MDF from wood
• Research questions:
o How to separate MDF from wood?
• Not possible using only infrared reflection, density, colour
• => classify using vision?
o How to classify it accurately enough?
o How to separate it fast enough?
Robotprogrammatie: enkele lessen uit de praktijk
Vision-based wood waste sorting
19
• Solution:
o Determine several features based on colour, texture, shape
o Train multilayer perceptrions to classify wood and MDF
o Onsorted input (45-55% MDF) � to sorted output with 5%
MDF in wood output and 2% wood in MDF output
o Capacity: 7.5 tons/hour
Robotprogrammatie: enkele lessen uit de praktijk
Outline
20
• ACRO activities: robots with vision and a plan
o Mobile, assistive robots with vision and a plan
o Agricultural robots with vision and a plan
o Industrial robots with vision and a plan
• Programming robots: learnt lessons
o The RADHAR project as an example
o Requirements
o Solutions
• Trends and challenges for the future
Robotprogrammatie: enkele lessen uit de praktijk
The RADHAR project as an example
21
Vision: Robotic ADaptation to Humans Adapting to Robots
heterogeneous user groups
with (time)varying skills
dynamic, 3D environments
robots adapting to humans’
signals and responses
Requires life-long
adaptation between
two interacting
learning systems
(human & machine)
Robotprogrammatie: enkele lessen uit de praktijk
The RADHAR project as an example
22
Consortium
2 user
groupsNMSC, Nationaal
Multiple Sclerosis
Centrum V.Z.W.
Windekind,
school for
children with
disability
3 companies
3
universities
1 research
institute
Robotprogrammatie: enkele lessen uit de praktijk
The RADHAR project as an example
23
General overview of the framework
Robotprogrammatie: enkele lessen uit de praktijk
The RADHAR project as an example
24
Developed hardware:
Robotprogrammatie: enkele lessen uit de praktijk
Outline
25
• ACRO activities: robots with vision and a plan
o Mobile, assistive robots with vision and a plan
o Agricultural robots with vision and a plan
o Industrial robots with vision and a plan
• Programming robots: learnt lessons
o The RADHAR project as an example
o Requirements
o Solutions
o Problems encountered
• Trends and challenges for the future
Robotprogrammatie: enkele lessen uit de praktijk
Requirements/challenges
26
o Hardware level:
• Be able to deal with different sensors, wheelchairs, interfaces,
people
• Real-time (predictable, fast) control even if data-intensive
o Algorithmic level:
• Easily transfer algorithms to other platforms
• Safety
• Calibration
• Debugging
o Robotics-related challenges:
• How to program the robot to perform what it should do?
• Environment: uncertain/unknown, changing, 3D, soft
• Robot: kinematics and dynamics, slippage, castors, size
• Concurrent tasks/processes/threads
Robotprogrammatie: enkele lessen uit de praktijk
Requirements/challenges
27
o System level: large-scale collaborative research and
development
• Several research groups across Europe develop software
• Portability, different platforms (Windows/Linux, Java/C++)
• Little time during integration weeks (tests on the hardware)
• Runtime robustness: stopping/crashing a module of a running
robot application should not result into an overall
stopping/crashing of the robot application
• Runtime flexibility: be able to change algorithms of modules at
runtime, extend the robot application with new modules at run-
time, probe ingoing and outgoing data of modules at runtime.
Robotprogrammatie: enkele lessen uit de praktijk
Outline
28
• ACRO activities: robots with vision and a plan
o Mobile, assistive robots with vision and a plan
o Agricultural robots with vision and a plan
o Industrial robots with vision and a plan
• Programming robots: learnt lessons
o The RADHAR project as an example
o Requirements
o Solutions
• Trends and challenges for the future
Robotprogrammatie: enkele lessen uit de praktijk
Solutions (lessons learnt)
29
1. Hardware abstraction and algorithmic level:
• Use of object-oriented programming (C � C++):
• C++ has more features => more complex, but more readable and less maintenance cost
• C++ might be slightly slower
• Timely reaction, real-time control is very important =>
maintainability, reusability is sometimes ignored (nicely
encapsulated software runs slower)
• => lesson learnt: avoid optimisation until it is needed
• use of GPU � results were not that spectacular (< 10x improvement)
• ... sometimes just use a faster PC
Robotprogrammatie: enkele lessen uit de praktijk
Solutions (lessons learnt)
30
• Use of design patterns,
• e.g. to abstract implementation details away from the functionality (e.g. Factory Method): => easily adopt “higher-level” code on different wheelchairs, user interfaces, sensors, ...
Robotprogrammatie: enkele lessen uit de praktijk
Solutions (lessons learnt)
31
• Component-based software engineering:
• Compose sw from off-the-shelf and custom-built components
• Well-defined external interface that hides its internals, independently developed from where it is going to be used, clear specification of what it requires and provides and depends on => it can be composed
• 5C’s principle of separation of concerns separating the communication, computation, coordination, configuration, and composition aspects in the overall software functionality. Design patterns exist to decouple these aspects …
- Composition: group entities together, model interactions (= an art)
- Computation: algorithmic part (the “useful” part of the sw)
- Configuration: change settings of a system
- Coordination: how do entities work together, life-cycle FSM
- Communication
Robotprogrammatie: enkele lessen uit de praktijk
Solutions (lessons learnt)
32
• Evolution towards a more complex but very structured and motivated Composition Pattern as the basic building block
Vanthienen, Klotzbücher, Bruyninckx, "The 5C-based architectural Composition Pattern: lessons
learned from re-developing the iTaSC framework for constraint-based robot programming", Journal
of Software Engineering for Robotics, pp. 17-35, May 2014.
Robotprogrammatie: enkele lessen uit de praktijk
Solutions (lessons learnt)
33
2. Robotics related level:
• Probabilistic robotics: explicitly model uncertainty on all
information sources, and use that uncertainty when taking
decisions
• For intention estimation (plan recognition)
• For shared control
Robotprogrammatie: enkele lessen uit de praktijk
Plan recognition
34
• How to represent intentions?
1. We assume: users wish to reach a certain end pose pe (xe,
ye, θe) with an end velocity ve (ve, ωe)”, e.g.:
Robotprogrammatie: enkele lessen uit de praktijk
Plan recognition
35
2. Furthermore, we assume that users have a certain mental
trajectory in mind to arrive at an end state, e.g.:
i1
Robotprogrammatie: enkele lessen uit de praktijk
Plan recognition
36
• Intention estimation scheme:
1. Generate plan hypotheses i based on local paths or paths to learned end poses or end poses indicated on an a priori map.
Robotprogrammatie: enkele lessen uit de praktijk
Plan recognition
37
2. A probability distribution is maintained over the set of
possible user plans i. Initially, a uniform distribution is
adopted;
3. This distribution is updated every time new user signals uk
are obtained according to Bayes’ rule.
Robotprogrammatie: enkele lessen uit de praktijk
Plan recognition
38
• Example of plan recognition performance:
Robotprogrammatie: enkele lessen uit de praktijk
Shared control
39
• The probability distribution over user plans may be
multi-modal
• Nevertheless, decisions should be made at each
time instant regarding navigation assistance
• 3 approaches have been proposed to make these
decisions:
o Maximum Likelihood (ML)
o Maximum A Posteriori (MAP)
o Greedy Partially Observable Markov Decision Process
(greedy POMDP)
Robotprogrammatie: enkele lessen uit de praktijk
Shared control
40
• Maximum likelihood versus Maximum a posteriori
versus POMDP
Robotprogrammatie: enkele lessen uit de praktijk
Shared control
41
• Example of shared control performance:
o Benchmark test: visit the goal locations 4 – 9 – 6 – 11 – 2 –
14 – 5 – 10 and back to position 4
o Execute this in user control mode, in ML shared control
mode, in POMDP shared control mode
o In shared control, the wheelchair can drive farther than in
user control mode, with the danger of making wrong
decisions and driving too far
Robotprogrammatie: enkele lessen uit de praktijk
Shared control
42
• Example of shared control performance:
Robotprogrammatie: enkele lessen uit de praktijk
Solutions (lessons learnt)
43
3. System level:
• we adopted ROS (Robot Operating System), an open-source
middleware initiative that allows:
• Easy and +/- efficient data exchange
• Tools (rosbags) for data collection and replay
• Dynamic configuration of components’ (nodes) parameters
• Interactive GUI for data visualization
• ... And other (multi-language support – C++/Python), support for distributed systems
Robotprogrammatie: enkele lessen uit de praktijk
Solutions (lessons learnt)
44
• We developed a module base class, built on top of ROS
• Used to share common functionality between modules (reuse)
• This implemented a finite-state-machine: stop/start a component, be robust against errors, ...
• Health monitor: each component sends a heart-beat message to
let a central controller know about the wellbeing of a component
• The central controller sends a heart-beat to a hardware watchdog
• A GUI allows to display the health state of all modules, to
activate/deactivate modules
• This worked for us, but:
• Required lots of integration work and programming
• Will probably be duplicated by others
Robotprogrammatie: enkele lessen uit de praktijk
Outline
45
• ACRO activities: robots with vision and a plan
o Mobile, assistive robots with vision and a plan
o Agricultural robots with vision and a plan
o Industrial robots with vision and a plan
• Programming robots: learnt lessons
o The RADHAR project as an example
o Requirements
o Solutions
• Trends and challenges for the future
Robotprogrammatie: enkele lessen uit de praktijk
Trends and challenges for the future
46
• Probabilistic programming
o Trend towards probabilistic robots (due to many successes)
o Can we incorporate these probabilistic and machine learning
concepts into the programming language?
• E.g. prob distributions as new data type, automatically tune code
using built-in learning functions (program fails => indicate the
correct behaviour and learn from this example)
o Initial successes: much fewer lines of code; e.g. Darpa grand
challenge (autonomous car): 100.000 lines of code
o But: how can it be debugged (randomness plays a crucial
role)
Robotprogrammatie: enkele lessen uit de praktijk
Trends and challenges for the future
47
• Insurance and liability
o Robots should be safe, e.g. implementing obstacle avoidance
� but how reliable are these? Bugs keep popping up, in
unpredictable way (e.g. memory leaks)
o E.g. by feeding random input or pseudo-random input and
see if the system crashes - a sort of automated unit tests
rather than manually crafted unit tests
• Currently, typically only “nominal” behaviour is evaluated; for
safety “worst-case” behaviour is important
• Reliable communication
Robotprogrammatie: enkele lessen uit de praktijk
Trends and challenges for the future
48
• Plug-and-play
o there are no standard methods for connecting sensors,
motors, actuators, cameras and other components to robots
• E.g. plug a device in, detect the device, install a driver, know what
can be performed with it, ...
o It’s starting (Orocos, ROS, OpenCV), but this requires still
much programming at lower levels
• Software deployment support:
o Robot “apps” should be made to work easily on different
platforms, have example programs available and good
documentation
Recommended