1

Prepare for Variation – Towards Using AI/Robotics Methods for Handling

Uncertainty in Agriculture Applications

Joachim Hertzberg

Osnabrück University and

DFKI Robotics Innovation Center (RIC), Osnabrück Branch

IROS Workshop on Agri-Food Robotics, Hamburg, Oct 2nd, 2015

2

1.  Uncertainty in AI

2.  Small Example: Sensor Data Interpretation

3.  Larger Example: Combine Harvester Control

4.  Wrap-up

3

Uncertainty in AI – Generic Methods

Part I Artificial Intelligence Part II Problem Solving Part III Knowledge and Reasoning Part IV Uncertain Knowledge and Reasoning

13 Quantifying Uncertainty 14 Probabilistic Reasoning 15 Probabilistic Reasoning over Time 16 Making Simple Decisions 17 Making Complex Decisions

Part V Learning Part VII Communicating, Perceiving, and Acting Part VIII Conclusions

3rd Ed., 2010

2005

… and in AI/Robotics

4

Dealing with Uncertainty – the AI Arsenal

•  Probabilistic representation & reasoning –  Bayesian networks –  Graphical models

•  Logical r&r about defaults/prototypes and preferences –  Answer set programming

•  R&r about time and space –  Constraint-based quantitative methods, e.g., STPs –  Interval-based qualitative calculi, e.g. interval algebras

Key problem for building applications: combine •  representational efficiency, learnability •  inferential efficiency •  integration of different reasoners & their representations

5

Uncertain ≠ Chaotic!

• Literature (AI, Robotics) often mentions “unstructured environments”

• Nearly always they mean “not 150% controlled”: variety, lack of knowledge, clutter, occlusion, dynamics, … – but normally a high degree of structure! Structure is often known and/or learnable!

• R&r can exploit this structure, if part of a closed-loop control! (and so can learning, perception, action, …)

6

1.  Uncertainty in AI

2.  Small Example: Sensor Data Interpretation

3.  Larger Example: Combine Harvester Control

4.  Wrap-up

7

Own Contributions: Interpreting 3D Point Clouds

… but until now mostly in non-agriculture applications

8

Rigor in Structure – Variation in Detail

Photos: Wikipedia/english “Grape” Image: Wikipedia/deutsch “Weinrebe”

9

Model-Based Phenotyping Steinhage et al.: A Model-Based Approach to High Performance Phenotyping, 2012 Univ. Bonn, CROP.SENSe

Grammar-Based Stem-Skeleton Interpretation Relational Growth Grammar (cf. skeleton models)

Probabilistic Sampling of 3D Scan Structuring Hypotheses Monte-Carlo-based sampling in hypothesis space

10

Model-Based Phenotyping, ctd.

Result Hypotheses •  Length of rachis: 8.50 cm

•  Av. pedicel length: 0.95 cm

•  # berry groups: 18

•  # nodes on rachis: 9

•  # twigs with sub-twigs: 3

•  # terminal pedicel twigs: 9

If this is often enough good-enough (“satisficing”) information …

11

1.  Uncertainty in AI

2.  Small Example: Sensor Data Interpretation

3.  Larger Example: Combine Harvester Control

4.  Wrap-up

12

Cooperative Corn Harvesting Scenario

S. Scheuren, S. Stiene, R. Hartanto, J. Hertzberg, M. Reinecke. Spatio-temporally Constrained Planning for Cooperative Vehicles in a Harvesting Scenario. KI – Künstliche Intelligenz, 27(4):341-346, 2013

Harvested area Non-harvested area Combine harvester Tractor Truck

Driving Paths

Rendezvous point

13

Route Planning for the Tractor Outer field border Inner field border

Driving traces Headland trace Unloading spot

Route graph Overloading window

14

Motion Planning, Structure

Tractor Harvester

Motion Planning

Approximate nominal path

SBPL lattice planner2

Graph search

Approximate nominal path

SBPL lattice planner2

15

Motion Plan for the Tractor

Outer field border Inner field border

Driving traces Headland trace Unloading spot Path of tractor

Overloading window

16

Variety #1: Machine Parameters

Motion Planning ?�Steering system

Path

Sensor data

max.

No AI – just engineering!

17

Variety #2: Change of Yield/Zone

•  Planning based on prior estimation (yield/zone in the past)

•  Significant variation happens –  less yield is no problem –  more yield may require earlier overload

•  No robust solution based on ad-hoc methods

Solution Approach

•  Hierarchical hybrid (causal, temporal, spatial, resource) planning •  CHIMP: Conflict-driven Hierarchical Meta-CSP Planner •  S. Stock, M. Mansouri, F. Pecora, J. Hertzberg. Online Task Merging with a

Hierarchical Hybrid Task Planner for Mobile Service Robots. IROS 2015

18

How to reduce this huge hybrid search space?

Reason about diffe- rent knowledge forms in an integrated way!

HTN Task Decomposition

Meta-Constraint reasoning1

1Cesta, Oddi, Smith: “A constraint-based method for project scheduling with time windows”, 2002. �

Hybrid Search Space

19

•  High-level requirements modelled as meta-constraints on top of a ground-CSP1�

•  Hybrid planning as enforcing consistency in a family of constraint networks2�

•  Meta-CSP framework3 allows to implement solvers for problems that can be cast as a meta-CSP�

1Cesta, Oddi, Smith: “A constraint-based method for project scheduling with time windows”, 2002. �2Mansouri, Pecora: “More knowledge on the table: Planning with space, time and resources for robots”, 2014 �3http://metacsp.org�

Meta-Constraint Reasoning

20

•  Integrates HTN-Planning into meta-CSP reasoning �•  Represents initial state and resulting plan in constraint networks�•  Creates partially-ordered plans �•  Plans may contain actions that can be executed in parallel�•  Allows online task merging of new goals into an existing plan

during plan execution �•  Based on the meta-CSP Framework1�

Freely available at http://sebastianstock.github.io/chimp

1http://metacsp.org

CHIMP

21

… and How Can CHIMP Go Corn Harvesting?

•  Model constraints in hybrid search space –  temporal (e.g., durations, duration variations), –  spatial (e.g., tracks), –  resource (e.g., capacity, inflow)

•  Model plan change strategies as meta-constraints

•  Apply plan change strategies at execution time if needed; keep to domain representation

Work in progress!

22

1.  Uncertainty in AI

2.  Small Example: Sensor Data Interpretation

3.  Larger Example: Combine Harvester Control

4.  Wrap-up

23

Wrap-Up

•  AI has an arsenal of established generic methods for dealing with variations

•  Extending them is subject of intensive current research

•  Their integration may be challenging if more than one type of representation is needed

•  Yet what are the alternatives? –  Ad-hoc methods lack understanding/rigor –  Specific methods are hard to integrate

•  May be a win-win for agricultural applications and AI!

24

CHIMP paper: cf. IROS 2015 proceedings CHIMP planner: http://sebastianstock.github.io/chimp/ �

Meta-CSP framework: http://metacsp.org

Thanks to … … my Staff •  Kai Lingemann •  Stephan Scheuren •  Stefan Stiene •  Sebastian Stock •  Astrid Ullrich •  Thomas Wiemann

… the marion Project (2010–13) •  Ronny Hartanto •  CLAAS and staff •  PT DLR •  BMWi

… the SOILAssist Project (2015–18) •  Thünen Institut •  PT Jülich •  BMBF

… the RACE Project (2011–14) •  Masumeh Mansouri •  Federico Pecora •  Örebro U •  Hamburg U •  EU FP7