Task Learning by Instruction: Benefits and Challenges for Intelligent Interactive Systems

1USC INFORMATION SCIENCES INSTITUTE Tailor, Spring 07

Task Learning by Instruction:Benefits and Challenges

for Intelligent Interactive Systems

Jim Blythe, Prateek Tandon and Mandar TilluUSC Information Sciences Institute


Overview

Complex tasks in intelligent assistants

Overview of task learning by instruction in Tailor

Integration with other task-related capabilities

Challenges and new approaches in specifying tasks


Handling complex tasks in an intelligent assistant

Assistant

UI

Task models

Domain models

Travel

Contracts

Medication Regime

Vacation Activities


Specialized capabilities for complex tasks

Assistant

UI

Task models

Domain models

ApplicationUser invocationAdvice/ PreferencesExplanationMonitoringTrouble-shootingAutonomyOpportunism/ Synergy….

Plan creationLearning/ InstructabilitySharing tasks and goals


Specialized capabilities for complex tasks

Assistant

UI

Task models

Domain models

ApplicationUser invocationAdvice/ PreferencesExplanationMonitoringTrouble-shootingAutonomyOpportunism/ Synergy….

Plan creationLearning/ InstructabilitySharing tasks and goals


Modifying a procedure with Tailor:“Task Learning by Instruction”

User enters text instruction, e.g. “You don’t need authorization when the cost is below $2000.”

Tailor interprets by

(1) recognizing type of modification,

(2) mapping terms to its task and world ontology,

(3) analyzing the proposed modification.

To purchase equipment: 1. find two bids 2. fill out form 10C 3. request authorization from two managers 4. if authorization granted place order with purchasing

To purchase equipment: 1. find two one bids 2. fill out form 10C 3. if it costs > $500 request authorization from two managers 3½. record purchase on web site 4. if authorization granted place order with purchasing

{defProcedure PurchaseEquip cue: [do: (purchase $obj_id)] precond: (and (user $user) (pSelId $obj $obj_id)) body: [seq: [do: (find_bids $obj $bid)] [context: (use_form $form $obj) do: (complete_form $form $obj)] [select: (and (total_price $bid $price) (< $price 2000)) [do: (get_authorization $obj $user)]]]}


Modifying a procedure with Tailor compared with learning from observation/demonstration

Benefits:

Challenges:

Users directly describe the desired change Users describe steps without taking them Users can ignore implementation details

Map user instruction to procedure knowledge using context (e.g. other intentions)

Navigating a set of complex alternatives Follow-up questions to resolve ambiguity and

help fix introduced problems

[Blythe, IUI 05; AAAI 05]


User describes a new taske.g. “list hotels within +miles of +meeting”


The first step

“List each hotel name”


0:1:(Price $sel’n $p+), ..2: (and (HasDisplay $sel’n $d) (Size $d $s)), ..

Aggregating DP approach to find compositions of tasks

ComputerTotalPrice,ShippingCost, RAM, …

HasDisplay

Size

0:$selection (GPA)“the selected laptop”

0:US (constant)1:

<, >, …

Vendor

0:1:(Vendor $sel’n $v+)

Country

2:(and (Vendor $sel’n $v) (Country $v US))

match

DialCode

User instruction: “you don’t need form B when the seller is in the US”

Tailor finds matching expression: (and (vendor $selection $v) (country $v US))

Shown as: “when the country of the vendor of the selected laptop is the US”

laptop

company

0:1:(HasDisplay $sel’n $d+)

displaycountry

number

Variable and description from GPA

Object from user instruction

Synonym fromWordNet: seller


Recent improvements: extend DP with dynamic grouping of partial matches

Group partial expressions by datatype and set of instruction words matched invariant wrt. propagation and matching

Propagate groups, not individual expressions Exponential reduction in propagations, and in iterations

required to find a match

0:$selection (GPA)“the selected laptop”

Vendor

0:1:(Vendor {lapt: Ø})

laptopcompany

0:US (constant)1:

Country

2:(and {cmpny: seller} (Country ? US))

match

country$selection ε {laptop: Ø}

(Vendor {laptop: Ø}) ε {company: seller}

… ε {country: seller, US}


The last step and resulting procedure


Overview

Task learning in desktop systems

Overview of task learning by instruction

Integration with other task-related capabilities

Challenges and new approaches in specifying tasks


Challenges highlighted by Tailor prototype

Users unsure of the “next step” More context-dependent menus and wizards Fewer options, although there are several valid alternatives

Many alternative input sets for primitive procedures Full list is overwhelming: be selective and summarize Constraints on inputs of primitive procedures

Procedure hard to understand Combining tasks built by users – inappropriate information Hide substeps Hide parameters Allow drill-down


Task-related capabilities: communication and reasoning

Communicating with user

InvocationExplanationAdvice/ preferences

Reasoning

Learning by instruction/ demo Autonomy

Plan creationMonitoringLearning by observationOpportunismCollaboration


Select modification: Add step

Add input

In order to find available hotels near the meeting with input a meeting:

No steps currently defined

Done

Context-related guidance

Add a step


Describe step to add:

In order to find available hotels near the meeting with input a meeting:

get hotels OK

get hotel chains with frequent flyer miles for American

Optionsget hotel chains with frequent flyer miles for American

get hotel chains with frequent flyer miles for United

get hotel chains with frequent flyer miles for Delta

get hotels from onlineReservationz

Hiding steps and parameters

Hide parameters: focus on distinct

actions



In order to find available hotels near the meeting with input a meeting, a street and a zip:

list hotel if distance < 2 OK

get hotels from onlineReservationz for each hotel: get Yahoo distance if distance < 2 add the hotel to the list

OptionsOptions

Heuristic: show external actions (e.g. info access), hide internal book-keeping

Show steps that determine

information access

Highlight new inputs that are automatically

suggested



In order to find available hotels near the meeting with input a meeting, a street and a zip:

list hotel if distance < 2 OK


OptionsOptions

distance between the street and zip, and the hotel street and zip

options that don’t use the hotel street and zip

options that don’t use new input variables

Show parameters when the user asks to see alternatives

Explore only through the inputs, not how

they are used

Show more detail about steps when the user explores the alternatives


Reduce search space using constraints on inputs

Given 3 locations:

YahooDistance(street1, zip1, street2, zip2): 81 alternatives

Given: ‘location’ = tuple(street, zip) YahooDistance(loc1, loc2): 9 alternatives

Given: output is a distance metric: symmetric, d(X,X) = 0 YahooDistance(loc1, loc2): 3 alternatives


Constraints may be reusedfrom task ontology

Action

Information-generating action

Information-generating action:output includes a distance betweentwo inputs A and B

YahooDistance(A:loc, B:loc) -> distance_miles

Symmetric: only consider one orderingconstraints

Identity: d(A,A) = 0, so ignore B = A.

loc = compound(street, zipcode)


Done


Output is the list of hotels

Select modification:

In order to find available hotels near the meeting given a meeting, a street and zip:

High-level organization of outputs

Add step before this

Add test

Remove this step

Mark step as output

Change step inputs

Show details

Output is created in list form because the constructor is embedded in an iteration


Done


Output is the list of hotels

Select modification:

In order to find available hotels near the meeting with input a meeting, a street and zip:

Instead of

Result: simpler description and control


Current work

Creating primitive procedures while creating top-level procedure, not before.

Using examples from previously-build procedures to select steps, bindings and follow-ups

Constraints: representation, input and use


Summary

Learning complex activities from instruction. Similar issues in modules that communicate about tasks

Many potential matches for an instruction

Describing learned tasks: find the ‘right’ level of initial detail and allow drill-down

Utilize constraints on possible procedure inputs

Documents

Task Learning by Instruction: Benefits and Challenges for Intelligent Interactive Systems