L C S

Supporting Technology for Group Interaction

Howard Shrobe

MIT AI Lab

Oxygen Workshop, January, 2002

L C SA Three Pronged Attack on Collaborative Knowledge Management

• Ubiquitous human-centered, perceptually enabled environments.

• An adaptive infrastructure that understands the context and content of problem solving discourse (at least a little)– It accurately indexes all information.

– It distributes information to those who can use it.

– It can find resources (information and people) to help a collaboration.

• Tools informed by an understanding of the domain and the organization.– An “almost expert” (apprentice) system

* Which knows when to ask questions:

* What it doesn’t understand is often what’s most important

* What isn’t obvious is what needs to be documented and distributed

• Documentation almost for free by embedding a helpful computational system in our normal work flow

L C SAn Example of Group Interaction and Shared Plans

ClarifyInterface

Propose

Code

Document

Brainstorm

Critique

The side-tracker interface would be much clearer if it worked this way!

I think I can make it work that way, but does it affect anything else!

That should work, as long as it doesn’t break the special hack for managers.

I’ll look into that, you all can start the coding and doc updates

Plan

L C S

Plan

KB

GoalPlan

KnowledgeBase

Haystacks

Knowledgeacquisition

Facilitator

SharedPlans

SharedInfo

Shared Semantic Web

Inference

ResourceDiscovery

Facilitator Agent

Knowledge-Based Infrastructure Helps People Collaborate:

• By acting as an assistant to meeting facilitators it can help to capture important information such as:– Issue, positions, arguments for and against positions– Commitments, action items– Video and audio transcripts of meetings

• By understanding the role of individuals within the organization it can help to decide who should see what information.

• By understanding the technical capabilities of individuals it can help to match “who can do what” to “what is required”.

L C S

• Information nodes:• Goals of the project• Proposed methods for the goals• Arguments in favor and against methods• Documents supporting these arguments

• Node format:• Much of the content is opaque to the system

(e.g. Multi-media fragments)• Some slots are understood by the system• And, short natural language annotations can be

attached, parsed and understood by START• Formal representations are also possible

• Relational information:• Links showing the relationships between the

information nodes• Link types are meaningful to the system

• Organizational descriptions: • Resource descriptions including capabilities,

roles and interests (for personnel)• Process goals and plans

Meeting Manager

Plan

KB

GoalPlan

KnowledgeBase

Haystacks

Knowledgeacquisition

Facilitator

SharedPlans

SharedInfo

Shared Semantic Web

Inference

ResourceDiscovery

email web

The Oxygen Collaboration Infrastructure:Knowledge Based Collaboration Webs

L C S

Oxygen Can Support Human Collaboration by Making Simple Inferences

• Oxygen deduces new information using:– The background knowledge base– The types of the links– The slots of node structures that are understood– The semantic content of the node annotations– Descriptions of the resources and people in the organization

• Oxygen routes information and discussion topics to those whose:– Organizational role requires it– Interests suggest it– Capabilities and skills might be useful

• Oxygen posts new goals and initiates new discussion processes to address these goals.

• Oxygen supports human interactions in a manner relevant to the organization’s problem solving context

L C SKey Oxygen Components Make This Much Easier by Bringing Computers Into Our World

Microphone Array

Tracking Cameras

Video Displays

Pointing Camera

Pointing Camera

L C SA Meeting Web

Video/Audio Transcript (quicktime movie)

Position 1Start time;End time:

Position 2Start time;End time:

Discourse Structure

Issue 1Start time;End time:

Issue 2Start time;End time:

Supporting Argument

Start time;End time:

Refuting Argument

Start time;End time:

Agenda ItemStart time;End time:

Agenda ItemStart time;End time:

Agenda ItemStart time;End time:

Meeting Structure

CommitmentStart time;End time:Who:DeadlineDiscourse

CommitmentStart time;End time:Who:DeadlineDiscourse

Action Items

CommitmentStart time;End time:Who:DeadlineDiscourse

People

L C S5 Keys Challenges for Adaptive Interfaces

• Providing a practical level of knowledge representation that enables groups interactions and grounding in the real world of space and time

• Providing services in a multi-user environment while making optimal use of the currently available resources

• Recovering from equipment failures, information attacks, etc.

• Coordinating and fusing information from many sensors and modalities

• Capitalizing and recognizing context

• Maintaining Security and Privacy and trading these off against other goals

L C S

Challenge 1: Grounding in Real-World Semantics

• We want to build applications that service many individuals and groups of individuals

• These people will move among many physical spaces

• The devices and resources they use change as time progresses

• The context shifts during interactions

• The relevant information base evolves as well.

L C S

Research Agenda: Knowledge Representations

• People– Interests, skills, responsibilities, organizational role

• Organizations– Members, structure

• Spaces– Location– Subspaces– Devices and resources

• Resources• Information nodes

– Topic area, place in ontology, format• Services

– Methods, parameter bindings, resource requirements• Agents

– Capabilities, society, acting on behalf of whom• Events

– People identification, motion in new space, gestures

L C S

Challenge 2: Adaptive Resource Management

• In most systems, applications are written in terms of specific resources – (e.G. The left projector in michael’s office, or worse yet physical

address).

• This is in conflict with– Portability across physical contexts

– Changes in equipment availability across time

– Multiple applications demanding similar resources

– Need to take advantage of new resources

– Need to integrate mobile devices as they migrate into a space

– Need to link two or more spaces

• What is required is a more abstract approach to resources in which no application needs to be tied to a specific device.

L C S

AbstractService

ServiceControl Parameters

User’sUtility

Function

The binding of parameters has a value to the user

Resource1,1

Resource1,2

Resource1,j

Each method requires different resources

The system selects the method which maximizes net benefit

User requests A

service with certain parameters

ResourceCost

Function

The resourcesUsed by the methodHave a cost

Net benefit

Each method binds the settings of The control parameters in a different way

Method1

Method2

Methodn

Each service can beProvided by several

Methods

The system adapts by having many plans for each service

L C S

Security and Privacy Issues Are Addressed by Factoring in the Cost of Violating a Security or Privacy Policy

ResourceCost

Function

Resource1,1

Resource1,2

Resource1,j

AbstractService

User’sUtility

Function

Net Benefit

Method1

Method2

Methodn

Security PoliciesSecurity Policies

Challenge 6: Security and PrivacyIs Addressed by This Infrastructure

L C S

Challenge 3: Robustness and Recovery From Failures

• The intelligent room renders services by translating them into plans involving physical resources– Physical resources have known failure modes

• Each plan step accomplishes sub-goals needed by succeeding steps– Each sub-goal has some way of monitoring whether it has been

accomplished

– These monitoring steps are also inserted into the plan

• If a sub-goal fails to be accomplished, model-based diagnosis isolates and characterizes the failure

• A recovery is chosen based on the diagnosis– It might be as simple as “try it again”, we had a network glitch

– It might be “try it again, but with a different selection of resources”

– It might be as complex as “clean up and try a different plan”

L C S

I need to ask a question of a systems

wizard

Plan 1: • Locate A systems wizard in the E21

• Monitor: check that person is still there• Turn on the selected projector

• Monitor: check that projector turned on• Project the message• Done

• Monitor: check that the person noticed the message

I don’t see light on the screen

I see sally by the screen

Projector-1 must be broken.We’ll try again, but using

Projector-3.

PlanBreakdown

Recovering From Failures

L C S

Frobulate the sidetracker

White BoardPlace Manager

Sally is near The Whiteboard

If A Person approaches A device ?x and Grammar ?y is relevant to ?xThen Activate Grammar ?y

Activate the Drawing Grammar

The Drawing GrammarIs Relevant to the Whiteboard

Challenge 4: Exploitation of Context

• The attention of the system should be focused by what people in the facility do and say.

• Speech recognition should be biased in favor of things going on in the facility– Speech system is made up of many “ grammar fragments”

– Grammar fragments are activated (and deactivated) when perceptual events (visual or speech) suggest they should be

• Visual Interpretation should also be biased by context

L C SChallenge 5: Perceptual Coordination

• Separate the implementation of perceptual tasks from the uses to which perception is put

• All perceptual modules advertise the class of “behavioral events” they are capable of recognizing and signaling– These events are organized into a taxonomy– Some events are “ close to the physics” (e.g. an object was

observed at location <x,y>)– Some events are more abstract (e.g. a person is near the white-

board)– The same event can be signaled by quite different perceptual

modules (e.g. both face and voice recognition can localize a person).

• Other modules register their interest in certain classes of events– Requests at a higher level in the taxonomy subsume lower level

events

• Modules which receive low-level events may register for and collate many different classes of events – They combine these and signal higher-level events

L C SResearch Agenda: A Dynamic Event Bus For Perceptual Coordination

VisualTracke

r

Signal body motion

VoiceIdentification

Interested in body motionSignal the location of individuals

FaceRecognition

Interested in face locationI signal the location of individuals

White BoardContext Manager

Interested in the location of individualsSignal people approaching the whiteboard

FaceSpotter

Interested in body motionI signal face location

L C S

Plan

KB

GoalPlan

KnowledgeBase

Haystacks

Knowledgeacquisition

Facilitator

SharedPlans

SharedInfo

Shared Semantic Web

Inference

ResourceDiscovery

Facilitator Agent

Oxygen’s Knowledge-Based Infrastructure Helps People Collaborate:

• By acting as an assistant to meeting facilitators it can help to capture important information such as:– Issue, positions, arguments for and against positions– Commitments, action items– Video and audio transcripts of meetings

• By understanding the role of individuals within the organization it can help to decide who should see what information.

• By understanding the technical capabilities of individuals it can help to match “who can do what” to “what is required”.

• By understanding the context within which people act, it can react in appropriate ways.