QuASI: Question Answering using

Statistics, Semantics, and Inference

Marti Hearst, Jerry Feldman, Chris Manning, Srini Narayanan

Univ. of California-Berkeley / ICSI / Stanford University

Dynamic Probabilistic Inference for event

structure Srini NarayananJerry Feldman

ICSI and UC BerkeleyJan-June 2003

Scenario Question (CNS data) How has Al-Qaida conducted its efforts to acquire WMD

capability and what are the results of this endeavor? Even with perfect parsing, to answer this question, we

have to go beyond words in the input in at least the following ways: Multiple sources (reports, evidence, news)

• Fusing information from unreliable sources (P(Information = true | source))

• Non-monotonicity. Previous assertions or predictions may have to be retracted in the light of new evidence.

Modeling complex events• Evolving events with complex dynamics including sequence,

concurrency, coordination, interruptions and resources.

Reasoning about Events for QA Reasoning about dynamics

Complex event structure• Multiple stages, interruptions, resources

Evolving events• Conditional events, presuppositions.

Nested temporal and aspectual references• Past, future event references

Metaphoric references• Use of motion domain to describe complex events.

Reasoning with Uncertainty Combining Evidence from Multiple, unreliable sources Non-monotonic inference

• Retracting previous assertions• Conditioning on partial evidence

Previous work Models of event structure that are able to deal

with the temporal and aspectual structure of events

Based on an active semantics of events and a factorized graphical model of complex states. Models event stages, embedding, multi-level

perspectives and coordination. Event model based on a Stochastic Petri Net

representation with extensions allowing hierarchical decomposition.

State is represented as a Temporal Bayes Net (T(D)BN).

Factorized Inference

Quantifying the model

Pilot System Results

Captures fine grained distinctions needed for interpretation Frame-based Inferences (COLING02) Aspectual Inferences (Cogsci98, IJCAI 99,

COLING02) Metaphoric Inferences (AAAI 99)

Sufficient Inductive bias for verb learning (Bailey97, CogSci99), construction learning (Chang02, to Appear)

Model for DAML-S (WWW02, Computer Networks 03)

Extensions to Pilot System

Scalable Data Resources Language Resources/Ontology

• Lexicon (Open Source, WordNet, FrameNet)• Conceptual Relations:

• Schemas, Maps, Frames, Mental Space • General Principle: Use Semantic Web resources• (DAML, DAML-S, OpenCYC, IEEE SUMO)

Language Analyzer Construction Parser (ICSI/EML) Statistical techniques (UCB/Stanford,

CU,UTD) Scalable Domain Representation

Coordinated Probabilistic Relational Models

Problems with DBN Scaling up to relational structures Supports linear (sequence) but not

branching (concurrency, coordination) dynamics

Structured Probabilistic Inference

Probabilistic inference for QA

Filtering• P(X_t | o_1…t,X_1…t)• Update the state based on the observation sequence

and state set MAP Estimation

• Argmaxh1…hnP(X_t | o_1…t, X_1…t)• Return the best assignment of values to the hypothesis

variables given the observation and states Smoothing

• P(X_t-k | o_1…t, X_1…t)• modify assumptions about previous states, given

observation sequence and state set Projection/Prediction/Reachability

• P(X_t+k | o_1..t, X_1..t)• Predict future states based on observation sequence

and state set

PRM (and DBN) inference is hard Exact Inference Techniques (NP):

Variable Elimination (VE) Junction-Tree Methods

Approximate inference (NP): Variational Approximations Loopy propagation (loses information)

Tractable inference and net topology Polytree-inference is tractable (Pearl

1990) Proportional to Network Size

SCFG-inference can be modeled as extended Polytree inference (Narayanan 99)

For more complicated models, exploit relational structure (Pfeffer 99, Kohler et al 00, 02).

Probabilistic Relation Inference

Scalable Representation of States, domain knowledge, ontologies

• (Pfeffer 2000, Koller et al. 2001) Merges relational database technology with

Probabilistic reasoning based on Graphical Models. Domain entities and relations. Inter-entity relations are probabilistic

functions Can capture complex dependencies with

both simple and composite slot (chains). Inference exploits structure of the domain

Inference With PRMsSVE inference for a PRM P with q query variables and N attributes is

O(Nkbk(m+2)bq) (Pfeffer 2000) k is the maximum number of interface

variables q is the number of query variables m is the maximum tree width for any

object in P (related to the markov blanket).

Controlling PRM inference The number of interface variables, k, is related to

the number of relations that a variable participates in as well as the number of slot chains that the variable participates in Careful selection of relations (only part-of) can

make inference tractable. The tree width m depends on the markov blanket

of an attribute. Control of network topology can reduce this.

Adding Time to PRM’s Since time is another relation, doesn’t increase expressive

power. Significant impact of inference tractability since both k

and m may become quite large. New Algorithm: Exploit the structure of time using the

interface and frontier algorithm (Murphy 2002). Variables at slice t with links to variables at t+1 form

the interface Interface variables d-separate the past (< t) from the

future slices (> t). Allows for on-line inference algorithms similar to inside-

outside algorithm for SCFG’s.

The CPRM algorithm Combines insights from

the SVE algorithm for PRMs (Pfeffer 2000) the frontier algorithms for temporal models

(Murphy 2002) and Inference algorithms for complex, coordinated

events (Narayanan 1999) Expressive Probabilistic Modeling paradigm with

relations and branching dynamics. Offers principled methods to bound inferential

complexity.

Status of CPRM inference

Spring-Summer 2003• Design Dynamic Probabilistic Relational Models

(DPRM)• Initial Design of CPRM inference algorithm• Integrate Parser with existing Pilot System

• Steve Sinha Summer/Fall 2003

• Implement CPRM to replace Pilot System• Nathaniel Smith, Eva Mok

• Test CPRM for QA (UTD) Related Work

• Probabilistic OWL (PrOWL)• Probabilistic FrameNet

Conclusion QA with complex scenarios (such as the CNS

scenario/data) needs complex inference that deals with Relational Structure Uncertain source and domain knowledge Complex dynamics and evolving events

We have developed a representation and inference algorithm that is capable of tractable inference for a variety of domains.

We are collaborating with UTD (Sanda Harabagiu) to apply these techniques to QA systems.