AQUAINT R&D Program: “State of the Program” Phase I 18-Month Workshop 9-12 June 2003 Dr. John D. Prange AQUAINT Program Manager [email protected] 301-688-7092

AQUAINT R&D Program:“State of the Program”

Phase I 18-Month Workshop9-12 June 2003

Dr. John D. PrangeAQUAINT Program Manager

[email protected]

http://www.ic-arda.org

AQUAINT Ph I 18-Month Wkshp – 9-12 June 2003 2

Focus of Workshop

• Technical Talks– AQUAINT Project Highlights; Not Status Reports– Written Project Status Reports included on Workshop CD

• Two Separate Demos Sessions

• Exercise Focused on Advanced Question Answering Scenarios

• Program Updates on:– Data - Testbed - Evaluations

• Birds of a Feather Sessions Wed Evening

• Update on Phase II Planning and Schedule


Still More to Talk About

• AQUAINT Program Symposium Series

• NRRC Summer Workshops

• Other QA Related Activities– AAAI Spring Symposium Workshop– NAACL-HLT 2003 Conference– Continuing evolution of the TREC QA Track– Question Answering Books in the works– Coordination with other Related Programs– The Learning Federation’s Question Answering Roadmap– Vulcan Inc. HALO Projects

• AQUAINT Program Phase I 24-Month Workshop

• A Couple of Special Recognitions


AQUAINT Symposium Series

• Need for Something More than Program Workshops

– Program is sponsoring multiple approaches to Advanced Question Answering

– Significant Progress is being made is multiple areas

– My own very positive experience from Project Reviews/Site Visits

– Smaller setting; comprehensive overview of project activities, approaches and successes; highly interactive setting

• 20-30 minute overview presentations in Workshop setting just aren’t the same

• The Idea of a Symposium Series is born


AQUAINT Spring Symposium - 2003

• Rapidly Organized by ARDA with help from NIST (Donna Harman, Ellen Voorhees) and ARDA’s Northwest Regional Research Center (Rich Quadrel)

• Initial Symposium held at NIST, Gaithersburg, MD on Mon-Tues 5-6 May 2003

• Format:

– 4 half-day detailed technical overview presentations by four AQUAINT Program teams

– Attendance limited to AQUAINT Program PI’s, a few Government personnel and invited guests

– Encouraged high degree of interaction between presenters and audience


AQUAINT Spring Symposium – 2003 (con’t)

• Presenters:– Language Computer Corporation (LCC) (Combination of 3

Projects)– University of Southern California / Information Science

Institute (TextMap Project)– IBM T. J. Watson Research Center (PIQUANT Project)– Cycorp Inc. (QUIRK Project)

• Results– One significant glitch with Cycorp’s Internet connection– Lively “Discussions” & Interactions between researchers

from different teams – It was great to see & hear!– Format proved to be right on the money– CD with all presentations will be published by NWRRC– Sessions videotaped; Future distribution is a possibility


AQUAINT Fall Symposium - 2003

• Hugh Success of Spring Symposium implies we’ll try it again

• Tentative Dates:– 8-9 October 2003 (Wed & Thurs) (Note: Yom Kippur is on Tues 7

Oct – Any other known conflicts??)

– West Coast location likely – NWRRC is searching for a location

• Symposium Focus– Focus of Spring Symposium was on “End-to-end Systems”

– What should the focus of Fall Symposium be?

– Volunteers by teams to make Presentation?

• Symposium Format– Keep things pretty much the same – increase size only slightly to

accommodate on a case-by-case basis more than just the PI from a given Project

– Honorarium again used to cover travel costs


Northeast Regional Research Centerhttp://nrrc.mitre.org

• Conduct 6-8 week workshops on multiple ARDA-related challenge problems– Three AQUAINT Focused Workshops conducted during Spring /

Summer 2002– Three AQUAINT Focused Workshops to be conducted during

Summer 2003

• Analytic Disciplines Day (3 Feb 2003) / Intelligence Analysis Futures Day (27 June 2003)– Periodic day-long set of talks focused on some aspect of

Analysis– Held at MITRE-Bedford but video teleconferenced to a number of

other MITRE locations

• Participation/Attendance open to interested contractors and to Government researchers/technologists/analysts

Hosted By MITRE, Bedford, MA


FY2002 NRRC Wkshp Challenge Problems

1. Temporal Issues

– Generate Sequence of events and activities along evolving timeline, resolving multiple levels of time references across series of documents/sources. Developed:

1. TimeML: Definition and design of a Metadata Standard for Markup of events, their temporal anchoring, and how they are related to each other in News articles.

2. TIMEBANK: Creation of a gold standard corpus of 300 articles marked up for temporal expressions, events, and basic temporal relations, based on the specification of TimeML.

– Leader: James Pustejovsky, Brandeis University

TERQAS Web Site: http://time2002.org


3. Re-Use of Accumulated Knowledge– Investigate strategies for structuring and maintaining

previously generated knowledge for possible future use. E.g. previous knowledge might include questions and answers (original and amplified) as well as relevant and background information retrieved and processed.

– Leaders: Marc Light, MITRE and Abraham Ittycheriah, IBM


2. Multiple Perspectives

– Develop approaches for handling situations where relevant information is obtained from multiple sources on the same topic but generated from different perspectives (e.g. cultural or political differences).

– Leader: Jan Wiebe, University of Pittsburgh


2. Graphical Annotation Toolkit for TimeML

– Address some specific annotation and user interface problems that arose in the context of the TERQAS workshop in 2002:

1. Development of a Text Segmented Closure Algorithm

2. Creation of a Semi-graphical Annotation Tool

– Leader: James Pustejovsky, Brandeis University


1. Toward More Reliable Information Retrieval Technology

– Explore issues in IR performance and QA relating to performance variability as a function of topic, collection, queries & other contextual aspects

– Leader: Donna Harman, NIST


3. Preparing to Explore a New Paradigm in Information Access: A Scenario Approach to Question-Answering

– Jointly Explore with QA Researchers and Subject Matter Experts the nature and implications of the emerging QA Scenario Paradigm and report back on:

1. The new paradigm from the intelligence analysts’ perspective,

2. The more advanced capabilities being developed beyond simple factoid QA

3. The scenarios proposed by the Subject Matter Experts,

4. The range of QA capabilities needed to make scenario-based QA a reality.

– Leader: Liz Liddy, Syracuse University



Regional Research Centers:FY 2004 Workshop Challenge Problems

• Resource Enhancement Program’s Regional Research Centers (ARDA POC: Penny Lehtola)– Northeast: MITRE, Bedford, MA

• Mark Maybury Director http://nrrc.mitre.org

– Northwest: Battelle PNNL, Richland, WA • Rich Quadrel, Director

– Southern CA: SPAWAR, San Diego, CA• John Audia, Director

• Call for Workshop Proposals in Fall 2003 Timeframe– Process concerning Call for Proposals is being refined

– Strongly Encourage AQUAINT-Focused Workshops

– Talk to me if you are interested; if you have an idea; if you want to brainstorm an idea

– Will send out email to AQUAINT PI’s when Call for Proposal Process has been finalized


Related External Workshops

• Workshop on New Directions in Question Answering

AAAI Spring Symposium

24-26 March 2003

Stanford University

Palo Alto, CA

Mark Maybury, MITRE

Workshop Chair

http://www.aaai.org/Symposia/Spring/2003/sss-03.html


Related External Conferences

• HLT (Human Language Technology) Conferences– Sponsored by NAACL & Government Funders DARPA, NSF, and

ARDA; SIGIR and ISCA represented on HLT Advisory Committee

– NAACL/HLT-2003: Edmonton, Alberta, Canada 27 May - 1 June 2003

http://www.hlt03.org

– Solid participation from AQUAINT Program• Number of Technical Talks• Pre-Conference Tutorial on TimeML• Post-Conference Workshop on Gazetteer Exploitation

– NAACL/HLT-2004: Continuation endorsed by both NAACL and Government Funders

• Likely location: Boston, MA• Likely Timeframe: 2nd Week in May 2004


Related External Conferences

• Text REtrieval Conferences (TREC) Question Answering (QA) Track

– TREC managed by NIST

– Co-Sponsored by DARPA and ARDA

– TREC-2003: NIST, Gaithersburg, MD 18-21 November 2003

http://trec.nist.gov

– QA Track Question Types - 2003• Factoid Questions

• List Questions

• Definition Questions

– Evolving Nature of QA Track in TREC & AQUAINT Program Evaluations


Question Answering Books

• TERQAS Book– Outgrowth of NRRC 2002 Summer Workshop– James Pustejovsky, Brandeis University, Editor

• Future Directions in Question Answering– Outgrowth of AAAI Spring Symposium 2003– To be published by MIT Press– Mark Maybury, MITRE, Editor

• Advanced Question Answering Topics – Focus on newly emerging approaches to advanced QA– To be published by Kluwer Press– Co-Authors:

• Tomek Strzalkowski, SUNY Albany

• Sanda Harabagiu, University of Texas at Dallas


Coordination with Other Government Sponsors of Information Technology R&D

• Periodic Coordination Meetings

• Program Managers Involved– DARPA: Charles Wayne, Ted Senator– NSF: Gary Strong– ITIC: Art Becker– ARDA: John Prange, Lucy Nowell– Intelligence Community: Steve Dennis; J.K. Davis

• Programs Covered– DARPA:TIDES, EARS, EELD, Info Awareness + Others– NSF: Information Technology Research - ITR– ITIC: KD-D– ARDA: AQUAINT, NIMD + Others– IC: ACE + Others

• Some Coordination with DARPA’s DAML and RKF Programs


The Learning FederationQuestion Answering Roadmap

• Question Generation and Answering Systems R&D for Technology-Enabled Learning Systems– “An important component of advanced learning systems will be

tools for answering learner queries. A key R&D issue is how to use what we know about question generation and question answering to design learning environments that guide learners to actively construct knowledge.”

– Published a draft Roadmap Document:

http://www.learningfederation.org/workshops/wks2/index.html

– POC: Kay Howell; [email protected]

– Affiliated with Federation of American Scientists

• Workshops:– Oct 2002, Univ. of Memphis, Memphis, TN

– April 2003, MITRE, Bedford, MA


Plans for AQUAINT Program Phase I 24-Month Workshop

• When & Where– 2-5 Dec 2003 (Tues-Friday)– Opening Reception Evening of 1 Dec– MITRE, McLean, VA

MITRE 1 Conference Facility (formerly Hayes Building)7525 Colshire DriveMcLean, VA 22102-7508www.mitre.org/about/locations.html

• Workshop Focus

– Final Overview Summary of Phase I

– Project Summary Presentation (30-60 minutes each)• Highlight significant research activities and results

• Focus on Contributions to and Impact on the AQUAINT Program

• Final System Demonstrations

MITRE 1


Special Recognition:AQUAINT Program Executive Committee

• AQUAINT Program Executive Committee formed in January 2002; Meets Monthly

• Intelligence Community Members:– ARDA (John Prange; Heather McCallum-Bayliss-SETA; Paul Matthews-

SETA)

– CIA (John Donelon, Steve Maiorano, Jean-Michel Pomarede-SETA)

– DIA (Kelcy Allwein)

– NIMA (Duncan McCarthy)

– NSA (Carol Van Ess-Dykema, J.K. Davis, Mike Blair-SETA)

• Other Members/Advisors:– NIST (Donna Harman, Ellen Voorhees)

– MITRE (Scott Mardis, John Burger)

– Tarragon Consulting (Richard Tong)


Special Recognition ! !

Uppsala University24 Jan 2003Web Cam

“I have … climbed the Matterhorn, driven a Land Rover from London to Capetown, gotten attacked by a thousand people in Egypt, gotten stuck in quicksand in the interior of Iceland, flown in a Russian cargo plane to Timbuktu, followed orangutans around the Borneo rain forest, narrowly avoided being kidnapped in Yemen, etc.”

• Three Decades of work in AI and Computational Linguistics

• Information Seeking Dialogs• FASTUS

• Scandinavia• Venerable Swedish University• Annual Site of Nobel Prize Ceremony

First Some Hints . . .


Special Recognition ! !

First Some Hints . . .

But What’s The Connection?

Uppsala University24 Jan 2003Web Cam • Three Decades of work in AI

and Computational Linguistics• Information Seeking Dialogs• FASTUS

Jerry HobbsUSC / ISI

• Scandinavia• Venerable Swedish University• Annual Site of Nobel Prize Ceremony

“I have … climbed the Matterhorn, driven a Land Rover from London to Capetown, gotten attacked by a thousand people in Egypt, gotten stuck in quicksand in the interior of Iceland, flown in a Russian cargo plane to Timbuktu, followed orangutans around the Borneo rain forest, narrowly avoided being kidnapped in Yemen, etc.”


CONGRATULATIONS ! !

• On January 24, 2003, Uppsala University in Sweden bestowed the title of Doctor of Philosophy honoris causa on Jerry Hobbs of USC / ISI and formerly of SRI International

• The AQUAINT Family extends our Congratulations to Jerry ! !

• And if we haven’t embarrassed Jerry enough, here are some more pictures! **

** From his colleagues at USC / ISI


Gets ready

Receives degree

The laurel wreath

Our hero rehearses

AQUAINT Ph I 18-Month Wkshp – 9-12 June 2003 26Anna Sagvall-Heinof Uppsala University

The Banquetin the Castle

Jerry bought the artillery shell fired in his honor


Questions / Comments


Contact Information

Dr. John Prange, AQUAINT Program Director

• ARDA Web Pages: http://www.ic-arda.org

• Email [email protected] [email protected]

• Phones: 301-688-7092800-276-3747301-688-7410 (Fax)

• Mailing: ARDA (RA)Room 12A69 NBP#1

STE 66449800 Savage Road

Fort Meade, MD 20755-6644

Documents

AQUAINT R&D Program: “State of the Program” Phase I 18-Month Workshop 9-12 June 2003 Dr. John D. Prange AQUAINT Program Manager [email protected] 301-688-7092