View
218
Download
0
Category
Tags:
Preview:
Citation preview
GATEOverview and Demo
University of WashingtonCLMA Treehouse Presentation
October 8, 2010Prescott Klassen
Overview
• Summary of GATE information and documentation found at gate.ac.uk
• GATE Developer features, components, and plug-ins
• IDE Demo• Embedded GATE• Using GATE with Condor on Patas• GATE code samples
Background• Sheffield Natural Language Processing Group at the University of
Sheffield• Released 1996 – re-written and re-released 2002 • Latest Release GATE 5.2.1 (May 6, 2010) – Windows, Linux,
Solaris, and Mac OS• Beta Release GATE 6.0 (Beta 1 – August 21, 2010)• 100% Java Reference Implementation• Compatible with IBM Unstructured Information Management
Architecture (UIMA)• Open Source (GNU Library General Public License)• XML Corpus Encoding Standard (XCES) format, used by the
American National Corpus
What is GATE?
• An architecture describing how language processing systems are made up of components.
• A framework (class library) written in Java and tested on Linux, Windows and Solaris.
• A graphical development environment built on the framework (IDE for NLP)
GATE Products• GATE Developer
– IDE for language processing components bundled with the ANNIE (A Nearly-New Information Extraction system) and plug-ins
• GATE Teamware– Web app for collaborative semantic annotation projects incorporating a
workflow engine and a backend service infrastructure • GATE Embedded
– Object library optimized for inclusion in applications• GATE Services
– Hosted services for cloud application development• GATE Wiki
– Wiki/CMS• GATE Cloud
– Cloud computing solution for hosted large-scale text processing
GATE Components
• Language Resources (LRs)—documents, corpora and ontologies
• Processing Resources (PRs)—parsers, stemmers, co-reference resolvers, ML components, etc.
• Visual Resources (VRs)—IDE components that provide a visual interface (GUI) to GATE components and plug-ins
Language Resources
• Documents, corpora, and ontologies• Can persist in Java Serial Store or Lucene Serial Data Store• Document = content + annotations + features• “Stand-off” Markup• Annotations as Directed Acyclic Graphs (start Node, end
Node, ID, type, Feature Map, pointers into the sources document—character offsets)
• Input Formats: Plain Text, HTML,SGML,XML, RTF, Email, PDF, Microsoft Word
• Ontology support (Sesame2,OWLIM3)
Processing Resources
• ANNIE (a Nearly-New Information Extraction System)– Document Reset– Tokeniser– Gazetteer– Sentence Splitter– RegEx Sentence Splitter– Part of Speech Tagger– Semantic Tagger – Orthographic Coreference (OrthoMatcher)– Pronominal Coreference
Processing Resources
• JAPE (Java Annotation Pattern Engine): – Regular expressions over annotations– Finite state transduction over annotations based on
regular expressions– Not against strings but against annotation graphs– Non-deterministic
• ANNIC: ANNotations-In-Context– full-featured annotation indexing and retrieval system– Searchable Serial DataStore– Based on Lucene
Processing Resources
• The Annotation Diff Tool– enables two sets of annotations in one or two
documents to be compared– figures are generated for precision, recall, F-
measure• Corpus Benchmark Tool– Apply evaluation across an entire corpus
• Balance Distance Measure (BDM) Ontology Tool
Processing Resources (PlugIns)
• OntoGazetteer• HashGazetteer• Gazetteer List Collector• Large KB Gazetteer• Ontology-Aware JAPE Transducer• Batch Learning PR (LibSVM, PAUM algorithm,
Weka interface)• Machine Learning PR (Maxent, Weka and SVM
Light)
Resources on the Web sitegate.ac.uk
• User Guide• Movie Tutorials• Developer’s Guide/API docs• NLP Application Programmer’s Guide• Research Papers• GATE project descriptions• Demos• Plug-in Info• Commerical/Academic partnerships• Etc…
IDE Demo
What is GATE Embedded?
• Everything in GATE IDE without the GUI• A Java framework for many different types of
NLP solutions• A complex assortment of core functionality
and plug-ins• Extensible and Composable– GATE can be included as a component in other
Java Frameworks and vice-versa
Example Application with a GATE Embedded Component
Running GATE (“Hello World”)import gate.*;import gate.creole.*;
public class Main {
public static void main(String[] args) throws Exception {
Gate.setGateHome(new File(<Path to GATE>));Gate.setPluginsHome(new File(<Path to Plugins>));
Gate.init(); // start GATE}
Registering Directories
Gate.getCreoleRegister().registerDirectories(new File(Gate.getPluginsHome(), "ANNIE").toURL());
Gate.getCreoleRegister().registerDirectories(new File(Gate.getPluginsHome(), "Information_Retrieval").toURL());
Gate.getCreoleRegister().registerDirectories(new File(Gate.getPluginsHome(), "Stemmer_Snowball").toURL());
Creating Processing ResourcesSerialAnalyserController annieController =
(SerialAnalyserController) Factory.createResource("gate.creole.SerialAnalyserController",Factory.newFeatureMap(),Factory.newFeatureMap(), "ANNIE");
FeatureMap params = Factory.newFeatureMap();
annieController.add((ProcessingResource) Factory.createResource("gate.creole.annotdelete.AnnotationDeletePR", params));annieController.add((ProcessingResource) Factory.createResource("gate.creole.tokeniser.DefaultTokeniser", params));annieController.add((ProcessingResource) Factory.createResource("stemmer.SnowballStemmer", params));annieController.add((ProcessingResource) Factory.createResource("gate.creole.gazetteer.DefaultGazetteer", params));annieController.add((ProcessingResource) Factory.createResource("gate.creole.splitter.RegexSentenceSplitter", params));annieController.add((ProcessingResource) Factory.createResource("gate.creole.POSTagger", params));annieController.add((ProcessingResource) Factory.createResource("gate.creole.ANNIETransducer", params));annieController.add((ProcessingResource) Factory.createResource("gate.creole.orthomatcher.OrthoMatcher", params));
FeatureMap coRefParams = Factory.newFeatureMap();coRefParams.put("resolveIt", "true");
annieController.add((ProcessingResource) Factory.createResource("gate.creole.coref.Coreferencer", coRefParams));
Creating Language ResourcesCorpus corpus = Factory.newCorpus("DUC Queries");
@SuppressWarnings("static-access")File topicsFile = new File(ConfigMgr.getTopicFilePath() + "topics.xml");gate.Document topicDoc = Factory.newDocument(topicsFile.toURL());
corpus.add(topicDoc);annieController.setCorpus(corpus);
annieController.execute();
Iteration and CleanupAnnotationSet defaultAnnotations = topicDoc.getAnnotations();AnnotationSet originalMarkup = topicDoc.getAnnotations("Original markups");AnnotationSet topicAnnotationSet = originalMarkup.get("TOPIC");
for (Annotation topicAnnotation : topicAnnotationSet) { ArrayList<Query> topicQueryArrayList = new ArrayList<Query>();
if (ConfigMgr.isQueryBreakdown()) {topicQueryArrayList = Utilities.buildTopicMultiQuery(topicAnnotation,
originalMarkup, defaultAnnotations, config); } else {
topicQueryArrayList = Utilities.buildTopicQuery(topicAnnotation, originalMarkup, defaultAnnotations, config); }
String topicKey = null;
topicKey = topicQueryArrayList.get(0).getDucTopicName(); globalQueryHash.put(topicKey, topicQueryArrayList);}
topicDoc.cleanup();Factory.deleteResource(topicDoc);corpus.cleanup();Factory.deleteResource(corpus);
Iterating through Annotations public static AnnotationSet getChildAnnotationSet(
String childAnnotationSetName, Annotation annotation, AnnotationSet parentAnnotationSet) throws NullPointerException {
AnnotationSet childAnnotationSet = null;
// traverse nested Annotation Set for named annotation using parent offsets to delimit rangetry { childAnnotationSet = parentAnnotationSet.get(childAnnotationSetName,
annotation.getStartNode().getOffset(), annotation.getEndNode().getOffset()); if (childAnnotationSet == null) {
throw new NullPointerException(); }} catch (Exception e) { System.err.println(e.getMessage());}
return childAnnotationSet; }
Example Script for Compiling on Patas#! /bin/bash
javac -classpath .:/NLP_TOOLS/tool_sets/gate/gate-5.1/bin/gate.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/activation.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/ant-contrib-1.0b2.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/ant.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/ant-junit.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/ant-launcher.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/jdom.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/antlr.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/ant-nodeps.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/ant-trax.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/Bib2H‚Ñ¢L.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/commons-discovery-0.2.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/commons-fileupload-1.0.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/commons-lang-2.4.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/commons-logging.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/concurrent.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/gate-asm.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/gate-compiler-jdt.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/gateHmm.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/geronimo-ws-metadata_2.0_spec-1.1.1.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/GnuGetOpt.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/icu4j.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/jakarta-oro-2.0.5.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/javacc.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/jaxb-api-2.0.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/jaxen-1.1.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/jaxws-api-2.0.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/junit.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/jwnl.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/log4j-1.2.14.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/lubm.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/lucene-core-2.2.0.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/mail.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/nekohtml-1.9.8+2039483.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/ontotext.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/orajdbc3.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/PDFBox-0.7.2.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/pg73jdbc3.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/poi-2.5.1-final-20040804.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/spring-beans-2.0.8.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/spring-core-2.0.8.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/stax-api-1.0.1.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/tm-extractors-0.4.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/wstx-lgpl-3.2.3.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/xercesImpl.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/xml-apis.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/xmlunit-1.2.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/xpp3-1.1.3.3_min.jar:/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/xstream-1.2.jar:edu.mit.jwi_2.1.5.jar ling573extractive/*.java
GATE Condor Scriptuniverse = javaexecutable = ling573extractive/Main.classarguments = ling573extractive.Mainoutput = ling573extractive.outputerror = ling573extractive.errorjar_files =
/NLP_TOOLS/tool_sets/gate/gate-5.1/bin/gate.jar,/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/junit.jar,/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/ant-junit.jar,/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/jdom.jar,/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/commons-lang-2.4.jar,/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/gate-asm.jar,/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/gate-compiler-jdt.jar,/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/log4j-1.2.14.jar,/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/lucene-core-2.2.0.jar,/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/nekohtml-1.9.8+2039483.jar,/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/ontotext.jar,/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/PDFBox-0.7.2.jar,/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/orajdbc3.jar,/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/wstx-lgpl-3.2.3.jar,/NLP_TOOLS/tool_sets/gate/gate-5.1/lib/xercesImpl.jar,edu.mit.jwi_2.1.5.jar
java_vm_args = -Xmn100M -Xms500M -Xmx500M+RequiresWholeMachine = TrueRequirements = ( Memory > 0 && TotalMemory >= (7*1024) )queue
Discussion
Recommended