Feature Generation and Selection in SRL

Alexandrin Popescul & Lyle H. Ungar

Presented By Stef Schoenmackers

Overview

• Structural Generalized Linear Regression (SGLR) Overview

• Design Motivations

• Experiments

• Conclusions

SGLR Overview

• Adds statistical methods to ILP SQL as the logical language Generalized Linear Regression as statistical method

• Uses clustering to generate new relations

• Builds discriminative models Targeted at large problems where generative models

impossible

• Integrates feature generation and problem modeling

SGLR Loop

SGLR Method

• Clusters data and adds clusters as new relations

• Searches the space of SQL query refinements Features are numerical SQL aggregates Test feature with statistical measure (e.g. AIC, BIC) Add only significantly predictive features Examine each feature only once Use current set of features to guide search

Overview

Structural Generalized Linear Regression (SGLR) Overview

• Design Motivations

• Experiments

• Conclusions

SQL Motivation

• Most of the world’s data is in relational databases Can exploit schema and meta-information

• SQL uses a fairly expressive language Non-recursive first-order logic formulas

• Relational DBs have been studied and optimized for decades, so should be more scalable than other alternatives

Clustering Motivation

• Dimensionality reduction• Clusters are added as relations (new first-

class concepts) Increases expressivity of the language

describing patterns in the data Can lead to a more rapid discovery of

predictive features

• Done as a pre-processing step cost(clustering) << cost(feature search)

Aggregation Motivation

• Summarizes the information in a table into scalar values usable by a statistical model average, max, min, count, average, empty/exists (0/1)

• Exploits database work into making them efficient

• Provides a richer space of features to choose from

Dynamic Feature Generation

• Most features do not provide useful information• In large domains, feature generation is

expensive, and precomputing all possible features is far too time consuming

• Solution: Use a smarter search strategy and dynamically generate features. Let the features already selected influence which features are added

• Focuses only on the promising areas in the search space

Feature Streams

• Put features into different evaluation queues

• Choose next feature from the ‘best’ stream

• If feature in multiple streams, only evaluate once

• Stream design can use prior knowledge/bias

Refinement Graphs (in ILP)

• Start with most general rule, and ‘refines’ it to produce more specific clauses Single variable substitution Add predicate involving 1+ existing variables

• Uses top-down breadth-first search to find the most general rule that covers only positive examples

• Performs poorly in noisy domains

Refinement Graphs (in SGLR)

• Adds one relation to a query and expands it into all possible configurations of equality conditions of new attributes with a new or old attribute Contains at least one equality condition between a

new and old attribute Any attribute can be set to a constant High-level variable typing/classes are enforced

• Not all refinements are most general, but simplifies pruning of equivalent subspaces (accounts only for the type and number of relations joined in a query)

Example Refinement Graph

Query(d)

Cites(d,d1) Author_of(d, a) Word_count(d, w, int)

Author_of(d, a=“Smith”)

Cites(d,d1),Cites(d1,d2)

Cites(d,d1), Author_of(d1, a)

Cites(d,d1), Author_of(d1, a=“Domingos”)

DB Tables

Overview

Structural Generalized Linear Regression (SGLR) Overview

Design Motivations

• Experiments

• Conclusions

Experiments

• Used CiteSeer data Citation(doc1, doc2), Author(doc, person),

PublishedIn(doc, venue), HasWord(doc,word) 60k Docs, 131k Authors, 173k Citations, 6.8M

Words

• Two Tasks Predict the publication venue Predict existence of a citation

Experiments

• Cluster all many-to-many relations K-means Added 6 new relations

• Use logistic regression for prediction

• BFS of search space

• 5k+/5k- examples for venue prediction

• 2.5k+/2.5k- examples for citation prediction

Results

Venue (87.2%) Citation (93.1%)

Dynamic Feature Generation

• Query expressions generated Breadth-First• Baseline puts all queries into one queue• Dynamic strategy enqueues queries into

separate streams Stream 1: exists and count over table Stream 2: other aggregates (counts of unique

elements in individual columns) Chooses next feature from stream where

(featuresAdded+1)/(featuresTried+1) is max Stop when a stream is empty

Results

Venue

Citation

Clusters No Clusters

Time Results

Venue

Citation

Clusters No Clusters

Domain Independent Learning

• Most citation prediction features are research-area generic

• Can we train a model for one area and test on another?

Domain Independent Results

• Used KDD-Cup 2003 data (High Energy Physics papers in arXiv)

Train On Test On Accuracy

CiteSeer arXiv 92.9%

CiteSeer CiteSeer 92.6%

arXiv arXiv 96.0%

Conclusions

• Cluster-based features add expressivity, and apply to any domain or SRL method

• Generating queries dynamically can reduce search time and increase accuracy

Questions?