Wrappers for feature subset selection

Amir Razmjou

Benefits of Feature Subset Selection

• Too many dimensions; Elimination of the curse of dimensionality problem

• Improved model and classifier performance• Simple Models and elimination of over-

fitting• Faster training times.

The Process

Feature Selection Methods

Wrapperuse a search algorithm to search through the

space of possible features and evaluate each subset by running a model on the subset

Risk of over fitting to the modelComputationally expensive

EmbeddedEmbedded in and specific to a model

FilterSimilar to Wrappers in the search approachSimpler filter is evaluated

Hypothetical Concept

Definition 2

Contradiction 2

Definition 3

Contradiction 3 All features are irrelevant

Definition 4

Contradiction 4every feature is relevant

Definition 5 – Strong Relevance

Definition 6 – Weak Relevance

Conclusion 1

• Forward selection methods: these methods start with one or a few features selected according to a method specific selection criteria. More features are iteratively added until a stopping criterion is met.

• Backward elimination methods: methods of this type start with all features and iteratively remove one feature or bunches of features.

Relief

• Evaluates the worth of an attribute by repeatedly sampling an instance and considering the value of the given attribute for the nearest instance of the same and different class. Can operate on both discrete and continuous class data.

Relief

• Relief does not help with redundant features. If most of the given features are relevant to the concept, it would select most of them even though only a fraction are necessary for concept description