Sparse Coding Trees with Application to Emotion Classification

Kevin H.C. ChenMarcus Z. ComiterH. T. KungBradley McDanel

AMFG 2015

Harvard University

Business Applications

User feedback systems, advertising, security systems

Politics

Automatic derivation of voter preferences, focus group testing

Medicine

Additional metrics for patient care, helping children with autism

Application Motivation

Emotion Classification for IoT and Beyond

Methodology Motivation

Machine Learning and Unsupervised Feature Extraction

Feature 1

Feat

ure

2

- Sparse coding makes data more linearly separable

- Labels are not required

Sparse Coding Pipeline for Classification

Sparse Coding

Classifier (e.g., SVM)

Transform data into feature representation

Prediction with simple classifiers such as SVM

x

z

label

Unsupervised

Supervised

Representation by Sparse Coding

argminz || x - Dz|| 2 + λ|| z|| 02

Express the input signal (x) as the weighted (z) sum of a few features (D)

Note: we can also penalize L1 norm instead of L0 norm

Dictionary Learning

argminD,Z || X - DZ|| 2 + λ|| Z || 0

Sparse Coefficients

- Finds common patterns in training data- Solved by alternating updates of D and Z

2

Our Enhancement to SC

Sparse Coding tree (SC-tree)to learn features with hierarchy

Non-negative constraintsto mitigate over-fitting in SC

Mirroringto increase variation tolerance

Sparse Coding TreeLearning Features for Hard Cases

- Some discriminating features can be subtle- Finding clusters within clusters, similar to how

hierarchical k-means works

Fear can be confused with happiness because they both display teeth

Constructing Sparse Coding Tree

Input

Sparse Coding

Classifier (e.g., SVM)

Group/Label Assignment

label

label label

If certain classes get confused consistently, put them through another layer of feature extraction

anger

contempt

sadness

disgust

fear

happiness

surprise

Branching in Sparse Coding TreeBased on the confusion matrix from the coarse predictor

Features Learned in SC-tree

Features learned in the root node

Features learned in a happy v.s. fear node

happy

Could be happy or fear

fear

Input

label label

label

maxpoolingsplit

Sparse coding(LASSO/OMP)

flip

Sparse coding(LASSO/OMP)

Mirroring for Reflection InvarianceUsing max pooling to capture the horizontal symmetry inherent in emotion classification

A reflected image would get the exact same representation

Improved Robustness with MirroringWith max pooling, we always pick up response from the side of face with stronger features

Nonnegative Sparse Coding

argminz || x - Dz|| 2 + λ|| z|| 02

s.t. D ≥ 0, z ≥ 0D with NN-constraintD without NN-constraint

Tends to learn regional components

Nonnegativity prevents cancelation of components, and therefore mitigates over-fitting

Datasets

Cohn-Kanade Extended Dataset (CK+)Emotions in the Wild Dataset (EitW)GENKI-4K DatasetAM-FED Dataset

CK+

GENKI

AM-FED

Multi-classMulti-class

BinaryBinary

after pre-processingoriginal data

Performance on Emotion Classification

Results reported in average recall

The sparse coding tree improves the performance of our pipeline consistently.

79.9

SC NNSC MNNSC

75.1

70.1

73.6

71.5

76.8

w/SC Tree

w/o SC Tree

CK+ dataset

Performance on Emotion Classification

Results reported in average recall

The sparse coding tree improves the performance of our pipeline consistently.

33.0

SC NNSC MNNSC

29.4

26.5

28.628.1

29.7

w/SC Tree

w/o SC Tree

EitW dataset

MNNSC Performance

Results reported in area under curve

with Mirroring and the non-negativity constraint, even greedy methods like OMP (L0) can be competitive

90.0

L0-min L1-min

best reported96.1

L0-min L1-min

best reported95.1

96.792.396.2

95.7

93.1

91.2

89.7

92.1

88.8

86.0

97.0

sparse coding

Non-negativity

Mirroring

GENKI-4K AM-FED

Applying Sparse Coding Tree to Action Recognition

Tested on KTH dataset

with SC Tree 92.13 %without: 86.57 %

ConclusionSparse coding, as an effective feature extraction method, can be enhanced by these techniques:

Sparse Coding tree (SC-tree)to learn features with hierarchy

Non-negative constraintsto mitigate over-fitting in SC

Mirroringto increase variation tolerance