Tell and Predict: Kernel Classifier Prediction for Unseen Visual Classesfrom Unstructured Text Descriptions

Mohamed Elhoseiny, Ahmed Elgammal, Babak Salehm.elhoseiny@cs.rutgers.edu, elgammal@cs.rutgers.edu, babaks@cs.rutgers.edu

AbstractIn this paper we propose a framework forpredicting kernelized classifiers in the vi-sual domain for categories with no trainingimages where the knowledge comes fromtextual description about these categories.Through our optimization framework, theproposed approach is capable of embed-ding the class-level knowledge from thetext domain as kernel classifiers in the vi-sual domain. We also proposed a distri-butional semantic kernel between text de-scriptions which is shown to be effectivein our setting. The proposed framework isnot restricted to textual descriptions, andcan also be applied to other forms knowl-edge representations. Our approach wasapplied for the challenging task of zero-shot learning of fine-grained categoriesfrom text descriptions of these categories.

1 Introduction

We propose a framework to model kernelized clas-sifier prediction in the visual domain for categorieswith no training images, where the knowledgeabout these categories comes from a secondary do-main. The side information can be in the formof textual, parse trees, grammar, visual representa-tions, concepts in the ontologies, or any form; seeFig 1. Our work focuses on the unstructured textsetting. We denote the side information as “priv-ileged” information, borrowing the notion from(Vapnik and Vashist, 2009).

Our framework is an instance of the conceptof Zero Shot Learning (ZSL)(Larochelle et al.,2008), aiming at transferring knowledge from seenclasses to novel (unseen) classes. Most zero-shotlearning applications in practice use symbolic ornumeric visual attribute vectors (Lampert et al.,2014; Lampert et al., 2009). In contrast, re-cent works investigated other forms of descrip-

Figure 1: Our setting where machine can predictunseen class from pure unstructured text

tions, e.g. user provided feedback (Wah and Be-longie, 2013), textual descriptions (Elhoseiny etal., 2013). It is common in zero-shot learningto introduce an intermediate layer that facilitatesknowledge sharing between seen classes, hencethe transfer of knowledge to unseen classes. Typi-cally, visual attributes are being used for that pur-pose, since they provide a human-understandablerepresentation, which enables specifying new cat-egories (Lampert et al., 2014; Lampert et al.,2009; Farhadi et al., 2009; Palatucci et al., 2009;Akata et al., 2013; Li et al., 2014). A funda-mental question in attribute-based ZSL models ishow to define attributes that are visually discrim-inative and human understandable. Researchershas explored learning attributes from text sources,e.g. (Rohrbach, 2014; Rohrbach et al., 2013;Rohrbach et al., 2010; Berg et al., 2010). Otherworks have explored interactive methodologies tolearning visual attribute that are human under-standable, e.g. (Parikh and Grauman, 2011).

There are several differences between our pro-posed framework and the state-of-the-art zero-shotlearning approaches. We are not restricted to useattributes as the interface to specify new classes.We can use any “privileged” information availablefor each category. In particular in this paper wefocus on the case of textual description of cate-gories as the secondary domain. This differenceis reflected in our zero-shot classification architec-

arX

iv:1

506.

0852

9v1

[cs

.CV

] 2

9 Ju

n 20

15

ture. We learn a domain transfer model betweenthe visual domain and the privileged informationdomain. This facilitates predicting explicit visualclassifiers for novel unseen categories given theirprivileged information. The difference in archi-tecture becomes clear if we consider, for the sakeof argument, attributes as the secondary domain inour framework, although this is not the focus of thepaper. In that case we do not need explicit attributeclassifiers to be learned as an intermediate layer astypically done in attribute-based ZSL e.g. (Lam-pert et al., 2009; Farhadi et al., 2009; Palatucci etal., 2009), instead the visual classifier are directlylearned from the attribute labels. The need to learnan intermediate attribute classifier layer in mostattribute-based zero-shot learning approaches dic-tates using strongly annotated data, where eachimage comes with attribute annotation, e.g. CU-Birds dataset (Welinder et al., 2010). In contrast,we do not need image-annotation pairs, and privi-leged information is only assumed at the categorylevel; hence we denote our approach weakly su-pervised. This also directly facilitates using con-tinuous attributes in our case, and does not assumeindependent between attributes.

Another fundamental difference in our case isthat we predict explicit kernel classifier in the formdefined in the representer theorem (Scholkopf etal., 2001), from privileged information. Explicitclassifier prediction means that the output of ourframework is classifier parameters for any newcategory given text description, which can appliedto any test image to predict its class. Predictingclassifier in kernelized form opens the door for us-ing any kind of side information about classes, aslong as kernels can be defined on them. The im-age features also do not need to be in a vectorizedformat. Kernelized classifiers also facilitates com-bining different types of features through a multi-kernel learning (MKL) paradigm, where the fusionof different features can be effectively achieved.

We can summarize the features of our proposedframework, hence the contribution as follows: 1)Our framework explicitly predicts classifiers; 2)The predicted classifiers are kernelized; 3) Theframework facilitates any type of “side” informa-tion to be used; 4) The approach requires the sideinformation at the class level, not at the imagelevel, hence, it needs only weak annotation. 5)We propose a distributional semantic kernel be-tween text description of visual classes that we

show its value in the experiments. The structureof the paper is as follows. Sec 2 describes the re-lation to existing literature. Sec 3 and 4 explainsthe learning setting and our formulation. Sec 5presents the proposed distributional semantic ker-nel for text descriptions. Sec 6 shows our experi-mental results.

2 Related Work

We already discussed the relation to the zero-shotlearning literature in the Introduction section. Inthis section, we focus on the relations to other vol-umes of literature.

There has been increasing interest recently inthe intersection between Language and ComputerVision. Most of the work on this area is fo-cused on generating textual description from im-ages (Farhadi et al., 2010; Kulkarni et al., 2011;Ordonez et al., 2011; Yang et al., 2011; Mitchellet al., 2012). In contrast, we focus on generatingvisual classifiers from textual description or otherside information at the category level.

There are few recent works that involved unan-notated text to improve visual classification orachieve zero-shot learning. In (Frome et al., 2013;Norouzi et al., 2014) and (Socher et al., 2013),word embedding language models (e.g. (Mikolovet al., 2013b)) was adopted to represent classnames as vectors. Their framework is based onmapping images into the learned language modethen perform classification in that space. In con-trast, our framework maps the text information toa classifier in the visual domain, i.e. the appo-site direction of their approach. There are sev-eral advantages in mapping textual knowledge intothe visual domain. To perform ZSL, approachessuch as (Norouzi et al., 2014; Frome et al., 2013;Socher et al., 2013) only embed new classes bytheir category names. This has clear limitationswhen dealing with fine-grained categories (suchas different bird species). Most of fine-grainedcategory names does not exist in current semanticmodels. Even if they exist, they will end up closeto each other in the learned language models sincethey typically share similar contexts. This limitsthe discriminative power of such language models.In fact our baseline experiment using these modelsperformed as low as random when applied to fine-grained category; described in Sec 6.4. Moreover,our framework directly can use large text descrip-tion of novel categories. In contrast to (Norouzi etal., 2014; Frome et al., 2013; Socher et al., 2013)

which required a vectorized representation of im-ages, our framework facilitates non-linear classifi-cation using kernels.

In (Elhoseiny et al., 2013), an approach wasproposed to predict linear classifiers from textualdescription, based on a domain transfer optimiza-tion method proposed in (Kulis et al., 2011). Al-though both of these works are kernelized, a closelook reveals that kernelization was mainly used toreduce the size of the domain transfer matrix andthe computational cost. The resulting predictedclassifier in (Elhoseiny et al., 2013) is still a lin-ear classifier. In contrast, our proposed formula-tion predicts kernelized visual classifiers directlyfrom the domain transfer optimization, which is amore general case. This directly facilitates usingclassifiers that fused multiple visual cues such asMultiple Kernel Learning (MKL).

3 Problem Definition

We consider a zero-shot multi-class classificationsetting on domain X as follows. At training, be-sides the data points from X and the class labels,each class is associated with privileged informa-tion in a secondary domain E in particular, how-ever not limited to, a textual description. We as-sume that each class yi ∈ Ysc(training/seen la-bels), is associated with privileged informationei ∈ E . While, our formulation allows mul-tiple pieces of privileged information per class(e.g. multiple class-level textual descriptions), wewill use one per class for simplicity. Hence,we denote the training as Dtrain = {Sx ={(xi, yi)}N , Se = {yj , ej}Nsc}, where xi ∈ X ,yi ∈ Ysc, yj ∈ Ysc, and Nsc and N are thenumber of the seen classes and the training ex-amples/images respectively. We assume that eachof the domains is equipped with a kernel func-tion corresponding to a reproducing kernel Hilbertspace (RKHS). Let us denote the kernel for Xby k(·, ·) and the kernel for E by g(·, ·). At thezero-shot time, only the privileged information ez∗is available for each novel unseen class z∗; seeFig 1.

The common approach for multi-class classifi-cation is to learn a classifier for each class againstthe remaining classes (i.e., one-vs-all). Accordingto the generalized representer theorem (Scholkopfet al., 2001), a minimizer of a regularized empir-ical risk function over an RKHS could be repre-sented as a linear combination of kernels, evalu-

ated on the training set. Adopting the representertheorem on classification risk function, we definea kernel-classifier of class y as follows

fy(x∗) =

N∑i=1

βiyk(x

∗, xi) + b = βyTk(x∗), (1)

where x∗ ∈ X is the test point, xi ∈ Sx,k(x∗) = [k(x∗, x1), · · · , k(x∗, xN ), 1]T, βy =

[β1y · · ·βNy , b]T. Having learned fy(x∗) for each

class y (for example using SVM classifier), theclass label of the test point x∗ can be predicted as

y∗ = argmaxy

fy(x∗) (2)

It is clear that fy(x∗) could be learned for allclasses with training data y ∈ Ysc = y1 · · · yNsc

,since there are examples Sx for the seen classes;we denote the kernel-classifier parameters of theseen classes as Bsc = {βy}Nsc , ∀y ∈ Ysc. How-ever, it is not obvious how to predict fz∗(x∗) fora new unseen class z∗ ∈ Yus = z1 · · · zNus

. Ourmain notion is to use the privileged informationez∗ ∈ E , associated with unseen class z∗, and thetraining data Dtrain to directly predict the unseenkernel-classifier parameters. Hence, the classifierof z∗ is a function of ek∗ and Dtrain; i.e.

fz∗(x∗) = β(ez∗ ,Dtrain)

T · k(x∗), (3)

fz∗(x∗) could be used to classify new points that

belong to an unseen class as follows: 1) one-vs-all setting fz∗(x

∗) ≷ 0 ; or 2) in a Multi-classprediction as in Eq 2.

4 Approach

Prediction of β(ez∗ ,Dtrain), which we denote asβ(ez∗) for simplicity, is decomposed into training(domain transfer) and prediction phases.

4.1 Domain TransferDuring training, we firstly learn Bsc as SVM-kernel classifiers based on Sx. Then, we learna domain transfer function to transfer the privi-leged information e ∈ E to kernel-classifier pa-rameters β ∈ RN+1 in X domain. We call thisfunction βDA(e), which has the form of TTg(e),where g(e) = [g(e, e1) · · · g(e, eNsc)]

T; T is anNsc × N + 1 matrix, which transforms e to ker-nel classifier parameters for the class e represents.

We aim to learn T, such that g(e)TTk(x) >l if e and x correspond to the same class,g(e)TTk(x) < u otherwise. Here l controls sim-ilarity lower-bound if e and x correspond to same

class, and u controls similarity upper-bound if eand x belong to different classes. In our setting,the term TTg(ei) should act as a classifier pa-rameter for class i of the training data. There-fore, we introduce penalization constraints to ourminimization function if TTg(ei) is distant fromβi ∈ Bsc, where ei corresponds to the class thatβi classifies. Inspired by domain adaptation op-timization methods (e.g. (Kulis et al., 2011)), wemodel our domain transfer function as follows

T∗ = argminTL(T) = [

1

2r(T) + λ1

∑k

ck(GTK)+

λ2

Nsc∑i=1

‖βi − TTg(ei)‖2(4)

where, G is an Nsc×Nsc symmetric matrix, suchthat both the ith row and the ith column are equalto g(ei), ei ∈ Se; K is an N +1×N matrix, suchthat the ith column is equal to k(xi), xi ∈ Sx.ck’s are loss functions over the constraints definedas ck(GTK)) = (max(0, (l − 1T

iGTK1j)))2for same class pairs of index i and j, or = r ·(max(0, (1T

iGTK1j − u)))2 otherwise, where 1iis an Nsc × 1 vector with all zeros except at indexi, 1j is an N × 1 vector with all zeros except atindex j. This leads to ck(GTK) = max(0, (l −g(ei)TTk(xj)))2 for same class pairs of index iand j, or = r·(max(0, (g(ei)TTk(xj)−u)))2 oth-erwise, where u > l, r = nd

ns such that nd and nsare the number of pairs (i, j) of different classesand similar pairs respectively. Finally, we used aFrobenius norm regularizer for r(T).

The objective function in Eq 4 controls the in-volvement of the constraints ck by the term multi-plied by λ1, which controls its importance; we callit Cl,u(T). While, the trained classifiers penaltyis captured by the term multiplied by λ2; we callit Cβ(T). One important observation on Cβ(T)is that it reaches zero when T = G−1BT, whereB = [β1 · · ·βNsc

], since it could be rewritten asCβ(T) = ‖BT −GT‖2F .

One approach to minimize L(T) is gradient-based optimization using a quasi-Newton opti-mizer. Our gradient derivation of L(T) leads tothe following form

δL(T)δT

= T+λ1 ·∑i,j

g(ei)k(xj)Tvij+r ·λ2 ·(G2T−GB)

(5)

where vij = −2 · max(0, (l − g(ei)TTk(xj)))if i and j correspond to the same class, 2 ·max(0, (g(ei)TTk(xj) − u) otherwise. Another

approach to minimize L(T) is through alternatingprojection using Bregman algorithm (Censor andZenios, 1997), in which T is updated with respectto a single constraint every iteration.

4.2 Classifier PredictionWe propose two ways to predict the kernel-classifier. (1) Domain Transfer (DT) Prediction,(2) One-class-SVM adjusted DT Prediction.Domain Transfer (DT) Prediction: Constructionof an unseen category is directly computed fromour domain transfer model as follows

βDT (ez∗) = T∗Tg(ez∗) (6)

One-class-SVM adjusted DT (SVM-DT) Pre-

diction: In order to increase separability againstseen classes, we adopted the inverse of the idea ofthe one class kernel-svm, whose main idea is tobuild a confidence function that takes only posi-tive examples of the class. Our setting is the op-posite scenario; seen examples are negative exam-ples of the unseen class. In order introduce ourproposed adjustment method, we start by present-ing the one-class SVM objective function. The La-grangian dual of the one-class SVM (Evangelistaet al., 2007) can be written as

β∗+ =argminβ

[βTK

′β − βTa

]s.t. : βT1 = 1, 0 ≤ βi ≤ C; i = 1 · · ·N

(7)

where K′

is an N × N matrix, K′(i, j) =

k(xi, xj), ∀xi, xj ∈ Sx (i.e. in the training data),a is an N × 1 vector, ai = k(xi, xi), C is a hyper-parameter . It is straightforward to see that, if β isaimed to be a negative decision function instead,the objective function becomes in the form

β∗− =argminβ

[βTK

′β + βTa

]s.t. : βT1 = −1,−C ≤ βi ≤ 0; i = 1 · · ·N

(8)

While β∗− = −β∗

+, the objective function inEq 8 of the one-negative class SVM inspires uswith the idea to adjust the kernel-classifier param-eters to increase separability of the unseen kernel-classifier against the points of the seen classes,which leads to the following objective function

β(ez∗) =argminβ

[βTK

′β − ζβDT (ez∗)

TK′β + βTa

]s.t. : βT1 = −1, β

T

DTK′β > l,−C ≤ βi ≤ 0;∀i

C, ζ, l: hyper-parameters,(9)

where βDT is the first N elements in βDT ∈RN+1, 1 is an N × 1 vector of ones. The ob-jective function, in Eq 9, pushes the classifier ofthe unseen class to be highly correlated with thedomain transfer prediction of the kernel classi-fier, while putting the points of the seen classesas negative examples. It is not hard to see thatEq 9 is a quadratic program in β, which could besolved using any quadratic solver; we used IBMCPLEX. It is worth to mention that, the approachin (Elhoseiny et al., 2013) predicts linear classi-fiers by solving an optimization problem of sizeN + dX +1 variables (dX +1 linear-classifier pa-rameters and N slack variables); a similar limita-tion can be found in (Frome et al., 2013; Socheret al., 2013). In contrast, our objective func-tion in Eq 9 solves a quadratic program of onlyN variables, and predicts a kernel-classifier in-stead, with fewer parameters. Hence, if very high-dimensional features are used, they will not affectour optimization complexity.

5 Distributional Semantic (DS) Kernelfor text descriptions

When E domain is the space of text descrip-tions, we propose a distributional semantic ker-nel g(·, ·) = gDS(·, ·) to define the similarity be-tween two text descriptions . We start by dis-tributional semantic models by (Mikolov et al.,2013c; Mikolov et al., 2013a) to represent the se-mantic manifold Ms, and a function vec(·) thatmaps a word to a K × 1 vector in Ms. Themain assumption behind this class of distribu-tional semantic model is that similar words sharesimilar context. Mathematically speaking, thesemodels learn a vector for each word wn, suchthat p(wn|(wn−L, wn−L+1, · · · , wn+L−1, wn+L)is maximized over the training corpus, where 2×Lis the context window size. Hence similarity be-tween vec(wi) and vec(wj) is high if they co-occurred a lot in context of size 2×L in the train-ing text-corpus. We normalize all the word vectorsto length 1 under L2 norm, i.e., ‖vec(·)‖2 = 1.

Let us assume a text description D thatwe represent by a set of triplets D ={(wl, fl, vec(wl)), l = 1 · · ·M}, where wl isa word that occurs in D with frequency fland its corresponding word vector is vec(wl)in Ms. We drop the stop words from D.We define F = [f1, · · · , fM ]T and V =[vec(w1), · · · , vec(wM )]T, where F is an M × 1

vector of term frequencies and V is an M × Kmatrix of the corresponding term vectors.

Given two text descriptions Di and Dj whichcontains M1 and M2 terms respectively. We com-pute Fi (Mi × 1) and Vi (Mi × K) for Di andFj (Mj × 1) and Vj (Mj × K) for Dj . FinallygDS(Di, Dj) is defined as

gDS(Di, Dj) = FTi ViVT

jFj (10)

One advantage of this similarity measure is that itcaptures semantically related terms. It is not hardto see that the standard Term Frequency (TF) sim-ilarity could be thought as a special case of thiskernel where vec(wl)Tvec(wm) = 1 if wl = wm,0 otherwise, i.e., different terms are orthogonal.However, in our case the word vectors are learntthrough a distributional semantic model whichmakes semantically related terms have higher dotproduct (vec(wl)Tvec(wm)).

6 Experiments

6.1 Datasets and Evaluation MethodologyWe validated our approach in a fine-grained settingusing two datasets: 1) The UCSD-Birds dataset(Welinder et al., 2010), which consists of 6033 im-ages of 200 classes. 2) The Oxford-Flower dataset(Nilsback and Zisserman, 2008), which consistsof 8189 images of 102 flower categories. Bothdatasets were amended with class-level text de-scriptions extracted from different encyclopediaswhich is the same descriptions used in (Elhoseinyet al., 2013); see samples in the supplementarymaterials. We split the datasets to 80% of theclasses for training and 20% of the classes for test-ing, with cross validations. We report multiplemetrics while evaluating and comparing our ap-proach to the baselines, detailed as follows

Multiclass Accuracy of Unseen classes (MAU):Under this metric, we aim to evaluate the perfor-mance of the unseen classifiers against each oth-ers. Firstly, the classifiers of all unseen categoriesare predicted. Then, an instance x∗ is classified tothe class z∗ ∈ Yus of maximum confidence for x∗

of the predicted classifiers; see Eq 2.AUC: In order to measure the discriminative

ability of our predicted one-vs-all classifier foreach unseen class, against the seen classes, we re-port the area under the ROC curve. Since unseenclass positive examples are few compared to nega-tive examples, a large accuracy could be achievedeven if all unseen points are incorrectly classified.

Hence, AUC is a more consistent measure. In thismetric, we use the predicted classifier of an un-seen class as a binary separator against the seenclasses. This measure is computed for each pre-dicted unseen classifier and the average AUC is re-ported. This is the only measure addressed in (El-hoseiny et al., 2013) to evaluate the unseen classi-fiers, which is limiting in our opinion.|Nsc| to |Nsc + 1|Recall: Under this metric, we

aim to check how the learned classifiers of the seenclasses confuse the predicted classifiers, whenthey are involved in a multi-class classificationproblem of Nsc + 1 classes. We use Eq 2 to pre-dict label of an instance x∗, such that the unknownlabel y∗ ∈ Ysc ∪ lus, such that lus is the label of theunseen class. We compute the recall under this set-ting. This metric is computed for each predictedunseen classifier and the average is reported.

6.2 Comparisons to Linear ClassifierPrediction

We compared our proposed approach to (Elho-seiny et al., 2013), which predicts a linear clas-sifier for zero-shot learning from textual descrip-tions ( E space in our framework). The aspectsof the comparison includes 1) whether the pre-dicted kernelized classifier outperforms the pre-dicted linear classifier 2) whether this behavioris consistent on multiple datasets. We performedthe comparison on both Birds and Flower dataset.For these experiments, in our setting, domain X isthe visual domain and domain E is the textual do-main, i.e. , the goal is to predict classifiers frompure textual description. We used the same fea-tures on the visual domain and the textual domainsas (Elhoseiny et al., 2013). That is, for the vi-sual domain, we used classeme features (Torre-sani et al., 2010), extracted from images of theBird and the Flower datasets. Classeme is a 2569-dimensional features, which correspond to confi-dences of a set of one-vs-all classifiers, pre-trainedon images from the web, as explained in (Torre-sani et al., 2010), not related to either the Bird northe Flower datasets. The rationale behind usingthese features in (Elhoseiny et al., 2013) was thatthey offer a semantic representation. For the tex-tual domain, we used the same textual feature ex-tracted by (Elhoseiny et al., 2013). In that work,tf-idf (Term-Frequency Inverted Document Fre-quency)(Salton and Buckley, 1988) features wereextracted from the textual articles were used, fol-

Table 1: Recall, MAU, and average AUC onthree seen/unseen splits on Flower Dataset and aseen/unseen split on Birds dataset

Recall-Flower improvement Recall-Birds improvementSVM-DT kernel-rbf 40.34% (+/- 1.2) % 44.05 %

Linear Classifier 31.33 (+/- 2.22)% 27.8 % 36.56 % 20.4 %

MAU-Flower improvement MAU-Birds improvementSVM-DT kernel-rbf 9.1 (+/- 2.77) % 3.4 %

DT kernel-rbf 6.64 (+/- 4.1) % 37.93 % 2.95 % 15.25 %Linear Classifier 5.93 (+/- 1.48)% 54.36 % 2.62 % 29.77 %Domain Transfer 5.79 (+/- 2.59)% 58.46 % 2.47 % 37.65 %

AUC-Flower improvement AUC-Birds improvementSVM-DT kernel-rbf 0.653 (+/- 0.009) 0.61

DT kernel-rbf 0.623 (+/- 0.01) % 4.7 % 0.57 7.02 %Linear Classifier 0.658 (+/- 0.034) - 0.7 % 0.62 -1.61%Domain Transfer 0.644 (+/- 0.008) 1.28 % 0.56 8.93%

lowed by a CLSI (Zeimpekis and Gallopoulos,2005) dimensionality reduction phase.

We denote our DT prediction and one classSVM adjust DT prediction approaches as DT-kernel and SVM-DT-kernel respectively. Wecompared against the linear classifier predictionby (Elhoseiny et al., 2013). We also comparedagainst the direct domain transfer (Kulis et al.,2011), which was applied as a baseline in (El-hoseiny et al., 2013) to predict linear classifiers.In our kernel approaches, we used Gaussian rbf-kernel as a similarity measure in E and X spaces(i.e. k(d, d′) = exp(−λ||d− d′||)).

Recall metric : The recall of our approach is44.05% for Birds and 40.34% for Flower, whileit is 36.56% for Birds and 31.33% for Flower us-ing (Elhoseiny et al., 2013). This indicates that thepredicted classifier is less confused by the classi-fiers of the seen compared with (Elhoseiny et al.,2013); see table 1 (top part)

MAU metric: It is worth to mention that themulticlass accuracies for the trained seen classi-fiers are 51.3% and 15.4% using the classeme fea-tures on Flower dataset and Birds dataset1, re-spectively. Table 1 (middle part) shows the av-erage MAU metric over three seen/unseen splitsfor Flower dataset and one split on Birds dataset,respectively. Furthermore, the relative improve-ments of our SVM-DT-kernel approach is reportedagainst the baselines. On Flower dataset, it is in-teresting to see that our approach achieved 9.1%MAU, 182% improvement over the random guessperformance, by predicting the unseen classifiersusing just textual features as privileged informa-tion (i.e. E domain). We also achieved also 13.4%,268% the random guess performance, in one of thesplits (the 9.1% is the average over 3 seen/unseen

1Birds dataset is known to be a challenging dataset forfine-grained, even when applied in a regular multiclass settingas it is clear from the 15.4% performance on seen classes

10 20 30 40 50 60

0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

AUC for different class in Flower dataset

AU

C

Object Class Index

Figure 2: AUC of the 62 unseen classifiers theflower data-sets over three different splits (bottompart) and their Top 10 ROC-curves (top part)

splits). Similarity on Birds dataset, we achieved3.4% MAU from text features, 132% the randomguess performance (further improved to 224% innext experiments).

AUC metric: Fig 2 (top part) shows the ROCcurves for our approach on the best predicted un-seen classes from the Flower dataset. Fig 2 (bot-tom part) shows the AUC for all the classes onFlower dataset (over three different splits). Moreresults and figures are attached in the supplemen-tary materials. Table 1 (bottom part) shows theaverage AUC on the two datasets, compared to thebaselines.

Looking at table 1, we can notice that the pro-posed approach performs marginally similar to thebaselines from AUC perspective. However, thereis a clear improvement in MAU and Recall met-rics. These results show the advantage of pre-dicting classifiers in kernel space. Furthermore,the table shows that our SVM-DT-kernel approachoutperforms our DT-kernel model. This indicatesthe advantage of the class separation, which is ad-justed by the SVM-DT-kernel model. More de-tails on the hyper-parameter selection are attachedin the supplementary materials.

6.3 Multiple Kernel Learning (MKL)Experiment

This experiment shows the added value of propos-ing a kernelized zero-shot learning approach. Weconducted an experiment where the final kernel onthe visual domain is produced by Multiple KernelLearning (Gonen and Alpaydin, 2011). For the vi-sual domain, we extracted kernel descriptors for

Table 2: MAU on a seen-unseen split-BirdsDataset (MKL)

MAU improvementSVM-DT kernel-rbf (text) 4.10 %

Linear Classifier 2.74 % 49.6 %

Birds dataset. Kernel descriptors provide a prin-cipled way to turn any pixel attribute to patch-level features, and are able to generate rich fea-tures from various recognition cues. We specifi-cally used four types of kernels introduced by (Boet al., 2010) as follows: Gradient Match Kernelsthat captures image variation based on predefinedkernels on image gradients. Color Match Kernelthat describes patch appearance using two kernelson top of RGB and normalized RGB for regu-lar images and intensity for grey images. Thesekernels capture image variation and visual apper-ances. For modeling the local shape, Local BinaryPattern kernels have been applied.

We computed these kernel descriptors on lo-cal image patches with fixed size 16 x 16 sam-pled densely over a grid with step size 8 in a spa-tial pyramid setting with four layers. The densefeatures are vectorized using codebooks of size1000. This process ended up with a 120,000 di-mensional feature for each image (30,000 for eachtype). Having extracted the four types of de-scriptors, we compute an rbf kernel matrix foreach type separately. We learn the bandwidth pa-rameters for each rbf kernel by cross validationon the seen classes. Then, we generate a newkernel kmkl(d, d

′) =∑4

i=1 wiki(d, d′), such that wi

is a weight assigned to each kernel. We learnthese weights by applying Bucak’s Multiple Ker-nel Learning algorithm (Bucak et al., 2010). Then,we applied our approach where the MKL-kernel isused in the visual domain and rbf kernel on the textTFIDF features.

To compare our approach to (Elhoseiny et al.,2013) under this setting, we concatenated all ker-nel descriptors to end up with 120,000 dimen-sional feature vector in the visual domain. Ashighlighted in the approach Sec 4, the approachin (Elhoseiny et al., 2013) solves a quadratic pro-gram of N +dX +1variables for each unseen class.Due to the large dimensionality of data (dX =

120, 000), this is not tractable. To make this set-ting applicable, we reduced the dimensionality ofthe feature vector into 4000 using PCA. This high-lights the benefit of our approach since it does notdepend on the dimensionality of the data. Table 2shows MAU for our approach under this setting

Table 3: MAU on a seen-unseen split-BirdsDataset (CNN features, text description)

MAU improvementSVM-DT kernel (X -rbf, E-DS kernel) 5.35 %

SVM-DT kernel (X -rbf, E-rbf on TFIDF) 4.20 % 27.3%Linear Classifier (TFIDF text) 2.65 % 102.0%

(Norouzi et al., 2014) 2.3% 132.6%

against (Elhoseiny et al., 2013). The results showthe benefits of having a kernel approach for zeroshot learning where kernel methods are applied toimprove the performance.

6.4 Multiple Representation Experiment andDistributional Semantic(DS) Kernel

The aim of this experiment is to show thatour approach also work on different representa-tions of text and visual domain. In this exper-iment, we extracted Convolutional Neureal Net-work(CNN) image features for the Visual domain.We used caffe (Jia et al., 2014) implementationof (Krizhevsky et al., 2012). Then, we extractedthe sixth activation feature of the CNN since wefound it works the best on the standard classifica-tion setting. We found this consistent with the re-sults of (Donahue et al., 2014) over different CNNlayers. While using TFIDF feature of text descrip-tion and CNN features for images, we achieved2.65% for the linear version and 4.2% for the rbfkernel on both text and images. We further im-proved the performance to 5.35% by using ourproposed Distributional Semantic (DS) kernel inthe text domain and rbf kernel for images. In thisDS experiment, we used the distributional seman-tic model by (Mikolov et al., 2013c) trained onGoogleNews corpus (100 billion words) resultingin a vocabulary of size 3 million words, and wordvectors of K = 300 dimensions. This experimentshows both the value of having a kernel versionand also the value of the proposed kernel in oursetting. We also applied the zero shot learning ap-proach in (Norouzi et al., 2014) which performsworse in our settings; see Table 3.

6.5 Attributes Experiment

We emphasis that our main goal is not attributeprediction. However, it was interesting for us tosee the behavior of our method where side infor-mation comes from attributes instead of text. Incontrast to attribute-based models, which fully uti-lize attribute information to build attribute classi-fiers, our approach do not learn attribute classi-fiers. In this experiment, our method uses onlythe first moment of information of the attributes

Table 4: MAU on a seen-unseen split-BirdsDataset (Attributes)

MAU improvementSVM-DT kernel-rbf 5.6 %

DT kernel-rbf 4.03 % 32.7 %Lampert DAP 4.8 % 16.6 %

(i.e. the average attribute vector). We decided tocompare to an attribute-based approach from thisperspective. In particular, we applied the (DAP)attribute-based model (Lampert et al., 2014; Lam-pert et al., 2009), widely adopted in many appli-cations (e.g., (Liu et al., 2013; Rohrbach et al.,2011)), to the Birds dataset. Details weak attributerepresentation in E space are attached in the sup-plementary materials due to space. For visual do-main X , we used classeme features in this experi-ment (like table 1 experiment)

An interesting result is that our approachachieved 5.6% MAU (224% the random guess per-formance); see Table 4. In contrast, we get 4.8%multiclass accuracy using DAP approach (Lam-pert et al., 2014). In this setting, we also measuredthe Nsc to Nsc + 1 average recall. We found therecall measure is 76.7% for our SVM-DT-kernel,while it is 68.1% on the DAP approach, whichreflects better true positive rate (positive class isthe unseen one). We find these results interesting,since we achieved it without learning any attributeclassifiers, as in (Lampert et al., 2014). Whencomparing the results of our approach using at-tributes (Table 4) vs. textual description (Table 1)2

as the privileged information used for prediction,it is clear that the attribute features gives betterprediction. This support our hypothesis that themore meaningful the E domain, the better the per-formance on X domain.

7 Conclusion

We proposed an approach to predict kernel-classifiers of unseen categories textual descrip-tion of them. We formulated the problem as do-main transfer function from the privilege spaceE to the visual classification space X , while sup-porting kernels in both domains. We proposed aone-class SVM adjustment to our domain transferfunction to improve the prediction. We validatedthe performance of our model by several exper-iments. We applied our approach using differentprivilege spaces (i.e. E lives in a textual space or anattribute space). We showed the value of propos-

2We are refering to the experiment that uses classeme asvisual features to have a consistent comparison to here

ing a kernelized version by applying kernels gen-erated by Multiple Kernel Learning (MKL) andachieved better results. We also compared our ap-proach with state-of-the-art approaches and inter-esting findings have been reported.

References[Akata et al.2013] Zeynep Akata, Florent Perronnin,

Zaid Harchaoui, and Cordelia Schmid. 2013.Label-embedding for attribute-based classification.In CVPR.

[Berg et al.2010] Tamara L Berg, Alexander C Berg,and Jonathan Shih. 2010. Automatic attribute dis-covery and characterization from noisy web data. InECCV.

[Bo et al.2010] L. Bo, X. Ren, and D. Fox. 2010. Ker-nel descriptors for visual recognition. In NIPS.

[Bucak et al.2010] Serhat Selcuk Bucak, Rong Jin, andAnil K. Jain. 2010. Multi-label multiple kernellearning by stochastic approximation: Applicationto visual object recognition. In NIPS.

[Censor and Zenios1997] Y. Censor and S.A. Zenios.1997. Parallel Optimization: Theory, Algorithms,and Applications. Oxford University Press, USA.

[Donahue et al.2014] Jeff Donahue, Yangqing Jia, OriolVinyals, Judy Hoffman, Ning Zhang, Eric Tzeng,and Trevor Darrell. 2014. Decaf: A deep convo-lutional activation feature for generic visual recog-nition. In ICML.

[Elhoseiny et al.2013] Mohammad Elhoseiny, BabakSaleh, and Ahmed Elgammal. 2013. Write a clas-sifier: Zero shot learning using purely text descrip-tions. In ICCV.

[Evangelista et al.2007] Paul F. Evangelista, Mark J.Embrechts, and Boleslaw K. Szymanski. 2007.Some properties of the gaussian kernel for one classlearning. In ICANN.

[Farhadi et al.2009] Ali Farhadi, Ian Endres, DerekHoiem, and David A. Forsyth. 2009. Describingobjects by their attributes. In CVPR.

[Farhadi et al.2010] Ali Farhadi, Mohsen Hejrati, Mo-hammad Amin Sadeghi, Peter Young, CyrusRashtchian, Julia Hockenmaier, and David Forsyth.2010. Every picture tells a story: Generating sen-tences from images. In ECCV.

[Frome et al.2013] Andrea Frome, Gregory S. Cor-rado, Jonathon Shlens, Samy Bengio, Jeffrey Dean,Marc’Aurelio Ranzato, and Tomas Mikolov. 2013.Devise: A deep visual-semantic embedding model.In NIPS.

[Gonen and Alpaydin2011] Mehmet Gonen and EthemAlpaydin. 2011. Multiple kernel learning algo-rithms. JMLR.

[Jia et al.2014] Yangqing Jia, Evan Shelhamer, JeffDonahue, Sergey Karayev, Jonathan Long, RossGirshick, Sergio Guadarrama, and Trevor Darrell.2014. Caffe: Convolutional architecture for fast fea-ture embedding. In ACM Multimedia.

[Krizhevsky et al.2012] Alex Krizhevsky, IlyaSutskever, and Geoffrey E Hinton. 2012. Im-agenet classification with deep convolutional neuralnetworks. In Advances in neural informationprocessing systems (NIPS).

[Kulis et al.2011] B. Kulis, K. Saenko, and T. Darrell.2011. What you saw is not what you get: Domainadaptation using asymmetric kernel transforms. InCVPR.

[Kulkarni et al.2011] Girish Kulkarni, Visruth Premraj,Sagnik Dhar, Siming Li, Yejin Choi, Alexander CBerg, and Tamara L Berg. 2011. Baby talk: Under-standing and generating simple image descriptions.In CVPR.

[Lampert et al.2009] Christoph H. Lampert, HannesNickisch, and Stefan Harmeling. 2009. Learningto detect unseen object classes by betweenclass at-tribute transfer. In In CVPR.

[Lampert et al.2014] C.H. Lampert, H. Nickisch, andS. Harmeling. 2014. Attribute-based classificationfor zero-shot visual object categorization. TPAMI,36(3):453–465, March.

[Larochelle et al.2008] Hugo Larochelle, Dumitru Er-han, and Yoshua Bengio. 2008. Zero-data learningof new tasks. In AAAI.

[Li et al.2014] Zhenyang Li, Efstratios Gavves,Thomas Mensink, and Cees GM Snoek. 2014.Attributes make sense on segmented objects. InECCV.

[Liu et al.2013] Jingen Liu, Qian Yu, Omar Javed, SaadAli, Amir Tamrakar, Ajay Divakaran, Hui Cheng,and Harpreet Sawhney. 2013. Video event recogni-tion using concept attributes. In WACV.

[Mikolov et al.2013a] Tomas Mikolov, Kai Chen, GregCorrado, and Jeffrey Dean. 2013a. Efficient estima-tion of word representations in vector space. ICLR.

[Mikolov et al.2013b] Tomas Mikolov, Ilya Sutskever,Kai Chen, Greg S Corrado, and Jeff Dean. 2013b.Distributed representations of words and phrasesand their compositionality. In Advances in NeuralInformation Processing Systems, pages 3111–3119.

[Mikolov et al.2013c] Tomas Mikolov, Ilya Sutskever,Kai Chen, Greg S Corrado, and Jeff Dean. 2013c.Distributed representations of words and phrasesand their compositionality. In Advances in NeuralInformation Processing Systems, pages 3111–3119.

[Mitchell et al.2012] Margaret Mitchell, Jesse Dodge,Amit Goyal, Kota Yamaguchi, Karl Stratos, XufengHan, Alyssa Mensch, Alexander C. Berg, Tamara L.

Berg, and Hal Daume III. 2012. Midge: Generatingimage descriptions from computer vision detections.In EACL.

[Nilsback and Zisserman2008] M-E. Nilsback andA. Zisserman. 2008. Automated flower classifica-tion over large number of classes. In ICVGIP.

[Norouzi et al.2014] Mohammad Norouzi, TomasMikolov, Samy Bengio, Yoram Singer, JonathonShlens, Andrea Frome, Greg S Corrado, and Jef-frey Dean. 2014. Zero-shot learning by convexcombination of semantic embeddings. In ICLR.

[Ordonez et al.2011] Vicente Ordonez, Girish Kulka-rni, and Tamara L Berg. 2011. Im2text: Describingimages using 1 million captioned photographs. InNIPS.

[Palatucci et al.2009] Mark Palatucci, Dean Pomerleau,Geoffrey E. Hinton, and Tom M. Mitchell. 2009.Zero-shot learning with semantic output codes. InNIPS.

[Parikh and Grauman2011] Devi Parikh and KristenGrauman. 2011. Interactively building a discrimi-native vocabulary of nameable attributes. In CVPR.

[Rohrbach et al.2010] Marcus Rohrbach, MichaelStark, Gyorgy Szarvas, and Bernt Schiele. 2010.Combining language sources and robust semanticrelatedness for attribute-based knowledge transfer.In Parts and Attributes Workshop at ECCV 2010.

[Rohrbach et al.2011] Marcus Rohrbach, MichaelStark, and Bernt Schiele. 2011. Evaluating knowl-edge transfer and zero-shot learning in a large-scalesetting. In CVPR.

[Rohrbach et al.2013] Marcus Rohrbach, Sandra Ebert,and Bernt Schiele. 2013. Transfer learning in atransductive setting. In NIPS.

[Rohrbach2014] Marcus Rohrbach. 2014. Combin-ing visual recognition and computational linguistics:linguistic knowledge for visual recognition and nat-ural language descriptions of visual content.

[Salton and Buckley1988] Gerard Salton and Christo-pher Buckley. 1988. Term-weighting approachesin automatic text retrieval. IPM.

[Scholkopf et al.2001] Bernhard Scholkopf, Ralf Her-brich, and Alex J. Smola. 2001. A generalized rep-resenter theorem. In COLT.

[Socher et al.2013] Richard Socher, Milind Ganjoo,Hamsa Sridhar, Osbert Bastani, Christopher D.Manning, and Andrew Y. Ng. 2013. Zero shotlearning through cross-modal transfer. In NIPS.

[Torresani et al.2010] Lorenzo Torresani, Martin Szum-mer, and Andrew Fitzgibbon. 2010. Efficient objectcategory recognition using classemes. In ECCV.

[Vapnik and Vashist2009] Vladimir Vapnik and AkshayVashist. 2009. A new learning paradigm: Learningusing privileged information. Neural Networks.

[Wah and Belongie2013] Catherine Wah and Serge Be-longie. 2013. Attribute-based detection of unfamil-iar classes with humans in the loop. In CVPR.

[Welinder et al.2010] P. Welinder, S. Branson, T. Mita,C. Wah, F. Schroff, S. Belongie, and P. Perona.2010. Caltech-UCSD Birds 200. Technical report,California Institute of Technology.

[Yang et al.2011] Yezhou Yang, Ching Lik Teo, HalDaume III, and Yiannis Aloimonos. 2011. Corpus-guided sentence generation of natural images. InProceedings of the Conference on Empirical Meth-ods in Natural Language Processing, pages 444–454. Association for Computational Linguistics.

[Zeimpekis and Gallopoulos2005] Dimitrios Zeim-pekis and Efstratios Gallopoulos. 2005. Clsi:A flexible approximation scheme from clusteredterm-document matrices. In SDM.