Robust Student Network Learning - PKUci.idm.pku.edu.cn/TNNLS19.pdfconvolution and depth-wise separable convolution. Deep-fried convnets [31] introduced a novel Adaptive Fastfood transform

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 1

Robust Student Network LearningTianyu Guo, Chang Xu , Shiyi He, Boxin Shi, Member, IEEE, Chao Xu, and Dacheng Tao , Fellow, IEEE

Abstract— Deep neural networks bring in impressive accuracyin various applications, but the success often relies on heavynetwork architectures. Taking well-trained heavy networks asteachers, classical teacher–student learning paradigm aims tolearn a student network that is lightweight yet accurate. In thisway, a portable student network with significantly fewer parame-ters can achieve considerable accuracy, which is comparable tothat of a teacher network. However, beyond accuracy, the robust-ness of the learned student network against perturbation is alsoessential for practical uses. Existing teacher-student learningframeworks mainly focus on accuracy and compression ratios,but ignore the robustness. In this paper, we make the studentnetwork produce more confident predictions with the help of theteacher network, and analyze the lower bound of the perturbationthat will destroy the confidence of the student network. Twoimportant objectives regarding prediction scores and gradients ofexamples are developed to maximize this lower bound, to enhancethe robustness of the student network without sacrificing theperformance. Experiments on benchmark data sets demonstratethe efficiency of the proposed approach to learning robust studentnetworks that have satisfying accuracy and compact sizes.

Index Terms— Deep learning, knowledge distillation (KD),teacher–student learning.

I. INTRODUCTION

RECENT years have witnessed the marked progress ofdeep learning. Since the breakthrough in 2012 ImageNet

competition [1] achieved by AlexNet [2] using five convo-lutional layers and three fully connected layers, a series ofmore advanced deep neural networks have been developed tokeep rewriting the record, e.g., VGGNet [3], GoogLeNet [4],and ResNet [5]. However, their excellent performances requirethe support of a huge amount of computation. For instance,AlexNet [3] contains about 232 million parameters and needs

Manuscript received September 28, 2018; revised March 24, 2019 andJune 22, 2019; accepted July 5, 2019. This work was supported in part bythe National Natural Science Foundation of China under Grant 61876007and Grant 61872012, and in part by ARC under Grant FL-170100117,Grant DP-180103424, Grant IH-180100002, and Grant DE-180101438.(Corresponding author: Chang Xu.)

T. Guo, S. He, and C. Xu are with the Key Laboratory of MachinePerception, Ministry of Education, Peking University, Beijing, 100871, China,and also with the Coopertative Medianet Innovation Center, Department ofMachine Intelligence, Peking University, Beijing, 100871, China (e-mail:[email protected]; [email protected]; [email protected]).

C. Xu and D. Tao are with the UBTECH Sydney Artificial IntelligenceCenter, Faculty of Engineering, University of Sydney, Darlington, NSW 2008,Australia, and also with the School of Computer Science, Faculty of Engi-neering, University of Sydney, Darlington, NSW 2008, Australia (e-mail:[email protected]; [email protected]).

B. Shi is with the National Engineering Laboratory for Video Tech-nology, Department of Computer Science and Technology, Peking Univer-sity, Beijing 100871, China, and also with the Peng Cheng Laboratory,Shenzhen 518040, China (e-mail: [email protected]).

Color versions of one or more of the figures in this article are availableonline at http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/TNNLS.2019.2929114

7.24×108 multiplications to process an image with resolutionof 227 × 227. Hence, the potential power of deep neuralnetworks can only be fully unlocked on high-performanceGPU servers or clusters. In contrast, the majority of the mobiledevices used in our daily life usually have rigorous constraintson the storage and computational resource, which preventsthem from fully taking advantages of the deep neural network.As a result, networks with smaller hardware demanding whilestill maintaining similar accuracies are of great interests tomachine learning and computer vision community.

Compressing convolutional neural networks can be achievedby vector quantization [6], decomposing weight matrices [7],and encoding with hashing tricks [8]. Unimportant weightscan be pruned to achieve the same goal by removing thesubtle weights [9], [10], reducing the redundancy betweenweights in the frequency domain [11], and using the binarynetworks [12], [13]. Another straightforward approach is todesign a compact network directly, e.g., ResNeXt [14], Xcep-tion network [15], and MobileNets [16]. These networks areoften deep and thin with fewer parameters in each layer, andthe nonlinearity of these networks are strengthened by increas-ing the number of layers, which guarantees the performanceof the network.

Teacher–student learning framework, introduced in knowl-edge distillation (KD) [17], is one of the most popularapproaches to realize model compression and accelera-tion [11], [12]. Taking a heavy neural network, such asGoogleNet [4] or ResNet [5], that has already been welltrained with massive data and computing resources asthe teacher network, a student network of light architec-ture can be better learned under the teacher’s guidance.To inherit the advantages of the teacher network, differ-ent methods [18]–[21] have been proposed to encourage theconsistency between teacher and student networks.

These aforementioned algorithms have achieved impressiveexperimental results. However, they were mainly developedin ideal scenarios, where all data are implicitly assumedto be clean. In practice, given examples with perturbation,the training process of the network can be seriously influenced,and the resulting network would not be confident as beforeto make predictions of examples. The teacher network mightmake some mistakes since it is difficult for the teacher networkto be familiar with all examples fed into the student network.This is consistent with student–teacher learning in the realworld. An excellent student is expected to solve practicalproblems in changeable circumstances, where there might bequestions even not known by teachers.

To solve this problem, in this paper, we introduce a robustteacher–student learning algorithm. The framework of theproposed method is illustrated in Fig. 1. We enable the student

2162-237X © 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

https://orcid.org/0000-0002-4756-0609

https://orcid.org/0000-0001-7225-5449


2 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS

Fig. 1. Framework of the proposed algorithm. Constraint LS(NS) imposed on the output part ensures the student network to have higher confidence in theprediction than that of the teacher network. Constraint LG(NS) imposed on the gradients encourages the student network to preserve its confidence in theprediction if there is perturbation on the data. f (x) represents the network’s prediction for the ground-truth label y of input x.

network to be more confident in its prediction with the help ofa teacher network. Perturbations on examples might seriouslyinfluence the learning of the student network. We derive thelower bound of the perturbations that can make the studentmore vulnerable than a teacher through rigorous theoreticalanalysis. New objectives in terms of prediction scores andgradients of examples are further developed to maximize thelower bound of the required perturbation. Hence, the overallrobustness of the student network to resist perturbations onexamples can be improved. Experimental results on benchmarkdata sets demonstrate the superiority of the proposed methodfor learning compact and robust deep neural networks.

We organized the rest of the paper as follows. In Section II,we summarize related works on learning convolutionalneural networks with fewer parameters by different meth-ods. Section III introduces the previous work we based on.In Section IV, we formally introduce our robust studentnetwork learning method in detail, including mathematicalproof to the proposed theorem, the calculation method of theloss function, and the training strategy. Section V providesthe results of our algorithm obtained on various benchmarkdata sets to prove the effectiveness of the proposed method.Section VI concludes this paper.

II. RELATED WORKS

In this section, we briefly introduce related works onlearning an efficient convolutional neural network with fewerparameters. There are two different categories of methodsaccording to their techniques and motivations.

A. Network TrimmingNetwork trimming aims to remove redundancy in heavy

networks to obtain a compact network with fewer parametersand less computational complexity, whereas the accuracyof this portable network is close to that of the original

large model. Gong et al. [6] utilized the benefits of vectorquantization to compress neural networks, and a cluster centerof weights was introduced as the representation of similarweights. Denton et al. [7] implemented singular value decom-position to the weight matrix of a fully connected layer toreduce the number of parameters. Chen et al. [8] attemptedto explore hash encoding to improve the compression ratio.Courbariaux et al. [12] and Rastegari et al. [13] implementedbinary networks. All weights previously storied as 32-bit float-ing, are converted to binary ({−1, 0, 1} or {−1, 1}). Moreover,Wang et al. [11] and Han et al. [9] exploited weight pruningto achieve the same goal. In particular, Han et al. [9] focusedon removing subtle weights to reduce the parameters whileminimizing the impact of removing them. Over 80% of subtleweights were dropped without the accuracy drop. Furthermore,Han et al. [10] integrated several neural network compressiontechniques, i.e., pruning, quantization, and Huffman coding,to further compress the network. Wang et al. [11] showedthat redundancy exists not only subtle weights but also largeweights. It converted convolutional kernels into the frequencydomain to reduce the redundancy contained in larger weightsand thereby compress networks with a higher compressionratio. Also, Wang et al. [22] focused on the redundancy infeature maps instead of network weights, which can also beconsidered as a modification of network architecture. Althoughthe network trimming method brings a considerable compres-sion and speedup ratio, due to the highly sparse parametersand the irregular network architectures, the actual accelerationeffect is often heavily dependent on the customized hardware.

B. Design Small NetworksDirectly designing a new deep neural network of light

size is a straightforward approach to realize efficient deeplearning. Most of these methods increase the depth of networkswith much lower complexity compared to simply stacking


GUO et al.: ROBUST STUDENT NETWORK LEARNING 3

convolution layers. For example, ResNet [5] introduced anovel residual block that obtained a significant performancewith only slightly computation costs. ResNeXt [14] exploredgroup convolutions into the building blocks to boost perfor-mance. Flattened networks [23] introduced fully factorizedconvolutions and designed an extremely factorized network.Almost at the same time, Factorized Networks [24] introducedtopological convolution that treats sections of tensors sepa-rately. SqueezeNet [25] designed a portable network with abottleneck architecture. SENet [26] proposed a novel archi-tecture named SE block, which focuses on the relationshipbetween channels. Moreover, depthwise separable convolutionis introduced in [27] to obtain a great gain in the speed, andthe size of networks. With the help of depthwise separableconvolutions, Inception models [28], [29] reduced the com-plexity of the first few layers of the network. Later, Xcep-tion network [15] outperformed Inception model by scalingup depthwise separable convolutional filters. Subsequently,the MobileNets [16] combined channel-wise decompositionof convolutional filters with depthwise separable convolu-tions and achieved state-of-the-art results among portablemodels. ShuffleNet [30] introduced a novel form of groupconvolution and depth-wise separable convolution. Deep-friedconvnets [31] introduced a novel Adaptive Fastfood transformto reduce the computation of networks. Structured transformnetworks [32] offered considerable accuracy–compactness–speed tradeoffs based on the new notions rooted in the theoryof structured matrices.

C. Teacher–Student LearningAs one way of training a portable network, teacher–student

learning aims to learn a student network with much fewerparameters compared with the teacher network. There aremany heavy networks with a huge number of parameters.With the help of large amounts of data, these heavy networksare well trained and will achieve state-of-the-art performanceon many challenging data sets. However, the high demandsplaced on the storage space and computing power caused bythe huge number of parameters limit the application scenariosof these giant networks. This allows them to be deployedonly on high-performance computers with GPUs. Learninga small network directly has proven to be hard to trainand difficult to achieve satisfactory accuracy. Teacher–studentlearning introduces a method of learning a small network bybuilding a bridge between the heavy network and the smallnetwork. By regarding the well-trained heavy network as ateacher and the portable network as a student, this method issuccessful in utilizing the intrinsic information captured by theteacher to conduct the learning progress of the student. Thereare lots of methods beneficial to learning deep networks andimproving proposed to distillate knowledge from the teacherand learning a portable student network. Ba and Caruana [18]introduced a method that the student network mimics thefeatures extracted from the last layer of the teacher networkto assist the training progress of student networks, therebyincreasing the depth of the student network. KD [17] pointedout that, for two networks with huge structural differences,it is difficult to directly mimic features. Therefore, KD [17]

proposed to minimize the relaxed output of softmax layersof the two networks. This strategy can further deepen thestudent network. FitNet [33], based on KD, minimized thedifference between the features extracted from the middlelayers of the student and teacher networks. They added severallayers of multilayer perceptrons at the middle layer of theteacher network to match the dimensions of the features ofthe student network. By establishing a connection betweenthe middle layers of two networks, the student network canbe further deepened with fewer parameters. McClure andKriegeskorte [19] attempted to minimize the distance betweenpairs of samples to reduce the difficulty of training the studentnetwork. You et al. [20] proposed utilizing multiple teachernetworks to provide more guidance for the training of studentnetworks. They leverage a voting strategy to balance themultiple guidance from each teacher network. Wang et al. [21]regarded the student network as a generator which is a partof generative adversarial network [34], as well as utilized adiscriminator as an assistant of teacher for forcing the studentto generate features which are difficult to distinguish fromfeatures of the teacher.

Compared to the network trimming algorithm, the student–teacher learning framework has more flexibility, no specialrequirements on hardware, and a more structured networkstructure. Compared to the direct design of a deeper network,guidance from the teacher is beneficial to learning deepnetworks and improving the performance of the student. How-ever, existing student–teacher algorithms pay more attentionto improving the performance of the student network onpure data sets. The instability caused by the large reductionin parameters makes the performance degradation under theperturbation settings not yet studied. Therefore, a more robustlearning algorithm for improving student network performanceunder perturbed conditions is desired. This paper proposed amethod under the teacher–student learning and KD frameworkto improve the robustness of the student network.

III. PRELIMINARY OF TEACHER–STUDENT LEARNING

To make this paper self-contained, we briefly introducesome preliminary knowledge of teach-student learning here.

The teacher network NT has a complicated architecture, andit has already been well trained to achieve sufficiently highperformance. We aim to learn a student network NS , which isdeeper yet thinner than the teacher network NT but still has asatisfying accuracy. Let X be the example space and Y be itscorresponding k-label space. Outputs of these two networksare defined as

oT = softmax(aT ), oS = softmax(aS) (1)

where aT and aS are the features produced by presoftmaxlayers of teacher and student networks, respectively.

The teacher network NT is usually trained on a relativelylarge data set and consists of a large number of parametersso that the teacher network usually achieves high accuracy inthe classification task. Given significantly fewer parametersand numbers of multiplication operations, if adopting thesame training strategies as the teacher network, the studentnetwork NS is difficult to achieve high performance. It is,



therefore, necessary to improve the student network perfor-mance by investigating the assistance of the teacher network.A straightforward method is to encourage the features of animage extracted from these two networks to be similar [18].The objection function can be written as

L(NS) = 1

n

n∑i=1

[H(

oiS, yi)+ λ

2

∥∥oiS − oi

T

∥∥2]

(2)

where the second term helps the student network to extractknowledge from the teacher, H refers to the cross-entropyloss, oi

S indicates the output of the i th example in X by thestudent network, yi refers to the corresponding label, and λis the coefficient to balance two terms in the function. Theteacher and student networks can be significantly different inarchitecture, and thus, it is difficult to expect features extractedby these two networks for the same example to be the same.Hence, KD [17], as an effective alternative, was proposed todistill knowledge from classification results to minimize

LKD(NS) = 1

n

n∑i=1

[H(oi

S, yi)+ λH(τ(oi

S

), τ

(oi

T

))](3)

where the second term H(τ (oiS), τ (oi

T )) aims to enforce thestudent network to learn from the softened output of theteacher network. τ (·) is a relaxation function defined asfollows:

τ (oT ) = softmax( aT

τ

)

τ (oS) = softmax( aS

τ

). (4)

τ is introduced to make sure that the second term in equa-tion (3) can play a different role compared with the first one.This is because oT might be extremely similar to the onehot code representation of the ground truth labels, while asoftened version of output is different from the true labels.Moreover, the softened version of output could also providemore information to guide the learning of the student, as thecross-entropy loss and soften version output will enhance theinfluence of classes other than the true label one.

Although KD loss in (3) allows the student network toaccess the knowledge from the teacher network, the significantreduction in the number of parameters decreases the capabilityof the student network and makes it more vulnerable toinput disturbances. The learned student network might achievereasonable performance on clean data, but it would suffer froma serious performance drop when encountering perturbation onthe data in real-world applications. To solve this issue, it is,therefore, necessary to enforce the robustness of the studentnetwork when applied to a practical scenario.

IV. ROBUST STUDENT NETWORK LEARNING

We take a multi-class classification problem over k classesas an example to introduce our robust student network learn-ing. Given a teacher network NT and a student network NS ,an example x can then be classified by two networks oT (x) =NT (x) and oS(x) = NS(x), respectively. Denote o j

T (x) ando j

S(x) as the j th value of the k-dimensional vectors oT (x)

and oS(x), respectively. Then, we define fT (x) = oyT (x) and

fS(x) = oyS(x) as the scores produced by two networks for the

ground truth label y of example x, respectively. If a classifierhas more confidence in its prediction, the predicted score willbe higher. With the help of the teacher network, the studentnetwork is supposed to be more confident in its prediction,so that

fS(x) > fT (x). (5)

Confidence, here, refers to the predicted score of the neuralnetwork for an input image. If the network classifies the imageto a category with a higher perdition score, we suggest thenetwork is confident in its prediction.

A. Theoretical AnalysisThe above relationship holds in an ideally noise-free sce-

nario. In the practical scenario, perturbations on examples areunavoidable, and the student network is expected to resist theunexpected influence and bring in robust prediction

fS(x + δ) > fT (x + δ), �δ�2 ≤ R and x + δ ∈ C (6)

where δ is a perturbation added to x. The perturbation cancome from various sources. There could be electrical noise,photon noise, thermal noise, and so on. As noise degrades thequality of an image, the performance of neural networks inimage classification task could be seriously influenced. Theproposed robust student network aims to handle unexpectednoises in images and to preserve consistent decisions with orwithout noises. To achieve this, we restrict this perturbationin a spherical space of radius R, and C is a constraint set thatspecifies some requirements for the input, e.g., an image inputshould be in [0, 1]d , where d is the dimension. We define theball as Bp(x, R) = {z ∈ R

d | �x − z�p ≤ R}.We aim to discover a student network that stands on the

shoulder of the teachers to make a confident prediction notonly for clean examples but also for examples with perturba-tions. The perturbation δ exists on examples without influenc-ing their corresponding ground truth labels. However, with theincrease of perturbation intensity, the learning process of thestudent network would be seriously disturbed. Taking (6) asan auxiliary constraint in training the student network can behelpful for improving the robustness of the network. However,it is difficult and impossible to enumerate and try everypossible δ to form the constraint. To make the optimizationproblem tractable, we seek for some alternatives and proceedto study the maximum perturbation that can be defended bythe system. Fig. 1 shows the framework of our approach.

Theorem 1: Let x ∈ Rd be an example in X . fS(x) and

fT (x) are functions adapted from the student and teachernetworks to predict the label y of example x, respectively.Given fS(x) > fT (x), for any δ ∈ R

d with

�δ�q ≥ fS(x)− fT (x)

maxz∈Bp(x,R) �∇ fT (z)−∇ fS(z)�p(7)

we have fT (x + δ) > fS(x + δ).Proof: By the main theorem of calculus, we have

fS(x + δ) = fS(x)+∫ 1

0�∇ fS(x + tδ), δ�dt (8)



and

fT (x + δ) = fT (x)+∫ 1

0�∇ fT (x + tδ), δ�dt . (9)

If the perturbation δ is so significant that fT (x + δ) >fS(x + δ), we get

0 < fS(x)− fT (x) ≤∫ 1

0�∇ fT (x + tδ)−∇ fS(x + tδ), δ�dt .

(10)

Consider the fact that∫ 1

0�∇ fT (x + tδ −∇ fS(x + tδ), δ�dt

≤ �δ�q∫ 1

0�∇ fT (x + tδ)−∇ fS(x + tδ)�pdt (11)

where holder inequality is applied and q-norm is dual to thep-norm with 1

p+ 1q = 1. By combining (10) and (11), we have

fS(x)− fT (x)

≤∫ 1

0�∇ fT (x + tδ −∇ fS(x + tδ), δ�dt

≤ �δ�q∫ 1

0�∇ fT (x + tδ)− ∇ fS(x + tδ)�pdt . (12)

Then, we can upper bound the left term of above equationas following:

fS(x)− fT (x) ≤ �δ�q∫ 1

0�∇ fT (x + tδ)−∇ fS(x + tδ)�pdt .

(13)

Furthermore, we can obtain that

�δ�q ≥ fS(x)− fT (x)∫ 1

0�∇ fT (x + tδ)− ∇ fS(x + tδ)�pdt

. (14)

where the denominator can be further upper bounded usingthe following inequality:∫ 1

0�∇ fT (x + tδ −∇ fS(x + tδ)�pdt

≤ maxz∈Bp(x,R)

�∇ fT (z)−∇ fS(z)�p. (15)

As the 2-norm of δ is upper bounded by R which is the radiusof the Ball, and the value of x + tδ can get hardly out of thisball. The right inner term represents the max value in thisball. As a result, considering the left term of (15), the innerterm of it is definitely less or equal than the inner term of theright term of equation (15). Integrated by t will result in theright term. After that, we can finally conclude equation (15).Moreover, the lower bound for the q-norm of δ to break therobust prediction of the student network (i.e., equation (6)) istherefore

�δ�q ≥ fS(x)− fT (x)

maxz∈Bp(x,R) �∇ fT (z)−∇ fS(z)�p(16)

which completes the proof. �

According to Theorem 1, maximizing the value of fS(x)−fT (x) while minimizing the value of maxz∈Bp(x,R) �∇ fT (z)−∇ fS(z)�p, the lower bound over δ will be enlarged, so thatthe student network is able to tolerate more severe perturba-tion and become more robust to make confident prediction.L2-norm is widely used in the optimization field because it iseasier to calculate than other norms (such as L0, L1, and L p),and more importantly, it is easier to be optimized by gradientdescent in deep learning. In addition, in the field of the imageprocess, some concepts are often defined in the Euclideanspace, or the form of a power of 2, such as mean-square errorand peak signal-to-noise ratio. Considering these above facts,we set p = q = 2.

B. MethodBased on the analysis above, two new objectives are intro-

duced into the teacher–student learning paradigm to achieve arobust student network. To encourage fS(x) > fT (x), we planto minimize the loss function

LS(NS) = 1

n

n∑i=1

max(0, γ + fT (x i)− fS(x i)) (17)

where γ > 0 is a constant margin. fS(x) is supposed to begreater than fT (x) + γ ; otherwise, there will be a penaltyfor the student network. It is difficult to explicitly calculatethe value of maxz∈B2(x,R) �∇ fT (z) − ∇ fS(z)�2, due to theexistence of the max operation. However, by appropriatelysetting the radius R and considering the sufficiently largetraining set, the data point in the ball B2(x, R) to reach themaximum value of �∇ fT − ∇ fS�2 would often have someclosed examples in the training set. Hence, to minimize thevalue of maxz∈B2(x,R) �∇ fT (z) − ∇ fS(z)�2, we propose tominimize the difference between gradients of the student andthe teacher networks with respect to the training examples as

LG(NS) = 1

n

n∑i=1

�∇τ ( fS(x i))−∇τ ( fT (x i))�2 (18)

where τ is the relaxation function explained in (3). In addition,we take the KD loss [17] into consideration, the resultingobjective function of our robust student network learningalgorithm can be written as

L(NS) = LKD(NS)+ C1LG(NS)+ C2LS(NS) (19)

where C1 and C2 are the balanced coefficients of LG(NS)and LS(NS), respectively.

The process of training the student network can be foundin Algorithm 1. After initialization of the student network,we train the student network according to the proposed algo-rithm. Next, we explain in detail the calculation of loss. Forconvenience, we set the batch size as 1, i.e., we first selecta sample {x, y} from the data set X and Y as input forforwarding propagation of the teacher network and the studentnetwork. Then, we calculate outputs of the two networksoT (x) and oS(x). Combining outputs oT (x) and oS(x) withthe corresponding label y, we can calculate the first termin (19) LKD according to (3) and (4). oT and oS are bothk-dimensional vectors, which is the prediction score of the



Algorithm 1 Robust Student Network LearningInput: A given neural network NT ; training data set X with

n instances; the corresponding k-label set Y; parameters:λ, β, and τ .

1: Initialize a neural network NS , where the number ofparameters in NS is significantly fewer than that in NT ;

2: repeat3: Select an instance x and its label y randomly;4: Employ the teacher network: oT ← NT (x),5: Employ the student network: oS ← NS(x);6: Calculate the loss function L(NS) w.r.t. equation (18);7: Update weights in the student network NS;8: until reaching the limitation of training epoch;

Output: A robust student network NS .

network for k categories. With the help of label y, we canget the predicted scores for the label, fT and fS and calculatethe second term in (19) Ls according to (17). To get the valueof Lg , we first calculate the derivative of fT and fS withrespect to the input sample x. Automatic derivation tool inte-grated into PyTorch is applied here. Same as backpropagationalgorithm, we can apply the chain rule to get these results.By doing so, we have obtained the gradients of fS and fT

with respect to the input x , and now the loss mentionedin (17) can be easily calculated. After getting fT and fS ,we backpropagate the obtained loss through the network,which equals a second gradient, and we also leverage theautomatic derivation tool of PyTorch to achieve this. Finally,the parameters in the student network are updated with thegradients obtained by the backpropagation algorithm.

In the literature, the overall noise produced by a digitalsensor is usually considered as a stationary white additiveGaussian noise [35]. We report robustness of the learnednetworks against Gaussian noise in experiments. In addition,we also evaluate the performance of the learned networksagainst combinations of different types of noise on trainingand test sets, since it is difficult to know what types of noisecould be before the test stage.

V. EXPERIMENTS

In this section, we experimentally investigate the effec-tiveness of the proposed robust student network learningalgorithm. The learned student network is compared withthe original teacher network, and student networks arelearned through KD [17] and FitNet [33]. The experimentsare conducted on three benchmark data sets: MNIST [36],CIFAR-10 [37], and CIFAR-100 [37].

A. Data Sets and SettingsMNIST [36] is a handwritten digit data set (from 0 to 9)

composed of 28 × 28 greyscale images from ten categories.The whole data set of 70 000 images is split into 60 000 and10 000 images for training and test, respectively. Followingthe setting in [33], we trained a teacher network of maxoutconvolutional layers reported in [38], which contains threeconvolutional maxout layers and a fully connected layer

with 48-48-24-10 units, respectively. Later, we design thestudent network of six convolutional maxout layers and afully connected layer, which is twice as deep as the teachernetwork. However, the parameters of the student network areonly roughly 8% of the parameters of the teacher network.As reported in Table I, the architectures of the teacher andstudent networks were shown in detail in the first two columns.

CIFAR-10 [37] is a data set that consists of 32 × 32RGB color images draw from ten categories. There are60 000 images in the CIFAR-10 data set which are splitinto 50 000 training and 10 000 testing images. According to[38] and [33], we preprocessed the data using global con-trast normalization (GCN) and zero-phase component analy-sis whitening and augmented the training data via randomflipping. We followed the architecture used in Maxout [38]and FitNet [33] to train a teacher network with three maxoutconvolutional layers of 96-192-192 units. For a fair compari-son, we designed a student network with a structure similar toFitNet which has 17 maxout convolutional layers followed bya maxout fully connected layer and a top softmax layer, and wealso investigate KD method with the same architecture. Thedetailed architecture of the teacher was shown in the “Teacher(CIFAR-10)” column of Table I, and that of the student wasshown as “Student 4” column.

CIFAR-100 data set [37] has images of the same size andformat as those in CIFAR-10, except that it has 100 categorieswith only one-tenth as labeled images per category. More cat-egories and fewer labeled examples per category indicate thatclassification task on CIFAR-100 is more challenging than thaton CIFAR-10. We preprocessed images in CIFAR-100 usingthe same methods for CIFAR-10, and the teacher network andthe student network share the same hyperparameters with thoseon the CIFAR-10 data set. In addition, the architecture of theteacher is also the same as that used for CIFAR-10, except thatthe number of units in the last softmax layer was changed to100 to adapt to the number of categories.

The hyperparameters are tuned by minimizing the error ona validation set consisting of the last 10 000 training exampleson each data set. Following the setting in FitNet [33], we seta batch size as 128, max training epoch as 500, learning rateas 0.17 for linear layers and 0.0085 for convolutional layers,and momentum as 0.35. According to the hint layer proposedin FitNet [33], we pretrained a classifier using the features inthe middle layer of the teacher network, and then we applythe classifier with the student network features.

B. Robustness of Student Networks

We evaluated the robustness of the student network learnedthrough different algorithms under different intensities ofperturbation. Since it is difficult and sometimes impossibleto know what test data can be in practice, the augmentation oftraining data with certain noise cannot be very helpful to resistthe perturbation. Hence, we trained all networks using cleantraining set and introduced white Gaussian Noise (WGN) intotest data as the perturbation. The intensity of the introducednoise was measured in terms of signal-to-noise ratio (SNR).We trained the proposed algorithm and compared it with



TABLE I

MODEL ARCHITECTURE FOR DATA SETS

Fig. 2. Accuracies obtained by different networks trained on three data sets and under various values of SNR. (a) MNIST. (b) CIFAR-10. (c) CIFAR-100.

the teacher network [38], and the student network from KDnetwork [17] and FitNet [33] methods.

In Fig. 2, we investigated the accuracy of these networkson three data sets with different SNR values. As the classifi-cation task on MNIST is relatively easier, lower SNRs werechosen from 9 to 1. Lower SNR value indicates that moreperturbations are added. It can be found from Fig. 2(a) thatthe accuracy of the proposed robust student network is superiorto the other three networks nearly under all SNR values. WhenSNR equals 2, two student networks from KD and FitNetperform even worse than the original teacher network. Howour proposed algorithm achieves obviously leading 98.17%accuracy. When SNR was down to 1, the accuracy drops of theteacher network and the student network from KD and FitNetare serious, up to 5.65%, 7.25%, and 3.23%, respectively.In contrast, the accuracy of our robust student network onlydrops 2.23%. Our method achieves better performance andshows more robustness when there was perturbation in theinput.

A similar phenomenon can be observed in Fig. 2(b) and (c)on the CIFAR-10 and CIFAR-100 data sets. With the decreaseof SNR, the accuracy of KD network and FitNet droppedfaster than that of the teacher network, especially during theperiod when SNR drops from 12 to 10. Given the significantreduction in network complexity, the capacity of the studentnetwork can be seriously weakened, and the student networkwould be more vulnerable to perturbations on data if there isno appropriate response action. However, the student networklearned from the proposed algorithm can be robust to seriousperturbations.

In Fig. 3, we reported the predicted scores of exampleimages by different methods on the CIFAR-10 data set. Theclean image without noise looks fuzzy, since the images fromthe CIFAR-10 data set only a resolution of 32×32. As shownin Fig. 3(c), all student networks can confidently predict theground truth class “cat” of the image. However, given the sameimage added with SNR=10 noise in Fig. 3(b) and (d) showsthat though student networks from KD and FitNet methods



Fig. 3. Example images (left column) and their corresponding prediction scores by different networks (right columns). (a) and (e) Pure images. (c) and (g) Theircorresponding prediction scores. (b) and (f) Disturbed images. (g) and (h) Their corresponding prediction scores.

reluctantly made the correct prediction, KD also thought theimage is similar to “deer,” and FitNet trusted “bird” as theprediction with a higher confidence level. On the contrary,our robust student network confidently insists on its correctprediction even the quality of the image has been seriouslyinfluenced by the perturbation. In addition, given the “ship”image, the teacher network can stand against the perturbation,due to its strong capability coming from the complicatednetwork structure. The KD method mistakes it as a “deer”image, while FitNet assigned a higher score to label “deer” forthis “ship” image. By encouraging more confident predictionswith help of the teacher network during the training stage,we derive the robust student network that can not only keepthe highest prediction score on the “ship” label but alsosuppress the predictions on wrong categories [see label “cat”in Fig. 3(g) and (h)].

We conduct experiments on a much challengeable imageclassification data set ImageNet ILSVRC 2012 [1]. There aremore than 1.28 M images for training and 50 K images forvalidation. In this part, we choose ResNet-101 [39], whichcontains about 128 M parameters as the teacher networkand Inception-BN [40] which contain 32 M parameters asthe student network. Referring to [41], we set the initiallearning rate as 0.1, and it was divided by 10 at 30, 60,and 90 epochs, respectively. We optimize the network usingNesterov accelerated gradient with weight decay as 10−4 andmomentum as 0.9. We train all models with clean data. In thetest phase, we follow the method used in Section V-B to

TABLE II

NOISED VALIDATION RESULTS ON IMAGENET

introduce noise into test data and validate these models with1-crop. We report the Top-1 results in Table II. The teachernetwork achieves 28.77% Top-1 error. The proposed methodachieves 30.15% Top-1 error, which outperforms student net-works trained by comparison algorithms. In this data set, allstudent networks failed to outperform the teacher but stillshowed acceptable results which are not far away from theteacher.

C. Comparison Under Different Perturbation

A neural network might handle noisy test data if similarnoise also exists in the training set. However, in practice,it is difficult to guarantee the test data to have the sametype of perturbation as the training data. We next proceed toevaluate the performance of different methods under differentcombinations of noisy training and test sets. The accuracies



TABLE III

PERFORMANCE COMPARISON ON DIFFERENT TRAINING AND TEST SETS. DATA SET WAS SPLIT AS TRAINING/TEST. “C” REPRESENTSCLEAN DATA, “G” REPRESENTS DATA WITH GAUSSIAN NOISE, AND “P” REPRESENTS DATA WITH POISSON NOISE

TABLE IV

PERFORMANCE COMPARISON ON DIFFERENT BLOCK SIZES

in different settings are presented in Table III. The first linein Table III is a description of the experiment settings. Thefirst capital letter indicates the noise type introduced to thetraining set, and the second letter indicates that of the testset. It should be noted that the results of the Robust networklisted in this table are all trained under the clean data set,but tested under the corresponding type of noise indicatedby the first line. If both training and test data are clean, allnetworks can achieve more than 90% accuracy, as shown in thefirst column of Table III. If networks are trained on the cleandata and tested on the data with Gaussian noise, the teacherand student networks from KD and FitNet will be seriouslyinfluenced and can only achieve less than 86% accuracy.However, the proposed robust student can still own more than90% accuracy. A similar phenomenon can be observed whenthe networks are trained with Gaussian noise but tested withPoisson noise and the other way round, as shown in the fourthand last columns of Table III, respectively. If both training andtest data are polluted with the same type of Gaussian noise, allnetworks would try to fit the noisy data as far as possible andreceive only a slight performance drop. However, this rigorousconstraint overtraining and test data cannot always be satisfiedin real-world applications. It shows that adding limited kindsof noise to the training set is difficult to improve the robustnessof neural network when facing various unexpected types ofnoises existing in practical applications.

Moreover, Table III shows that the accuracy of the teachernetwork is worse than that of “Robust” Network when trainingand test sets are both with Gaussian noise. The distributions oftraining data and test data would not be significantly differentif they are both polluted by the Gaussian noise. Hence,the general teacher network and student network can well

fit the noisy data and receive reasonable accuracy. However,the student network achieves some performance improvement,because of its deeper architecture than that of the teachernetwork. The depth encourages the reuse of features and leadsto more abstract and invariant representations at higher layers.The proposed robust student network can successfully traina deeper network by exploiting information from the teachernetwork.

D. Complex PerturbationIn real-world applications, noise is not the only perturbation

that may be encountered. Some more complex perturbationalso challenges the robustness of neural networks. In thissection, we investigate the robustness of the student networkobtained by our proposed method on two more complexperturbations, i.e., image occlusion and domain adoption.

1) Image Occlusion: Considering the target object in thereal environment is often blocked, and such perturbation oftenresults in the loss of information in a continuous area, the per-formance of neural networks will be influenced more seriouslyby image occlusion. In order to investigate the robustness ofour method under this disturbance, we took image occlusion asa more complex perturbation. To simulate the occlusion in thereal-world applications, we randomly selected a small rectan-gular area in an image and set pixels covered by the rectangleas zeros. Five different block sizes, i.e., 2×2, 4×4, 6×6, 8×8,and 10 × 10, are used in experiments. We implemented thisexperiment on the CIFAR-10 data set and the CIFAR-100 dataset. The results are shown in Table IV. Given 4 × 4 blocks,teacher, KD, FitNet, and the proposed network, respectively,have accuracies of 89.03%, 89.53%, 89.94%, and 90.79%.Given 8× 8 blocks, the corresponding accuracies are 85.65%,



TABLE V

DOMAIN ADAPTATION RESULTS

83.37%, 84.62%, and 85.92%, respectively. According to theseresults, larger blocks indicate more serious perturbations ofimages, which will degrade the performance of neural net-works. However, the student network obtained by the proposedmethod stably stays ahead, because of its robustness.

2) Domain Adaptation: In practical applications, not onlyunexpected noise and occlusion but also the unexpected distri-bution shift could challenge the robustness of neural networks.It is also an important indicator to evaluate the adaptability ofthis algorithm in the task of domain adaptation.

In this experiment, we took the USPS data set obtainedfrom the scanning of handwritten digits from envelopes bythe U.S. Postal Service. The images in this data set are all16×16 grayscale images, and the values have been normalized.The whole data set has 9298 handwritten numeric images,of which 7291 are for training, and the remaining 2007 arefor validation. Similar to MNIST, the USPS data set has tencategories, but it has different numbers of samples per cate-gory. In addition, considering the picture size in the MNISTdata set is 28×28, for convenience, we pad the images in theUSPS data set to the same size. Moreover, we preprocessedthe USPS data sets in the same way as MNIST.

In this section, we train student networks on the MNISTdata set and test them on the USPS data set. Similarly, we trainnetworks on the USPS data set and test them on MNIST.The results are shown in Table V. The first two columnsshow the result of adapting MNIST to USPS, and the per-formance of adapting USPS to MNIST was reported in thelast two columns of this table. According to the results, theproposed algorithm achieves an accuracy of 95.02%, whilethe comparison methods KD and FitNet only get 93.25%and 94.12%, respectively. This demonstrates that the proposedrobust student network can preserve its robustness advantageover comparison methods when faced with more complexperturbation of data in domain adaptation task. The simi-lar phenomenon can be observed in the results of “USPSto MNIST.” With the similar accuracy on USPS data set,the Robust Network outperforms networks obtained by theother algorithms. Moreover, the results tested on USPS dataset while trained on MNIST data set are much better thanthose tested on MNIST and trained on USPS. This is becausethe number of pictures in the MNIST data set is much largerthan that of USPS. The networks trained by MNIST data setcould extract more useful information from a larger amountof data and thus has better generalization capabilities.

3) Difference With Image Recovery: The mainstream visualrecovery algorithms focus on recover images which are pol-luted, such as image denoising [44], [45], image deblur [46],

and image inpainting [47], [48]. These methods aim to restoreimages from polluted images with the help of prior knowledge.Combining with these visual recovery algorithms, classifica-tion networks enjoy higher performance as the polluted inputimages are cleaned. The proposed method does not includerecovery images directly, but show robustness for pollutedimages in the classification task. As the analysis in thispaper, the proposed method trains the student network withina Ball not only at data points, which provides the studentnetwork the ability to classify the polluted image with highaccuracy.

E. Comparison With State-of-the-Art MethodsAlthough the main purpose of this paper is to improve

the robustness of the student network, instead of focusingon the performance of the student network on clean data,we also compared the proposed approach with a state-of-the-art teacher–student learning methods on clean data sets.For the clean data, the proposed algorithm can still achievecomparable accuracy as compared to others in Tables VI–VIII.Table VII summarizes the obtained results on three datasets: MNIST, CIFAR-10, and CIFAR-100. On the MNISTdata set, the teacher network got 99.45% accuracy. Withthe assistance of KD, the student network achieved 99.46%accuracy. FitNet generated a slightly better student networkwith 99.49% accuracy, which has outperformed the teachernetwork. Although the proposed algorithm aims to enhancethe robustness of the learned student network, it can alsoachieve comparable or even better accuracy than those of state-of-the-art methods. The accuracy obtained by the proposedmethod increased to 99.51% on the MNIST data set. Table VIshows results on the CIFAR-10 data sets, and the baselineteacher network achieved 90.25% accuracy. Without help fromthe teacher network, the student network finally achieved86.78% accuracy. Accuracy of the student network generatedby KD and FitNet was 91.07% and 91.64%, respectively. Theproposed student network obtained 91.93% accuracy as shownin Table VI, which outperforms other student networks andteacher. This suggests that the proposed method is able toenhance the stability of the student network and then improvethe performance of the network.

CIFAR-100 is similar but more challenging than CIFAR-10because of its 100 categories. The accuracy obtained by theteacher network is only 63.49%. As a comparison, the accu-racy of the teacher on CIFAR-10 is 90.25%, which is muchbetter than that on CIFAR-100. The robust student networkachieved 65.28% accuracy, which outperforms student net-works trained by other strategies, i.e., the network trained byKD obtains a test accuracy of 64.13%, and the accuracy ofFitNet is 64.86%. When compared to other methods, the stu-dent network generated by the proposed method providesnearly state-of-the-art performance. This result demonstratesthat the proposed method succeeds in assisting to learn astudent network with considerable performance.

F. Analysis on Structures of Student NetworkWe followed the experimental setting in FitNet [33] and

designed four student networks with different configurations



TABLE VI

CLASSIFICATION ACCURACIES OF DIFFERENT NETWORKS ON CIFAR-10 AND CIFAR-100 DATA SETS

TABLE VII

CLASSIFICATION ACCURACIES ON THE MNIST DATA SET

of parameters and layers. The teacher network has the samestructure as that used on the CIFAR-10 data set. We designfour student networks of different sizes and structures, and thedetailed structure of these networks can be found in Table I.From “Student 1” to “Student 4,” the volume of the networkhas gradually increased, and the performance of the networkhas gradually increased as well. Table IX reported the perfor-mance of four student networks and the teacher network onthe CIFAR-10 data set. The compression ratio and speed-upratio compared with the teacher, and the number of parametersand multiplications can also be found in Table IX.

From Table IX, we find that the proposed robust studentnetwork outperforms FitNet under all four different studentstructures. Although there is no perturbation on the data,the proposed method can achieve higher accuracy, whichindicates the effectiveness of encouraging the student networkto make confident predictions with the help of the teachernetwork. In addition, the smallest network Student 1 hasthe biggest compression and speed-up ratios, but it can stillachieve a test accuracy of 89.62%, which is fairly close to the90.25% of teacher and outperforms the 89.07% obtained byFitNet. Student 1 contains significantly fewer parameters thanthose of the teacher, improving the accuracy of such a network

Fig. 4. Accuracy of robust student network obtained by differenthyperparameters.

with limited capacity is challenging, which in turn reflects theeffectiveness of the proposed method.

Although there are significantly fewer parameters containedin the student network learned by the proposed method, thesestudent networks are still regular networks which can befurther compressed and accelerated by existing sparsity-baseddeep neural network compression technologies, such as deepcompression [10] and feature compression [11].

G. Hyperparameters and Running Time AnalysisWe choose hyperparameters following the principle of

cross-validation. First, a validation set is sampled from thetraining set. Then, when the training progress is converged,the student network is evaluated on this validation set. Hyper-parameters which provide the best accuracy on the validationset will be chosen.

Fig. 4 shows the accuracies obtained on the CIFAR-10 dataset with different values of hyperparameters C1 and C2. It canbe found in Fig. 4 that the too large or too small value ofthese hypermeters will lead to a network of lower accuracy.This is because smaller values of hypermeters will reduce theinfluence of losses proposed in this paper, and larger valuescould lead the student network to ignore the classification lossfunction. We choose C1 and C2 with the best performance onthe validation set.



TABLE VIII

TEN-CLASS CLASSIFICATION RESULTS OF DIFFERENT NETWORKS ON THE CIFAR-10 DATA SET

TABLE IX

PERFORMANCE OF THE PROPOSED METHOD ON STUDENT NETWORKS WITH VARIOUS ARCHITECTURES

TABLE X

EXECUTION TIMES OBTAINED ON THE CIFAR-10 DATA SET

We conduct our experiments with PyTorch on an NVIDIA1080 Ti GPU. Table X reports the execution times of dif-ferent networks on the CIFAR-10 data set. We find that thecomputation demand of the student network is much less thanthat of the teacher network. Since comparison methods sharethe same architecture of the student network, they have asimilar test time. The training time of FitNet and the proposedmethod are more than that of the teacher network and KDmethod. FitNet introduced additional networks to teach thestudent network, while the proposed method needs to calculategradients of input images and weights in an iteration. Althoughthe training cost of the proposed algorithm is larger, ourlearned student network receives better forward cost, as shownin Table X. In addition, teacher–student learning mechanismaims to learn a student network that has fewer parameters andfaster forward speed. Once the student network has been welltrained, we can keep the benefits of less computation, storagecost, and faster forward speed, and do not need to update anyparameters of the network during the inference.

VI. CONCLUSION

We proposed to learn a robust student network with theguidance of the teacher network. The proposed method pre-vented the student network from being disturbed by theperturbations on input examples. Through rigorous theoretical

analysis, we proved a lower bound of perturbations that willweaken the confidence of the student network in its prediction.We introduced new objectives based on prediction score andgradients of examples to maximize this lower bound andthen improved the robustness of the learned student networkto resist perturbations on examples. Experimental results onseveral benchmark data sets demonstrate the proposed methodis able to learn a robust student network with satisfyingaccuracy and compact size.

REFERENCES

[1] O. Russakovsky et al., “ImageNet large scale visual recognition chal-lenge,” Int. J. Comput. Vis., vol. 115, no. 3, pp. 211–252, Dec. 2015.

[2] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classificationwith deep convolutional neural networks,” in Proc. Adv. Neural Inf.Process. Syst., 2012, pp. 1097–1105.

[3] K. Simonyan and A. Zisserman, “Very deep convolutional networks forlarge-scale image recognition,” Apr. 2014, arXiv:1409.1556. [Online].Available: https://arxiv.org/abs/1409.1556

[4] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinkingthe inception architecture for computer vision,” in Proc. IEEE Conf.Comput. Vis. Pattern Recognit. (CVPR), Jun. 2016, pp. 2818–2826.

[5] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning forimage recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit.(CVPR), Jun. 2016, pp. 770–778.

[6] Y. Gong, L. Liu, M. Yang, and L. Bourdev, “Compressing deep convolu-tional networks using vector quantization,” Dec. 2014, arXiv:1412.6115.[Online]. Available: https://arxiv.org/abs/1412.6115

[7] E. L. Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fergus, “Exploit-ing linear structure within convolutional networks for efficient evalua-tion,” in Proc. Adv. Neural Inf. Process. Syst., 2014, pp. 1269–1277.

[8] W. Chen, J. Wilson, S. Tyree, K. Weinberger, and Y. Chen, “Compress-ing neural networks with the hashing trick,” in Proc. 32nd Int. Conf.Mach. Learn., Jun. 2015, pp. 2285–2294.

[9] S. Han, J. Pool, J. Tran, and W. Dally, “Learning both weights andconnections for efficient neural network,” in Proc. Adv. Neural Inf.Process. Syst., 2015, pp. 1135–1143.

[10] S. Han, H. Mao, and W. J. Dally, “Deep compression: Compress-ing deep neural networks with pruning, trained quantization andHuffman coding,” Feb. 2015, arXiv:1510.00149. [Online]. Available:https://arxiv.org/abs/1510.00149

[11] Y. Wang, C. Xu, S. You, D. Tao, and C. Xu, “CNNpack: Packingconvolutional neural networks in the frequency domain,” in Proc. Adv.Neural Inf. Process. Syst., 2016, pp. 253–261.



[12] M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv, and Y. Bengio,“Binarized neural networks: Training deep neural networks with weightsand activations constrained to+1 or-1,” Mar. 2016, arXiv:1602.02830.[Online]. Available: https://arxiv.org/abs/1602.02830

[13] M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi, “XNOR-Net:ImageNet classification using binary convolutional neural networks,”in Proc. Eur. Conf. Comput. Vis., Sep. 2016, pp. 525–542.

[14] S. Xie, R. Girshick, P. Dollar, Z. Tu, and K. He, “Aggregated residualtransformations for deep neural networks,” in Proc. IEEE Conf. Comput.Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 1492–1500.

[15] F. Chollet, “Xception: Deep learning with depthwise separableconvolutions,” Apr. 2017, arXiv:1610.02357. [Online]. Available:https://arxiv.org/abs/1610.02357

[16] A. G. Howard et al., “MobileNets: Efficient convolutional neural net-works for mobile vision applications,” Apr. 2017, arXiv:1704.04861.[Online]. Available: https://arxiv.org/abs/1704.04861

[17] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in aneural network,” Mar. 2015, arXiv:1503.02531. [Online]. Available:https://arxiv.org/abs/1503.02531

[18] J. Ba and R. Caruana, “Do deep nets really need to be deep?” in Proc.Adv. Neural Inf. Process. Syst., 2014, pp. 2654–2662.

[19] P. McClure and N. Kriegeskorte, “Representational distance learningfor deep neural networks,” Frontiers Comput. Neurosci., vol. 10, p. 131,Dec. 2016.

[20] S. You, C. Xu, C. Xu, and D. Tao, “Learning from multiple teachernetworks,” in Proc. 23rd ACM SIGKDD Int. Conf. Knowl. DiscoveryData Mining, Aug. 2017, pp. 1285–1294.

[21] Y. Wang, C. Xu, C. Xu, and D. Tao, “Adversaial learning of portablestudent networks,” in Proc. 32nd AAAI Conf. Artif. Intell., Apr. 2018,pp. 1–8.

[22] Y. Wang, C. Xu, C. Xu, and D. Tao, “Beyond filters: Compact featuremap for portable deep model,” in Proc. 34th Int. Conf. Mach. Learn.,Aug. 2017, pp. 3703–3711.

[23] J. Jin, A. Dundar, and E. Culurciello, “Flattened convolutional neuralnetworks for feedforward acceleration,” Nov. 2014, arXiv:1412.5474.[Online]. Available: https://arxiv.org/abs/1412.5474

[24] M. Wang, B. Liu, and H. Foroosh, “Factorized convolutional neuralnetworks,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Oct. 2016,pp. 545–553.

[25] F. N. Iandola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally,and K. Keutzer, “SqueezeNet: AlexNet-level accuracy with 50x fewerparameters and <0.5MB model size,” Nov. 2016, arXiv:1602.07360.[Online]. Available: https://arxiv.org/abs/1602.07360

[26] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,”in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2018,pp. 7132–7141.

[27] V. Vanhoucke, “Learning visual representations at scale,” ICLR InvitedTalk, vol. 1, p. 2, Apr. 2014.

[28] C. Szegedy et al., “Going deeper with convolutions,” in Proc. IEEEConf. Comput. Vis. Pattern Recognit., Jun. 2015, pp. 1–9.

[29] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deepnetwork training by reducing internal covariate shift,” Mar. 2015,arXiv:1502.03167. [Online]. Available: https://arxiv.org/abs/1502.03167

[30] X. Zhang, X. Zhou, M. Lin, and J. Sun, “ShuffleNet: An extremelyefficient convolutional neural network for mobile devices,” Dec. 2017,arXiv:1707.01083. [Online]. Available: https://arxiv.org/abs/1707.01083

[31] Z. Yang et al., “Deep fried convnets,” in Proc. IEEE Int. Conf. Comput.Vis. (ICCV), Dec. 2015, pp. 1476–1483.

[32] V. Sindhwani, T. Sainath, and S. Kumar, “Structured transforms forsmall-footprint deep learning,” in Proc. Adv. Neural Inf. Process. Syst.,2015, pp. 3088–3096.

[33] A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta,and Y. Bengio, “FitNets: Hints for thin deep nets,” Dec. 2014,arXiv:1412.6550. [Online]. Available: https://arxiv.org/abs/1412.6550

[34] I. Goodfellow et al., “Generative adversarial nets,” in Proc. Adv. NeuralInf. Process. Syst., 2014, pp. 2672–2680.

[35] T. Julliand, V. Nozick, and H. Talbot, “Image noise and digital imageforensics,” in Proc. Int. Workshop Digit. Watermarking, Mar. 2016,pp. 3–17.

[36] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-basedlearning applied to document recognition,” Proc. IEEE, vol. 86, no. 11,pp. 2278–2324, Nov. 1998.

[37] A. Krizhevsky and G. Hinton, “Learning multiple layers of features fromtiny images,” Tech. Rep., 2009.

[38] I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, andY. Bengio, “Maxout networks,” Sep. 2013, arXiv:1302.4389. [Online].Available: https://arxiv.org/abs/1302.4389

[39] K. He, X. Zhang, S. Ren, and J. Sun, “Identity mappings in deepresidual networks,” in Proc. Eur. Conf. Comput. Vis., Sep. 2016,pp. 630–645.

[40] S. Ioffe and C. Szegedy, “Batch Normalization: Accelerating deepnetwork training by reducing internal covariate shift,” Mar. 2015,arXiv:1502.03167. [Online]. Available: https://arxiv.org/abs/1502.03167

[41] Z. Huang and N. Wang, “Like what you like: Knowledge distillvia neuron selectivity transfer,” Jul. 2017, arXiv:1707.01219. [Online].Available: https://arxiv.org/abs/1707.01219

[42] M. Lin, Q. Chen, and S. Yan, “Network in network,” Mar. 2013,arXiv:1312.4400. [Online]. Available: https://arxiv.org/abs/1312.4400

[43] C.-Y. Lee, S. Xie, P. Gallagher, Z. Zhang, and Z. Tu, “Deeply-supervisednets,” Artif. Intell. Statist., vol. 21, pp. 562–570, Feb. 2015.

[44] D. Liu, B. Wen, X. Liu, Z. Wang, and T. S. Huang, “When imagedenoising meets high-level vision tasks: A deep learning approach,”in Proc. 27th Int. Joint Conf. Artif. Intell., Jul. 2018, pp. 842–848.

[45] U. Schmidt and S. Roth, “Shrinkage fields for effective image restora-tion,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2014,pp. 2774–2781.

[46] X. Tao, H. Gao, X. Shen, J. Wang, and J. Jia, “Scale-recurrent networkfor deep image deblurring,” in Proc. IEEE Conf. Comput. Vis. PatternRecognit. (CVPR), Jun. 2018, pp. 8174–8182.

[47] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros,“Context encoders: Feature learning by inpainting,” in Proc. IEEE Conf.Comput. Vis. Pattern Recognit. (CVPR), Jun. 2016, pp. 2536–2544.

[48] J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang, “Generativeimage inpainting with contextual attention,” in Proc. IEEE Conf. Com-put. Vis. Pattern Recognit. (CVPR), Jun. 2018, pp. 5505–5514.

Tianyu Guo received the B.E degree from TianjinUniversity, Tianjin, China, in 2016. He is currentlypursuing the Ph.D. degree with the Key Labora-tory of Machine Perception, Ministry of Education,Peking University, Beijing, China.

His current research interests include machinelearning and computer vision.

Chang Xu received the Ph.D. degree from PekingUniversity, Beijing, China.

He is currently a Lecturer and the ARC DECRAFellow with the School of Computer Science, Uni-versity of Sydney, Sydney, NSW, Australia. He hasauthored or coauthored more than 50 papers in pres-tigious journals and top tier conferences, includingthe IEEE TRANSACTIONS ON PATTERN ANALYSIS

AND MACHINE INTELLIGENCE (T-PAMI), the IEEETRANSACTIONS ON NEURAL NETWORKS ANDLEARNING SYSTEMS (T-NNLS), the IEEE TRANS-

ACTIONS ON IMAGE PROCESSING (T-IP), ICML, NIPS, IJCAI, and AAAI.His current research interests include machine learning algorithms and relatedapplications in computer vision.

He was a recipient of several paper awards, including the DistinguishedPaper Award in IJCAI 2018. He severed as a senior PC for many conferences,including AAAI and IJCAI. He has been recognized as Top Ten DistinguishedSenior PC in IJCAI 2017.



Shiyi He received the B.E degree from theBeijing University of Posts and Telecommunica-tions, Beijing, China, in 2016. She is currently pur-suing the master’s degree with the Key Laboratory ofMachine Perception, Ministry of Education, PekingUniversity, Beijing.

Her research interests include machine learningand computer vision.

Boxin Shi (M’14) received the B.E. degree fromthe Beijing University of Posts and Telecommu-nications, Beijing, China, in 2007, the M.E. degreefrom Peking University, Beijing, in 2010, and thePh.D. degree from the University of Tokyo, Tokyo,Japan, in 2013.

From 2013 to 2016, he with the MIT Media Lab,Singapore University of Technology and Design,Nanyang Technological University, Singapore,where he was involved in was postdoctoralresearch. From 2016 to 2017, he was a Researcher

with the National Institute of Advanced Industrial Science and Technology,Tokyo. He is currently Boya Young Fellow Assistant Professor with PekingUniversity, where he leads the Camera Intelligence Group.

Dr. Shi was a recipient of the Best Paper Runner-Up Award at InternationalConference on Computational Photography in 2015. He has served as theArea Chair for ACCV 2018, BMVC 2019, and 3DV 2019.

Chao Xu received the B.E. degree from TsinghuaUniversity, Beijing, China, in 1988, the M.S. degreefrom University of Science and Technology ofChina, Hefei, China, in 1991, and the Ph.D. degreefrom the Institute of Electronics, Chinese Academyof Sciences, Beijing, in 1997.

From 1991 and 1994, he was an Assistant Profes-sor with the University of Science and Technologyof China, Hefei. Since 1997, he has been withSchool of Electronics Engineering and ComputerScience (EECS), Peking University, Beijing, where

he is currently a Professor. His current research interests include image andvideo coding, processing, and understanding. He has authored or coauthoredmore than 80 publications and 5 patents in these fields.

Dacheng Tao (F’15) is a Professor of computerscience and the ARC Laureate Fellow with theSchool of Computer Science, Faculty of Engineer-ing, and the Inaugural Director of the UBTECHSydney Artificial Intelligence Center, University ofSydney, Darlington, NSW, Australia.

His research results in artificial intelligence haveexpounded in 1 monograph and 200+ publications atprestigious journals and prominent conferences, suchas IEEE TRANSACTIONS ON PATTERN ANALYSIS

AND MACHINE INTELLIGENCE (T-PAMI), the IEEETRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS

(T-NNLS), IJCV, JMLR, AAAI, IJCAI, NIPS, ICML, CVPR, ICCV, ECCV,ICDM, and KDD, with several best paper awards.

Dr. Tao is a fellow of the Australian Academy of Science. He was a recipientof the 2018 IEEE ICDM Research Contributions Award.

Documents

Robust Student Network Learning - PKUci.idm.pku.edu.cn/TNNLS19.pdfconvolution and depth-wise separable convolution. Deep-fried convnets [31] introduced a novel Adaptive Fastfood transform