Learning to Write Stylized Chinese Characters by Reading a ... · sults even if a small bias is introduced in Chinese character generation. To address this issue, we integrate the

Learning to Write Stylized Chinese Charactersby Reading a Handful of Examples

Danyang Sun∗, Tongzheng Ren∗, Chongxuan Li, Hang Su†, Jun Zhu†Department of Computer Science and Technology, Tsinghua Lab of Brain and Intelligence

State Key Lab for Intell. Tech & Sys., BNRist Lab, Tsinghua University, 100084, China{sundy16, rtz14, lcx14}@mails.tsinghua.edu.cn; {suhangss, dcszj}@tsinghua.edu.cn

Abstract

Automatically writing stylized characters is an at-tractive yet challenging task, especially for Chi-nese characters with complex shapes and struc-tures. Most current methods are restricted to gener-ate stylized characters already present in the train-ing set, but require to retrain the model when gen-erating characters of new styles. In this paper, wedevelop a novel framework of Style-Aware Vari-ational Auto-Encoder (SA-VAE), which disentan-gles the content-relevant and style-relevant compo-nents of a Chinese character feature with a novelintercross pair-wise optimization method. In thiscase, our method can generate Chinese charactersflexibly by reading a few examples. Experimentsdemonstrate that our method has a powerful one-shot/few-shot generalization ability by inferring thestyle representation, which is the first attempt tolearn to write new-style Chinese characters by ob-serving only one or a few examples.

1 IntroductionThe field of character generation remains relatively under-explored compared with the automatic recognition of char-acters [Pal and Chaudhuri, 2004; Zhong et al., 2016]. Thisunbalanced progress is disadvantageous for the developmentof information processing of all languages, especially Chi-nese, one of the most widely used languages that has its char-acters incorporated into many other Asian languages, suchas Japanese and Korean. Learning to write stylized Chinesecharacters by reading a handful of examples (as shown inFig. 1) is also a good testbed for machine intelligence as thetask can be finished effectively by humans while still remain-ing challenging for machines.

Though great efforts have been made on synthesizing sim-ple English or Latin characters [Graves, 2013; Lake et al.,2015], only a little exploration has been done on Chinese

* equal contribution; † corresponding authors.

Figure 1: Illustration of Chinese character generation task based onour method. Given a few examples with specified styles (e.g., thesignatures), we infer the latent vectors corresponding to differentstyles. Afterwards, we generate the stylized Chinese characters (“Tobe or not to be, that is a question”) by recognizing their contents andtransferring the corresponding styles (“Shakespeare”) to the targetedcharacters.

character generation. This is not surprising as Chinese char-acters pose new challenges due to their complexity and diver-sity. First, Chinese characters have a much larger dictionarycompared with the alphabetic writings. For instance, thereare 27,533 unique Chinese characters in the official GB18030charset, with daily used ones up to 3,000. Recently, Zhanget al. [2017] propose to generate Chinese characters basedon Recurrent Neural Networks [Hochreiter and Schmidhu-ber, 1997], which learns a low dimensional embedding as thecharacter label to solve the issue of large dictionary. How-ever, this method only generates characters without any styleinformation, and heavily relies on the online temporal infor-mation of the strokes, which cannot generalize to the off-lineapplications.

Moreover, Chinese characters have more complex shapesand structures than other symbolic characters such as Latinalphabets. Some attempts have been made on Chinese char-acter generation by assembling components of radicals andstrokes [Xu et al., 2009; Zong and Zhu, 2014; Lian et al.,2016]. However, these models rely on the preceding pars-ing, which requires each character to have a clear structure.Therefore, they cannot deal with a joined-up writing style , inwhich strokes are connected and cursive.

Finally, the generation of stylized Chinese characters is

Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI-18)

920

also challenged by the complexity in writing styles. Therecent “Rewrite” [Tian, 2016] and its advanced version“zi2zi” [Tian, 2017] based on the “pix2pix” [Isola et al.,2016] implement font style transfer of Chinese characters bylearning to map the source style to a target style with thou-sands of character pairs. However, these methods have tore-train a model from scratch when observing some Chinesecharacters with a new style, failing to re-utilize the knowledgelearned before.

1.1 Our ProposalTo address the aforementioned challenges, we proposea novel framework named Style-Aware Variational Auto-Encoder (SA-VAE) to conduct the stylized Chinese char-acter generation. Compared with the alternative populardeep generative model of Generative Adversarial Networks(GAN) [Goodfellow et al., 2014], Variational Auto-Encoder(VAE) [Kingma and Welling, 2014] naturally has the abilityof posterior inference to reveal the underlying factors, usuallyyielding better inference effects than inference extensions toGAN [Donahue et al., 2016; Dumoulin et al., 2016]. In thispaper, we make a feature-disentangled extension to VAE bothin the model level and algorithm level.

It is imperative to extract a pure and accurate style rep-resentation for the purpose of learning a new writing stylegiven one or a few examples, which therefore drives us todisentangle the style and content features from the given sam-ples. To our best knowledge, the most relevant study to solvethis two-factor disentanglement problem is using a bilinearmodel [Tenenbaum and Freeman, 1997]. However, the dis-entanglement based on matrix decomposition limits its appli-cabilities in the complicated situations with highly nonlinearcombinations. Therefore, in the model level, we leverage thepowerful capabilities of deep learning for non-linear functionapproximation to design a style inference network and a con-tent recognition network, which encode the style and contentinformation respectively. Meanwhile, in the algorithm level,we propose a novel intercross pair-wise optimization method,which encourages the extracted style embedding more pureand reliable.

Unlike the general image generation like faces [Ziwei Liuand Tang, 2015] or bedrooms [Yu et al., 2015], Chinese char-acter generation is also challenged by the large dictionary andrigorous structures. It may lead to confusing or incorrect re-sults even if a small bias is introduced in Chinese charactergeneration. To address this issue, we integrate the charac-ter knowledge into our framework by considering the specificconfiguration and radicals information of Chinese characters,which offers significant benefits when we generate charac-ters with complex structures. In order to generate charac-ters with an unseen style, our model learns the characteris-tics of numerous printing and handwritten styles as our stylebank. Relying on the powerful generalization capabilities ofour model, the proposed model makes a reasonable inferenceand conducts effective generalization on new styles withoutthe need to retrain the model.

Extensive experiments demonstrate that our method cangenerate stylized Chinese characters by reading only one ora few samples with diverse styles. To the best of our knowl-

edge, this is the first attempt to learn to write Chinese charac-ters of new styles in this one-shot/few-shot scenario. To sumup, our major contributions are as follows:• We propose a novel intercross pair-wise optimization

method to disentangle the style and content features ofChinese characters, which is a general technique to solvetwo-factor disentanglement problems with supervision.• We introduce the domain knowledge of Chinese charac-

ters into our model a priori to guide the generation.• Our proposed framework (SA-VAE) can achieve desir-

able style inference and generate stylized Chinese char-acters in the one-shot/few-shot setting, even for the un-seen styles in the training stage.

2 MethodologyIn this section, we first formally present our task and the ar-chitecture of the whole framework. Then, we present ourintercross pair-wise optimization method in detail. Finally,we show how to conduct one-shot/few-shot stylized Chinesecharacter generation with our model.

2.1 Problem FormulationTo frame the task formally, we make the mild assumptionthat all Chinese characters are independently decided by thecontent factor and the style factor, which can be denoted as:

xi,j ← (si, cj), (1)

where si represents style i, cj represents content j, and xi,jis the character with style i and content j.

Based on this assumption, we propose to infer the stylerepresentation when observing the characters, and integratethese style features with the content features to generate tar-get characters. As the characters do not have an explicitseparation between style and content, we thus have to do“reverse-engineering”— recognize the content and disentan-gle the style information from given characters, and combinethe style with target contents to generate the stylized Chinesecharacters.

To this end, we first build a dataset containing all the Chi-nese characters with a diverse set of styles as our style-bank,which is denoted as:

X = {xi,j}, i = 1, 2, ...,M , j = 1, 2, ...,N , (2)

whereM is the number of styles andN is the total number ofunique Chinese characters in the training dataset (See Sec. 3.1for details).

Afterwards, we build a Content Recognition Network Cand a Style Inference Network S to respectively obtain thedisentangled features of content cj and style si for each Chi-nese character xi,j . The characters are then generated via ageneration network G. In the one-shot/few-shot setting, weallow the model to observe only one or a few characters witha new style denoted as {xM+k,l:l+m} where k > 0,m ≥ 0.Based on these characters, the style feature sM+k can beinferred by S , which is combined with any content featurecn ∈ {c1:N} to generate the desirable stylized Chinese char-acter xM+k,n via G.


921

Figure 2: Our proposed SA-VAE framework mainly consist of three subnets, including a Content Recognition Network C, a Style InferenceNetwork S and a Character Generation Network G. S and C extract the style feature and the content feature respectively and G combinesthese two features to conduct the generation. Besides, we introduce the domain knowledge of Chinese characters K to get a more informativecontent representation. The training process is executed by an intercross pair-wise way.

2.2 Model Architecture

Here, we present the details of our model. The model mainlyconsists of three modules of a Content Recognition NetworkC, a Style Inference Network S and a Character GenerationNetwork G as illustrated in Fig. 2. The whole process canbe separated as two stages — inference and generation. Inthe inference stage, we first learn to disentangle the latentfeatures into content-related and style-related componentsbased on the Content Recognition Network and Style Infer-ence Network respectively. In the generation stage, we takethe content and style vectors as inputs via a deconvolutionalnetwork, such that the stylized characters are well recon-structed with the style feature that is appropriately capturedin the inference stage. The training process is executed byan intercross pair-wise way (See Sec. 2.3) for a reliabledisentanglement.

Content Recognition Network. Content recognition aimsto recognize the content of every Chinese character image,which provides a correct content label to the style inferencenetwork and generation network. We refer to the previousworks on state-of-the-art character recognition [Xiao et al.,2017; Zhong et al., 2016] to satisfy our need and formulate itas:

C : y = fη(x), (3)

where x denotes the character image, y is the content label,and η denotes the network’s parameters.

Character Structure Knowledge. Utilizing the domainknowledge of Chinese characters is essential to develop acompact model and improve the generation quality. Previ-ous deep generative models [Radford et al., 2015; Li et al.,2017] usually cast each category as a one-hot embedding,and concatenate it to all the feature maps in every layer. Thismethod works well for some small datasets such as MNISTand SVHN which have only ten classes but fails to solve the

Figure 3: A diagram to illustrate our encoding method, which con-tains the configuration information and the radical information.

Chinese character generation with thousands of classes1. Itmay lead to an explosive growth of model parameters. Toaddress this issue, we propose a more informative characterencoding method by introducing the configuration and radi-cal information of Chinese characters into our framework andbuild a corresponding index table K.

Compared with the one-hot embedding, our encodingmethod can reuse the configuration and radical informationshared among all the Chinese characters. The one-hot em-bedding y of every character can be transformed to a uniquecontent hashing code through a index table K as

K : c = T [y]. (4)Our encoding method is shown in Fig. 3, which is much

shorter than the one-hot embedding (3000 bits in total). Thefirst 12 bits are used to identify the 12 common configura-tions in Chinese characters, including up-down, left-right,surroundings and so on. The middle 101 bits are used toidentify 100 frequently used radicals and the case of missingradicals. The last 20 bits are as a binary index when theformer two parts are the same. Proved by the experimentalresults, our proposed encoding method compresses theembedding dimension while does not harm the model’scapacity compared with other embedding methods.

Style Inference Network. The Style Inference Network isthe key module to make our framework have the capacity of

1The number of classes is comparable to that of the ImageNet.


922

achieving accurate style inference. To automatically disen-tangle the style feature from the content information, we feedthe content hashing code c to this network besides the charac-ter image x. It is realized as a diagonal Gaussian distributionparameterized by the convolutional neural network. Hence,we formulate the Style Inference Network as:

S : qφ(s|x, c) = N (s|µφ(x,c), diag(σ2φ(x,c))), (5)

where µφ(x,c) and σ2φ(x,c) correspondingly denote the mean

and variance of the Gaussian distribution, and φ denotes theparameters of the convolutional neural network.

On the other hand, for a reliable disentanglement, a goodStyle Inference Network should try its best to satisfy thisequality:

qφ(si|xi,j , cj) = qφ(si|xi,k, ck), (6)

which requires the Style Inference Network to provide thesame posterior distribution given different characters with thesame style. This target implicitly ensures that the Style Infer-ence Network can extract the style information because thecontent information of the inputs are different. This propertyis very important as our main purpose is to disentangle thestyle information and achieve one-shot inference. We willexplain how to achieve Eq. (6) by the intercross pair-wiseoptimization method in detail in Sec. 2.3.

Character Generation Network. In order to generate dif-ferent characters with diverse styles flexibly, the CharacterGeneration Network takes the content hashing code c and thestyle feature s as two independent inputs, and these two dis-entangled factors jointly decide the generated characters. Weuse a Bernoulli distribution parameterized with a deconvo-lutional neural network [Zeiler et al., 2010; Radford et al.,2015] to model the binary image of Chinese characters, whichcan be formulated as:

G : pθ(x|s, c) = Bern(x|µθ(s,c)), (7)

where µθ(s,c) denotes the logits of Bernoulli distribution andθ denotes the parameters of deconvolutional neural network.

2.3 Intercross Pairwise OptimizationGenerally, latent variable generative models are usuallytrained by maximizing the marginal log-likelihood, but it isoften intractable to implement the optimization directly. Toaddress this issue, variational inference [Wainwright et al.,2008] can be applied by optimizing a variational lower boundparameterized with qφ(z|x) instead of the log-likelihood.

In our Chinese character generation task, the style andcontent of Chinese characters are independent as in Eq.(1).Meanwhile, the content code c can be obtained by the well-trained C. Therefore, we can get the conditional variationallower bound of vanilla VAE:

log pθ(x|c) ≥ log pθ(x|c)−KL [qφ(s|x, c)‖pθ(s|x, c)]=LELBO(x; c, θ,φ), (8)

where KL denotes the Kullback Leibler Divergence andpθ(s|x, c) is the true posterior from the generative model.

Figure 4: The comparison between vanilla VAE and our intercrosspair-wise training, in which dotted lines and solid lines denote thestyle providers and the content providers, respectively.

However, directly optimizing the Eq. (8) cannot ensure thatcharacters with the same font share a similar style embed-ding. This is because the vanilla variational inference treatseach character separately and there is no strong correlationbetween different characters, especially those with the samestyle. Hence, we propose a new objective of intercross pair-wise optimization, which is still a lower bound of the log-likelihood. Instead of reconstructing the input as in the vanillaVAE, we try to produce the target character based on twocharacters with the same style information but different con-tents, that is to say, the generated character obtains the styleinformation by another character with the same style, as isillustrated in Fig. 4.

Specifically, we consider a pair of characters (x,x′), whichshare the same style s but with different contents c and c′. Wedefine the objective function as

L(x,x′,c, c′; θ,φ)

= log pθ(x|c)−KL[qφ(s|x′, c′)‖pθ(s|x, c)]. (9)

By maximizing this objective, we can achieve the target ofEq.(6) which is the key point of disentanglement and one-shotinference. Specifically, from Eq.(9), we know that maximiz-ing L is equivalent to minimizing the KL term, since the datalikelihood is invariant. Meanwhile, as we randomly samplethe training pairs, the dual pairs will also appear. Hence, itsimultaneously minimizes:

KL[qφ(s|x′, c′)‖pθ(s|x, c)], (10)

KL[qφ(s|x, c)‖pθ(s|x′, c′)]. (11)

With the assumption in Eq.(1), we know that the real posteriordistribution of the style is identical for x and x′, that is

pθ(s|x, c) = pθ(s|x′, c′). (12)

Ideally, if two KL terms in Eq.(10) and Eq.(11) are optimizedto zero, the point of equilibrium is arrived and Eq.(6) holds bysubstituting Eq.(12). Therefore, this derivation explains whyour objective function can achieve a reliable disentanglement.

Furthermore, this objective function can be rewritten as:

L(x,x′, c, c′; θ,φ) (13)

=Eqφ(s|x′,c′)[log pθ(x|s, c)]−KL[qφ(s|x′, c′)‖pθ(s)],

where pθ(s) denotes the prior distribution of s. This form ismore explicit to derive our training algorithm, in which thefirst term means a reconstruction loss and the second termmeans a regularization. Note that our proposed intercross


923

Algorithm 1: Training AlgorithmInput: Chinese characters dataset X = {xi,j}Output: The model parameters : θ , φ , ηRandomly initialize θ , φ , η.η ← Pre-train a Content Recognition Network Crepeat

x← Randomly select a mini-batch from Xx′ ← Randomly select a mini-batch correspondinglyshares the same font with xc← Get the content code of x using C and Kc′ ← Get the content code of x′ using C and Kg←∇θ,φL(x,x′, c, c′; θ,φ) (Gradient of (13))θ,φ← Update parameters using gradients g

until Converge

pair-wise optimization method is a general technique for two-factor disentanglement problems [Tenenbaum and Freeman,1997], which is not only restricted in the task of stylized Chi-nese character generation. The details of the intercross pair-wise optimization is shown in Alg. 1.

2.4 One-shot/Few-shot Character GenerationWhen testing with one-shot setting, we observe a new stylewith only one character, then we model other characters usingthe following way:p(xi,1:n|c1:n,xi,j) =∫

pθ(xi,1:n|c1:n, si)qφ(si|xi,j ,T [fη(xi,j)])dsi,(14)

in which xi,j is the observed character as the style-provider,cj is the content code of xi,j which can be obtained fromContent Recognition Network C, si is the new style embed-ding inferred by our Style Inference Network S from the char-acter xi,j , c1:n means the content codes of characters we wantto generate and xi,1:n are the target characters with the newstyle generated by Character Generation Network G.

If we can observe a few more characters, which means test-ing with few-shot setting, we use the averaged embeddingvectors:

si =1

n

n∑j=1

s(j)i , where s

(j)i ∼ qφ(si|xi,j , cj), (15)

where j = 1...n with n being the number of observed char-acters; xi,j are corresponding characters we observed andcj are the content code of xi,j . s

(j)i are independent ran-

dom variables correspondingly obey the posterior distributionqφ(si|xi,j, cj). With Eq.(6), we can get V ar[si] = 1

nV ar[si],which means that we can obtain more stable estimation of thestyle information by averaging over multiple characters.

3 ExperimentsIn this section, we first introduce the dataset we build.Then we present the one-shot/few-shot generation results andpresent some experimental results to verify that our modelhas a strong ability of style generalization and disentangle-ment. Finally, we show the necessity and advantage of usingthe proposed encoding method with human knowledge.

(a) An excerpt from an classical essay Lou Shi Ming.

(b) An excerpt from the Analects of Confucius.

Figure 5: The generation results in the one-shot and few-shot setting.The characters in the first and third rows are generated by our SA-VAE framework compared with the ground truth in the second andfourth rows. The characters in the dotted frame are used to specifieda a new targeted style for the model.

3.1 Data PreparationAs our main purpose is to generalize over new styles basedon learning sufficient support styles, we need to get enoughstyles for our model to learn. As no existing datasets sat-isfy our goal, we build a new one. The dataset consists ofChinese characters in 200 styles collected from the Internet.We randomly select 80% of these styles (i.e., 160 styles) tobe used in the style-bank for our training and the rest 20%(i.e., 40 styles) to be used for test. We choose the most fre-quently used 3000 Chinese characters from each style in thestyle-bank to build our training set.

3.2 One-shot/Few-shot Generation ResultsFig. 5 shows the one-shot and few-shot generation results.Both results are presented with a printing style and a hand-written style, which demonstrates that our model has a ca-pacity of dealing with diverse styles.

Meanwhile, we make a comparison with some base-line models of font style transfer (”Rewrite” [Tian, 2016],”zi2zi” [Tian, 2017]) as shown in Table 1. As the formersetting, our SA-VAE generates the new-stylized characters inthe table by observing only one character. As for other meth-ods, if the target style is in the training set (1000 samples usedto train here), the results are close to ours. However, if the tar-get style is new (unseen in the training stage) and the amountof style providers is small (10 characters with the new styleused to finetune here), they hardly capture anything about thenew style. Therefore, our SA-VAE presents a clear advantagein the one-shot/few-shot inference.

3.3 Style GeneralizationThe style generalization is the key point in our SA-VAEframework, which proves our model has a capacity of gener-alizing to a new style. To show that our model can achievethis goal, we do the interpolation between two disparatestyles. Besides, we also find the nearest neighbors of our one-shot generalization results to show our method is not find-


924

Setting Method Results

Seen style(1000 characters)

Rewrite

zi2zi

SA-VAEGround Truth

New style(Extra 10 characters)

Rewritezi2zi

SA-VAEGround Truth

Table 1: Comparison of generation results obtained by our SA-VAEand other font style transfer methods in different settings.

(a) Interpolation (b) NN results

Figure 6: (a) is the interpolation of two styles. (b) is the nearestneighbor of one-shot generation. The first characters in the dottedframe are style sources and the second ones are generated. The thirdand the forth are the two nearest neighbors in the training set.

ing the most similar style in the training set. The results areshown in Fig. 6.

3.4 Evaluation of DisentanglementSuccessfully disentangling the style information from thecontent information is another necessity to achieve one-shotgeneration with given styles. First, we show a qualitative ex-perimental result as shown in Fig 7. We can see comparedwith our SA-VAE, the style embedding extracted without in-tercross pair-wise optimization method contains too muchcontent information and leads to blurry results.

Meanwhile, We also conduct a quantitative experiment.Based on the intuition that a pure style embedding shouldcontain enough information to represent its style while lit-

Figure 7: We show the one-shot generation results produced bywhether using proposed intercross pair-wise optimization or not.

Methods Style Acc Content Acc

Vanilla VAE optimization 43.44% 28.55 %Our Pair-wise optimization 47.42% 1.28 %

Table 2: Classification accuracy using the style embedding

tle information to represent its content, we choose a simpleclassifier to do some auxiliary classification tasks using thestyle embeddings obtained by our intercross pair-wise opti-mization method and vanilla VAE’s method. To simplify thetraining, we only use 100 unique characters per style as thetraining data to conduct style classification and use 100 stylesper character to conduct character content classification, thentest the prediction accuracy over the rest of samples. The re-sults are shown in Table 2.

Both the qualitative and quantitative experiment resultsprovide a strong proof that our intercross pair-wise optimiza-tion method has a powerful ability of disentanglement.

3.5 Structure Knowledge of Chinese CharactersOur encoding method with domain knowledge instead of theone-hot embedding provides a solution for the large dictio-nary issue of Chinese characters. To show the effectivenessof knowledge guidance, we compare the converge speed ofthree encoding methods – our content code, the one-hot em-bedding and the binary embedding. The results are shown inFig. 8.

Depending on the cost of expanding model capacity, theone-hot embedding may provide a comparable convergespeed with our knowledgeable content code. However, thisembedding method will make the model parameters explodedin a larger Chinese characters dictionary. The binary embed-ding consumes shorter length (dlog2Ne bits) , but it hardlyconverges to a meaningful result.

Figure 8: The training curves of three encoding methods using dif-ferent numbers of unique characters, where the y-axis shows thenegative lower bound and the x-axis shows the number of epochsduring training. Notice that when we add the character numbers to500, the model parameters using one-hot embedding will explodeand we cannot show the corresponding curve.


925

4 ConclusionsWe have presented a novel Style-Aware VAE framework,which achieves the stylized Chinese character generation byreading only one or a few characters. Our model disentanglesthe style information and the content information with the in-tercross pair-wise optimization method and shows a powerfulone-shot or few-shot generalization ability with new styles.Experiments demonstrate the impressive generation resultson characters of both printing and handwritten styles.

AcknowledgementsThe work is supported by the National NSF of China (Nos.61571261, 61620106010, 61621136008, and 61332007),Beijing Natural Science Foundation (No. L172037), Ts-inghua Tiangong Institute for Intelligent Computing and theNVIDIA NVAIL Program, and partially funded by MicrosoftResearch Asia and Tsinghua-Intel Joint Research Institute.Our project is built on ZhuSuan [Shi et al., 2017], which is adeep probabilistic programming library based on Tensorflow.

A Generation Results of Symbolic AlphabetsWe also apply our SA-VAE framework for other simple sym-bolic characters including digits, English letters and Japanesekanas. The experimental results are shown in Fig. 9. Similarto the setting of Fig. 5, the characters in the dotted frame areas the style providers and the others in the same row are theone-shot generation results with the specified style. Contrastto the Chinese character generation, we use one-hot embed-ding to label these symbolic characters because there is nocomplex structural information in them. The experimental re-sults imply that our SA-VAE framework is not only restrictedin Chinese character generation, but also a general techniquefor the disentanglement and one-shot generalization.

References[Donahue et al., 2016] Jeff Donahue, Philipp Krahenbuhl,

and Trevor Darrell. Adversarial feature learning. arXivpreprint arXiv:1605.09782, 2016.

[Dumoulin et al., 2016] Vincent Dumoulin, Ishmael Belg-hazi, Ben Poole, Alex Lamb, Martin Arjovsky, OlivierMastropietro, and Aaron Courville. Adversarially learnedinference. arXiv preprint arXiv:1606.00704, 2016.

[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen-erative adversarial nets. In Advances in neural informationprocessing systems, pages 2672–2680, 2014.

[Graves, 2013] Alex Graves. Generating sequences with re-current neural networks. arXiv preprint arXiv:1308.0850,2013.

[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter andJurgen Schmidhuber. Long short-term memory. Neuralcomputation, 9(8):1735–1780, 1997.

[Isola et al., 2016] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou,and Alexei A. Efros. Image-to-image translation with

(a) One-shot digit generation

(b) One-shot English character generation

(c) One-shot English character generation

Figure 9: The characters in the dotted frames are as the styleproviders and the others are generated characters in each row. Asthe size of their dictionary is not large, we use one-shot embeddingto label each character. The style is consistent in each row.

conditional adversarial networks. CoRR, abs/1611.07004,2016.

[Kingma and Welling, 2014] Diederik P Kingma and MaxWelling. Auto-encoding variational bayes. stat, 1050:10,2014.

[Lake et al., 2015] Brenden M Lake, Ruslan Salakhutdinov,and Joshua B Tenenbaum. Human-level concept learn-ing through probabilistic program induction. Science,350(6266):1332–1338, 2015.

[Li et al., 2017] Chongxuan Li, Jun Zhu, and Bo Zhang.Max-margin deep generative models for (semi-) super-vised learning. IEEE Transactions on Pattern Analysis andMachine Intelligence, 2017.


926

[Lian et al., 2016] Zhouhui Lian, Bo Zhao, and JianguoXiao. Automatic generation of large-scale handwritingfonts via style learning. In SIGGRAPH ASIA 2016 Techni-cal Briefs, page 12. ACM, 2016.

[Pal and Chaudhuri, 2004] U Pal and BB Chaudhuri. Indianscript character recognition: a survey. pattern Recognition,37(9):1887–1899, 2004.

[Radford et al., 2015] Alec Radford, Luke Metz, andSoumith Chintala. Unsupervised representation learningwith deep convolutional generative adversarial networks.arXiv preprint arXiv:1511.06434, 2015.

[Shi et al., 2017] Jiaxin Shi, Jianfei Chen, Jun Zhu,Shengyang Sun, Yucen Luo, Yihong Gu, and Yuhao Zhou.Zhusuan: A library for bayesian deep learning. arXivpreprint arXiv:1709.05870, 2017.

[Tenenbaum and Freeman, 1997] Joshua B Tenenbaum andWilliam T Freeman. Separating style and content. InAdvances in neural information processing systems, pages662–668, 1997.

[Tian, 2016] Yuchen Tian. Rewrite: Neural style trans-fer for chinese fonts, 2016. https://github.com/kaonashi-tyc/Rewrite.

[Tian, 2017] Yuchen Tian. zi2zi: Master chinese calligraphywith conditional adversarial networks, 2017. https://github.com/kaonashi-tyc/zi2zi.

[Wainwright et al., 2008] Martin J Wainwright, Michael IJordan, et al. Graphical models, exponential families, andvariational inference. Foundations and Trends R© in Ma-chine Learning, 1(1–2):1–305, 2008.

[Xiao et al., 2017] Xuefeng Xiao, Lianwen Jin, YafengYang, Weixin Yang, Jun Sun, and Tianhai Chang. Buildingfast and compact convolutional neural networks for offlinehandwritten chinese character recognition. arXiv preprintarXiv:1702.07975, 2017.

[Xu et al., 2009] Songhua Xu, Tao Jin, Hao Jiang, and Fran-cis CM Lau. Automatic generation of personal chinesehandwriting by capturing the characteristics of personalhandwriting. In IAAI, 2009.

[Yu et al., 2015] Fisher Yu, Ari Seff, Yinda Zhang, Shu-ran Song, Thomas Funkhouser, and Jianxiong Xiao.Lsun: Construction of a large-scale image dataset usingdeep learning with humans in the loop. arXiv preprintarXiv:1506.03365, 2015.

[Zeiler et al., 2010] Matthew D Zeiler, Dilip Krishnan, Gra-ham W Taylor, and Rob Fergus. Deconvolutional net-works. In Computer Vision and Pattern Recognition(CVPR), 2010 IEEE Conference on, pages 2528–2535.IEEE, 2010.

[Zhang et al., 2017] Xu-Yao Zhang, Fei Yin, Yan-MingZhang, Cheng-Lin Liu, and Yoshua Bengio. Drawing andrecognizing chinese characters with recurrent neural net-work. IEEE Transactions on Pattern Analysis and Ma-chine Intelligence, 2017.

[Zhong et al., 2016] Zhao Zhong, Xu-Yao Zhang, Fei Yin,and Cheng-Lin Liu. Handwritten chinese character recog-nition with spatial transformer and deep residual networks.In Pattern Recognition (ICPR), 2016 23rd InternationalConference on, pages 3440–3445. IEEE, 2016.

[Ziwei Liu and Tang, 2015] Xiaogang Wang Ziwei Liu,Ping Luo and Xiaoou Tang. Deep learning face attributesin the wild. In Proceedings of International Conference onComputer Vision (ICCV), 2015.

[Zong and Zhu, 2014] Alfred Zong and Yuke Zhu. Stroke-bank: Automating personalized chinese handwriting gen-eration. In AAAI, pages 3024–3030, 2014.


927

Documents

Learning to Write Stylized Chinese Characters by Reading a ... · sults even if a small bias is introduced in Chinese character generation. To address this issue, we integrate the