10.1261/rna.030916.111Access the most recent version at doi: 2012 18: 1279-1288 originally published online April 24, 2012RNA

Lana Xia Garmire and Shankar Subramaniam dataEvaluation of normalization methods in mammalian microRNA-Seq

MaterialSupplemental http://rnajournal.cshlp.org/content/suppl/2012/03/21/rna.030916.111.DC1.html

References http://rnajournal.cshlp.org/content/18/6/1279.full.html#ref-list-1

This article cites 26 articles, 12 of which can be accessed free at:

serviceEmail alerting

click heretop right corner of the article orReceive free email alerts when new articles cite this article - sign up in the box at the

http://rnajournal.cshlp.org/subscriptions go to: RNATo subscribe to

© 2012 RNA Society

Cold Spring Harbor Laboratory Press on January 11, 2013 - Published by rnajournal.cshlp.orgDownloaded from

http://rnajournal.cshlp.org/lookup/doi/10.1261/rna.030916.111http://rnajournal.cshlp.org/content/suppl/2012/03/21/rna.030916.111.DC1.htmlhttp://rnajournal.cshlp.org/content/18/6/1279.full.html#ref-list-1http://rnajournal.cshlp.org/cgi/alerts/ctalert?alertType=citedby&addAlert=cited_by&saveAlert=no&cited_by_criteria_resid=rna;18/6/1279&return_type=article&return_url=http://rnajournal.cshlp.org/content/18/6/1279.full.pdfhttp://rnajournal.cshlp.org/cgi/adclick/?ad=37257&adclick=true&url=http%3A%2F%2Fwww.invitrogen.com%2Fsite%2Fus%2Fen%2Fhome%2Fbrands%2Fambion%2Fambion-publication-rewards-program.html%3FCID%3DBN-CO110930-RNA-728x90http://rnajournal.cshlp.org/subscriptionshttp://rnajournal.cshlp.org/http://www.cshlpress.com

METHOD

Evaluation of normalization methods in mammalian

microRNA-Seq data

LANA XIA GARMIRE1 and SHANKAR SUBRAMANIAM1

Department of Bioengineering, Jacobs School of Engineering, University of California at San Diego, La Jolla, California 92093-0412, USA

ABSTRACT

Simple total tag count normalization is inadequate for microRNA sequencing data generated from the next generationsequencing technology. However, so far systematic evaluation of normalization methods on microRNA sequencing data islacking. We comprehensively evaluate seven commonly used normalization methods including global normalization, Lowessnormalization, Trimmed Mean Method (TMM), quantile normalization, scaling normalization, variance stabilization, andinvariant method. We assess these methods on two individual experimental data sets with the empirical statistical metrics ofmean square error (MSE) and Kolmogorov-Smirnov (K-S) statistic. Additionally, we evaluate the methods with results fromquantitative PCR validation. Our results consistently show that Lowess normalization and quantile normalization perform thebest, whereas TMM, a method applied to the RNA-Sequencing normalization, performs the worst. The poor performance ofTMM normalization is further evidenced by abnormal results from the test of differential expression (DE) of microRNA-Seq data.Comparing with the models used for DE, the choice of normalization method is the primary factor that affects the results of DE.In summary, Lowess normalization and quantile normalization are recommended for normalizing microRNA-Seq data, whereasthe TMM method should be used with caution.

Keywords: microRNA-Seq; next generation sequencing; statistical normalization; high-throughput data analysis

INTRODUCTION

The next generation sequencing (NGS) technology hasbeen widely used recently to study a variety of biologicalproblems, such as quantifying the mRNA transcripts expres-sion (RNA-Seq), chromosome DNA–DNA bound proteininteractions (CHIP-Seq), and small microRNA expression.It has been shown that NGS provides higher reproduc-ibility, wider range, and better quality of data, comparedwith the microarray method. Moreover, the open-platformof NGS enables discoveries of new mRNA transcripts andnew microRNA strands.

Mammalian microRNAs are small RNAs of z22 nt inlength (Bartel 2004). They are regarded to destabilize targetmRNAs or inhibit translation machinery by binding tospecific regions of mRNA transcripts such as 39 UTRs (Baeket al. 2008). Though new microRNAs are being discoveredcontinuously, the total number of known microRNAs is

much smaller than mRNAs. For example, so far there are

to mRNA-Seq, there has been no unbiased evaluation ofnormalization methods on microRNA-Seq data. Moreover,it remains questionable whether the normalization methodused for mRNA-Seq data can be adequately adapted toprocess microRNA-Seq data, given the fact that the totalnumber of mRNA transcripts is magnitudes larger than thetotal number of microRNA strands.

To answer these questions, we systematically evaluated sevencommonly used normalization methods for high-through-put data, namely global normalization, Lowess normalization,Trimmed Mean Method (TMM), quantile normalization,scaling normalization, variance stabilization (VSN), and in-variant method (INV). These methods make different assump-tions about the true biological difference and the random noisein order to be able to estimate the systematic variation. Theycan be classified into two categories, according to the applica-tion of linear scaling or not. (1) The first category includesscaling, global, Lowess, and TMM. Scaling normalizationassumes the ranges of data are the same and that the noiseand the stochastic variations of microRNAs are proportional tothe signal intensity (Smyth et al. 2003). Global normalization isanother linear scaling approach that scales all the data of theexperimental condition against the control condition by afactor of the difference in the means of two data (Smyth et al.2003). Lowess normalization does not use a global scalingfactor; instead, it calculates local scaling factors within a certainwindow size (Smyth et al. 2003). TMM, a more recent normal-ization method applied to mRNA-Seq data, also assumes themajority of the mRNAs in NGS output are similar, except thedata points that lie within the extreme M-value and A-valueranges. It derives a simple scaling factor after trimming thedata points located in extreme M-value and A-value ranges(Robinson et al. 2010). (2) The second category includesquantile, VSN, and INV. Quantile normalization is nonscalingand assumes that the overall distribution of signal intensitydoes not change (Bolstad et al. 2003). VSN assumes that mostmicroRNAs do not change and transform the data such thatthe transformed variance is constant among different expres-sion levels. Therefore, it allows better precision in low ex-pression regions, which generally suffer from greater variance

(Huber et al. 2002). INV assumes that a subpopulation ofexpressed microRNAs does not change, and it learns a set of‘‘invariants’’ through algorithms, instead of assigning ‘‘house-keeping genes’’ subjectively (Perkins et al. 2007; Pradervandet al. 2009).

We carried out the pairwise comparisons on two publiclyavailable microRNA-Seq profiling data sets. One is the com-parison between activated versus inactivated natural killer cells(abbreviated as F-data) (Fehniger et al. 2010), and the other isthe comparison between pro-B cells and pre-B cells (abbreviatedas K-data) (Kuchen et al. 2010). These data sets are chosenbecause of the availability of quantitative PCR results for theassessment of sensitivity and specificity. We used a combi-nation of criteria to evaluate the performance of eachnormalization method, including metrics such as meansquare error (MSE) and Kolmogorov-Smirnov (K-S) statis-tic, validation from quantitative PCR data, and the abnor-mality diagnosis based on the results of DE test.

RESULTS

Necessity of normalization beyond simple tagcount normalization

MA-plot is a plot of log-intensity ratios (M-values) versuslog-intensity averages (A-values). It is commonly used toillustrate the dependency on intensities in the high-through-put data. Currently, most studies use a simple scaling factor,which is equal to the ratio of summed tag counts betweentwo conditions, to normalize the experimental lane to thecontrol lane. Using MA-plots, we evaluated such normaliza-tion on the two public data sets (see Materials and Methods),denoted as F-data (Fehniger et al. 2010) and K-data (Kuchenet al. 2010). Figure 1 shows the distribution of M-valuesbetween two comparison samples after the standard normal-ization procedure of accounting for the total tag countsamong known, detectable microRNAs. The centers of thedistributions of M-values are significantly deviated from zero,with a median of 0.57 for F-data, and �0.47 for K-data,indicating that additional normalization procedure is needed.

FIGURE 1. MA-plots after simple tag count normalization. MA-plots show the distribution of microRNAs in paired samples of comparison afterthe simple total tag count normalization among the known, detectable microRNAs. The horizontal lines denote the mean of the M-values, whichare significantly deviated from zero in both data sets, with a median of 0.57 for F-data (left) and �0.47 for K-data (right).

Garmire and Subramaniam

1280 RNA, Vol. 18, No. 6

Cold Spring Harbor Laboratory Press on January 11, 2013 - Published by rnajournal.cshlp.orgDownloaded from

http://rnajournal.cshlp.org/http://www.cshlpress.com

In the following sections, we systematically evaluate sevennormalization methods, namely global normalization, Lowessnormalization, TMM, quantile normalization, scaling normal-ization, VSN, and INV. These methods were previously appliedto high-throughput data, such as microarray and mRNA-Seqdata.

Effect of normalization on data distribution

We first examined the effects of different normalization onthe distribution of data using the MA-plots that are com-monly done in microarray analysis, exemplified by F-data. Asshown in Figure 2A, the greatest change among MA dataoccurs in global normalization, Lowess normalization, scaling,quantile, and VSN. Global normalization shifts the center ofM-values of data to 0, and thus the observed changes on alllevels of A-values are expected. Similarly, the scaling methodenforces equal median absolute deviations in both M- andA-values. Lowess, quantile, and VSN present bigger changesin normalized log2 tag counts in the lower A-value range (Fig.2A,B). Lowess takes account of local weighting to adjust thedata points, and quantile normalization assumes the quantiledistribution of data in two conditions is the same. They bothredistribute normalized M-values around M-values prior tonormalization, especially at the lower level of A-values. Onthe other hand, VSN aims to have uniform variance acrossdifferent expression levels, and shifts M-values at lower levelsof A-values toward a higher A-value region (Fig. 2B). Althoughless obvious, a similar pattern was observed in the K-data(Supplemental Fig. 1), except that many normalization meth-ods pull the M-values toward more positive directions due tothe fact that the control condition has larger mean M-value(Fig. 1B).

Evaluation of normalization basedon empirical statistics

MSE is a comparison criterion that is widely used tomeasure statistical models, such as the alternative normal-ization methods in this study and others (Xiong et al.2008). MSE can be decomposed into the summation ofvariance and the square of bias. Small MSE indicates betternormalization overall, within which variance is a metric forprecision and bias is a measurement of accuracy. In thisreport, we use MSE to calculate the difference betweenM-values and the center of M-values. We present the resultsusing this metric to compare all normalization methods onboth F-data and K-data (Fig. 3). As mentioned before, theaverage of M-values in the data without normalization isdeviated from 0, resulting in bias in MSE. Expectedly,global normalization eliminates the bias of M, resulting insmaller MSE. Using the MSE of global normalization as thereference point, Lowess, quantile, and VSN normalizationsconsistently produce smaller MSE. These smaller MSEs can

be decomposed into smaller variances and smaller biasescomparing with those of unnormalized data. On the otherhand, TMM, a method that trims data strongly affected bytreatment conditions, and INV are worse than globalnormalization as evidenced by even greater MSEs than nonormalization. TMM also produces the largest biasesamong all, most likely because it throws away valuabledata information through trimming M-values by 30%and A-values by 5% by default (Robinson et al. 2010).Although this approach could be beneficial while dealingwith thousands of mRNAs in the mRNA-Seq normaliza-tion, it can be harmful among the small body of hundredsof microRNAs.

The K-S test is a goodness-of-fit test to measure similar-ities between two distributions by counting the largest devi-ation between the two accumulative distributions. Based onthe rationale that an effective normalization method wouldgenerate two similar distributions and thus a small K-Sstatistic, we also measured the K-S statistics that were calcu-lated from the M-values before and after different normali-zations on both data sets and show the results in Figure 3.Similar results to the MSEs are observed. Global normaliza-tion does decrease the K-S statistic compared with no normal-ization. Lowess and quantile normalization produce smalleror equal K-S statistics relative to the global normalization.Both VSN and scaling methods give similar K-S values to theglobal normalization. On the other hand, TMM and INVsconsistently generate larger K-S statistics relative to the globalnormalization. Similar to the largest MSE from TMM, K-Sstatistics from TMM is also the largest.

Evaluation of normalization with quantitativePCR results

Quantitative PCR (QPCR) is an effective alternativemethod to assess the expression profile of microRNAs.Over 100 QPCR experiments were conducted in parallelwith the microRNA-Seq results from F-data, making theman ideal set to evaluate gains in sensitivity and specificityafter normalization. The ‘‘true positives’’ from QPCRresults were assigned to microRNAs with at least twofoldchanges in the activated versus inactivated states. We foundthere is more noise in the miRs that are of low tag counts inthe microRNA-Seq data, when comparing microRNA-SeqDE data (twofold cutoff) with the assumed ‘‘truth’’ ofQPCR DE data (twofold cutoff). Both false positive andfalse negative microRNAs have lower tag counts (data notshown). To compare the sensitivity and specificity of thedifferent methods, we obtained standard receiver operatingcharacteristic (ROC) plots where the area under the curvescan be used for evaluations (Fig. 4A). Some normalizationmethods improved the sensitivity and specificity comparedwith no normalization, whereas others did not. Consistentwith the results of MSE and K-S statistics, quantile andLowess normalization perform the best, followed by VSN,

microRNA-Seq data normalization evaluation

www.rnajournal.org 1281

Cold Spring Harbor Laboratory Press on January 11, 2013 - Published by rnajournal.cshlp.orgDownloaded from

http://rnajournal.cshlp.org/http://www.cshlpress.com

global, and scaling normalization, all of which are betterthan no normalization. INV appears to have a similar ROCcurve to that of no normalization. Clearly, TMM has the

worst under-curve coverage even when compared with nonormalization. This result is consistent with the conclu-sions based on MSE and K-S statistics.

FIGURE 2. Effect of normalization on F-data distribution. (A, top) MA-plots before and after applying different normalization schemes to F-datadescribed in the text (Fehniger et al. 2010). Except the raw data plot, in all the other MA-plots, black circles are data before normalization, and redcircles are after normalization. (B, bottom) Box plots list the transformed log2 counts in the treatment condition after normalization (except rawdata) separated by quartiles: Q1, Q2, Q3, and Q4. Q1 is the lowest quartile and Q4 is the highest quartile. The color codes for box plots from leftto right are as follows: black (raw data), blue (global normalization), purple (Lowess normalization), brown (TMM normalization), orange(scaling normalization), gray (quantile normalization), green (VSN normalization), and red (INV normalization).

Garmire and Subramaniam

1282 RNA, Vol. 18, No. 6

Cold Spring Harbor Laboratory Press on January 11, 2013 - Published by rnajournal.cshlp.orgDownloaded from

http://rnajournal.cshlp.org/http://www.cshlpress.com

The QPCR results are available for 12 microRNAs inthe K-data. Though the ROC plot is not possible, correla-tion study of microRNA-Seq data versus QPCR of the 12microRNAs is achievable, as shown in Figure 4B. As ex-pected, global normalization performs slightly better thanno normalization. It has better correlation coefficient (CC)and better R-square (R^2) value from a linear regressionbetween log2 transformed microRNA-Seq data and log2transformed QPCR data. Lowess normalization has the bestcorrelation among all, with a CC = 0.677 and R^2 = 0.459from the linear regression, closely followed by quantilenormalization which has a CC of 0.652 and R^2 of 0.426.VSN produces slightly better CC and R^2, compared withglobal normalization. TMM and INVs do not show muchdifference in correlation from no normalization, whereasscaling had the worst correlations among all methods.Overall, the correlation results of K-data are also consistentwith the previous conclusions based on MSE and K-Sstatistics, in the sense that Lowess and quantile normaliza-tion have the highest correspondence to the QPCR results.

Test of differential expression

It was shown in mRNA-Seq studies that normalization isa primary factor affecting the test of DE (Taslim et al. 2009;Anders and Huber 2010; Bullard et al. 2010; Robinson andOshlack 2010). We therefore also used the results in DE to

reflect the performance of the normalization methods.Based on previous studies (Taslim et al. 2009; Bullardet al. 2010; Robinson and Oshlack 2010), we used threedifferent tests of DE, namely x2 test, Poisson distribution,and binomial distribution. We define significantly changedmicroRNAs as those that have P-values

confirms that the normalization method, rather than themodel of DE, is a primary factor affecting the results of DE.

DISCUSSION

Analyzing microRNA profiles with NGS is becominga new trend of microRNA-related discoveries in many

different organisms. The high volume and digitized in-formation make microRNA-Seq highly competitive to theprobe-based microRNA-array method. It is generally be-lieved that statistical normalization is beneficial comparedwith no normalization. So far, most normalizations for RNAsgenerated by NGS use simple total tag count normalizationto remove differences in sequencing depths between li-

FIGURE 4. Evaluation of normalization methods with QPCR results. (A) ROC plot of sensitivity and specificity of the various normalizationmethods, based on F-data. The color codes of ROC curves for the normalization methods are the same as those in Figure 2. A ‘‘true difference’’ ofvalue 1 is assigned to the microRNAs whose QPCR expression ratios are at least twofold different between activated and inactivated state, but0 otherwise. A ‘‘predicted difference’’ is the absolute value of the normalized M-value of microRNA-Seq tag counts. Note: the ROC curves of thescaling and global normalization methods are identical and the global normalization (blue) is superimposed on the scaling method (orange). (B)Linear regression of microRNA-Seq log2 fold change results versus QPCR log2 fold change results based on K-data, over various normalizationmethods. The correlation coefficient (CC) and R-square (R^2) are two metrics to measure the correlation between the two types of data. The linesare the best linear regression fits to the data. For comparison, all x- and y-axes are uniformized to the same scales.

Garmire and Subramaniam

1284 RNA, Vol. 18, No. 6

Cold Spring Harbor Laboratory Press on January 11, 2013 - Published by rnajournal.cshlp.orgDownloaded from

http://rnajournal.cshlp.org/http://www.cshlpress.com

FIGURE 5. Test of differential expression based on different normalizations. (A) Bar graphs show the results of DE based on x2/Fisher’s test (chi-square), Poisson model (poisson), and binomial model (binomial) for different normalization schemes. The color codes for the tests of DE are asfollows: black bar (x2), gray bar (Poisson), and empty bar (binominal). The normalization methods upon which the DE tests are performed arelisted above the bars. ‘‘Up-regulated’’ microRNAs are plotted above y = 0 and ‘‘down-regulated’’ microRNAs below y = 0. The top plot is fromF-data, and the bottom plot from K-data. (B) Heat maps show the hierarchical clustering results of all significant microRNAs that are defined in A.The values are transformed from P-values of specific microRNAs. Blue color represents down-regulation whereas brown color represents up-regulation. pois, Poisson model; bi, binomial model; and chi, x2/Fisher’s test.

Cold Spring Harbor Laboratory Press on January 11, 2013 - Published by rnajournal.cshlp.orgDownloaded from

http://rnajournal.cshlp.org/http://www.cshlpress.com

braries. There is an urgent need to propose more sophisti-cated normalization methods. However, currently mostmicroRNA-Seq experiments do not use biological repli-cates, thus impeding the estimation of true biological vari-ations. This problem will likely disappear in a year or twodue to the competition of lowering the cost of deep se-quencing. Nevertheless, the library normalization issue stillneeds to be addressed before any test of DE. It seems rea-sonable to assume that the majority body of the microRNAsare expressed similarly, on the closely related biological sam-ples, such as the activated versus inactivated natural killercells and pro-B cells versus pre-B cells that are exemplifiedin this study. Normalization could remove the technicalartifacts arising from unintended noise; however, it main-tains the true differences between the samples.

These above normalization methods and their assumptionswere evaluated by multiple independent data sets on severallevels. Although each level of evaluation was based on somespecific assumptions, the consensus of multiple levels ofevaluations helps to draw unbiased conclusions. First, we usedthe generic, empirical statistics MSE and K-S statistics tomeasure the fitness of normalization methods. The MSEmetric is based on the rationale that better normalizationmethods should create small variations and trivial bias withinthe data. And K-S statistics is based on the assumption thatgood normalization methods could maximize the distributionsimilarities between two data sets. Due to these reasons, theMSE and K-S statistics may bias toward the normalizationmethods whose assumptions favor to minimize them (such asquantile normalization), as observed and discussed by others(Xiong et al. 2008). These empirical metrics need to be usedtogether with other evaluation criteria to draw unbiasedconclusions. We did so using two other approaches: QPCRvalidation and results from test of DE. QPCR is an orthogonal,quantitative method to RNA-Seq. We assumed that a betternormalization method for the microRNA-Seq data shouldyield better correlations between the QPCR and microRNA-Seq, and evaluated the methods with ROC plots and linearregression. We found that Lowess and quantile normalizationsare consistently superior to other methods, whereas TMMnormalization performs the poorest. Lastly, we also comparedresults of DE, as ways to reveal the consequences of normal-ization and diagnose abnormalities in the normalized data.These evaluations gave consistent results over all. Lowess andquantile normalizations are the best among tested methods,whereas TMM behaved abnormally and extremely. Similar toour results, Bullard and colleagues also found that, in mRNA-Seq experiments, the quantile-based method yields betterconcordance with qRT-PCR methods than the linear totalscaling method. Our study supports the speculation on theadvantage of quantile normalization over scaling in smallRNA-Seq (Bullard et al. 2010), while discouraging the usage ofTMM application to microRNA-Seq as recently proposed(McCormick et al. 2011). More broadly speaking, microRNA-Seq data are a portion of the bigger data set that is generated

from the small RNA (

tween activated and inactivated state. And the ‘‘predicted difference’’is actually the absolute value of the normalized M-value (or, anabsolute value of the Z-score calculated from the M-value) ofmicroRNA-Seq tag counts.

Test of differential expression

Test of DE was modeled by x2/Fisher’s exact test, binomial test,and Poisson tests, similar to others (Taslim et al. 2009; Bullardet al. 2010; Robinson and Oshlack 2010). Briefly, in the x2/Fisher’sexact test, each microRNA is associated with a 2 3 2 contingencytable, with the tag counts of specific microRNAs in control versustreatment condition, as well as with the summed tag counts of allother microRNAs in the population. If all tag counts are abovefive, x2 test was applied. Otherwise, Fisher’s exact test was appliedfor accuracy. A microRNA is called ‘‘up-regulated’’ (or ‘‘down-regulated’’) when the observed tag counts are greater (or less) thanthe expected tag counts, with a Bonferroni-corrected P-value

Smyth GK. 2005. Limma: Linear models for microarray data. InBioinformatics and computational biology solutions using R and bio-conductor (ed. R Gentleman et al.), pp. 397–420. Springer, New York.

Smyth GK, Yang YH, Speed TP. 2003. Statistical issues in microarraydata analysis. Methods Mol Biol 224: 111–136.

Srivastava S, Chen L. 2010. A two-parameter generalized Poissonmodel to improve the analysis of RNA-seq data. Nucleic Acids Res38: e170. doi: 10.1093/nar/gkq670.

Taslim C, Wu J, Yan P, Singer G, Parvin J, Huang T, Lin S, HuangK. 2009. Comparative study on ChIP-seq data: Normalizationand binding pattern characterization. Bioinformatics 25: 2334–2340.

Xiong H, Zhang D, Martyniuk CJ, Trudeau VL, Xia X. 2008. UsingGeneralized Procrustes Analysis (GPA) for normalization of cDNAmicroarray data. BMC Bioinformatics 9: 25. doi: 10.1186/1471-2105-9-25.

Garmire and Subramaniam

1288 RNA, Vol. 18, No. 6

Cold Spring Harbor Laboratory Press on January 11, 2013 - Published by rnajournal.cshlp.orgDownloaded from

http://rnajournal.cshlp.org/http://www.cshlpress.com