Differential ExpressionsClassical Methods

Lecture Topic 7

What’s your question?

• What are the genes that are different for the healthy versus diseased cells?– Gene discovery, differential expression

• Is a specified group of genes all up-regulated in a specified condition?– Gene set differential expression

• Can I use the expression profile of cancer patients to predict chemotherapy outcome?– Class prediction, classification

• Are there tumour sub-types not previously identified? Do my genes group into previously undiscovered pathways?– Class discovery, clustering

• This lecture covers first question - differential expression.

Some terms

• The following terms are used exchangeably:

• Differentially versus not differentially expressed• Active versus Inactive genes• Affected versus not affected genes

• We will first deal with the simplest case, comparing 2 conditions (say treatment and control).

• We will also mainly be interested in hypothesis testing (HT) aspect of Inference

Inference

• Lets first differentiate between the different schools of thought:

• Classical Inference• Bayesian Inference• Empirical Bayes’ methods

The MAIN Difference

• The Classical School believes that PARAMETERS are unknown but FIXED quantities and the data are random

• The Bayesians believe after observing, data can be considered fixed, whereas our knowledge about the parameter is truly random.

• This leads to two vastly different APPROACHES when it comes to hypothesis testing, though both approaches are inherently interested in the parameter.

Consider the following situation

qg= population mean difference in gene expression from treatment to control for gene g.

• Both approaches are interested in this. Particularly, both approaches are interested in seeing if the parameter is 0 or not, i.e. whether or not the genes are differentially expressed.

• This corresponds to information about}0{

gIvg

The difference

• Classical Statisticians tend to be drastic about the choice of I. It is either 0 or 1.

• Bayesians on the other hand tend to describe this value with a probability, just as if it were another parameter.

• The classical approach to (HT) in micro-array is to assume the negative hypothesis (null or H0) and try to find the absurdity probability of the null hypothesis.

• The Bayesian approach assumes that for each gene g, there is the unobservable function, vg, that defines the genes activation status. The idea here is to update the knowledge of vg in terms of a posterior probability. This is the probability that vg=0.

The Classical Procedure

• Here we have some claim or belief or knowledge or guess about a population parameter which we want to prove (or disprove) using our data for evidence.

•  • Hypothesis is always written as a pair.• Research: what we are trying to prove (Ha)

Null: that nullifies / negates our research.(H0)

The hypotheses

• H0g: gene g is NOT differentially expressed• H1g: gene g IS differentially expressed.

• We have n pairs of mutually exclusive hypothesis.

• The role of the two hypothesis is NOT symmetric. Classical Statistics assumes the null by default UNLESS enough contrary evidence is found.

• Hypothesis is a systematic procedure to summarize evidence in the data, in order to decide between the pair of hypothesis.

The Test Statistic

• The test statistic is a summary of the data used to evaluate the validity of the hypothesis.

• The summary chosen depends upon the question and circumstances.

• Typically the Test Statistic is chosen, so that when H0 is true the distribution of the test statistic has a certain known distribution.

The logic of hypothesis testing

• To actually test the hypothesis, what we try to do is to disprove or reject the null hypothesis.

• If we can reject the null, by default our Ha (which is our research) is true.

• How do we do this?• We take a sample and look at the test-statistic. Then we

try to quantify that, if the null was true, would our observed statistic be a likely value from the known distribution of test statistics.

• If our observed value is not a likely value, we reject the null.

• How likely or unlikely a value is, is determined by the sampling distribution of that statistic.

Error Rates

• Since we take our decisions about the parameter based on sample values we are likely to commit some errors.

• Type I error: Rejecting Ho when it is true• Type II error: Failing to reject Ho when Ha is true.

• In any given situation we want to minimize these errors.

• P(Type I error) = a, Also called size, level of significance.•  • P(Type II error) = ,b • Power = 1- , b HERE we reject Ho when the claim is true. We want

power to be LARGE

Terms in error control

• In hypothesis we have two main types of error:• Null hypothesis is wrongly rejected (FALSE POSITIVE)• Null hypothesis is wrongly accepted (failed to reject)

(FALSE NEGATIVE).

• Trade off is one cannot control both probabilities of both errors together.

• Size = Probability of a False Positive• Power= 1 – Probability of a False Negative

Decision Rule

• A decision rule is a method than translates the vector of observed values into a binary decision or REJECT or FAIL TO REJECT.

• The decision rule is made such that the error rate of Type I error is controlled.

• P-value: the absurdity value of the null hypothesis, or the probability of observing a number more extreme than the observed value, assuming that the null is true.

Classical Test for differential Expression concerning two populations

• The two sample t test

– Version 1: The Pooled t test– Version 2: The Satterwaite Welch’s t test– Version 3: Bootstrapped t tests– Version 4: Permutation t test– Wilcoxon Rank-Sum test (nonparametric alternative)– Likelihood Ratio Test

The Test Statistic: In general

• Notation: Let xgi1,…,xgik represent the observations (intensity/ log intensity/ normalized intensity etc) for gene g, for the ith condition with, i=1,2. Let the means of the two conditions, the standard deviations and the variance be given by

22g

21g21 , ., ., ssxx gg

t-test

• The test statistic is given by:

)2

1

1

1(

)( 21

kkse

xxt gg

tpooled ,)2(

)1()1(

21

222

211

kk

skskse

gg

This follows a t distribution with (k1+k2-2) df

The Satterwhite-Welch’s (SW) Test Statistic

222g1

21g

121g

122

2

21

2

22g

1

21g

2121

/s/s

/sc where

1)(kc)(11)c(k

1)-1)(k-(kdf

t with a follows

k

s

k

s

)()(

kk

k

xxt

gg

The distribution of the Test Statistic

• The pooled t assumes that the two variances are equal. The SW test does not.

• Both assume underlying normality for the observations xgi1…xgiki.

Bootstrapping and Permutation t Tests

• Before we do the tests lets talk about the procedures.

• The tests use the principles of Bootstrapping or Permutation and apply it in the case of differential expression for two conditions.

Bootstrapping

• The term “bootstrap” came from the legendary Baron Munchausen, who pulled himself out of a man-hole literally by grabbing his own bootstraps.

• Efron (1979) introduced the idea into Statistics, the idea is to “get yourself out of a hole” by re-using the data many times.

• It is essentially a re-sampling technique that simulates alternative values of the statistic under the null (in our case the t statistic), in order to calculate an empirical p-value.

• The idea is to create a large number of values from the statistic to get an idea of the distribution of the test statistic UNDER the null.

Bootstrapping contd…

• First I will use a small example for illustrating the procedure and then I will introduce notation.

• Lets say we have data on gene “g” for condition 1 and condition 2 as follows:

• Cond1 12 15 24 17• Cond2 15 32 26 24

Bootstrapping

• Calculate the observed t value for your data• First you pool the data:• Then samples of size 4 with replacement from the pooled sample, one

for sample 1 and the other for sample 2. Calculate the t statistic• Continue this for B samples.• Find what proportion of observations are BIGGER than your observed

value. This is your “estimated p-value” using bootstrap.

samples

obsstatvaluep

#

|||{|#

Example

s1 s2 comb s1b s2b

12 15 12 15 17

15 32 15 17 24

24 24 24 12 17

17 17 17 24 24

15

32

24

17

boot

Frequency

3.01.50.0-1.5-3.0-4.5-6.0

180

160

140

120

100

80

60

40

20

0

-1.08

25

Histogram of Bootstrapped t values

Permutation Test

• Similar Idea to Bootstrapping.• Here instead of resampling WITH replacement we

“shuffle” the data to look at all possible permutations.• Hence, we pool the data, then sample without replacement

from the pooled data.• Same Idea, create a gamut of t statistics to get the

empirical distribution of the null.

samples

obsstatvaluep

#

|||{|#

Permutation Test Contd

• We should ONLY use permutation tests when the number of replicates are >10, else the number of possible resamples is quite small.

• Example when n1=4, n2=4. The possible permutation is • 8!/4!4!=70 possibilities, which is too small to estimate

empirical p-values.• Also, we need a firm assumption that the ONLY difference

between the two samples is due to location and they have the same shapes.

Wilcoxon Rank Sum Test

• The non-parametric alternative to the two-sample t.• Doesn’t assume underlying normality• Pool the data and rank all the observations in the pooled

data.• Then sum the ranks of ONE of the samples.• Idea is, if there is a location difference among the samples,

the sum of ranks for one sample will be either bigger or smaller than the other. Wilcoxon Rank Sum tables are generally available.

• Not very good for VERY few replicates. At least 4, preferably 6.

Large Sample Approx.

• Here n1 is size of the sample we summed up.

12/)1(

2/)1(

1212

121

nnnn

nnnW

Likelihood Ratio Test• The idea of Likelihood Ratio Test is simple and intuitive.• One basically looks at the ratio of the likelihood that the

gene is differentially expressed to that of the likelihood it isn’t.

• The method was used by Idekar et al (2000).• Model: xg1k = mg1 + mg1eg1k + dg1k

• xg2k = mg2 + mg2eg2k + dg2k

• Allows both additive and multiplicative error. Assumption, egik, eg2k = BVN( 0, 0, s2

eg1, s2eg2, re)

• dg1k, dg2k = BVN( 0, 0, s2dg1,

s2dg2, rd)

• Allows correlations.

The LR Statistic

• The parameters• b = (s2

eg1, s2eg1, r ,es2

dg1, s2dg1, rd) and m=(mg1,mg2) are

estimated via ML techniques optimizing:

The LR Statistic is

),|,( 11 1

1 kg

n

g

K

kkg xxP

21

212,1

21

),,(max

),(maxln(2

ggggg

gggggg L

L

The Issues of MULTIPLE Testing

• In Microarray setting there are often thousands of tests to be considered simultaneously.

• Obviously we have a pretty high chance of having false positives when you do thousands of tests at the same time.

• In these kinds of situations, the question of which error to control becomes an issue.

• We would like to control both Type I and Type II errors, but that’s not possible– If you make Type I error smaller, Type II goes up and vice versa.

The general practice is to fix Type I error at a low level (.05, .01, .001) and minimize the Type II error for that level.

Numbers of Correct and Incorrect Decisions

Declared Inactive

Declared Active

Total

Truly Inactive

U V =FP n0

Truly Active

T=FN S=TP n-n0

Total n-R R n

What Error Rates do we need to control?

• Most traditional methods focus on controlling the expected fraction of false positives out of the total number of TRUE null hypotheses.

• We want the expected fraction of false positives to be small. We want to control

• FPR = E ( FP / n0)

• However since we do not know n0 we really cannot control this rate. So we look at several measures that are related to this rate.

Per-comparison Error Rate (PCER):

• PCER = E(V)/n

• Controlling the expected number of False positives as a fraction of the total number of hypothesis tested.

• Here a level control achieved by performing each test in the family at a level alpha

• error arising from multiplicity ignored• false claims of significance are made at a higher rate than

Per-family Error Rate (PFER):

• PFER = E(V)

• The other end of the spectrum: controls the expected number of False positive per each hypothesis tested.

• This is the expected number of False Positives in a study.

• procedures that control this error rate are very strict• rarely used in microarray literature

Family-wise Error Rate (FWER):

• FWER = P(V ≥ 1)

Controls the probability of having a single Type I error in the study. Most commonly used, common examples are: Bonferroni, Tukey, Dunnett

• procedures based on this error rate are very conservative• generalized version of this error rate: k-FWER

False Discovery Rate (FDR):

• FDR = E(V/R | R ≥ 1)

• Controls the expected number of False positives OUT OF THE ONES DECLARED POSITIVE.

• FDR controlling procedures are liberal• widely used in microarray literature

The Methods Compared

• The hierarchy among techniques• PCER ≥ FDR ≥ FWER ≥ PFER

• PCER is the least conservative and PFER is the most conservative.

• These days people use either FWER or FDR

• These are all SINGLE STEP methods. However, Step-wise methods are getting more popular.

Stepwise methods

• Essentially a more adaptive form of testing.

• We can have step-down methods or step-up methods.• Step-down methods: rank observed p-values from the

smallest to the highest,– p(1)≤ … ≤p(m)

– Start at the smallest p-value, p(1).

• Step-Up Method– Rank as before – start with the largest p-vlaue

Step-Down: Bonferroni Holm Method

• rank observed p-values from the smallest to the highest,• p(1)≤ … ≤p(m) and define the corresponding H1….Hm

Start with the smallest one, reject H1 if p(1) < a/m and continue to Step 2 using p(2) and so on…

Else Fail to reject all H1….Hm.

• And reject H(j) if

• p(j)< /a (m - j + 1)

• This is more powerful than single-step procedures

Step-up: Hochberg and Hommel

• rank observed p-values from the smallest to the highest,• p(1)≤ … ≤p(m) and define the corresponding H1….Hm

Start with the LARGEST one, reject Hm if p(m) < a and all the corresponding H1….Hm

Otherwise Fail to reject and continue to Step 2 using p(m-1) and so on…

• And reject H(j) if

• this procedure rejects H(1),...,H(k) • if k =Max{j| p(j) ≤ j /a m} exists.

• This is more powerful than single-step procedures and the Step-Down procedure and controls FDR

Error Rates: used in microarrays

• FWER Bonferroni: Classify all genes as active if their p-value is less than /a n

• Step-up FWER(Hochberg): Let k be the largest g =1,…n, for which p(g) <= /a (n-g+1). Then reject all H(g) for g=1…k where H(g) is the null associated with the gth smallest p-value.

• Step-down FWER(Holmes): Let k be the smallest g =1,…n, for which p(g) >= /a (n-g+1). Then fail to reject all H(g) for g=1…k where H(g) is the null associated with the gth largest p-value

• FDR(Benjamini and Hochberg): Let k be the largest g =1,…n, for which p(g) <= ag/np0). Then reject all H(g) for g=1…k where H(g) is the null associated with the gth smallest p-value. (here p0 is the true proportion of inactive genes which is unknown, and generally replaced by 1 a conservative value)