Graphical Modeling of Multiple Sequence Alignment Jinbo Xu Toyota Technological Institute at Chicago...

Graphical Modeling of Multiple Sequence Alignment

Jinbo XuToyota Technological Institute at Chicago

Computational Institute, The University of Chicago

Two applications of MSA• Predict inter-residue interaction network (i.e.,

protein contact map) from MSA using joint graphical lasso– An important subproblem of protein folding

• Align two MSAs through alignment of two Markov Random Fields (MRFs)– Homology detection and fold recognition– Merge two MSAs into a larger one

Modeling MSAby Markov Random Fields

The generating probability of a sequence :

Infer , by maximum-likelihood encodes residue correlation relationshipA special case is Gaussian Graphical Model

Numeric Representation of MSA

… 0 0 … 1 0 …

21 elements for each column in MSA

Represent a sequence in MSA as a L×21 binary vector

Gaussian Graphical Model (GGM)

• a multiple sequence alignment (MSA)• Assume has Gaussian distribution where is the

covariance matrix• (inverse of ): the precision matrix, implying the

residue interaction pattern among all MSA columns

Covariance and Precision Matrix

L

L

The precision matrix has dimension 21L×21L

21×21

one residue pair

Larger values indicate stronger interaction

Today’s talk• Predict inter-residue interaction network (i.e.,

protein contact map) from MSA using joint graphical lasso– An important subproblem of protein folding

• Align two MSAs through alignment of two Markov Random Fields (MRFs)– Homology detection and fold recognition– Merge two MSAs into a larger one

Protein Contact Map(residue interaction network)

1

2

3

4

6.0

8.1

5.9

1 2 3 4

1 0 1 1 0

2 1 0 1 1

3 1 1 0 1

4 0 1 1 0

Two residues in contact if their Cα or Cβ distance < 8Å

3.8

Shorter distance Stronger interaction

Contact Matrix is Sparse

short range: 6-12 AAs apart along primary sequencemedium range: 12-24 AAs apartlong range: >24 AAs apart

#contacts is linear w.r.t. sequence length

Input:MEKVNFLKNGVLRLPPGFRFRPTDEELVVQYLKRKVFSFPLPASIIPEVEVYKSDPWDLPGDMEQEKYFFSTKEVKYPNGNRSNRATNSGYWKATGIDKQIILRGRQQQQQLIGLKKTLVFYRGKSPHGCRTNWIMHEYRLANLESNYHPIQGNWVICRIFLKKRGNTKNKEENMTTHDEVRNREIDKNSPVVSVKMSSRDSEALASANSELKKKASIIFYDFMGRNNSNGVAASTSSSGITDLTTTNEESDDHEESTSSFNNFTTFKRKIN

Protein Contact Prediction

Output:

With L/12 long-range native contacts, the fold of a protein can be roughly determined [Baker group]

Contact Prediction Methods

• Evolutionary coupling analysis (unsupervised learning) Identity co-evolved residues from multiple sequence alignment No solved protein structures used at all High-throughput sequencing makes this method promising e.g., mutual information, Evfold, PSICOV, plmDCA, GREMLIN

• Supervised machine learning Input features: sequence (profile) similarity, chemical properties

similarity, mutual information (implicitly) learn information from solved structures examples: NNcon, SVMcon, CMAPpro, PhyCMAP

Evolutionary Coupling (EC) AnalysisObservation: two residues in contact tend to co-evolve,i.e., two co-evolved residues likely to form a contact

http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0028766

Evolutionary Coupling (EC) Analysis (Cont’d)• Local statistical methods: examine the correlation

between two residues independent of the others Mutual information (MI): two residues in contact likely to have

large MI Not all residue pairs with large MI are in contact due to indirect

evolutionary coupling. If A~B and B~C, then likely A~C

• Global statistical methods: examine the correlation between two residues condition on the others

Need a large number of sequences Maximum-Entropy: Evfold Graphical lasso: PSICOV Pseudo-likelihood: plmDCA, GREMLIN

Single MSA-based Contact Prediction

• Given a protein sequence under prediction, run PSI-BLAST to detect its homologs and build an MSA

• Calculate the sample covariance matrix from the MSA

• is singular, so cannot calculate the precision matrix by

• Calculate by maximum-likelihood, i.e., maximize the occurring probability of observed seqs

Enforce sparse precision matrixWhy ?

Issues with Existing Methods Evolutionary coupling (EC) analysis works for proteins

with a large number of sequence homologs Focus on how to improve the statistical methods instead of

use of extra biological information/insight, e.g., relax the Gaussian assumption, consensus of a few EC methods,

Use information mostly in a single protein family Physical constraints other than sparsity not used

Our Work: contact prediction using multiple MSAs

Jointly predict contacts for related families of similar folds. That is, predict contacts using multiple MSAs. These MSAs share inter-residue interaction network to some degree

Integrate evolutionary coupling (EC) analysis with supervised learning EC analysis makes use of residue co-evolution information Supervised learning makes use of sequence (profile) similarity

Goal: focus on proteins without many sequence homologsStrategy: increase statistical power by information aggregation

Red: shared; Blue: unique to PF00116; Green: unique to PF13473

Observation: different protein families share similar contact maps

Joint evolutionary coupling (EC) analysis

Jointly predict contacts for a set of related protein families Predict contacts for a protein family using information in

other related families Enforce contact map consistency among related families Do not lose family-specific information

Joint graphical lasso for joint evolutionary coupling analysis

1. Given a protein family and its MSA, find related families and corresponding MSAs

Let be precision matrices

2. Estimate by joint log-likelihood as follows

Where the last term enforces sparse precision matrices

How to enforce contact map consistency?

Residue Pair/Submatrix Grouping

In total ≤L(L-1)/2 groups where L is the seq length

Enforce Contact Map Consistencyby Group Penalty

: the number of groups: the number of families Using group lasso to model family consistency:

Group conservation level

is defined as

Supervised Machine Learning

• Input features: sequence profile, amino acid chemical properties, mutual information power series, context-specific statistical potential

• Mutual information power series: – Local info: mutual information matrix (MI)– Partially global info: MI2, MI3, …, MI11

– Can be calculated much faster than PSICOV• Random Forests trained by 800-900 proteins

Joint EC Analysis with Supervised Prediction as Prior

max∑𝑘=1

𝐾

( log|𝛺𝑘|−tr (𝛺𝑘 �̂�𝑘 ) )

− 𝜆1∑𝑘=1

𝐾

¿|𝛺𝑘|∨¿1¿−∑𝑔=1

𝐺

𝜆𝑔∨¿𝛺𝑔∨¿2

− 𝜆2∑𝑘=1

𝐾

∑𝑖𝑗

‖𝛺𝑖𝑗𝑘‖1

𝑚𝑎𝑥 (𝑃 𝑖𝑗𝑘 ,0.3)

sparsity

contact map consistency among families

similarity with supervised prediction

Log-likelihood of K families

This optimization problem can be solved by ADMM to suboptimal

Accuracy on 98 Pfam families Medium-range Long-range L/10 L/5 L/2 L/10 L/5 L/2

CoinDCA 0.496 0.435 0.312 0.561 0.502 0.391

PSICOV 0.375 0.312 0.213 0.446 0.400 0.311PSICOV_b 0.388 0.306 0.199 0.462 0.400 0.294plmDCA 0.433 0.354 0.233 0.484 0.443 0.343

plmDCA_h 0.433 0.339 0.211 0.480 0.413 0.292GREMLIN 0.401 0.332 0.225 0.447 0.423 0.329

GREMLIN_h 0.391 0.316 0.204 0.428 0.400 0.301

Merge_p 0.303 0.246 0.178 0.370 0.328 0.253Merge_m 0.276 0.223 0.169 0.355 0.309 0.232

Voting 0.405 0.280 0.168 0.337 0.353 0.275

Accuracy vs. # Sequence Homologs(A) Medium-range (B) Long-range

X-axis: ln of the number of non-redundant sequence homologsY-axis: L/10 accuracy

Accuracy on 123 CASP10 targets

Medium-range Long-range L/10 L/5 L/2 L/10 L/5 L/2CoinDCA 0.500 0.440 0.340 0.412 0.351 0.279

Evfold 0.294 0.249 0.188 0.257 0.225 0.171PSICOV 0.310 0.259 0.192 0.276 0.225 0.168plmDCA 0.344 0.289 0.214 0.326 0.280 0.213

GREMLIN 0.343 0.280 0.229 0.320 0.278 0.159

NNcon 0.393 0.334 0.226 0.239 0.188 0.001CMAPpro 0.414 0.363 0.276 0.336 0.297 0.227

Accuracy vs. # sequence homologs (CASP10)

X-axis: ln of # non-redundant sequence homologsY-axis: L/10 long-range prediction accuracy

Accuracy vs. Contact Conservation Level

(A)Medium-range; (B) long-rangeX-axis: conservation level, the larger, the more conserved

Today’s Talk• Predict inter-residue interaction network (i.e.,

protein contact map) from MSA using joint graphical lasso– An important subproblem of protein folding

• Align two MSAs through alignment of two Markov Random Fields (MRFs)– Remote homology detection and fold recognition– Merge two MSAs into a larger one

Homology Detection & Fold Recognition

• Primary sequence comparison– Similar sequences -> very likely homologous– Sequence alignment method, e.g., BLAST, FASTA– works only for close homologs

• Profile-based method– Compare two protein families instead of primary sequences, using

evolutionary information in a family– Sequence-profile alignment & profile-profile alignment– Profile can be represented as a matrix (e.g., FFAS) or a HMM (e.g.,

HHpred, HMMER)– Sometimes works for remote homologs, but not sensitive enough

MSA to Sequence Profile

Two popular profile representations: (1) Position-specific scoring matrix (PSSM); (2) Hidden Markov Model (HMM)

Position-Specific Scoring Matrix (PSSM)

Taken from http://carrot.mcb.uconn.edu/~olgazh/bioinf2010/class10.html

Hidden Markov Model (HMM)

http://www.biopred.net/eddy.html

Our Work: Markov Random Fields (MRF) Representation

1) MRF encodes long-range residue interaction pattern while HMM does not;

2) Long-range interaction pattern encodes global information of a protein,So can deal with proteins of similar folds but divergent sequences

Protein alignment by aligning two MRFs

G R K - Y S A

G R K - Y S A

F L V - L Y I

K L V - L Y I

P T A K F R E

P T A K F R S

P T V P G Y E

P T V P G R S

MRF1

MRF2

Family 1

Family 2

Scoring function for MRF alignment

local alignment potential pairwise alignment potential

MRF1

MRF2

𝜃𝑖 , 𝑗𝑀

𝜃𝑘 ,𝑙𝑀

𝑍 𝑖 , 𝑗𝑀 =1 𝑍 𝑘 ,𝑙

𝑀 =1

NP-hard due to1) Gaps allowed2) Pairwise potential

Alternating Direction of Method Multiplier (ADMM)

Make a copy of z to y

Add a penalty term to obtain an augmented problem

ADMM (Cont’d)

Solve the above problem iteratively as follows:Step 1: Solve the optimization problem for a fixed Step 2: Update by subgradient and repeat 1) until convergence

Use a Lagrangian multiplier to relax the original problemand obtain a upper bound

ADMM(Cont’d)

(SP1) Where

(SP2)

Where

For a fixed plit the relaxation problem into two subproblems and solve them alternatively

Both subproblems can solved efficiently by dynamic programming!

Superfamily & Fold Recognition Rate

Superfamily level detection Fold level detection

Conclusion

• Joint evolutionary coupling analysis + supervised learning can significantly improve protein contact prediction by using information in multiple MSAs

• Long-range residue interaction encoded in an MSA helpful for remote homolog detection

Acknowledgements• RaptorX servers at http://raptorx.uchicago.edu• Students: Jianzhu Ma, Zhiyong Wang, Sheng Wang• Funding

– NIH R01GM0897532– NSF CAREER award and NSF ABI– Alfred P. Sloan Research Fellowship

• Computational resources– University of Chicago Beagle team– TeraGrid and Open Science Grid

Input:MEKVNFLKNGVLRLPPGFRFRPTDEELVVQYLKRKVFSFPLPASIIPEVEVYKSDPWDLPGDMEQEKYFFSTKEVKYPNGNRSNRATNSGYWKATGIDKQIILRGRQQQQQLIGLKKTLVFYRGKSPHGCRTNWIMHEYRLANLESNYHPIQGNWVICRIFLKKRGNTKNKEENMTTHDEVRNREIDKNSPVVSVKMSSRDSEALASANSELKKKASIIFYDFMGRNNSNGVAASTSSSGITDLTTTNEESDDHEESTSSFNNFTTFKRKIN

Protein Structure Prediction

Output:1. One of the most challenging problems in computational biology!

2. Improved due to better algorithms and large databases

3. Knowledge-based methods outperformsphysics-based methods

4. Big demand: our server processes> 800 jobs/week, >12k users in 3yrs

Performance in CASP9 (2010)A blind test for protein structure prediction

Server ranking tested on the 50 hardest TBM targets

Adapted from http://predictioncenter.org/casp9/doc/presentations/CASP9_TBM.pdf

Performance in CASP10 (2012)A blind test for protein structure prediction

The only server group among top 10

Adapted from http://predictioncenter.org/casp10/doc/presentations/CASP10_TBM_GM.pdf

The top 10 performing human/server groups on the hardest TBM targets

My WorkAnalyze large-scale biological data and build predictive models• Protein sequence and structure alignment• Homology detection and fold recognition• Protein structure prediction• Protein function prediction (e.g., interaction and binding site

prediction)• Biological network construction and analysis

Study computational methods that have applications beyond bioinformatics• Machine learning (e.g. probabilistic graphical model)• Optimization (discrete, combinatorial and continuous)

Homology Detection & Fold Recognition

• Homology detection & fold recognition– Determine the relationship between two proteins– Given a query, search for all homologs in a database

• Homology search/fold recognition useful for– Study protein evolutionary relationship– Functional transfer– Homology modeling (i.e., template-based modeling)

Two proteins are homologous if they have shared ancestry.Two proteins have the same fold if their 3D structures are similar.

Structure Prediction (Cont’d)• Template-based modeling (TBM)

– Using solved protein structures as template, e.g., homology modeling and protein threading

– Most reliable, but fails when no good templates• Template-free modeling (FM) or ab initio folding

– Not using solved protein structures as template– Mostly works only on some small proteins

• Subproblems– Loop modeling– Inter-residue contact prediction

Residue Pair Grouping

Precision Submatrix Grouping

for Family 1 for Family 2

Suppose that residue pair (2,4) in Family 1 aligned to pair (3,5) in Family 2

In total ≤L(L-1)/2 groups where L is the seq length

Performance on the 31 Pfam families with only distantly-related auxiliary families

Medium-range Long-range L/10 L/5 L/2 L/10 L/5 L/2

CoinFold 0.457 0.400 0.267 0.558 0.524 0.416PSICOV 0.413 0.360 0.252 0.494 0.465 0.377

PSICOV_p 0.320 0.295 0.212 0.396 0.355 0.290PSICOV_v 0.400 0.320 0.179 0.396 0.375 0.261

CoinFold: our workPSICOV: single-family method PSICOV_p: merge multiple families and apply single-family methodPSICOV_v: single-family method for each family and then consensus

Performance on the 13 Pfam families with closely-related auxiliary families

Medium-range Long-rangeL/10 L/5 L/2 L/10 L/5 L/2

CoinFold 0.501 0.395 0.251 0.462 0.413 0.293PSICOV 0.433 0.351 0.231 0.398 0.331 0.234

PSICOV_p 0.335 0.220 0.175 0.322 0.276 0.194PSICOV_v 0.423 0.320 0.188 0.386 0.384 0.301

CoinFold: our workPSICOV: single-family method PSICOV_p: merge multiple families and apply single-family methodPSICOV_v: single-family method for each family and then consensus

Our method vs. PSICOV

Our method vs. GREMLIN

L/10 top predicted long-range contacts are evaluated

Performance vs. family size

CoinFold: our workPSICOV: single-family method PSICOV_p: merge multiple families and apply single-family methodPSICOV_v: single-family method for each family and then consensus

Multiple Sequence Alignment (MSA) of One Protein Family

Top L/10 long-range prediction accuracy on 15 large Pfam families

PFAM ID MEFF CoinFold PSICOV PSICOV_p

PF00041 2981 0.767 0.767 0.667PF00595 3026 0.556 0.444 0.233PF03061 3334 0.375 0.250 0.500PF01522 3519 0.615 0.462 0.077PF00578 3733 0.308 0.308 0.231PF00059 3744 0.455 0.455 0.182PF07686 3801 0.917 0.583 0.667PF00034 4060 0.600 0.500 0.200PF00989 4596 0.583 0.250 0.500PF00144 4684 0.272 0.212 0.242PF00085 5075 0.636 0.545 0.182PF00168 6735 0.667 0.667 0.556PF00515 7230 0.500 0.500 0.250PF00089 9045 0.783 0.783 0.739PF00550 11476 0.857 0.857 0.714

Running Time

Average protein sequence length

Time (in seconds)

Performance: Alignment Accuracy

Performance: Homology Detection

Performance: Alignment Accuracy

Tmalign, Matt and DeepAlign represent three different ground truth

Joint Graphical Lasso Formulation

Log-likelihood: Regularization:

Rewrite the original problem as

Unconstrained optimization problem Both and are convex, so the objective is the difference of two convex functions Can be solved by the Convex-Concave Procedure [A. Yuille]

Alternating Direction of Method Multiplier (ADMM)

s.t. ZAdd a penalty term to obtain an augmented problem, which has the same solution but converges faster.

Make a copy of to , without changing the solution space

s.t. Z

Lagrangian Relaxation

min𝑈maxΩ ,𝑍

𝑓 (Ω)−𝑃 (𝑍 )−∑𝑘=1

𝐾

¿¿¿¿

Use a Lagrange multiplier for each constraint Obtain a dual problem to upper bound the augmented problem

Solve the dual problem iteratively by subgradient as follows.Step 1) fix and solve

Step 2) Update by and repeat 1) until convergence

ADMM (Cont’d)

For a fixed U, split the relaxation problem into two subproblems and solve them alternatively

(SP1)

(SP2)