© 2013 IBM Corporation
IBM Software GroupILOG CPLEX Optimization
Recent MIP Performance Improvements in
IBM ILOG CPLEX Optimization Studio
Andrea TramontaniILOG CPLEX Optimization Studio, IBM
January 22, 2014
© 2013 IBM Corporation2
IBM Software GroupILOG CPLEX Optimization
Outline
Main components of CPLEX MIP Solver
– Review of Branch and Bound (B&B)
– Main components of a (fast) B&B
– Performance impact of the main components
CPLEX 12.5.1 and CPLEX 12.6 performance improvements
– Benchmark results
– Overview of the main ingredients and estimated performance impact
– Concurrent root cut loops
– Lift and Project cuts
© 2013 IBM Corporation3
IBM Software GroupILOG CPLEX Optimization
Mixed Integer Programming and Branch and Bound (B&B)
A Mixed Integer (linear) Program (MIP) is a problem of the form
State of the art MIP solvers tackle a MIP through a Branch and Bound algorithm based on
the underlying Linear Programming (LP) relaxation:
integer allor some
)(
jx
uxl
bAxtoSubject
xczMinimizeMIP T
uxl
bAxtoSubject
xczMinimizeLP T
)(
© 2013 IBM Corporation4
IBM Software GroupILOG CPLEX Optimization
z* = 5.2Fathomed
Review of Branch and Bound (B&B)
Root;
x1=3.5
x2=2.3
Integer
z = 7x4=0.6
x3=0.3
Integer
z = 5
Infeas
x3=0.1
z* = 4.6z* = 5.4Fathomed
B&B algorithm:
– Enumerative solution scheme
based on the LP relaxation of MIP
Bounding:
– Nodes with z* > UB can be
fathomed without further
ramification.
Key points to avoid exponential
explosion of B&B tree:
– Strong LP relaxation
– Effective branching rules
– Primal heuristics
Upper Bound:
UB = 5
© 2013 IBM Corporation5
IBM Software GroupILOG CPLEX Optimization
Main components of a (fast) B&B: Overview
Presolve and probing
– Clean up the MIP formulation by removing redundant constraints/variables
– Strengthen the MIP formulation by tightening variable bounds, lifting constraints,
aggregating variables
– Derive cliques and implications to be used for cutting planes and/or branching
Cutting planes
– Valid inequalities for the MIP that cut off integer infeasible points of the LP relaxation
– Iteratively separated on the fly to strengthen the LP relaxation
– Cut separation must be combined with clever cut filtering and cut purging to avoid
• Numerical difficulties
• Node throughput slowdown
© 2013 IBM Corporation6
IBM Software GroupILOG CPLEX Optimization
Main components of a (fast) B&B: Overview (Cont.d)
Branching
– Divides the feasible region in a manner that all integer feasible solutions belong to one of
the branches
– Standard B&B: up and down branch on integer variables, chosen according to several
criteria:
• Strong branching, pseudo costs, pseudo reduced costs, conflicts, inferences
Primal heuristics
– Help finding quickly high quality feasible solutions
– Constructive heuristics (e.g., rounding heuristics)
– Improving heuristics (e.g., RINS heuristics)
Root node vs. B&B tree:
– Presolve, probing, cuts and primal heuristics heavily applied at the root node, before
resorting to branching
– Also applied at subsequent B&B nodes, but much less aggressively
© 2013 IBM Corporation7
IBM Software GroupILOG CPLEX Optimization
Main components of a (fast) B&B: Measuring performance impact
How important is each component?
Compare runs with feature turned on and off
– Solution time degradation (geometric mean)
– # of solved models
• Essential or just speedup?
– Number of affected models
• General of problem specific?
More detailed analysis in:
T. Achterberg and R. Wunderling, “Mixed Integer Programming: Analyzing 12 Years of
Progress”, in: Jünger and Reinelt (eds.) Facets of Combinatorial Optimization, Festschrift for
Martin Grötschel, pp.449-481, Springer, Berlin-Heidelberg (2013)
© 2013 IBM Corporation8
IBM Software GroupILOG CPLEX Optimization
Component Impact CPLEX 12.5.0 – Summary
Benchmarking setup
• 1769 models
• 12 core Intel Xenon 2.66 GHz
• Unbiased: At least one of all the
test runs took at least 10sec
99% 82% 91% 26%93%91% 46%83% 65%% affected
© 2013 IBM Corporation9
IBM Software GroupILOG CPLEX Optimization
Component Impact CPLEX 12.5.0 – Presolve
6.0
6
1.7
7
1.5
6
1.2
4
1.1
0
0
1
2
3
4
5
6
7
0
50
100
150
200
250
300
350
tim
e rati
o
ad
dit
ion
al
tim
eo
uts
© 2013 IBM Corporation10
IBM Software GroupILOG CPLEX Optimization
Component Impact CPLEX 12.5.0 – Cutting Planes
1.4
8
1.2
8
1.1
4
1.0
8
1.0
7
1.0
6
1.0
4
1.0
4
1.0
3
1.0
2
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
0
10
20
30
40
50
60
70
80
tim
e rati
o
ad
dit
ion
al
tim
eo
uts
© 2013 IBM Corporation11
IBM Software GroupILOG CPLEX Optimization
Main components of CPLEX MIP Solver
– Review of Branch and Bound (B&B)
– Main components of a (fast) B&B
– Performance impact of the main components
CPLEX 12.5.1 and CPLEX 12.6 performance improvements
– Benchmark results
– Overview of the main ingredients and estimated performance impact
– Concurrent root cut loops
– Lift and Project cuts
Outline
© 2013 IBM Corporation12
IBM Software GroupILOG CPLEX Optimization
0
200
400
600
800
1000
1200
14001
99
8
19
99
20
00
20
01
20
02
20
03
20
04
20
05
20
06
20
07
20
08
20
09
20
10
20
11
20
12
20
13
nu
mb
er o
f ti
meo
uts
0
50
100
150
200
250
tota
l sp
eed
up
CPLEX MIP Performance Evolution
10 sec
100 sec
1000 secv6.0
v6.5.3
v7.0v8.0
v9.0
v10.0
v11.0
v12.1 v12.2
v12.4v12.5
v12.6
Date: 28 September 2013
Testset: 3147 models (1792 in 10sec, 1554 in 100sec, 1384 in 1000sec)
Machine: Intel X5650 @ 2.67GHz, 24 GB RAM, 12 threads (deterministic since CPLEX 11.0)
Timelimit: 10,000 sec
© 2013 IBM Corporation13
IBM Software GroupILOG CPLEX Optimization
Date: 28 September 2013
Testset: 3116 models
Machine: Intel X5650 @ 2.67GHz, 24 GB RAM, 12 threads for parallel
Timelimit: 10,000 sec
MIP Performance Improvement – CPLEX 12.5.0 to CPLEX 12.6
1.0
0
0.8
7
0.8
1
1.0
0
0.8
1
0.7
3
1.0
0
0.7
2
0.6
2
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
>1s >10s >100s
CPLEX 12.5.0
CPLEX 12.5.1
CPLEX 12.6
1968
models
1210
models
678
models
Time limits:
96 / 49 / 35
Deterministic parallel MIP (12 threads)
1.23x 1.37x 1.61x
© 2013 IBM Corporation14
IBM Software GroupILOG CPLEX Optimization
Cuts
– Lift-and-project cuts
– Parallel cut loops to exploit variability
– Improvements in cut separation and purging
– Improvements in MIR cut aggregator
Presolve
– Probing enhancements
– Modified how to use multiple probing/presolve rounds
Branching
– Improvements in branching rule tie breaking
– Fixed a typo in branching rule
Other improvements
– Improvements in symmetry detection and exploitation
– Incremented default work memory from 128 MB to 2 GB
(t ≥ 100 sec)
8-13% speed-up
3-6% speed-up
2-10% speed-up
3-7% speed-up
3-8% speed-up
2-3% speed-up
4-7% speed-up
2-4% speed-up
1-10% speed-up
2-3% speed-up
Performance Improvements in CPLEX 12.5.1 and 12.6 – Summary
© 2013 IBM Corporation15
IBM Software GroupILOG CPLEX Optimization
Main components of CPLEX MIP Solver
– Review of Branch and Bound (B&B)
– Main components of a (fast) B&B
– Performance impact of the main components
CPLEX 12.5.1 and CPLEX 12.6 performance improvements
– Benchmark results
– Overview of the main ingredients and estimated performance impact
– Concurrent root cut loops
– Lift and Project cuts
Outline
© 2013 IBM Corporation16
IBM Software GroupILOG CPLEX Optimization
Root Cut Loop: Impact of the Initial LP Basis
Presolve/Probing + LP solve
Branch-and-cut
LP solution x* CutSEP (x*), Heur (x*)
Cut filtering
Add cuts to LP and resolve LP
Cut purging
Progress?YES
NO
LPs may have many alternative
optimal solutions/bases
Equivalent (but different) optimal
solutions/bases of the initial LP
relaxation may lead to
– Different cutting planes
– Different heuristic solutions
Reoptimizing with different cuts
amplifies the diversification
A small change in the initial LP basis
may have a big impact on the final
root node in terms of:
– Fractional solution x*
– Dual and primal bounds
– Cuts active in the LP
© 2013 IBM Corporation17
IBM Software GroupILOG CPLEX Optimization
M. Fischetti, A. Lodi, M. Monaci, D. Salvagnin, and A. Tramontani: ”Tree Search Stabilization
by Random Sampling”, Tech. Rep. DEI, University of Padova, 2013 (submitted for publication)
Exploit multi-threading and MIP performance variability at root node:
– Run K = 2 root cut loops in parallel, each starting from a different optimal LP basis
– Along the process, share cuts and feasible solutions among the K cut loops
Then do regular parallel branch-and-cut
Concurrent Root Cut Loops
Shake
Presolve/Probing + LP solve
Cut Loop #1 Cut Loop #2
Parellel Branch-and-cut
© 2013 IBM Corporation18
IBM Software GroupILOG CPLEX Optimization
Concurrent Root Cut Loops: Why K = 2?
Multi-threading is not for free and the overhead increases with K
CPLEX also applies other techniques in parallel at the root node
More cut loops means more cutting planes available,
– More cuts typically means better dual bound,
– Better dual bound does not mean less time to solve
– With the current cut selection strategy
• K = 3 cut loops is roughly a 5% performance degradation w.r.t. K = 2
Implementation details:
– Create diversification in the second parallel cut loop by changing
• The basis
• The random seed
• Other parameters
– Be conservative with the cuts:
• Sharing cuts between the two cut loops helps,
• But the cuts to be shared must be filtered aggressively.
© 2013 IBM Corporation19
IBM Software GroupILOG CPLEX Optimization
0.9
8
0.9
8
0.9
5
0.9
9
0.9
8
0.9
7
0.9
6
0.9
2
0.9
8
0.9
8
0.9
5
0.9
4
0.8
6 0.9
7
0.9
8
0
0.2
0.4
0.6
0.8
1
>1s >10s >100s
seed 1
seed 2
seed 3
seed 4
seed 5
Concurrent Root Cut Loops: Performance Impact in CPLEX 12.5.10.9
7
0.9
6
0.9
1
0.9
7
0.9
7
0.9
6
0.9
4
0.8
8
0.9
7
0.9
6
0.9
2
0.9
1
0.8
0 0.9
5
0.9
6
0
0.2
0.4
0.6
0.8
1
>1s >10s >100s
seed 1
seed 2
seed 3
seed 4
seed 5
All Models
Affected Models
≈1864
models
≈ 1089
models
≈ 565
models
≈ 1100 (59%)
models
≈ 686 (63%)
models
≈ 372 (66%)
models
1.04x 1.06x 1.09x
1.02x 1.04x 1.05x
Date: 23 May 2013
Testset: 3223 models
Machine: Intel X5650 @ 2.67GHz, 24 GB RAM, 12 threads
Timelimit: 10,000 sec
© 2013 IBM Corporation20
IBM Software GroupILOG CPLEX Optimization
Main components of CPLEX MIP Solver
– Review of Branch and Bound (B&B)
– Main components of a (fast) B&B
– Performance impact of the main components
CPLEX 12.5.1 and CPLEX 12.6 performance improvements
– Benchmark results
– Overview of the main ingredients and estimated performance impact
– Concurrent root cut loops
– Lift and Project cuts
Outline
© 2013 IBM Corporation21
IBM Software GroupILOG CPLEX Optimization
Lift and Project Cuts: Some references
Invented by Balas in the ‟70s:
– E. Balas, Disjunctive programming. Ann. Discrete. Math. 5, 3–51, 1979.
Successfully exploited in practice in the ‟90s:
– E. Balas, S. Ceria, and G. Cornuejols, A lift-and-project cutting plane algorithm for mixed-
integer programs. Mathematical Programming 58, 295-324, 1993.
– E. Balas, S. Ceria, and G. Cornuejols: Mixed 0-1 programming by lift-and-project in a
branch-and-cut framework. Management Science 42, 1229-1246, 1996.
Strong connection with Gomory Mixed Integer (GMI) cuts:
– E. Balas and M. Perregaard, A precise correspondence between lift-and-project cuts,
simple disjunctive cuts, and mixed integer gomory cuts for 0-1 programming.
Mathematical Programming 94, 221-245, 2003.
Effective approach to optimize over the Lift and Project closure:
– P. Bonami, On optimizing over lift-and-project closures. Math. Prog. Comp. 4, 151-179,
2012.
© 2013 IBM Corporation22
IBM Software GroupILOG CPLEX Optimization
Given:
The LP relaxation of a MIP, defined by the underlying polyhedron P:
A fractional solution x* in P (typically, a vertex of P) to cut off
A binary fractional variable xj*: xj in {0,1}, 0 < xj* < 1
Find a violated cut
α x ≥ β
using the simple split disjunction
xj ≤ 0 OR xj ≥ 1
Lift and Project Cuts: The Separation Problem
(P) A x ≥ b
x ≥ 0
© 2013 IBM Corporation23
IBM Software GroupILOG CPLEX Optimization
The Cut Generating LP (CGLP) of Balas, Ceria and Cornuejols
1. Consider the polyhedra P0 and P1 induced by the disjunction:
2. Separate a cut valid for the disjunctive polyhedron
Q = conv (P0 U P1)
by solving the following CGLP:
3. Strengthen the cut by Monoidal Strengthening (Balas and Jeroslow, Eur. J. Oper. Res. 1980)
P0
(u) A x ≥ b
(w) x ≥ 0
(u0) - xj ≥ 0
P1
(v) A x ≥ b
(z) x ≥ 0
(v0) xj ≥ 1
α = u A + w – u0 ej
β = u b
u, w, u0 ≥ 0
α = v A + z + v0 ej
β = v b + v0
v, z, v0 ≥ 0
min α x* - β
P0
P1
x*
αx ≥ β
© 2013 IBM Corporation24
IBM Software GroupILOG CPLEX Optimization
The CGLP is unbounded by construction (if a violated cut exists)
Can be normalized in many different ways: see, e.g.,
– Balas, Ceria and Cornuejols (Math. Prog.1993, Man. Science 1996)
– Ceria and Soares (Tech. Report, 1997)
The normalization is the real objective function and makes the difference: see, e.g.,
– Fischetti, Lodi and T. (Math. Prog. 2011)
Two popular normalization constraints are:
– The weak normalization (actually, the one used in CPLEX):
u0 + v0 = 1 (1)
– The strong normalization (Balas, Math. Prog. 1997):
1 u + 1 v + 1 w + 1 z + u0 + v0 = 1 (2)
CGLP: The Role of Normalization
© 2013 IBM Corporation25
IBM Software GroupILOG CPLEX Optimization
Balas and Perregaard (Math. Prog. 2003) adopt normalization (2) and:
1. Separate high-rank inequalities (x* is a basic solution of P)
2. Provide a precise correspondence between feasible CGLP bases and (possibly
infeasible) bases of the tableau of the original problem
3. Characterize Gomory Mixed Integer (GMI) cuts from the tableau as CGLP bases
4. Develop an efficient approach for solving the CGLP by a sequence of pivoting in the
tableau of the original space.
Different variants of the idea available in commercial and open-source solvers:
FICO XPRESS,
CBC (COIN-OR B&C), see Balas and Bonami (Math. Prog. Comp. 2009)
Lift and Project Cuts à la Balas-Perregaard (BP L&P cuts)
Warmstart CGLP
with GMI basis
Pivots in the space
A x – I s = b
Optimal CGLP
solution
© 2013 IBM Corporation26
IBM Software GroupILOG CPLEX Optimization
Bonami (Math. Prog. Comp., 2012):
1. Separate only rank-1 L&P cuts (x* is not a vertex of P)
2. Adopts normalization u0 + v0 = 1
3. Applies some algebra to get rid of u0 and v0 in the CGLP
4. Takes the dual of the simplified CGLP and gets the so called Membership LP (MLP):
5. Provides promising computational results on MIPLIB 3.0 and MIPLIB 2003 models
The (strengthened) L&P closure:
– Can be computed in practice
– Provides a good approximation of the split closure
Lift and Project Cuts à la Bonami (this is what we did!)
(MLP) max yj - δ*
(v) b δ* ≤ A y ≤ b δ* + s* (u)
(z) 0 ≤ y ≤ x* (w)
s* := A x* - b
δ* := xj*
© 2013 IBM Corporation27
IBM Software GroupILOG CPLEX Optimization
L&P cuts à la Balas-Perregaard:
– High-rank inequalities
– Strong normalization (2)
– L&P cuts as improved tableau GMI cuts
– Computationally more expensive
L&P cuts à la Bonami:
– Rank-1 inequalities
– Weak normalization (1)
– Computationally cheaper
L&P cuts à la Balas-Perregaard vs. L&P cuts à la Bonami
© 2013 IBM Corporation28
IBM Software GroupILOG CPLEX Optimization
Computational investigation in Fischetti, Lodi and T. (Math. Prog. 2011):
Normalization (2) is fundamental when high-rank inequalities are separated, because it:
– Breaks the correlation between optimization and separation
– Tends to produce low-rank inequalities
– Enforces the separation of cuts with sparse dual and primal support
Separating rank-1 cuts:
– Breaks the correlation between optimization and separation
– Enforces the separation of low-rank inequalities (not a big surprise)
– Tends to produce cleaner and sparser cuts
Lift and Project Cuts: Our Idea
Sounds like
normalization (2)
Separate some more rank-1 L&P cuts with u0 + v0 = 1
Keep tableau GMI cuts as they are (if your filter likes them enough), and
© 2013 IBM Corporation29
IBM Software GroupILOG CPLEX Optimization
Bonami MLP to separate rank-1 cuts only, plus some tricks:
Heuristic reduction of MLP to
– Speed up the separation
– Get rid of constraints you don„t want to use to produce the cut
Clever selection of the disjunctions to use
Lift and Project Cuts in CPLEX 12.5.1
© 2013 IBM Corporation30
IBM Software GroupILOG CPLEX Optimization
L&P Cuts in CPLEX 12.5.1: Performance Impact
Date: 23 May 2013
Testset: 3243 models
Machine: Intel X5650 @ 2.67GHz, 24 GB RAM, 12 threads
Timelimit: 10,000 sec
0.9
1
0.9
8
0.9
1
0.9
6
0.9
5
0.8
7
0.9
5
0.8
6
0.9
3
0.9
2
0.8
0
0.8
8
0.7
9
0.8
8
0.8
7
0
0.2
0.4
0.6
0.8
1
>1s >10s >100s
seed 1
seed 2
seed 3
seed 4
seed 5
0.9
5
0.9
9
0.9
5
0.9
8
0.9
8
0.9
3
0.9
7
0.9
2
0.9
6
0.9
5
0.8
7
0.9
3
0.8
6
0.9
2
0.9
2
0
0.2
0.4
0.6
0.8
1
>1s >10s >100s
seed 1
seed 2
seed 3
seed 4
seed 5
948 (51%)
models
617 (57%)
models
362 (64%)
models
All Models
Affected Models
1864
models
1089
models
567
models
1.03x 1.05x 1.10x
1.10x1.06x 1.16x
© 2013 IBM Corporation31
IBM Software GroupILOG CPLEX Optimization
Questions and Answers (Q&A)