52
Effective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale Problems & Stochastic Modelling, RICAM, Linz, Dec. 12-16, 2011 X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 1 / 47

Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Embed Size (px)

Citation preview

Page 1: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Effective Solvers for Reservoir Simulation

Xiaozhe Hu

The Pennsylvania State University

Numerical Analysis of Multiscale Problems & Stochastic Modelling,

RICAM, Linz, Dec. 12-16, 2011

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 1 / 47

Page 2: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Collaboration with:I ExxonMobil Upstream Research CompanyI China National Offshore Oil CooperationI Monix Energy Solutions

Main Collaborators:I James Brannick (PSU), Yao Chen (PSU), Chunsheng Feng (XTU),

Panayot Vassilevski (LLNL), Jinchao Xu (PSU),Chensong Zhang (CAS), Shiquan Zhang (Fraunhofer ITWM),and Ludmil Zikatanov (PSU)

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 2 / 47

Page 3: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Outline

Outline

1 Reservoir Simulation

2 Solvers for Reservoir Simulation

3 Applications & Numerical Tests

4 Ongoing Work and Conclusions

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 3 / 47

Page 4: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Reservoir Simulation

1 Reservoir Simulation

2 Solvers for Reservoir Simulation

3 Applications & Numerical Tests

4 Ongoing Work and Conclusions

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 4 / 47

Page 5: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Reservoir Simulation

World without Oil?

List of Oil Products:Antiseptics, Aspirin, AutoParts, Ballpoint pens,Candles, Cosmetics, Crayons,Eye Glasses, Fishing Line,Food Packaging, Glue, HandLotion, Insect Repellant,Insecticides, Lip Stick,Perfume, Shampoo, ShavingCream, Shoes, Toothpaste,Trash Bags, VitaminCapsules, Water Pipes, · · · .

Global oil discovery peaked in the late

1960s. Now we consume much more

than the amount we discover and this

gap is still growing!

(http://www.aspo-ireland.org/)

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 5 / 47

Page 6: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Reservoir Simulation

Oil Recovery

CSEF 2011 Kunming, China C.-S. Zhang

!"#$%&!'()"*(&+'$,

Antiseptics, Aspirin, Auto Parts, Ballpoint pens, Candles, Cosmetics, Crayons, Eye Glasses, Fishing Line, Food Packaging, Glue, Hand Lotion, Insect Repellant, Insecticides, Lip Stick, Perfume, Shampoo, Shaving Cream, Shoes, Toothpaste, Trash Bags, Vitamin Capsules, Water Pipes, !

List of Oil Products

CSEF 2011 Kunming, China C.-S. Zhang

-.)/.01%&+'$&210"31#4

!  Primary Recovery !  Secondary Recovery !  Enhanced Oil Recovery

Primary Recovery

Secondary Recovery

Enhanced Oil Recovery

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 6 / 47

Page 7: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Reservoir Simulation

Enhanced Oil Recovery

Enhanced Oil Recovery (EOR) isa generic term for techniques forincreasing the oil recovery ratefrom an oil field:

Thermal Method

Microbial Injection

Gas Injection

Chemical FloodingI AlkalineI PolymerI SurfactantI GelI · · ·· · ·

Recovery Rate:

Primary and Secondaryrecovery: 20-40%

Enhanced oil recovery:30-60%

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 7 / 47

Page 8: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Reservoir Simulation

Black Oil Model

Mass Conservation:

∂t

(So

Bo+

Rv Sg

Bg

)]= −∇ ·

(1

Bouo +

Rv

Bgug

)+ qO

∂t

(Sg

Bg+

RsSo

Bo

)]= −∇ ·

(1

Bgug +

Rs

Bouo

)+ qG

∂t

Sw

Bw

)= −∇ ·

(1

Bwuw

)+ qW

Darcy’s Law:

uα = −kkrα

µα(∇Pα − ραg∇z), α = o, g ,w

Constitutive Laws:

So + Sg + Sw = 1, Pcow = Po − Pw , Pcog = Pg − Po ,

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 8 / 47

Page 9: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Reservoir Simulation

Modified Black Oil Model

Add polymer and sodium chloride:

∂t

(φS∗

w CP

Br Bw

)+

∂t(ρr (1− φ)Ca) = −∇

(CP

BwuP

)+ qW CP

∂t

(φSw CN

Br Bw

)= −∇

(CN

BwuN

)+ qW CN

Effects on Viscosity:

uw = − kkrw

µw ,ef f Rk(∇Pw − ρwg∇z)

Add more components (brine, gel, surfactant, · · · )

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 9 / 47

Page 10: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Reservoir Simulation

Numerical Method

Discretization in time:I Fully Implicit Method (FIM)

(Douglas, Peaceman & Rachford 1959)I Implicit Pressure Explicit Saturation (IMPES)

(Sheldon, Zondek & Cardwell 1959; Stone & Garder 1961)I Sequential Solution Method (SSM)

(MacDonald & Coats 1970)I Adaptive Implicit Method (AIM)

(Thomas & Thurnau 1983)

Discretization in space: finite difference, finite volume, finiteelement, etc.

Linearization: Newton-Raphson Method.

Grid: Structured grids are still used by the main-stream; butunstructured grids are catching up.

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 10 / 47

Page 11: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Reservoir Simulation

Fully Implicit Method

1

∆t

(Sg

Bg+

RsSo

Bo

)]n+1

−[φ

(Sg

Bg+

RsSo

Bo

)]n

=

∇ · (T n+1g ∇Φn+1

α + Rn+1s T n+1

o ∇Φn+1o )

1

∆t

[(φSw

Bw

)n+1

−(φSw

Bw

)n]

= ∇ · (T n+1w ∇Φn+1

w )

1

∆t

[(φSo

Bo

)n+1

−(φSo

Bo

)n]

= ∇ · (T n+1o ∇Φn+1

o )

Potential:Φα := Pα − ραgz , α = w , o, g ,

Transmissibility:

Tα :=krαk

µαBα, α = w , o, g .

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 11 / 47

Page 12: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Reservoir Simulation

Newton-Raphson Method

In each Newton Iteration, we need to solve the following Jacobian system:

JgP JgSw JgSo

JwP JwSw JwSo

JoP JoSw JoSo

δPδSw

δSo

=

Rg

Rw

Ro

Different blocks have different properties, for example

JgP =1

∆tcgP −∇ · DgP∇− CgP · ∇ −RgP ,

usually the diffusion term in JgP is dominating, which makes JgP like an ellipticproblem. However,

JoSo =1

∆tcoSo − CoSo · ∇ −RoSo ,

here convection term is dominating, which makes JoSo like a transport problem.

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 12 / 47

Page 13: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Reservoir Simulation

Including Wells

Peaceman Model

One-dimentional radial flow

Single-phase flow in thewellbore

qi,α = −NperfX

j

2πhperf Kkrα

ln( rerw +s

)µα(Pj−Pbhp−Hwj)

Jacobian system with wells:

[JRR JRW

JWR JWW

] [δRδW

]=

[RR

RW

]

CSEF 2011 Kunming, China C.-S. Zhang

!"#$%&#'()*+,'-$)'!"#&)

∂t

φS∗

wCP

Bw

+

∂t(ρr(1− φ)Ca) = −∇

CP

BwuP

+ qW CP

uw = − kkrw

µw,effRk(∇P − ρwg∇z)

Add polymer(s) to the drive

Effects on viscosity

Add more components (polymer, brine, gel, surfactant, etc)

CSEF 2011 Kunming, China C.-S. Zhang

.&))'!"#&)/

•  One-dimensional radial flow into the wellbore; •  Single-phase flow; •  Pseudo steady-state pressure behavior.

Peaceman Model

o  Multi-segment o  Multi-lateral o  Complex wells o  Couple wells with reservoir

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 13 / 47

Page 14: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Reservoir Simulation

Numerical ChallengesStrongly coupled and complicated PDE system

High nonlinearity

Complicated geometry: irregular domain with faults, pinch-out and erosion

Heterogeneous porous media; large permeability jumps

Complex wells

Unstructured grids: corner point grid, anisotropic mesh

Highly non-symmetric and indefinite large-scale Jacobian system

Different properties of the physical quantities (the equation that describesthe pressure is mainly elliptic, the equation that describes saturation ismainly hyperbolic)

Commercial Simulators set a high bar: ECL2009 (Schlumberger), STARS(CMG), VIP (Halliburton), · · ·

Goal: develop fast solvers, deliver a solver package

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 14 / 47

Page 15: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Solvers for Reservoir Simulation

1 Reservoir Simulation

2 Solvers for Reservoir Simulation

3 Applications & Numerical Tests

4 Ongoing Work and Conclusions

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 15 / 47

Page 16: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Solvers for Reservoir Simulation

Solvers

“For a reservoir simulation with a number of gridblocks of order 100,000,about 80%− 90% of the total simulation time is spent on the solution oflinear systems with the Jacobian.”—- Chen, Huan, & Ma, Computational Methods for Multiphase Flows in Porous Media, 2005

Direct solvers: (Price & Coats 1974)

Incomplete factorization preconditioner: (Watts 1981, Behie & Vinsome 1982; Meyerink 1983;

Appleyard & Cheshire 1983)

Constrained Pressure Residual (CPR) preconditioner: (Wallis 1983; Wallis,Kendall, Little &

Nolen 1985; Aksoylu & Klie 2009)

Algebraic Multigrid (AMG) Method :

I Used in CPR preconditioner (pressure equation): (Klie 1997; Lacroix, Vassilevski &

Wheeler 2001, 2003; Scheichl, Masson & Wendebourg 2003; Cao, Tchelepi, Wallis & Yardumian 2005;

Hammersley & Ponting 2008; Dubois, Mishev & Zikatanov 2009; Al-Shaalan, Klie, Dogru & Wheeler 2009;

Jiang & Tchelepi 2009)

I For whole Jacobian system: (Papadopoulos & Tchelepi 2003; Stuben 2004; Clees & Ganzer

2007)

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 16 / 47

Page 17: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Solvers for Reservoir Simulation

Solvers

“For a reservoir simulation with a number of gridblocks of order 100,000,about 80%− 90% of the total simulation time is spent on the solution oflinear systems with the Jacobian.”—- Chen, Huan, & Ma, Computational Methods for Multiphase Flows in Porous Media, 2005

Direct solvers: (Price & Coats 1974)

Incomplete factorization preconditioner: (Watts 1981, Behie & Vinsome 1982; Meyerink 1983;

Appleyard & Cheshire 1983)

Constrained Pressure Residual (CPR) preconditioner: (Wallis 1983; Wallis,Kendall, Little &

Nolen 1985; Aksoylu & Klie 2009)

Algebraic Multigrid (AMG) Method :

I Used in CPR preconditioner (pressure equation): (Klie 1997; Lacroix, Vassilevski &

Wheeler 2001, 2003; Scheichl, Masson & Wendebourg 2003; Cao, Tchelepi, Wallis & Yardumian 2005;

Hammersley & Ponting 2008; Dubois, Mishev & Zikatanov 2009; Al-Shaalan, Klie, Dogru & Wheeler 2009;

Jiang & Tchelepi 2009)

I For whole Jacobian system: (Papadopoulos & Tchelepi 2003; Stuben 2004; Clees & Ganzer

2007)

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 16 / 47

Page 18: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Solvers for Reservoir Simulation

Auxiliary Space Preconditioning

1 construct preconditioner via auxiliary (simpler) problems

2 solve the auxiliary problems by efficient solvers (such as multigrid)

3 apply preconditioned Krylov subspace methods

Examples:

Fictitious Domain Methods (Nepomnyaschikh 1992)

Method of Subspace Correction (Xu 1992)

Auxiliary Space Method (Xu 1996)

Nodal Auxiliary Space Preconditioning in H(curl) and H(div) spaces(Hiptmair & Xu 2007)

· · · · · ·

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 17 / 47

Page 19: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Solvers for Reservoir Simulation

Auxiliary Space Method (Xu 1996)

Abstract problem: A : V → V is SPD

Au = f

Auxiliary spaces:V = V ×W1 × · · · ×WJ

Additive preconditioner:

B = S +J∑

j=1

Πj A−1j Π∗

j

With Πj : Wj 7→ V (transfer) and S : V 7→ V (smoother). It can beshown that under some assumptions

κ(BA) ≤ C

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 18 / 47

Page 20: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Solvers for Reservoir Simulation

Auxiliary Space Method for Reservoir SimulationFast Auxiliary Space Preconditioning

Auxiliary Spaces:

V = V ×WP ×WS ×Wwell

FASP Algorithm: Given u0, Bu0 := u4 where

u1 = u0 + ΠwellA−1wellΠ

∗well(f − Au0)

u2 = u1 + ΠSA−1S Π∗

S(f − Au1)

u3 = u2 + ΠPA−1P Π∗

P(f − Au2)

u4 = u3 + S(f − Au3)

Auxiliary problems Awell , AS and AP are solved approximately.

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 19 / 47

Page 21: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Applications & Numerical Tests

1 Reservoir Simulation

2 Solvers for Reservoir Simulation

3 Applications & Numerical Tests

4 Ongoing Work and Conclusions

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 20 / 47

Page 22: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Applications & Numerical Tests Single Phase Flow

Single Phase Flow (pressure equation)Intergrated Reservoir Performance PredictionExxonMobil Upstream Research Company

Possion-like Model Problem (After linearization):

−∇ · (a∇p) + cp = f , in Ω

p = gD , on ∂ΩD ,

(a∇p) · nnn = gN on ∂ΩN ,

Discretization: Hybrid-Mixed FEM (Kuznetsov & Repin 2003)

Linear system: Symmetric and Positive definite

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 21 / 47

Page 23: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Applications & Numerical Tests Single Phase Flow

Difficulties for the solver

Models are from real problems,the geometry is complicated

Mesh (hexahedral grid): Highlyunstructured; Degenerated,nonmatching, and distortedgridblock.

Permeability: Large jumps: varyform 10−4 to 104; Anisotropic.

Only two types of pinchout hexahedrons satisfy these assumptions. Dueto the terminology in [6], we call them the “Element H1” (the vertical edgeconnecting node 0 with node 4 vanishes) and “horizontal triangular prism”(two neighboring vertical edges (0, 4) and (1, 5) disappear), see Figure 4.

5

6

70, 4

1

23

6

70, 4

1, 5

3 2

Figure 4: Element H1 (left) and horizontal triangular prism (right)

The element H1 has seven vertexes, two triangular and four quadrilateralfaces, and 11 edges. The horizontal prism has six vertexes, two triangularand three quadrilateral faces.

13

Also, due to the presence of the so called fault surfaces, nonmatchingmeshes are possible. For instance, consider a case when we have a domaindivided into geological layers which allows us to construct a hexahedral meshfitting to the interfaces between layers. Suddenly, something happens, forinstance, an earthquake or any other natural disaster. In this case, certainhexahedral cells may change their positions, and some of the hexahedral cellsmay have two or more neighbors from the side of the fault. Therefore, weobtain a nonmatching mesh. Several examples of nonmatching hexahedralmeshes were considered in the previous report [2]. In this report, the side of ahexahedral cell which is adjacent to a fault interface was divided into severalparts. The number of this parts is equal to the number of the neighboring cellsfrom the other side of the fault interface. For instance, the green hexahedralcell shown in Figure 3 has three neighbors adjacent to its right side.

Figure 3: An example of a distorted nonmatching hexahedral mesh from theprevious project

11

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 22 / 47

Page 24: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Applications & Numerical Tests Single Phase Flow

AMG with ILU smoother?

Preconditioner #iter Setup time Solve timeILU(0) 3458 0.16 96.19AMG 362 0.85 40.32

ILU(0)/1+AMG 2255 1.00 305.17ILU(0)/2+AMG - 1.18 -

ILU or AMG alone does not work well

Using ILU as smoother in AMG may not work

Why AMG with ILU smoother does not work in this case?

Consider (B denotes ILU and S denotes AMG)

u ← u + B(f − Au), u ← u + S(f − Au), u ← u + B(f − Au).

This gives:I − BA = (I − BA)(I − SA)(I − BA).

B might not be positive definite, and cannot be applied to PCG

References about ILU smoother: Wittum 1989; Stevenson 1994

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 23 / 47

Page 25: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Applications & Numerical Tests Single Phase Flow

AMG with ILU smoother?

Preconditioner #iter Setup time Solve timeILU(0) 3458 0.16 96.19AMG 362 0.85 40.32

ILU(0)/1+AMG 2255 1.00 305.17ILU(0)/2+AMG - 1.18 -

ILU or AMG alone does not work well

Using ILU as smoother in AMG may not work

Why AMG with ILU smoother does not work in this case?

Consider (B denotes ILU and S denotes AMG)

u ← u + B(f − Au), u ← u + S(f − Au), u ← u + B(f − Au).

This gives:I − BA = (I − BA)(I − SA)(I − BA).

B might not be positive definite, and cannot be applied to PCG

References about ILU smoother: Wittum 1989; Stevenson 1994

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 23 / 47

Page 26: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Applications & Numerical Tests Single Phase Flow

AMG with ILU smoother?

Preconditioner #iter Setup time Solve timeILU(0) 3458 0.16 96.19AMG 362 0.85 40.32

ILU(0)/1+AMG 2255 1.00 305.17ILU(0)/2+AMG - 1.18 -

ILU or AMG alone does not work well

Using ILU as smoother in AMG may not work

Why AMG with ILU smoother does not work in this case?

Consider (B denotes ILU and S denotes AMG)

u ← u + B(f − Au), u ← u + S(f − Au), u ← u + B(f − Au).

This gives:I − BA = (I − BA)(I − SA)(I − BA).

B might not be positive definite, and cannot be applied to PCG

References about ILU smoother: Wittum 1989; Stevenson 1994

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 23 / 47

Page 27: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Applications & Numerical Tests Single Phase Flow

AMG + ILU!Consider

u ← u + S(f − Au), u ← u + B(f − Au), u ← u + S(f − Au).

This gives:I − BA = (I − SA)(I − BA)(I − SA).

Theorem (Falgout, H., Wu, Xu, Zhang, Zhang & Zikatanov 2011)

Assume that S : V → V satisfies ‖(I − SA)x‖A ≤ ‖x‖A, ∀x ∈ V andoperator B : V → V is SPD. Then the operator B is SPD.

Theorem (Falgout, H., Wu, Xu, Zhang, Zhang & Zikatanov 2011)

Assume that ((I − SA)v , v)A ≤ ρ(v , v)A, ρ ∈ [0, 1), B is SPD and thecondition number of BA is κ(BA) = m1/m0. If m1 > 1 ≥ m0 > 0. thenκ(BA) ≤ κ(BA), Furthermore, if ρ ≥ 1− m0

m1−1 , then κ(BA) ≤ κ(SA).

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 24 / 47

Page 28: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Applications & Numerical Tests Single Phase Flow

AMG + ILU!Consider

u ← u + S(f − Au), u ← u + B(f − Au), u ← u + S(f − Au).

This gives:I − BA = (I − SA)(I − BA)(I − SA).

Theorem (Falgout, H., Wu, Xu, Zhang, Zhang & Zikatanov 2011)

Assume that S : V → V satisfies ‖(I − SA)x‖A ≤ ‖x‖A, ∀x ∈ V andoperator B : V → V is SPD. Then the operator B is SPD.

Theorem (Falgout, H., Wu, Xu, Zhang, Zhang & Zikatanov 2011)

Assume that ((I − SA)v , v)A ≤ ρ(v , v)A, ρ ∈ [0, 1), B is SPD and thecondition number of BA is κ(BA) = m1/m0. If m1 > 1 ≥ m0 > 0. thenκ(BA) ≤ κ(BA), Furthermore, if ρ ≥ 1− m0

m1−1 , then κ(BA) ≤ κ(SA).

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 24 / 47

Page 29: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Applications & Numerical Tests Single Phase Flow

ExxonMobil Test Problems

Preconditioner #iter Setup time Solve timeILU(0) 3458 0.16 96.19AMG 362 0.85 40.32

ILU(0)/1+AMG 2255 1.00 305.17ILU(0)/2+AMG - 1.18 -

FASP (AMG+ILU) 41 0.99 5.83

Model 1 Model 2 Model 3 Model 4size 388,675 156,036 287,553 312,851nnz 4,312,175 1,620,356 3,055,173 3,461,087

#iter Setup time (s) Solve time (s) Total time (s)FASP(AMG+ILU) + CG

Model 1 16 2.12 4.63 6.75Model 2 41 0.99 5.83 6.82Model 3 47 1.97 12.81 14.78Model 4 11 1.75 2.68 4.43

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 25 / 47

Page 30: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Applications & Numerical Tests Single Phase Flow

ExxonMobil Test Problems

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 26 / 47

Page 31: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Applications & Numerical Tests Single Phase Flow

ExxonMobil Test Problems

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 27 / 47

Page 32: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Applications & Numerical Tests Single Phase Flow

ExxonMobil Large Test Problems

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 28 / 47

Page 33: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Applications & Numerical Tests Single Phase Flow

ExxonMobil Large Test Problems

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 29 / 47

Page 34: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Applications & Numerical Tests Enhanced Oil Recovery

(Modified) Black Oil ModelCNOOC and Monix: SOCF Simulator

Decoupling: Alternative Block Factorization (Bank, Chan, Coughran &

Smith 1989)

Auxiliary problems:I WP : pressure (+ bottom hole pressure)I WS : saturation (+ concentrations)I Wwell : wells (+ perforations)

Solvers for each auxiliary problems:I AP : AMG (+ILU)I AS : Block Gauss-Seidel with downwind ordering and crosswind blocks

(Bey & Wittum 1996; Hackbush & Probst 1997; Wang & Xu 1999; Kwok 2007)

I Awell : ILU/Direct solver

Smoother S : Block Gauss-Seidel with downwind ordering andcrosswind blocks.

Krylov iterative method: GMRes.

Ref: H., Liu, Qin, Xu, Yan and Zhang 2011, SPE 148388

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 30 / 47

Page 35: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Applications & Numerical Tests Enhanced Oil Recovery

Optimality

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 31 / 47

Page 36: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Applications & Numerical Tests Enhanced Oil Recovery

East Beverly Oil Field Test

Black Oil Model

Unstructured grid

83,592 grid blocks

169 wells

Faults: 994 non-localconnections

40 years of water flooding

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 32 / 47

Page 37: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Applications & Numerical Tests Enhanced Oil Recovery

East Beverly Oil Field Test

We use sequential version of FASP in this table. OpenMP version of FASPcosts 46 min (8-core, 2.8GHz).

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 33 / 47

Page 38: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Applications & Numerical Tests Enhanced Oil Recovery

JZ9-3 Oil Field Test

Polymer flooding(5-compoment)

Number of grids: 157x53x57

Lots of inactive grids

Several faults

37 wells

10 years simulation in total

3 years of polymer flooding

Simulator # Newton Iter Total CPU Time CPU Time/NewtonECL2009 1562 81.0 (min) 3.11 (sec)

SOCF(FASP) 1653 40.0 (min) 1.45 (sec)

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 34 / 47

Page 39: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Applications & Numerical Tests Enhanced Oil Recovery

JZ9-3 Results Comparison: Oil Production Rate

CSEF 2011 Kunming, China C.-S. Zhang

!"#$%&'()*+,)&-./0123).45&678'

!

CSEF 2011 Kunming, China C.-S. Zhang

!"#$%&'()*+,)&-./0123).45&698'

!

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 35 / 47

Page 40: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Applications & Numerical Tests Enhanced Oil Recovery

JZ9-3 Results Comparison: Gas Production Rate

CSEF 2011 Kunming, China C.-S. Zhang

!"#$%&'()*+,)&-./0123).45&678'

!

CSEF 2011 Kunming, China C.-S. Zhang

!"#$%&'()*+,)&-./0123).45&698'

!

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 36 / 47

Page 41: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Applications & Numerical Tests Enhanced Oil Recovery

JZ9-3 Results Comparison: Water Cut

CSEF 2011 Kunming, China C.-S. Zhang

!"#$%&'()*+,)&-./0123).45&67-8

!

CSEF 2011 Kunming, China C.-S. Zhang

9:;<$;&=$-./0.4(4,&&8(),

! !SOCF CMG

Field Oil Saturation

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 37 / 47

Page 42: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Applications & Numerical Tests Enhanced Oil Recovery

ld101-1 7-component Test

Polymer-Gelflooding

5 wells: 1 injectwell, 4 productwells

32 yearssimulation intotal

No wellILU FASP

Size # Iter Time (s) # Iter Time (s)40x40x4 37 2.20 12 0.47

40x40x20 153 19.59 15 2.8640x40x40 296 56.05 13 4.11

5 wellsILU FASP

# Iter Time (s) # Iter Time (s)40x40x4 55 2.70 13 0.50

40x40x20 236 28.21 15 2.9040x40x40 319 65.04 14 5.78

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 38 / 47

Page 43: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Applications & Numerical Tests Enhanced Oil Recovery

ld101-1 7-component Test

3/20/2011

75

4

149Isoquant contours of water saturation

SOCF CMG

4LD10-1

Polymer flooding Gel profile-control flooding

Isoquant contours of oil saturation150

3/20/2011

75

4

149Isoquant contours of water saturation

SOCF CMG

4LD10-1

Polymer flooding Gel profile-control flooding

Isoquant contours of oil saturation150

3/20/2011

76

4

151

Poly

mer

floo

ding

Gel

pro

file-

cont

rol

flood

ing

Oil profile

LD10-1

4

LD10-1

0 1000 2000 3000 4000 5000 6000 70000

50

100

150

200

250

300

350

400

Fiel

d O

il P

rodu

ctio

n R

ates

(SM

3)

Time(Day)

A=0 A=-3 A=-4

0 1000 2000 3000 4000 5000 6000 70000.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Time(Day)

Wat

er C

ut

A=0 A=-3 A=-4

Field Oil Production Rate (SM3) Water Cut152

0 2 31 2 3 4 1[1 ( ) exp( )]i

i i i i

spp w i p i p i p sep iA c A c A c c A t

3/20/2011

76

4

151

Poly

mer

floo

ding

Gel

pro

file-

cont

rol

flood

ing

Oil profile

LD10-1

4

LD10-1

0 1000 2000 3000 4000 5000 6000 70000

50

100

150

200

250

300

350

400

Fiel

d O

il P

rodu

ctio

n R

ates

(SM

3)

Time(Day)

A=0 A=-3 A=-4

0 1000 2000 3000 4000 5000 6000 70000.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Time(Day)

Wat

er C

ut

A=0 A=-3 A=-4

Field Oil Production Rate (SM3) Water Cut152

0 2 31 2 3 4 1[1 ( ) exp( )]i

i i i i

spp w i p i p i p sep iA c A c A c c A t

Oil Saturation (Left: Polymer; Right: Polymer/Gel)

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 39 / 47

Page 44: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Ongoing Work and Conclusions

1 Reservoir Simulation

2 Solvers for Reservoir Simulation

3 Applications & Numerical Tests

4 Ongoing Work and Conclusions

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 40 / 47

Page 45: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Ongoing Work and Conclusions Ongoing Work

Move to GPUNVIDIA

Parallel AMG algorithm suits GPU:

Regular sparsity pattern

Low complexity

Low setup cost

Fine-grain parallelism

Take advantage of hierarchical memory structure

Our choice: Unsmoothed Aggregation AMG (UA-AMG)

Controllable sparsity pattern

Low complexity (less fill-in)

Low setup cost (no triple matrices multiplication)

Large potential of parallelism

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 41 / 47

Page 46: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Ongoing Work and Conclusions Ongoing Work

UA-AMG with Nonlinear AMLI-cycleProblem: UA-AMG with V-cycle usually dose not converge uniformly!

Solution: UA-AMG with Nonlinear AMLI-cycle (Variable AMLI-cycle / K-cycle).

Nonlinear AMLI-cycle: Bk [·] : Vk → Vk .

Assume B1[f ] = A−11 f , and Bk−1[·] has been defined, then for f ∈ Vk

Pre-smoothing u1 = Rk f ;

Coarse grid correction u2 = u1 + Bk−1[Qk−1(f − Aku1)]

Post-smoothing Bk [f ] := u2 + Rtk(f − Aku2).

Nonlinear coarse grid correction: Bk−1[·] : Vk−1 → Vk−1

Defined by n iterations of a Krylov subspace iterative methods usingBk−1[·] as the preconditioner.

Ref: Axelsson & Vassilevski 1994; Kraus 2002; Kraus & Margenov 2005; Notay & Vassilevski

2008.

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 42 / 47

Page 47: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Ongoing Work and Conclusions Ongoing Work

UA-AMG with Nonlinear AMLI-cycleProblem: UA-AMG with V-cycle usually dose not converge uniformly!Solution: UA-AMG with Nonlinear AMLI-cycle (Variable AMLI-cycle / K-cycle).

Nonlinear AMLI-cycle: Bk [·] : Vk → Vk .

Assume B1[f ] = A−11 f , and Bk−1[·] has been defined, then for f ∈ Vk

Pre-smoothing u1 = Rk f ;

Coarse grid correction u2 = u1 + Bk−1[Qk−1(f − Aku1)]

Post-smoothing Bk [f ] := u2 + Rtk(f − Aku2).

Nonlinear coarse grid correction: Bk−1[·] : Vk−1 → Vk−1

Defined by n iterations of a Krylov subspace iterative methods usingBk−1[·] as the preconditioner.

Ref: Axelsson & Vassilevski 1994; Kraus 2002; Kraus & Margenov 2005; Notay & Vassilevski

2008.

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 42 / 47

Page 48: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Ongoing Work and Conclusions Ongoing Work

Convergence Analysis of Nonlinear AMLI-cycle (SPD)

Theorem (H., Vassilevski & Xu 2011)

Assume the convergence factor of V-cycle MG with fixed level differencek0 is bounded by δk0 ∈ [0, 1), if n is chosen such that (1− δn)δk0 + δn ≤ δhas a solution δ ∈ [0, 1), which is independent of k, then we have

‖v − Bk [Akv ]‖2Ak≤ δ‖v‖2

Akand ‖v − Bk [Akv ]‖2

Ak≤ δn‖v‖2

Ak,

A sufficient condition of n: n > 11−δk0

> 1.

This is a generalization of the result in Notay & Vassilevski 2008.

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 43 / 47

Page 49: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Ongoing Work and Conclusions Ongoing Work

Primary Results

1024×1024 2048×2048

Setup Solve Total Setup Solve Total

CPU 0.80 7/0.69 1.49 3.28 7/2.75 6.03

GPU (CUSP) 0.63 36/0.35 0.98 2.38 41/1.60 3.98

GPU (FASP) 0.13 19/0.47 0.60 0.62 19/2.01 2.63

Processor:

I CPU: Intel i7-2600 3.4 GHzI GPU: Tesla C2070

Algorithm:

I CPU: Classical AMG + V-cycle;I GPU(CUSP): Smoothed Aggregation AMG + V-cycle;I GPU(FASP): UA-AMG + Nonlinear AMLI-cycle

Ref: Brannick, Chen, H., Xu, Zikatanov, 2011

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 44 / 47

Page 50: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Ongoing Work and Conclusions Ongoing Work

Other ongoing work

Compositional Model (Equation of State)

Complex wells

Improve robustness of FASP solvers

Improve robustness of nonlinear iteration

Heterogeneous parallel computing

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 45 / 47

Page 51: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Ongoing Work and Conclusions Conclusions

Conclusions

We developed effective solvers for reservoir simulation, which workswell for reservoirs from real world.

We developed a solver package (FASP). Part of the solver packagehas be used for the chemical flooding simulator (SOCF) by MonixEnergy Solutions and CNOOC.

We are working on parallelization of FASP package, includingOpenMP, MPI and CUDA.

We are working on the robustness of our package.

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 46 / 47

Page 52: Effective Solvers for Reservoir Simulation - Ricam · PDF fileE ective Solvers for Reservoir Simulation Xiaozhe Hu The Pennsylvania State University Numerical Analysis of Multiscale

Ongoing Work and Conclusions Conclusions

Thank you!

X. Hu (Penn State) Effective Solvers for Reservoir Simulation Linz,Dec.14, 2011 47 / 47