ML: A Multilevel Preconditioning Package Copper Mountain Conference on Iterative Methods March 29-April 2, 2004 Jonathan Hu Ray Tuminaro Marzio Sala Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy under contract DE-AC04-94AL85000. Outline Overview Multigrid basics Available user options Configuring & building Interoperability with other packages Example program Conclusions ML Package Provides parallel multigrid preconditioning for linear solver methods Current developers: Ray Tuminaro, Jonathan Hu, Marzio Sala Former: Charles Tong (LLNL) Main methods Geometric Grid refinement hierarchy 2-level FE basis function domain decomposition AMG (algebraic multigrid based on aggregation) Smoothed aggregation* Edge-element AMG for Maxwells equations* n-level (smoothed) aggregation domain decomposition * Classical AMG Written primarily in C C++ interfaces to various Trilinos packages 102,411 lines of code (as of this afternoon) 26 example programs Downloadable as part of Trilinos CVS, bugzilla MG(f, u, k) { if (k == 1) u 1 = (A 1 ) -1 f 1 else { S k (A k, f k,u k ) //pre-smooth r k = f k A k u k f k-1 = R k-1 r k ; u k-1 = 0 //restrict MG(f k-1, u k-1, k-1) u k = u k + I k-1 u k-1 //interpolate & correct S k (A k, f k,u k ) //post-smooth } A k, R k, I k, S k required on all levels S k : smoothers R k : restriction operators I k : interpolation operators Recursive MG Algorithm (V Cycle) V Cycle to solve A 4 u 4 =f 4 k=4 k=1 W Cycle FMV Cycle ML Capabilities MG cycling: V, W, full V, full W Grid Transfers Several automatic coarse grid generators Several automatic grid transfer operators Coarse grid visualization capabilities Smoothers Jacobi, Gauss-Seidel, Hiptmair, Krylov methods, sparse approximate inverses, Chebyshev Variety of Serial & Parallel Direct Solvers SuperLU, Umfpack, KLU, MUMPS, etc. Kernels: matrix/matrix multiply, etc. Smoothed Aggregation Developed for linear elasticity (Vanek, Mandel, Brezina) Construct tentative prolongator t P Aggregation: group unknowns together Augment: interpolate null space (e.g., rigid body modes) Construct final prolongator P Smooth: P = (I D -1 A) T P lower energy in basis functions Take null space (e.g., const.) Smooth Split into local basis functions Smoothed Aggregation Uncoupled / MIS Aggregation Greedy algorithm Global graph partitioning Operates on global domain ParMETIS Aggregates can span processors Graph partitioning aggregation Local graph partitioning Processors work independently METIS 1 aggregate per processor ML Smoother Choices Jacobi, Point/Block Gauss Seidel MLS Based on Chebyshev polynomials of smoothed operator. Serial: competitive with true Gauss-Seidel Parallel: Performance is independent of # processors Hiptmair Distributed relaxation smoother for Maxwells Eqns. Other smoothers used within each projection step Aztec solvers Krylov methods, incomplete factorizations Direct solution (via Amesos interface): UMFPACK, KLU (serial) SuperLU MUMPS ML and Other Packages ML Epetra Accepts user data as Epetra objects Can be wrapped as Epetra_Operator TSF TSF interface exists Other matvecs Other solvers Accepts other solvers and MatVecs Amesos Amesos interface for direct solvers Aztecoo Meros Via Epetra & TSF Configuring and Building ML Builds by default when you configure & build Trilinos By default, you get Epetra & Aztecoo support Example suite ( Trilinos/packages/ml/examples ) Some options of interest (off by default) MPI support Graph partitioning aggregation (METIS, ParMETIS) Direct solvers (Amesos) Profiling configure --with-mpi-compilers=/usr/local/mpich/bin --with-ml_amesos \ --with-libs=-lamesos lsuperlu A Small Example: ml/examples/ml_example_epetra_preconditioner.cpp Linear Solver ML: multi- grid pre- cond. AztecOO (Epetra) ML (Teuchos) gmres AMG Solution component Example methods Packages used Solve Ax=b: A: advection/diffusion operator Linear solver: gmres Precond.: 2-level AMG, graph-partitioning aggregation ml_example_epetra_preconditioner.cpp Trilinos_Util_CrsMatrixGallery Gallery(recirc_2d", Comm); Gallery.Set("problem_size", 10000); // linear system matrix & linear problem Epetra_RowMatrix * A = Gallery.GetMatrix(); Epetra_LinearProblem * Problem = Gallery.GetLinearProblem(); // Construct outer solver object AztecOO solver(*Problem); // Set some solver options solver.SetAztecOption(AZ_solver, AZ_gmres); solver.SetAztecOption(AZ_output, 10); solver.SetAztecOption(AZ_kspace, 160); example (contd.) // Set up multilevel preconditioner ParameterList MLList; // parameter list for ML options MLList.set("max levels",2); MLList.set("aggregation: type", "METIS"); // graph partitioning MLList.set("aggregation: nodes per aggregate", 16); // set up aztecoo smoother MLList.set("smoother: type","aztec"); int options[AZ_OPTIONS_SIZE]; double params[AZ_PARAMS_SIZE]; AZ_defaults(options,params); options[AZ_precond] = AZ_dom_decomp; options[AZ_subdomain_solve] = AZ_ilut; MLList.set("smoother: aztec options", options); MLList.set("smoother: aztec params", params); MLList.set("coarse: type","Amesos_Superludist"); MLList.set("coarse: max processes", 4); ML_Epetra::MultiLevelPreconditioner * MLPrec = new // create preconditioner ML_Epetra::MultiLevelPreconditioner(*A, MLList, true); solver.SetPrecOperator(MLPrec); // tell solver to use ML preconditioner solver.Iterate(500, 1e-12); // iterate at most 500 times AMG for Common Problem Types Trilinos_Util_CrsMatrixGallery Gallery(laplace_3d", Comm); Gallery.Set("problem_size", 100*100*100); // linear system matrix & linear problem Epetra_RowMatrix * A = Gallery.GetMatrix(); Epetra_LinearProblem * Problem = Gallery.GetLinearProblem(); // Construct outer solver object AztecOO solver(*Problem); solver.SetAztecOption(AZ_solver, AZ_cg); // Set up multilevel precond. with smoothed aggr. defaults ParameterList MLList; // parameter list for ML options ML_Epetra::SetDefaults(SA,MLList); ML_Epetra::MultiLevelPreconditioner * MLPrec = new // create preconditioner ML_Epetra::MultiLevelPreconditioner(*A, MLList, true); solver.SetPrecOperator(MLPrec); // tell solver to use ML preconditioner solver.Iterate(500, 1e-12); // iterate at most 500 times Other problem types: DD, DD-ML, maxwell Collaborations Within Sandia ALEGRA Radiation Maxwell Electrostatic potential MPSalsa (Shadid, et al.) CEPTRE External users EM3D (Lawrence Berkeley NL) P. Arbenz (ETH Zurich) R. Geus (PSI) J. Fish, H. Waisman (RPI) Potential Users PREMO (Sandia) Getting Help Website:See Trilinos/packages/ml/examples See guide: ML Users Guide, ver. 3.0 (in ml/doc) Mailing listsBug reporting, enhancement requests via bugzilla: http://software.sandia.gov/bugzillahttp://software.sandia.gov/bugzillaus directly Conclusions ML 3.0 release this May Part of next Trilinos release Existing options Uncoupled / MIS aggregation Smoothers SuperLU New options with ML 3.0 Graph-based aggregation parameter list setup Amesos interface to direct solvers Visualization, aggregate statistics Use ML with Trilinos! ML as preconditioner is trivial Rich set of external libraries