PETSc algebraic solvers now run on Nvidia GPU systems. This sometimes provides an alternative high-performance, low-cost solution technique.
Nvidia GPU usage of PETSc is currently available in the current release, PETSc 3.2, but since things change rapidly in this arena, we recommend working with petsc-dev if you wish to utilize GPUs.
If you use the PETSc GPU code in your published work, please cite the following paper:@article{minden2010preliminary, title={Preliminary implementation of PETSc using GPUs}, author={Minden, V. and Smith, B.F. and Knepley, M.G.}, journal={Proceedings of the 2010 International Workshop of GPU Solutions to Multiscale Problems in Science and Engineering}, year={2010} }WARNING: Using GPUs effectively is difficult! You must be dedicated and willing to get into the guts of GPU usage if you are serious about using GPUs.
- Installing PETSc to use the Nvidia GPUs
- How the GPU solvers are implemented in PETSc
- Example that uses CUSP directly in the user function evaluation
- Presentation on some aspects of GPU usage from PETSc
-
Quick summary of usage:
-
The VecType
VECSEQCUSP
,VECMPICUSP
, orVECCUSP
may be used withVecSetType()
or -vec_type seqcusp,mpicusp, or cusp whenVecSetFromOptions()
is used. -
The MatType
MATSEQAIJCUSP
,MATMPIAIJCUSP
, orMATAIJCUSP
maybe used with MatSetType or -mat_type seqaijcusp,mpiaijcusp, or aijcusp whenMatSetOptions()
is used. -
The MatType
MATSEQAIJCUSPARSE
,MATMPIAIJCUSPARSE
, orMATAIJCUSPARSE
maybe used with MatSetType or -mat_type seqaijcusparse,mpiaijcusparse, or aijcusparse whenMatSetOptions()
is used. - If you are creating the vectors and matrices with a DM, you can use -dm_vec_type cusp and -dm_mat_type cusp
- It is useful to develop your code with the default vectors and then run production runs with the command line options to use the GPU since debugging on GPUs is difficult.
-
All of the Krylov methods except
KSPIBCGS
run on the GPU. We also providePCBICGSTABCUSP
, but it does not appear more efficient thanPCBICGSTAB
. -
The only preconditioners to run directly on the GPU are
PCJACOBI
,PCSACUSP
,PCSACUSPPOLY
, andPCAINVCUSP
. ThePCBJACOBI
andPCASM
are just containers, so if the subsolver runs on the GPU, they can also be considered to run on the GPU.
-
The VecType
-
Some GPU systems (for example many laptops) only run with single
precision; thus, PETSc must be built with the
./configure option --with-precision=single
- We could use your help in further developing PETSc for GPUs; see petsc-dev.