Shared and Dynamic Libraries

The Hopper system can support applications that use dynamic shared libraries (DSL) on the compute nodes.  Some "out-of-the-box" applications require DSLs and some popular applications like Python use DSLs as well.

Using System Shared and Dynamic Libraries

"System" DSLs include those that support software packages found in "typical" Linux distributions, e.g. Python and Perl. To build an application that will use system DSLs on Hopper, you must do three things:

• use the  -dynamic flag in the link step of your build
• set the environment variable  CRAY_ROOTFS to have the value DSL in your job script or before you run
• disable the xt-shmem module with command module unload xt-shem (unless you are making calls to the Cray SHMEM data-passing library)

The -dynamic flag instructs the compiler wrappers (i.e., ftn, cc, CC)  to prepare a dynamically linked executable. 

If your code does not use the Cray SHMEM data-passing library explicitly, and you do not unload the xt-shem module at both the compile time and run time you may receive a run-time error message such as (dmapp_dreg.c:391: _dmappi_dreg_register: Assertion `reg_cache_initialized' failed").

hopper% module unload xt-shmem           #if your code does not make Cray shmem library calls
hopper% ftn -dynamic -o dyn_example.x dyn_example.f90

Alternatively, you can set the environment variable XTPE_LINK_TYPE instead of using the compiler/linker flag.

Setting the environment variable CRAY_ROOTFS value of DSL tells the Cray runtime to make the system shared-object libraries available on the compute nodes.The syntax to set the CRAY_ROOTFS environment variable is

setenv CRAY_ROOTFS DSL #(for csh-like shells: csh, tcsh)
export CRAY_ROOTFS=DSL #(for sh-like shells: bash, sh, ksh)

An example batch script using the bash shell is

#!/bin/bash -l
#PBS -q debug
#PBS -l mppwidth=128
#PBS -l walltime=00:10:00
#PBS -N my_job
#PBS -j oe

cd $PBS_O_WORKDIR
module unload xt-shmem
export CRAY_ROOTFS=DSL

aprun -n 128 ./example.x

Notes:

Normally, the aprun command expects a (compiled) binary executable.  But you can also use aprun to launch a script on the compute nodes.  To execute a script on the compute nodes you need to add the option -a xt in your aprun command line. You would typically do this only if you want  your script to execute on a node or nodes dedicated to your job. An example of starting a python script on a compute node is

aprun -n 1 -a xt ./numpy_ex1.py

Using Your Own Shared and Dynamic Libraries

To build and use your own DSLs

• compile and build libraries with the -shared -fPIC compiler options
• link with the -dynamic flag

An example follows. First, compile several C source files and unload the xt-shmem module if you are not making calls to Cray SHMEM data-passing library routines:

hopper% module unload xt-shmem     
hopper% cc -shared -fPIC source1.c 
hopper% cc -shared -fPIC source2.c

Create a dynamic library, my_c_lib.so, from the C objects:

hopper% cc -fPIC -shared -o my_c_lib.so source1.o source2.o 

Link the dynamic library with a driver routine:

hopper% cc -fPIC -dynamic -o my_dyn_code.x driver.c my_c_lib.so

You can also create a dynamic library from Fortran source.

hopper% ftn -dynamic -fPIC -o my_f_lib.so stuff.f /opt/cray/xt-asyncpe/3.4/bin/ftn: INFO: linux target is being used

To run this code:

• Add a line to your batch script setting the LD_LIBRARY_PATH environment variable to the location of your shared objects. We highly recommend using a directory in $SCRATCH or $SCRATCH2 for performance reasons.
• At runtime, make sure you have loaded the same modules you loaded to build the code.

Here's an example batch script.

#PBS -S /usr/bin/csh
#PBS -q debug
#PBS -l mppwidth=128
#PBS -l walltime=00:10:00
#PBS -N my_job
#PBS -j oe

cd $PBS_O_WORKDIR

setenv CRAY_ROOTFS DSL #If you want the system DSLs

setenv LD_LIBRARY_PATH ${LD_LIBRARY_PATH}:${SCRATCH}/MY_DIRECTORY

module load AAA    #if AAA was used when building the executable

aprun -n 128 ./my_dyn_code.x

relocation truncated to fit: R_X86_64_PC32 against symbol `__hpf_quiet' defined in COMMON section in /opt/pgi/11.2.0/linux86-64/11.2/lib/libpgf90.a(initpar.o)

hopper% file a.out.dyn  
a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV),  
for GNU/Linux 2.6.4, dynamically linked (uses shared libs), not stripped