Event Tracing of the Earth System Modeling Framework with VampirTrace
Coupled Earth System Models exhibit architectural diversity. There has been very little written about the software engineering advantages of the different architectures and, to my knowledge, no one has performed a comprehensive comparison of ESM architectures. Toward this end, I have recently had a very good experience with a program for tracing/profiling high performance software called VampirTrace and a related product called Vampir for visualizing and analyzing traces. In this post, I will describe how to automatically add tracing instrumentation to Earth System Modeling Framework applications and show some sample analyses using the Vampir tool.
VampirTrace automatically instruments C, C++, and Fortran programs so that important events are recorded as the program executes. VampirTrace is designed to work with high performance applications, such as distributed memory parallel codes based on MPI. This works well for us since the vast majority of ESMs are based on this paradigm. When a program compiled with the VampirTrace compiler wrappers is executed, certain events will be recorded and, after program execution, the trace data will be available in a standardized format called Open Trace Format (OTF).
Lots of different events can be traced with fine grained timing data:
- MPI events such as message send, receive, and collective calls
- Function entry and exit
- System calls such as memory allocations and I/O calls
- Other user-defined events (requires manual code instrumentation)
A major advantage of using ESMF is that it provides a number of numerical modeling abstractions that would otherwise have to be coded by hand. Many of these abstractions hide the details of rather complex interactions, such as repartitioning field data from one decomposition (processor layout) to another and interpolating field data between grids with different resolutions. Because we are interested in what is going on within ESMF (not just within the user code that calls ESMF functions), I have instrumented ESMF itself using VampirTrace.
Instrumenting the ESMF library using VampirTrace
These are the software packages I am using:
- ESMF 5.2.0r
- Intel Fortran and C compilers (Composer XE 2011 SP1.6.233)
- OpenMPI 1.4.3
- NetCDF 4.1.3 (only required for I/O operations)
- VampirTrace 5.11.2
Although VampirTrace ships with the OpenMPI distribution, it seems that the version of VampirTrace packaged with OpenMPI lags behind the latest stable release. So, I use the latest version of VampirTrace and ignore the version packaged with OpenMPI. VampirTrace uses compiler wrappers for automatic code instrumentation. So, the ESMF build environment must be set up to use the VampirTrace wrappers instead of the usual C and Fortran compilers. The Intel compilers, OpenMPI, NetCDF, and VampirTrace packages are all set up using the standard build/install instructions that come with those packages.
Here are my ESMF environment variables:
export ESMF_COMM="openmpi" export ESMF_COMPILER="intel" export ESMF_DIR="<some directory>/esmf" export ESMF_CXX=vtcxx export ESMF_F90=vtf90 export ESMF_CXXCOMPILEOPTS="-vt:cxx mpicxx -DVTRACE" export ESMF_F90COMPILEOPTS="-vt:f90 mpif90 -DVTRACE"
With these environment variables, ESMF will be compiled using the VampirTrace wrappers vtcxx and vtf90 instead of the OpenMPI compiler wrappers mpicxx and mpif90. Make sure that VampirTrace, OpenMPI, and Intel compilers are all in the PATH or specify absolute locations for ESMF_CXX and ESMF_F90.
Now build ESMF and install it.
$ gmake $ gmake install
I do not recommend executing gmake check because all of the ESMF unit and system tests will be executed with tracing turned on. If you wish to check the build, set up your environment using the OpenMPI compiler wrappers, execute gmake to recompile and then run gmake check. If everything checks, then rebuild with the VampirTrace compiler wrappers. The only difference will be that ESMF is linked against the VampirTrace library instead of the regular MPI library. So, we’ll assume that the unit and system tests would pass in this case. If they did not, the bug would be within VampirTrace itself.
Okay, now that we have an ESMF build linked with the VampirTrace libraries, let’s run a small ESMF application and take a look at the resulting trace data. The easiest way to do this is to execute one or more of the system tests that ship with ESMF. For example:
$ cd $ESMF_DIR/system_tests/ESMF_ArrayRedist $ gmake $ cd $ESMF_DIR/test/testO/Linux.intel.64.openmpi.default $ mpirun -np 6 ./ESMF_ArrayRedistSTest
A couple notes:
- Line 2 will compile the ESMF_ArrayRedist system test. It will probably fail with an error File not found: ‘opari.tab.c’. To get around this, I’ve had to execute opari -table opari.tab.c manually. I’m not sure why it is not working automatically.
- The directory location in line 3 will change depending on the os, compiler, and MPI implementation you are using.
Line 4 will execute the instrumented system test. When it finishes, you should see several files produced in the same directory with the ESMF_ArrayRedistSTest executable: a series of files ESMF_ArrayRedistSTest.X.events.z (one for each of the six MPI processes), a file named ESMF_ArrayRedistSTest.0.def.z, and a file name ESMF_ArrayRedistSTest.otf. This set of files contains the trace data for the run.
Analysis with Vampir
Vampir is an analysis and visualization package capable of reading trace files in OTF format. It is not open source, but a demo license is available. Let’s examine the trace files from the array redistribution system test.
- Install Vampir (download here)
- Start Vampir
- Open the file ESMF_ArrayRedistSTest.otf that was generated from the system test run
In this example I have modified the system test to perform the same array redistribution 1000 times. This is a one-way repartitioning, so messages are sent from processes 0-3 and received by processes 4 and 5.
For now, this just gives an idea of the kinds of analyses supported by VampirTrace + Vampir. I have been impressed by the initial experience with the software. Potential issues may arise as the number of processes increases and the overall length of the run (this run took 7 seconds wall clock time), but so far so good.