Rolf Rabenseifner, Peter Gottschling, Wolfgang E. Nagel, and Stephan Seidl

Effective Performance Problem Detection of MPI Programs on MPP Systems: From the Global View to the Details

This paper presents an automatic counter instrumentation and profiling module added to the MPI library on Cray T3E systems. A detailed summary of the hardware performance counters and the MPI calls of any MPI production program is gathered during execution and written on a special syslog file. The user can get the same information on a different file. Statistical summaries are computed weekly and monthly. The paper describes experiences with this library on the Cray T3E systems at HLRS Stuttgart and TU Dresden. It focuses on the scalability aspects of the new interface: How to obtain the right amount of performance data to the right person in time, and how to draw conclusions for the further optimization process, e.g. with the trace-based profiling tool Vampir.

Stephan K.H. Seidl