MPI Forum Tools Working Group Details

Function Pointer Interception (QMPI) MPI Forum Tools Working Group September 2021 MPI Forum Meeting

Current State of the Art Name-shifted interface (PMPI) Relatively simple to use Supports a single tool at a time without resorting to non-standard workarounds (PnMPI) Tools implement their own versions of functions and intercept MPI calls with tricks like weak symbols Calls PMPI_<foo> when done

Desired Features Support for multiple, simultaneous tools Support for multiple copies of the same tool (e.g., one instance for rows and another for columns) A way for users to specify the set and order of tools Intercept all functions provided by an MPI library (including non- standard functions Interoperability with existing PMPI tools

Desired Features Support for multiple, simultaneous tools Support for multiple copies of the same tool (e.g., one instance for rows and another for columns) A way for users to specify the set and order of tools Intercept all functions provided by an MPI library (including non- standard functions Interoperability with existing PMPI tools QMPI!

QMPI Allows tools to insert themselves between the application and the MPI implementation Allows multiple tools to be used simultaneously Single tool in PMPI Useful to layer complementary tools at the same time. Replacement for PMPI Can co-exist with PMPI Regular MPI Calls Application MPI_Send MPI Library MPI Calls with PMPI PMPI Tool Application MPI Library PMPI_Send MPI_Send MPI Calls with PMPI & QMPI PMPI Tool QMPI Tool QMPI Tool MPI Library Application MPI_Send

Tool Life Cycle Tool Life Cycle Tool Specification QMPI_TOOL_LIST= foo,bar,foo mpiexec n 2 ./bar The user starts an MPI application Provides an environment variable to list desired QMPI tools Tools are uniquely identified by a string name Can have multiple copies of the same tool Implementations could potentially choose a different mechanism to identify tools The application still needs to be linked with QMPI libraries The application could also still be linked with a PMPI library Could also use any other tricks that are already used with PMPI (e.g., LD_PRELOAD) QMPI Tool Registration Application main() function QMPI Tool Initialization MPI QMPI Interception PMPI Interception Implementation

Tool Life Cycle Tool Life Cycle QMPI Tool Registration QMPI_TOOL_LIST= foo,bar,foo mpiexec n 2 ./bar The tool registers itself with a system-specific mechanism For example, adding the __constructor__ attribute works with most compilers and will call the function before main() The tool calls MPI_REGISTER_TOOL_NAME to tell MPI its own name (to be matched with the user string later) Provides a callback function in case the user asks for the tool to be loaded Must happen before any other MPI function is called QMPI Tool Registration Application main() function QMPI Tool Initialization MPI QMPI Interception PMPI Interception Implementation

Tool Life Cycle Tool Life Cycle Application Start QMPI_TOOL_LIST= foo,bar,foo mpiexec n 2 ./bar The application begins as normal No code changes are necessary to switch from PMPI to QMPI The next interesting thing happens when the application makes its first MPI call... QMPI Tool Registration Application main() function QMPI Tool Initialization MPI QMPI Interception PMPI Interception Implementation

Tool Life Cycle Tool Life Cycle QMPI Tool Initialization QMPI_TOOL_LIST= foo,bar,foo mpiexec n 2 ./bar No later than when the user makes the first MPI call... MPI calls the callback functions of all of the tools requested by the user in the order the user requested. Provides each tool a unique ID (int) The tool registers callback functions for each MPI(X) function it wants to intercept. The tool registers a storage location. Can stash instance-specific information in here. MPI keeps track of the ordering via the ID provided to the tool. QMPI Tool Registration Application main() function QMPI Tool Initialization MPI QMPI Interception PMPI Interception Implementation

Tool Life Cycle Tool Life Cycle PMPI Tool Interception QMPI_TOOL_LIST= foo,bar,foo mpiexec n 2 ./bar A PMPI tool (if one is linked in) is called before any QMPI tools are called for each MPI function. No code changes are required for PMPI tools The PMPI tool will still call the PMPI symbol when it is done The PMPI symbol will start the chain of QMPI tools QMPI Tool Registration Application main() function QMPI Tool Initialization MPI QMPI Interception PMPI Interception Implementation

Tool Life Cycle Tool Life Cycle QMPI Tool Interception QMPI_TOOL_LIST= foo,bar,foo mpiexec n 2 ./bar The first QMPI tool will be called by MPI The tool will be provided its context object and tool_id each time it is called Each tool can do whatever it wants internally, but any time it wants to call an MPI function, it needs to get the function pointer from MPI. This could be costly over time, so stashing the next pointer for each MPI function is advised. QMPI Tool Registration Application main() function QMPI Tool Initialization MPI QMPI Interception PMPI Interception Implementation

Tool Life Cycle Tool Life Cycle MPI Implementation QMPI_TOOL_LIST= foo,bar,foo mpiexec n 2 ./bar The function pointer given to the last tool will point back into the MPI implementation. The implementation fulfils its specification as usual and returns back up the stack when done. The QMPI/PMPI tools will be able to do any post-MPI-call activity as usual and will return back to the previous tool until control returns to the application. QMPI Tool Registration Application main() function QMPI Tool Initialization MPI QMPI Interception PMPI Interception Implementation

Registration API int MPI_Register_tool_name(char *tool_name, MPI_Tool_init_function *init_fn_ptr) Registers the tool s name and function pointer to be called to initialize (below) typedef void MPI_Tool_init_function(int tool_id); int MPI_Register_tool_storage(int tool_id, void *tool_storage) Register the tool s storage address for instance-specific information. int MPI_Register_tool_function(int tool_id, enum MPI_Functions_enum function_enum, void (*function_ptr) (void)) Register callback functions for each MPI function to intercept Uses an enum (specified in mpi.h) to list all MPI functions Function pointer matches the corresponding MPI function adding QMPI_Context context, int tool_id at the beginning.

Interception API int MPI_Get_next_tool_function(int tool_id, enum MPI_Functions_enum function_enum, void (*function_ptr) (void), int *next_tool_id) Gets the next function in the tool chain int MPI_Get_tool_storage(MPI_Context context, int tool_id, void *storage) Gets the tool s storage location out of the context object int MPI_Get_calling_address(MPI_Context context, void *address) Gets the address where the QMPI function will return after all of the tools are done. This isn t interoperable with PMPI. If both types of tools are used, this will return the address of the PMPI tool, not the application.

Example Usage Tool Registration and Initialization __attribute__((constructor)) void register_my_tool() { MPI_Register_tool_name("example", &example_init_function_pointer); } void example_init_function_pointer(int tool_id) { struct tool_struct * my_tool_struct = calloc(0, sizeof(struct tool_struct)); my_tool_struct->my_tool_id = tool_id; MPI_Register_tool_storage(tool_id, my_tool_struct); MPI_Register_function(tool_id, MPI_SEND_T, (void (*)(void)) &Example_Send); MPI_Register_function(tool_id, MPI_RECV_T, (void (*)(void)) &Example_Recv); }

1. Uses system-specific mechanism to call function 2. Register callback function to initialize Example Usage Tool Registration and Initialization __attribute__((constructor)) void register_my_tool() { MPI_Register_tool_name("example", &example_init_function_pointer); } void example_init_function_pointer(int tool_id) { struct tool_struct * my_tool_struct = calloc(0, sizeof(struct tool_struct)); 3. Construct tool- specific storage my_tool_struct->my_tool_id = tool_id; MPI_Register_tool_storage(tool_id, my_tool_struct); MPI_Register_function(tool_id, MPI_SEND_T, (void (*)(void)) &Example_Send); MPI_Register_function(tool_id, MPI_RECV_T, (void (*)(void)) &Example_Recv); } 5. Register each function the tool will intercept 4. Register the tool storage

Example Usage Application Tool Selection QMPI_TOOL_LIST= a,b,c,b Each tool will register a name and the environment variable will determine order. Tool registration and initialization will be separated into two steps

Example Usage Tool Interception (Send) int Example_Send(void *context, int tool_id, ...) { MPI_Send_t *next_fn; /* ...do things... */ MPI_Get_function(tool_id, MPI_SEND_T, (void (**)(void)) &next_fn, &next_tool_id); return (*next_fn)(context, next_tool_id, ...); }

Example Usage Tool Interception (Send) int Example_Send(void *context, int tool_id, ...) { 1. Get the next tool in the QMPI chain MPI_Send_t *next_fn; /* ...do things... */ MPI_Get_function(tool_id, MPI_SEND_T, (void (**)(void)) &next_fn, &next_tool_id); return (*next_fn)(context, next_tool_id, ...); } 2. Call the next function and return the error code back up the chain

Current Status Proposal text has gone through a first pass in the Tools Working Group: https://github.com/mpiwg-tools/mpi-standard/pull/10 Lots of internal notes to integrate into a second version of the text Example implementation is available in MPICH main branch Functions are prefixed with QMPI instead of MPI to avoid collisions and reduce confusion. Implementation has essentially zero overhead when running microbenchmarks like OSU bandwidth.