MoreDream

MPICH-GX: Extension of MPI functionality for the GRID

What is MPICH-GX?

MPICH-GX is a patch of MPICH-G2 to extend functionality required in the Grid. MPICH-G2 is a well-defined implementation of Grid-enabled MPI, but it is needed to be modified for supporting some requirements of Grid applications. MPICH-GX provides following functions.

1) Private IP Support
2) Fault Tolerant Support

Main Features

Basically, MPICH-GX is based on MPICH-G2. Thus, it has similar architecture of MPICH-G2 as shown in Figure 1. It consists of the GX device, a proxy daemon named nproxy, a local job manager called local-job-manager, a package of Java library called CM (Coallocator for MPICH-GX) and client tools based on Java. The GX device modifies ‘globus2 device’ of MPICH to add required functions. The nproxy relays the messages of MPI processes when the process is located in private IP clusters. The local-job-manager invokes a process and monitors the status of the process. The CM is a package of Java library for supporting the middlewares which are executing MPICH-GX applications. Lastly, we bring client tools based on Java. Client tools compose of gxrun as a tool for job submission and gxlrm as a deamon for local resource manager. The gxrun could submit a job through the gxlrm running in each distributed resource.

As shown in Figure 1, application developer can make MPICH-GX applications by using standard MPI API. It can be mapped to the globus2 device passing through ADI layer. We modify the globus2 device, so that some functions in MPICH-GX are different from MPICH-G2.

Figure 1. Architecture of MPICH-GX

Private IP Support and nproxy

It is a well-known problem that MPICH-G2 does not support private IP clusters. In MPICH-G2, processes communicate with each other based on IP addresses. Thus, it cannot support private IP clusters by the nature.

To support private IP clusters, MPICH-GX uses a communication relay scheme combining the NAT service with a user level proxy named ‘nproxy.’ In this approach, only incoming messages are handled by a user-level proxy to relay them into proper nodes inside the cluster, while the outgoing messages are handled by the NAT service at the front-end node of the cluster. Figure 2 show the conceptual diagram of our relay scheme.

(a) Between two private IP clusters

(b) Between a private IP cluster and a public IP cluster

Figure 2. Communication Relay Scheme in MPICH-GX

Fault Tolerant Support

We provide a checkpointing-recovery system for Grids. Our library requires no modifications of application source codes, and it affects the MPICH communication characteristics as less as possible. The features are that it supports the direct message transfer mode and that all of the implementation has been done at the low level, that is, the abstract device level.

Figure 3 describes the library structure of MPICH-GX for fault tolerant support. Although fault-tolerance module has been implemented at the abstract device level, please note that it is still user-level. It contains a checkpoint toolkit, an atomic message transfer management, and a connection re-establishment module. The checkpoint toolkit dumps the user-level memory image into the stable storage on the request. Messages that are in kernel memory or at network are assumed as in-transit messages. The atomicity of message transfer at the abstract device level realizes fine grained checkpoint timing compared to the higher-level approach, because it narrows the code area that should be exclusive against the checkpoint procedure.

Figure 3. The Structure of MPICH-GX Device for Fault Tolerant Support