University:
Boston University
Course:
MET CS 201 | Introduction to Programming
Academic year:

2023
Views:

406

Pages:

19
Author:

Rhianna Osborne

Introduction to MPI Outline • Brief overview of parallel computing and MPI • Basic MPI programming • Intermediate MPI programming Parallel Computing  Parallel computing is a type of computation in which many calculations are carried out simultaneously.  Speedup of a parallel program, p: number of processors (or cores), α: fraction of the program that is serial. • The picture is from: https://en.wikipedia.org/wiki/Parallel_computing Distributed and shared memory systems • Shared memory system • Distributed memory system • For example, a single node on a cluster • For example, multi nodes on a cluster • Open Multi-processing (OpenMP) or MPI • Message Passing Interface (MPI)  Figures are from the book Using OpenMP: Portable Shared Memory Parallel Programming MPI Overview  Message Passing Interface (MPI) is a standard for parallel computing on a computer cluster.  MPI is a Library. Includes routines in C, C++, and Fortran.  Computations are carried out simultaneously by multiple processes.  Data is distributed to multiple processes.  Data communication between processes is enabled by MPI subroutine/function calls.  Typically each process is mapped to one physical processor to achieve maximum performance.  MPI implementations: • OpenMPI • MPICH, MVAPICH, Intel MPI The first MPI program in C: Hello world! • Hello world in C #include main(int argc, char** argv){ int my_rank, my_size; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &my_rank); MPI_Comm_size(MPI_COMM_WORLD, &my_size); printf("Hello from %d of %d.\n", my_rank, my_size); MPI_Finalize(); } The first MPI program in Fortran: Hello world! • Hello world in Fortran program hello include 'mpif.h' integer my_rank, my_size, errcode call MPI_INIT(errcode) call MPI_COMM_RANK(MPI_COMM_WORLD, my_rank, errcode) call MPI_COMM_SIZE(MPI_COMM_WORLD, my_size, errcode) print *, 'Hello from ', my_rank, 'of', my_size, '.' call MPI_FINALIZE(errcode) end program hello Basic Syntax  Include the header file: mpi.h for C or mpif.h for Fortran  MPI_INIT: This routine must be the first MPI routine you call (it does not have to be the first statement).  MPI_FINALIZE: This is the companion to MPI_Init. It must be the last MPI call.  MPI_INIT and MPI_FINALIZE appear in any MPI program.  MPI_COMM_RANK: Returns the rank of the process. This is the only thing that sets each process apart from its companions.  MPI_COMM_SIZE: Returns the total number of processes.  MPI_COMM_WORLD: This is a communicator. Use MPI_COMM_WORLD unless you want to enable communication in complicated patterns.  The error code is returned to the last argument in Fortran, while it is returned to the function value in C. Exercise 1: hello world 1) Write an MPI hello-world code in either C or Fortran. Print the MPI ranks and size on all processes. 2) Compile the hello-world code. 3) Run the MPI hello-world program either in an interactive session or by submitting a batch job. Analysis of the output $ mpirun -np 4 ./hello Hello from 1 of 4. Hello from 2 of 4. Hello from 0 of 4. Hello from 3 of 4.  The MPI rank and size is printed by every process.  Output is “disordered”. The output order is random.  The output of all processes are printed on the session of the master process. Basic MPI programming  Point-to-point communication: MPI_Send, MPI_Recv  Exercise: Circular shift and ring programs  Synchronization: MPI_Barrier  Collective communication: MPI_Bcast, MPI_Reduce  Exercise: Compute the value of Pi  Exercise: Parallelize Laplace solver using 1D decomposition Point-to-point communication (1): Send  One process sends a message to another process.  Syntax: int MPI_Send(void* data, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm)  data: Initial address of send data.  count: Number of elements send (nonnegative integer).  datatype: Datatype of the send data.  dest: Rank of destination(integer).  tag: Message tag (integer).  comm: Communicator. Point-to-point communication (2): Receive  One process receives a matching massage from another process.  Syntax: int MPI_Recv (void* data, int count, MPI_Datatype datatype, int source, int tag, MPI_Comm comm, MPI_Status* status)  data: Initial address of receive data.  count: Maximum number of elements to receive (integer).  datatype: Datatype of receive data.  source: Rank of source (integer).  tag: Message tag (integer).  comm: Communicator (handle).  status: Status object (status). A C example: send and receive a number between two processes int my_rank, numbertoreceive, numbertosend; MPI_Init(&argc, &argv); MPI_Status status; MPI_Comm_rank(MPI_COMM_WORLD, &my_rank); if (my_rank==0){ numbertosend=36; MPI_Send( &numbertosend, 1, MPI_INT, 1, 10, MPI_COMM_WORLD); } else if (my_rank==1){ MPI_Recv( &numbertoreceive, 1, MPI_INT, 0, 10, MPI_COMM_WORLD, &status); printf("Number received is: %d\n", numbertoreceive); } MPI_Finalize(); A Fortran example: send and receive a number between two processes integer my_rank, numbertoreceive, numbertosend, errcode, status(MPI_STATUS_SIZE) call MPI_INIT(errcode) call MPI_COMM_RANK(MPI_COMM_WORLD, my_rank, errcode) if (my_rank.EQ.0) then numbertosend = 36 call MPI_Send( numbertosend, 1,MPI_INTEGER, 1, 10, MPI_COMM_WORLD, errcode) elseif (my_rank.EQ.1) then call MPI_Recv( numbertoreceive, 1, MPI_INTEGER, 0, 10, MPI_COMM_WORLD, status, errcode) print *, 'Number received is:', numbertoreceive endif call MPI_FINALIZE(errcode) What actually happened behind the scene? Operation 1: On process 0, MPI_Send copies the data to Send Queue/Buffer. Operation 2: MPI_Send moves the data from process 0’s Send Queue to process 1’s Receive Queue/Buffer. (The rank of the destination is an input argument of MPI_Send, so it knows where the data should go to.) Operation 3: On process 1, MPI_Recv checks whether the matching data has arrived (Source and tag are checked. But data type and counts are not checked). If not arrived, it waits until the matching data arrives. If arrives, it moves the data from the Receive Queue to process 1’s memory. This mechanism guarantees that the send data will not be “missed”. Process 1’s memory Process 0’s memory Data A Data A Operation 1 Red regions: save data. Blue regions: temporarily save data. Operation 3 Operation 2 Send Queue Send Queue Receive Queue Receive Queue Blocking Receives and Sends  MPI_Recv is always blocking.  Blocking means the function call will not return until the receive is completed.  It is safe to use the received data right after calling MPI_Recv.  MPI_Send try not to block, but don’t guarantee it.  If the size of send data is smaller than that of the Send Queue, MPI_Send is not blocking --the data is sent to the Receive Queue without waiting.  But if the size of send data is larger than that of the Send Queue, MPI_Send is blocking --- it first sends a chunk of data, then stops sending when the Send Queue is full and will restart sending when the Send Queue becomes empty again (for example when the chunk of data has been moved to the Receive Queue).  The later case often happens, so it is OK to think that MPI_Send is blocking. A deadlock due to blocking receives  An example: swapping arrays between two processes.  The following code meets a deadlock situation and will hang forever.  Both processes are blocked at MPI_Recv no matter how large the data size is. int n=10; // a small data size int my_rank, n_send1[n], n_send2[n], n_recv1[n], n_recv2[n]; MPI_Comm_rank(MPI_COMM_WORLD, &my_rank); if (my_rank==0){ MPI_Recv( &n_recv2, n, MPI_INT, 1, 11, MPI_COMM_WORLD, NULL); MPI_Send( &n_send1, n, MPI_INT, 1, 10, MPI_COMM_WORLD); } else if (my_rank==1){ MPI_Recv( &n_recv1, n, MPI_INT, 0, 10, MPI_COMM_WORLD, NULL); MPI_Send( &n_send2, n, MPI_INT, 0, 11, MPI_COMM_WORLD); } A deadlock due to blocking sends  If the sizes of the send arrays are large enough, MPI_Send becomes blocking, then the following code meets a deadlock situation.  Both processes are blocked at MPI_Send for a large data size. int n=5000; // a large data size int my_rank, n_send1[n], n_send2[n], n_recv1[n], n_recv2[n]; MPI_Comm_rank(MPI_COMM_WORLD, &my_rank); if (my_rank==0){ MPI_Send( &n_send1, n, MPI_INT, 1, 10, MPI_COMM_WORLD); MPI_Recv( &n_recv2, n, MPI_INT, 1, 11, MPI_COMM_WORLD, NULL); } else if (my_rank==1){ MPI_Send( &n_send2, n, MPI_INT, 0, 11, MPI_COMM_WORLD); MPI_Recv( &n_recv1, n, MPI_INT, 0, 10, MPI_COMM_WORLD, NULL); }