University:
Boston University
Course:
MET CS 201 | Introduction to Programming
Academic year:

2023
Views:

349

Pages:

20
Author:

Brendon Freeman

Introduction to MPI Outline • Brief overview on parallel computing and MPI • Using MPI on BU SCC • Basic MPI programming • Advanced MPI programming Parallel Computing  Parallel computing is a type of computation in which many computations are carried out simultaneously, based on the principle that large problems can often be divided into smaller ones, which are then solved at the same time.  Speedup of a parallel program, p: number of processors/cores, α: fraction of the program that is serial. • Figure from: https://en.wikipedia.org/wiki/Parallel_computing Distributed and shared memory systems • Shared memory system • Distributed memory system • For example, a single node on a cluster • For example, mutli nodes on a cluster • Open Multi-processing (OpenMP) or MPI • Message Passing Interface (MPI)  Figures from the book Using OpenMP: Portable Shared Memory Parallel Programming MPI Overview  Message Passing Interface (MPI) is a standard for parallel computing on a computer cluster.  MPI is a Library. Provides library routines in C, C++, and Fortran.  Computations are carried out simultaneously by multiple processes.  Data is distributed to multiple processes. There is no shared data.  Data communication between processes is enabled by MPI subroutine/function calls.  Typically each process is mapped to one physical processor to achieve maximum performance.  MPI implementations: • OpenMPI • MPICH, MVAPICH, Intel MPI The first MPI program in C: Hello world! • Hello world in C #include main(int argc, char** argv){ int my_rank, my_size; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &my_rank); MPI_Comm_size(MPI_COMM_WORLD, &my_size); printf("Hello from %d of %d.\n", my_rank, my_size); MPI_Finalize(); } The first MPI program in Fortran: Hello world! • Hello world in Fortran program hello include 'mpif.h' integer my_rank, my_size, errcode call MPI_INIT(errcode) call MPI_COMM_RANK(MPI_COMM_WORLD, my_rank, errcode) call MPI_COMM_SIZE(MPI_COMM_WORLD, my_size, errcode) print *, 'Hello from ', my_rank, 'of', my_size, '.' call MPI_FINALIZE(errcode) end program hello Basic Syntax  Include the header file: mpi.h for C or mpif.h for Fortran  MPI_INIT: This routine must be the first MPI routine you call (it does not have to be the first statement).  MPI_FINALIZE: This is the companion to MPI_Init. It must be the last MPI_Call.  MPI_INIT and MPI_FINALIZE appear in any MPI code.  MPI_COMM_RANK: Returns the rank of the process. This is the only thing that sets each process apart from its companions.  MPI_COMM_SIZE: Returns the total number of processes.  MPI_COMM_WORLD: This is a communicator. Use MPI_COMM_WORLD unless you want to enable communication in complicated patterns.  The error code is returned to the last argument in Fortran, while it is returned to the function value in C. Compile MPI codes on BU SCC  Use GNU compiler (default) and OpenMPI $ export MPI_COMPILER=gnu $ mpicc name.c -o name $ mpif90 name.f90 -o name Use Portland Gourp Inc. (PGI) compiler and OpenMPI $ export MPI_COMPILER=pgi $ mpicc name.c -o name $ mpif90 name.f90 -o name Compile MPI codes on BU SCC (continued)  Use Intel compiler and OpenMPI $ module load openmpi/1.10.1_intel2016 $ mpicc name.c –o name $ mpifort name.f90 –o name  Check what compiler and MPI implementation are in use $ mpicc --show $ mpif90 --show Interactive MPI jobs on BU SCC  Request an interactive session for MPI jobs $ qsh/qrsh/qlogin -pe mpi_12_tasks_per_node 24  Check which nodes and cores are requested $ qstat -u userID -t  Run an MPI executable $ mpirun -np $NSLOTS ./executable  Note: NSLOTS, representing the total number of requested CPU core, is an environmental variable provided by the job scheduler.  Check whether the job really runs in parallel $ top Submit a batch MPI job on BU SCC  Submit a batch job $ qsub job.sh  A typical job script is as following #!/bin/bash #$ -pe mpi_16_tasks_per_node 32 #$ -l h_rt=01:30:00 #$ -P project_name #$ -N job_name mpirun -np $NSLOTS ./executable  Note: No need to provide hostfile explicitly. The job scheduler automatically distributes MPI processes to the requested resources. Exercise 1: hello world 1) Write an MPI hello-world code in either C or Fortran. Print the MPI ranks and size on all processes. 2) Compile the hello-world code. 3) Run the MPI hello-world program either in an interactive session or by submitting a batch job. Analysis of the output $ mpirun -np 4 hello Hello from 1 of 4. Hello from 2 of 4. Hello from 0 of 4. Hello from 3 of 4.  Output is “disordered”. The output order is random. It depends on which process finishes its work first.  In a run on multiple nodes, the output of all nodes are printed on the session of the master node. This indicates implicit dada communication behind the scene. Basic MPI programming  Point-to-point communication: MPI_Send, MPI_Recv  Exercise: Circular shift and ring programs  Synchronization: MPI_Barrier  Collective communication: MPI_Bcast, MPI_Reduce  Exercise: Compute the value of Pi  Exercise: Parallelize Laplace solver using 1D decomposition Point-to-point communication (1): Send  One process sends a message to another process. Syntax: int MPI_Send(void* data, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm)  data: Initial address of send data.  count: Number of elements send (nonnegative integer).  datatype: Datatype of the send data.  dest: Rank of destination(integer).  tag: Message tag (integer).  comm: Communicator. Point-to-point communication (2): Receive  One process receives a matching massage from another process.  Syntax: int MPI_Recv (void* data, int count, MPI_Datatype datatype, int source, int tag, MPI_Comm comm, MPI_Status* status)  data: Initial address of receive data.  count: Maximum number of elements to receive (integer).  datatype: Datatype of receive data.  source: Rank of source (integer).  tag: Message tag (integer).  comm: Communicator (handle).  status: Status object (status). A C example: send and receive a number between two processes int my_rank, numbertoreceive, numbertosend; MPI_Init(&argc, &argv); MPI_Status status; MPI_Comm_rank(MPI_COMM_WORLD, &my_rank); if (my_rank==0){ numbertosend=36; MPI_Send( &numbertosend, 1, MPI_INT, 1, 10, MPI_COMM_WORLD); } else if (my_rank==1){ MPI_Recv( &numbertoreceive, 1, MPI_INT, 0, 10, MPI_COMM_WORLD, &status); printf("Number received is: %d\n", numbertoreceive); } MPI_Finalize(); A Fortran example: send and receive a number between two processes integer my_rank, numbertoreceive, numbertosend, errcode, status(MPI_STATUS_SIZE) call MPI_INIT(errcode) call MPI_COMM_RANK(MPI_COMM_WORLD, my_rank, errcode) if (my_rank.EQ.0) then numbertosend = 36 call MPI_Send( numbertosend, 1,MPI_INTEGER, 1, 10, MPI_COMM_WORLD, errcode) elseif (my_rank.EQ.1) then call MPI_Recv( numbertoreceive, 1, MPI_INTEGER, 0, 10, MPI_COMM_WORLD, status, errcode) print *, 'Number received is:', numbertoreceive endif call MPI_FINALIZE(errcode) What actually happened behind the scene? Operation 1: On process 0, MPI_Send copies the data to Send Queue/Buffer. Operation 2: MPI_Send moves the data from process 0’s Send Queue to process 1’s Receive Queue/Buffer. (The rank of the destination is an input argument of MPI_Send, so it knows where the data should go to.) Operation 3: On process 1, MPI_Recv checks whether the matching data has arrived (Source and tag are checked. But data type and counts are not checked). If not arrive, it waits until the matching data arrives. If arrives, it moves the data from the Receive Queue. This mechanism guarantees that the send data will not be “missed”. Process 1’s memory Process 0’s memory Data A Data A Operation 1 Red regions: save data A. Blue regions: temporarily save data A. Operation 3 Operation 2 Send Queue Send Queue Receive Queue Receive Queue