Introduction to MPI Collective Communication - MPI Scatter

Collective communication: MPI_Scatter  The root process sends chunks of an array to all processes. Each non-root process receives a chunk of the array and stores it in receive buffer. The root process also copies a chunk of the array to its own receive buffer.  Syntax: int MPI_Scatter (const void* send_data, int send_count, MPI_Datatype send_datatype, void* recv_data, int recv_count, MPI_Datatype recv_datatype, int root, MPI_Comm comm)  send_data: The send array that originally resides on the root process.  send_count: Number of elements to be sent to each process (i.e. the chunk size). It is often (approximately) equal to the size of the array divided by the number of processes.  send_datatype: Datatype of the send data.  recv_data: The receive buffer on all processes.  recv_count: Number of elements that the receive buffer can hold (i.e. the chunk size). It should be equal to send_count if send_datatype and recv_datatype are the same.  recv_datatype: Datatype of the receive data.  root: The rank of the root process. Collective communication: MPI_Gather  MPI_Gather is the inverse of MPI_Scatter.  Each non-root process sends a chunk of data to the root process. The root process receives chunks of data and stores them (including its own chunk) in the receive buffer in the order of MPI ranks.  Syntax: int MPI_Gather (const void* send_data, int send_count, MPI_Datatype send_datatype, void* recv_data, int recv_count, MPI_Datatype recv_datatype, int root, MPI_Comm comm)  send_data: The send data on each process.  send_count: Number of elements of the send data (i.e. the chunk size).  send_datatype: Datatype of the send data.  recv_data: The receive buffer on the root process.  recv_count: Number of elements of the receive data (i.e. the chunk size, not the size of the receive buffer) . It should be equal to send_count if send_datatype and recv_datatype are the same.  recv_datatype: Datatype of the receive data.  root: The rank of the root process. An example for MPI_Scatter and MPI_Gather  Compute the average of all elements in an array. int rank, nproc, i, m, n=100; double sub_avg=0., global_avg=0.; double * array = NULL, * sub_avgs = NULL; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &nproc); m = (int) n/nproc; // chunk size if(rank==0){ array = (double *) malloc(n*sizeof(double)); for(i=0; i< n; i++) array[i]=(double) i; } double * chunk = (double *) malloc(m*sizeof(double)); MPI_Scatter(array, m, MPI_DOUBLE, chunk, m, MPI_DOUBLE, 0, MPI_COMM_WORLD); for(i=0; i< m; i++) sub_avg += chunk[i]; sub_avg = sub_avg/(double)m; MPI_Barrier(MPI_COMM_WORLD); if(rank==0) sub_avgs = (double *) malloc(nproc*sizeof(double)); MPI_Gather(&sub_avg, 1, MPI_DOUBLE, sub_avgs, 1, MPI_DOUBLE, 0, MPI_COMM_WORLD); if(rank==0){ for(i=0; i< nproc; i++) global_avg += sub_avgs[i]; printf("The global average is: %f\n", global_avg/(double)nproc); } free(chunk); if(rank==0){ free(array); free(sub_avgs); } Collective communication: Allreduce, Allgather  MPI_Allreduce is the equivalent of doing MPI_Reduce followed by an MPI_Bcast. The root process obtains the reduced value and broadcasts it to all other processes.  MPI_Allgather is the equivalent of doing MPI_Gather followed by an MPI_Bcast. The root process gathers the values and broadcasts them to all other processes.  Syntax: int MPI_Allreduce (const void* send_data, void* recv_data, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) int MPI_Allgather (const void* send_data, int send_count, MPI_Datatype send_datatype, void* recv_data, int recv_count, MPI_Datatype recv_datatype, MPI_Comm comm) Quiz  What is the result of the following code on 4 processes?  Hints: Break down the code using MPI_Send and MPI_Recv, then analyze how the program steps forward. MPI_Comm_rank(MPI_COMM_WORLD, &rank); if (rank % 2 == 0) { // Even MPI_Allreduce(&rank, &evensum, 1, MPI_INT, MPI_SUM, MPI_COMM_WORLD); if (rank == 0) printf("evensum = %d\n", evensum); } else { // Odd MPI_Allreduce(&rank, &oddsum, 1, MPI_INT, MPI_SUM, MPI_COMM_WORLD); if (rank == 1) printf("oddsum = %d\n", oddsum); } a) evensum=2 oddsum=4 c) evensum=6 oddsum=6 b) evensum=6 oddsum=0 d) evensum=0 oddsum=0 Basic Datatypes  Basic datatype for C:  Basic datatype for Fortran:  MPI_CHAR --- Signed char  MPI_INTEGER --- INTEGER  MPI_SHORT --- Signed short int  MPI_REAL --- REAL  MPI_INT --- Signed int  MPI_LONG --- Signed long int  MPI_UNSIGNED_CHAR --- Unsigned char  MPI_UNSIGNED_SHORT --- Unsigned short  MPI_REAL8 --- REAL*8  MPI_DOUBLE_PRECISION --- DOUBLE PRECISION  MPI_COMPLEX --- COMPLEX  MPI_UNSIGNED --- Unsigned int  MPI_LOGICAL --- LOGICAL  MPI_UNSIGNED_LONG --- Unsigned long int  MPI_CHARACTER --- CHARACTER(1)  MPI_FLOAT --- Float  MPI_DOUBLE --- Double  MPI_LONG_DOUBLE --- Long double Derived Datatype  Derived datatype: for users to define a new datatype that is derived from old datatype(s).  Why derived datatype?  Noncontiguous messages  Convenience for programming  Possible better performance and less data movements  Declare and commit a new datatype:  MPI_Datatype typename : declare a new datatype  MPI_Type_commit(&typename): commit the new datatype before use it. Illustration of contiguous, vector, indexed and struct datatypes  Contiguous:  Vector: non-contiguous, fixed block length and stride.  Indexed: different block lengths and strides.  Struct: different block lengths, strides and old datatypes. Contiguous datatype  Allows replication of an old data type into contiguous locations.  For contiguous data.  Syntax: int MPI_Type_contiguous (int count, MPI_Datatype oldtype, MPI_Datatype * newtype)  count: replication count (nonnegative integer)  oldtype: old data type  newtype: new data type Vector datatype  Allows replication of an old datatype into locations that consist of equally spaced blocks.  Each block is a concatenation of the old datatype.  The block length and the stride are fixed.  Syntax: int MPI_Type_vector ( int count, int blocklength, int stride, MPI_Datatype oldtype, MPI_Datatype * newtype)  count: number of blocks (nonnegative integer)  blocklength: number of elements in each block (nonnegative integer)  stride: number of elements between start of each block (integer)  oldtype: old data type  newtype: new data type Indexed datatype  Allows replication of an old datatype into a sequence of blocks.  The block lengths and the strides may be different.  Syntax: int MPI_Type_indexed ( int count, const int * blocklength, const int * displacements, MPI_Datatype oldtype, MPI_Datatype * newtype)  count: number of blocks (nonnegative integer)  blocklength: number of elements in each block (array of nonnegative integer)  displacements: displacement of each block in multiples of oldtype (array of integer)  oldtype: old data type  newtype: new data type A C example for contiguous, vector and indexed datatypes int n=18; int blocklen[3] = {2, 5, 3 }, disp[3] = { 0, 5, 15 }; MPI_Datatype type1, type2, type3; MPI_Type_contiguous(n, MPI_INT, &type1); MPI_Type_commit(&type1); MPI_Type_vector(3, 4, 7, MPI_INT, &type2); MPI_Type_commit(&type2); MPI_Type_indexed(3, blocklen, disp, MPI_INT, &type3); MPI_Type_commit(&type3); if (rank == 0){ for (i=0; i< n; i++) buffer[i] = i+1; MPI_Send(buffer, 1, type1, 1, 101, MPI_COMM_WORLD); MPI_Send(buffer, 1, type2, 1, 102, MPI_COMM_WORLD); MPI_Send(buffer, 1, type3, 1, 103, MPI_COMM_WORLD); } else if (rank == 1) { MPI_Recv(buffer1, 1, type1, 0, 101, MPI_COMM_WORLD, &status); MPI_Recv(buffer2, 1, type2, 0, 102, MPI_COMM_WORLD, &status); MPI_Recv(buffer3, 1, type3, 0, 103, MPI_COMM_WORLD, &status); } Struct datatype  Allows each block to consist of replications of different datatypes.  The block lengths, the strides and the old datatypes may be different.  Give users full control to pack data.  Syntax: int MPI_Type_struct ( int count, const int * blocklength, const MPI_Aint * displacements, MPI_Datatype oldtype, MPI_Datatype * newtype)  count: number of blocks (nonnegative integer)  blocklength: number of elements in each block (array of nonnegative integer)  displacements: displacement of each block in multiples of bytes (array of integer)  oldtype: old data type  newtype: new data type Pack size  Returns the upper bound on the amount of space needed to pack a message.  Syntax: int MPI_Pack_size ( int incount, MPI_Datatype datatype, MPI_Comm comm, int *size)  count: Count argument to packing call (integer)  datatype: Datatype argument to packing call  comm: Communicator  size: Upper bound on size of packed message, in unit of bytes (integer) A C example for struct datatype int psize; int blocklens[3] = { 2, 5, 3 }; MPI_Aint disp[3] = { 0, 5*sizeof(int), 5*sizeof(int)+10*sizeof(double) }; MPI_Datatype oldtypes[3], newtype; oldtypes[0] = MPI_INT; oldtypes[1] = MPI_DOUBLE; oldtypes[2] = MPI_CHAR; MPI_Type_struct( 3, blocklens, disp, oldtypes, &newtype ); MPI_Type_commit( &newtype ); MPI_Pack_size( 1, newtype, MPI_COMM_WORLD, &psize ); printf("pack size = %d\n", psize); Exercise 5: Laplace Solver (version 2)  Rewrite an MPI program to solve the Laplace equation based on 2D decomposition.  Analysis: 1. Decompose the grids into sub-grids. Divide both rows and columns. Each process owns one sub-grid. 2. Define necessary derived datatypes (e.g. MPI_contiguous and MPI_vector). 3. Pass necessary data between processes. (e.g. use MPI_Send and MPI_Recv). Be careful to avoid dead locks. 4. Pass “shared” data between the root process and all other processes (e.g. use MPI_Bcast and MPI_Reduce). What is not covered ……  Communicator and topology  Single-sided Communications  Remote Memory Access  Hybrid Programming: MPI + OpenMP, MPI + OpenACC, MPI + CUDA, ……  MPI-based libraries  MPI I/O  MPI with other languages: python, perl, R, ……