Introduction to Matlab Parallel Computing Tooblox Pt 3

Array Distributions  Objective is to distribute data, hence workload, to workers for shorter walkclock time.  Data distribution utilities: • – Data distributed from client and access readily on client. Data always distributed along the last dimension • – Data distributed within • Distribute along arbitrary dimension • Distribute in parallel • Distributed array created directly (locally) on worker • No communication required to distribute (from client) • Non-distributed variables and arrays created within belongs to this class Spring 2012 25 How To Distribute Arrays Ways to distribute data:  Partitioning a larger array Often, this larger array already exist on Client. Distributing it to labs will take up further memory as well as run time.  Building from smaller arrays Array sections are constructed directly on individual labs resulting in smaller memory footprint.  Created with MATLAB constructor function (rand, zeros, . . .) Supported by and Spring 2012 26 Array Distribution Usage Examples ' Spring 2012 ' 27 Array Distribution Usage Examples (2) >> whos Name A B N aa ab ad al b br n Spring 2012 Size Bytes 8x8 8x8 1x1 1x8 8x8 8x8 1x8 8x8 8x8 1x8 512 512 8 2041 2077 2077 2041 2077 2077 2041 Class Attributes double double double Composite distributed distributed Composite distributed distributed Composite 28 Data Parallel Example – Matri Multipl Computations of distributed arrays need not be in an spmd region. The operations, however, are governed by spmd paradigm. >> >> >> >> >> >> >> >> >> matlabpool open 4 n = 3000; A = rand(n); C = A * B; % maxNumCompThreads(1);% C1 = A * B; % a = distributed(A); % b = distributed(B); % c = a * b; % matlabpool close C1 = A * B (1 thread) B = rand(n); run with 4 threads set threads to 1 run on single thread distributes A, B along columns on client a, b on workers; accessible from client run on workers; c is distributed C=A*B (4 threads) 12.06 a = distribute(A) b = distribute(B) 3.25 c=a*b (4 workers) 2.15 3.89 • The cost for distributing the matrices is recorded separately as this is incurred only once over the life of the distributed matrices. • Time required to distribute matrices is not significantly affected by matrix size. Spring 2012 29 Additional Ways to Distribute Matrices More efficient to distribute arrays within spmd:  All workers participate in distribution (co-distribute)  No need to decompose array on client, then copy to individual workers as with distributed. >> matlabpool open 4 >> A = rand(3000); B = rand(3000); >> spmd p = rand(n, codistributor1d(1)); % 2 ways to directly create q = codistributed.rand(n); % distributes rand array by col. s = p * q; % run on workers; s is codistributed % distribute matrix after it is created u = codistributed(A, codistributor1d(1)); % by row v = codistributed(B, codistributor1d(2)); % by column w = u * v; % run on workers; w is distributed end >> matlabpool close Spring 2012 30 A Case Of vs % First, consider the serial code N = 40; B = rand(N); C = rand(N); p = ones(N,1); v = p; a = rand(N,1); for i = 1:1000 % iterative loop for j=1:N vn(j)=foo(v,p(j),a,B(j,:),C(j,:)); % compute new v end if max(abs(vn-v))< 1e-7 % check for convergence fprintf('****** Converged at iteration %5d\n',i); break; end v = vn; % update v end Spring 2012 31 The version matlabpool open 4 N = 40; B = rand(N); C = rand(N); p = ones(N,1); v = p; a = rand(N,1); for i = 1:1000 % iterative loop parfor j=1:N vn(j)=foo(v,p(j),a,B(j,:),C(j,:)); % compute new v end if max(abs(vn-v))< 1e-7 % check for convergence fprintf('****** Converged at iteration %5d\n',i); break; end v = vn; % update v end matlabpool close Parfor and other parallel paradigms have high overhead cost to start. Avoid using parfor within an iterative loop. Spring 2012 32 version matlabpool open 4 N = 40; spmd % Single Program Multiple Data parallel paradigm B=codistributed.rand(N,codistributor('1d',1)); C=codistributed.rand(N,codistributor('1d',1)); p=codistributed.ones(N,1,codistributor()); v = p; % initial guess; v is distributed by virtue of p for i = 1:1000 % iterative loop vn=foo(v,p,a,B,C); % compute new v if max(abs(vn-v))< 1e-7 % check for convergence printf('****** Converged at iteration %5d\n',i); break; end v = vn; % update v end end % spmd matlabpool close SPMD need be started once. Distributed arrays remain in the worker space until spmd region ends. Spring 2012 33 Preferred Way to Distribute Matrices ? For matrix-matrix multiply, there are 4 combinations on how to distribute the 2 matrices (by row or column). While all 4 ways lead to the correct solution, some perform better than others. n = 3000; A = rand(n); B = rand(n); spmd ar = codistributed(A, codistributor1d(1)) % distributed by row ac = codistributed(A, codistributor1d(2)) % distributed by column br = codistributed(B, codistributor1d(1)) % distributed by row bc = codistributed(B, codistributor1d(2)) % distributed by column crr = ar * br; crc = ar * bc; ccr = ac * br; ccc = ac * bc; end Wall clock times of the four ways to distribute A and B C (row x row) 2.44 C (row x col) 2.22 C (col x row) 3.95 C (col x col) 3.67 The above is true for on-node or across-nodes runs. Across-nodes array distribution and runs are more likely to be slower than on-node ones due to slower communications. Specifically for MATLAB, Katana uses 100-Mbits Ethernet for across-nodes communications instead of Gigabit InfiniBand. Spring 2012 34