University:
Boston University
Course:
MET CS 201 | Introduction to Programming
Academic year:

2023
Views:

216

Pages:

21
Author:

granitt3c2r

Outline       Parallel Examples Parallel Strategies Hardware Processes and threads Libraries & your own code Parallelization pitfalls Common Parallel Libraries Library Parallelization Notes Python multiprocessing Processes Standard language library. Matlab parpool Implicit parallelism Processes Threads Standard language library. Some operations will automatically multi-thread. R parallel foreach Threads Processes Standard language libaries. BLAS (SCC: blis or openblas modules) Threads Linear algebra. Widely used, for example by R and Python (via the numpy library). Intel Math Kernel Library (MKL) Threads Linear algebra and a lot more. Widely used. FFTW Threads Fast Fourier Transforms. OpenCV Threads Image processing. Tensorflow Threads Machine learning. PETSc Processes and threads Partial differential equation solver, multi-compute node. Hadoop and Spark Processes and threads Multi-compute node, includes a parallel file system. MPI Processes Low-level library for multi-node communication. OpenMP Threads Low-level library (C/C++/Fortran) for multi-threading. Example: BLAS  The Basic Linear Algebra Subprograms library provides a variety of functions for linear algebra type calculations.  This underlies a staggering number of algorithms and computations in every area of computing.  High performance threaded BLAS libraries continue to be an active area of computer science research.   SCC benchmark. Note poor performance of default Linux system BLAS library! Enable OpenMP Threading Libraries on the SCC  Most software on the SCC that uses multiple cores are built with the OpenMP threading library.  Including BLAS routines as commonly used by R and Python.  The number of threads that will be used by your program can be set using the environment variable OMP_NUM_THREADS  If a program uses the Intel MKL library the threads can be set with MKL_NUM_THREADS instead  The SCC sets OMP_NUM_THREADS=1 by default.  NEVER request more threads than there are cores for the job. #$/bin/bash -l # Request 8 cores for this job # The queue will set the variable # NSLOTS to 8 #$ -pe omp 8 # We know a priori that this multithreads # with OpenMP module load nobel_winner/1.0 # Allow for OpenMP threading. export OMP_NUM_THREADS=$NSLOTS # Using NSLOTS means we will never ask # for more threads than cores. # Now run the program...is it faster? nobel_winner ...etc... Enable OpenMP on non-SCC computers  Environment variables can be set in various ways on different operating systems.  The OpenMP library looks for OMP_NUM_THREADS regardless of the operating system.  Mac users: the BLAS library used by R, Python, etc. is likely to be the Apple Accelerate library. In that case try setting the variable VECLIB_MAXIMUM_THREADS. Know your software  OpenMP is hardly the last word in multithreading.  Different software may have different mechanisms for enabling threaded or multiprocess calculations such as configuration options or command line flags.  Read the documentation! The Message Passing Interface (MPI)  With the right software tools processes can be run on multiple computers simultaneously and communicate with each other across a network.  The MPI library is the most successful system for this in high performance computing.  On the SCC we standardized on the OpenMPI implementation: module avail openmpi  Used on the world’s largest clusters with thousands of cores over hundreds of compute nodes for single programs. MPI  Since MPI uses separate processes, the programmer has to decide how and when data is shared between them.  MPI provides routines for communication, parallel file I/O, gathering and reducing data from processes, and many more. Using MPI in your software  OpenMPI libraries are typically available for C, C++, Fortran, and Java.  Wrappers libraries for MPI are readily available. These will typically work with whichever MPI implementation is available  OpenMPI, MVAPICH, Intel MPI, etc. Language Library Python mpi4py R Rmpi Julia MPI.jl C# MPI.NET  MPI programming is an advanced programming skill. RCS is happy to help – email us! 3 compute nodes, 4 cores each. mpirun MPI process mpirun -np 12 my_mpi_prog 1 MPI process per compute node will run.  MPI programs have a special program to launch them, mpirun  OpenMPI’s mpirun has many options that control how MPI processes are started and where they run. mpirun -np 3 my_mpi_prog 3 MPI processes will run…all on node 0.  Try module help modulename on the SCC for MPI-based modules  On the SCC the configuration of compute nodes for mpirun is handled by the MPI process with 1 OpenMP thread queue. OpenMP thread mpirun -np 3 --map-by node my_mpi_prog 3 MPI processes will run, one per node export OMP_NUM_THREADS=4 mpirun -np 3 –-map-by node my_mpi_prog 3 MPI processes will run, one per node, with 4 threads Network Type SCC MPI Nodes  Request MPI-specific nodes on the SCC with the qsub option:  -pe mpi_16_tasks_per_node N  Where N is a multiple of 16  N=48  4 16-core nodes  NSLOTS  48  -pe mpi_28_tasks_per_node M  Where M is a multiple of 28  The only way to use multiple compute nodes for a job on the SCC is to use the MPI queues. Bandwidth (Gbit/sec) Latency (ms) 10gig Ethernet 10 12.5 FDR Infiniband 13.64 0.7 EDR Infiniband 25 0.5  These jobs run on dedicated compute nodes connected with an Infiniband network. There are a couple of versions on the SCC.  Latency is how quickly a data transfer can be initiated. For MPI computations this is often the limit, not the bandwidth.  MPI jobs on a single compute node should use the regular “-pe omp N” queues. Parallel Speedup  There are many ways to parallelize code.  …is it worth the effort and how much will it benefit you? Amdahl’s Law  The speedup ratio S is the ratio of time between the serial code (T1) and the time when using N workers (TN): 𝑇1 𝑆= = 𝑇𝑁 𝑇1 1−𝑓 𝑓+ 𝑇1 𝑁 N = number of threads or processes f = fraction of program that is serial  This is the theoretical best speedup achievable with parallelization. Intel Xeon CPU E5-2650 v2 @ 2.60GHz. 16 physical cores, 2 sockets (scc-pi2)  For small matrix sizes, using any number of threads >1 is slower.  Thread coordination takes longer than the parallel speedup.  Larger matrices have diminishing returns for higher numbers of threads. 1-  For any given code you’ll likely find a range above which more threads doesn’t help.  You have to test…test…and test some more!  SCC suggestion – try 4 threads/cores. AMD EPYC 7702 CPU @2 GHz. 64 physical cores, 1 socket  Running on a 64-core system the computation actually gets slower with too many threads.  It may be that some parts of your code benefit from more threads than others – try to pick a sensible number.  The ideal thread number may change if you change the CPU manufacturer, CPU model, BLAS library, and so on. Outline       Parallel Examples Parallel Strategies Hardware Processes and threads Libraries & your own code Parallelization pitfalls Parallelization Difficulties  Some code cannot be parallelized – it must be computed in order.  Some loops or function calls can have dependencies on other loop iterations that make it impossible to parallelize.  Sometimes you can alter a loop with additional copies of data to make it parallelize.  Trading off memory usage for computation time. x = np.random.random([100]) # This cannot be parallelized. for i in range(1,x.shape[0]-1): x[i] = x[i] - x[i-1] + x[i+1]  Choose your battles wisely  Use profiling to identify code that is worth improving. Parallelization Difficulties  Random number generation is not straightforward.  Computing RNG’s in parallel requires different random seeds for each worker.  Do not assume that different workers, if seeded by default or from the system clock, will be generating different sequences of RNGs. Parallelization Difficulties  Be careful about the amount of I/O your workers are performing.  Disks, networks, etc. have bandwidth limits.  Excess workers can overload resources, turning the problem from CPU-bound to I/O bound. Apache web server How Many Workers*? ….  I/O-bound programs may run hundreds or thousands of workers Hundreds of copies of itself handle incoming web traffic  These spend a lot of time waiting for data from the network, the disk, the user, etc.  CPU-bound programs should run one worker per physical core.  Memory-bound programs often use fewer workers than cores. LAMMPS molecular dynamics code 4 cores – 4 workers What happens with too many workers?  For CPU-bound problems, use no more than 1 worker per core.  More than 1 results in workers competing with each other for access to the cores and memory bandwidth.  Performance will suffer significantly with excess workers.  Watch for mixing multiple processes and multithreading (like OpenMP): each process can end up launching many threads, overloading the cores!