University:
Boston University
Course:
MET CS 201 | Introduction to Programming
Academic year:

2023
Views:

267

Pages:

18
Author:

Carina Watson

Outline       Parallel Examples Parallel Strategies Hardware Processes and threads Libraries & your own code Parallelization pitfalls Hardware for Parallel Computation Lenovo ThinkSystem HPC cluster iPhone motherboard  Parallel computing is used on systems of all sizes, from your smartphone to clusters with thousands of processors. CPUs and cores  In the beginning...a CPU plugged into a socket in the computer.  The term “core” wasn’t in use but we’d call this a 1-core CPU today.  Multiple CPU computers had multiple CPU sockets.  In 2001 IBM introduced their POWER4 CPU which embedded 2 “cores” into one physical CPU package. AMD K5 in a Socket 7 (1996) core core  The two cores are manufactured on the same physical semiconductor die.  1 socket POWER4 circuit view Modern configurations   Quad Intel Xeon CPUs Up to 28 cores per CPU   Dual AMD Epyc CPUs Up to 64 cores per CPU   Single Intel CPU 4 cores (Core-i3, ~$100)    For PC and server hardware the high end has very high core counts. Entry-level systems still have multiple cores. Parallel capabilities are everywhere these days. CPUs and cores 1 CPU, 1 core 1 program at a time 1 CPU, 2 cores 2 programs simultaneously 1 CPU, 16 cores 16 programs simultaneously  “CPU” typically refers today to the physical packaging of multiple cores.  CPU, processor, and core are sometimes used interchangeably to mean “core”. Hyperthreading (Intel trademark) Logical cores “Logical Cores” or “hardware threads” Progam A Progam B  CPUs with this feature have some additional hardware that lets a program have its execution state pre-loaded onto a core while another program is actually executing on that core.  The hardware allows the OS to switch the physical core to run the other program very quickly. Real physical core  For many sets of programs (especially I/O bound) this makes better use of the physical core. Logical cores No logical cores Off in main memory Progam B Progam A Progam B Progam A Prog A is executing Real physical core  For regular CPUs the program switching is slower.  For parallel CPU or compute bound programs the “extra cores” are of no benefit and typically degrade overall system performance.  Hyperthreading or logical cores does not double the computational resources. Hyperthreaded Intel i5-9300h CPU  A linear algebra matrix-matrix multiply.  4 real cores, 4 logical cores.  Note performance increases stop for cores > 4.  CPU-bound programs can only take advantage of real cores. Count Your Cores  Operating system utilities are the easiest way.  Windows Task Manager  Linux command: lscpu  Mac OSX: [~] sysctl -n hw.logicalcpu 8 [~] sysctl -n hw.physicalcpu 4 Final Comments  On your personal or lab computers, check to see if logical cores are present.  If so, beware of using all of them for a parallel computation.  It’s best to use just the physical cores if CPU-bound.  On the SCC any compute node that supports logical cores has this feature disabled.  All SCC core counts are real physical cores. Outline       Parallel Examples Parallel Strategies Hardware Processes and threads Libraries & your own code Parallelization pitfalls Types of Parallelization  On the SCC: queue parallelization.  You have N files to process. Submit N jobs.  Or, one job array that launches N jobs.  This often requires little to no changes to your code…  Parallel Libraries  Use a library that internally implements some kind of parallelization.  Multiple Processes  Your program launches several copies of itself (or other programs) to solve the computational problem.  On one computer or many.  Multiple Threads  Your program creates threads, which are parts of the same program that can execute independently of each other. Process  A program running on a computer.  Processes can start other processes.  Properties:  A private (non-shared) memory space  A process ID  Can exchange data with other processes via files, pipes, network connections, system shared memory, etc.  The operating system schedules the process so that it shares computational time with other processes. Process Scheduling by Analogy  Consider this 4-burner stove.  There are 17 pots of soup to cook.  Some must be kept very hot (lots of time on a burner)  The chef swaps pots frequently to share the burners as fairly as possible. Process Scheduling by Analogy  Consider this 4-burner stove.  Consider a 4-core CPU  There are 17 pots of soup to cook.  There are 17 processes to run  Some must be kept very hot (lots of time on a burner)  The chef swaps pots frequently to share the burners as fairly as possible.  Some require a lot of CPU time  The OS swaps processes on and off the cores to share computation time as fairly as possible. Threads  A part of a process that can be scheduled to run independently of the rest of the process.  Are created, run, and destroyed by a process.  Properties:  Shares memory with each other and the original process.  Does not have a separate process ID.  Can exchange data with other threads or with other processes.  Python running threads on 22 cores.  In the top program 100% of CPU means 1 core is 100% busy.  2186% means 22 cores are busy.  The cooking analogy equivalent would be a large pot that covered several burners. Monitoring with the top tool  On the SCC, use top  To see your processes only: top -u username  100% of CPU means 1 core is 100% occupied.  200% means 2 cores are used, etc.  The RES column is the amount of RAM actively in use by the process.  VIRT is the virtual memory – essentially the maximum amount of RAM the process might request. Parallelize with Processes or Threads?  You can add parallelism to your program through changing your source code or by calling libraries that implement parallel algorithms.  Process-based parallelism:  Keeps memory separated.  Can potentially run on multiple computers and communicate via a network.  Avoids issues with non-thread-safe code.  Thread-based:  All the program memory is accessible by all threads.  Higher performance intra-thread communication compared with processes.  More complicated parallelization patterns can be implemented.  Easy to start & stop threads.