Basic Introduction to HPC and Parallel Computing

So, I've been pursuing my Masters at ANU for the last 2 years and have been doing some really neat stuff which might interest you. Basically it involved doing some C/C++ programming and got to use the supercomputer there. I also got to do some advanced AI, Neural Networks, HCI stuff as well as HPC & Parallel computing. If you are looking to speed up your code or Algorithm's the HPC & Parallel computing, techniques may be quite useful.

So first and foremost you must understand Amdahl's law. - https://en.wikipedia.org/wiki/Amdahl%27s_law

Speed up =(1/ (1-P)+ (P/N))

Where, P = Parallel portion of code and (1-P) is the sequential portion of the code.

       N = No. of Processors.

A video walkthrough is available here https://youtu.be/WdRiZEwBhsM

A more detailed description can be found here- https://www.shodor.org/media/content/petascale/materials/UPModules/beginnersGuideHPC/moduleDocument_pdf.pdf

Loop Unrolling- https://www.cs.umd.edu/class/fall2001/cmsc411/proj01/proja/loop.html

Vector Processing using SSE (Links given below)

One of the things I've been looking at is SIMD processor programming. I've fallen in love with MPI and CUDA, There are some really cool blogs on this out there, I'll post a few, I found useful here.

Eric Holk's post- https://theincredibleholk.wordpress.com/2012/10/26/are-gpus-just-vector-processors/

If you want to try CUDA, SSE or MPI. I highly recommend trying respective documentation and software, as it come's in great detail.

CUDA- https://developer.nvidia.com/cuda-zone(Has amazingly good documentation)

JCUDA- https://www.jcuda.org/

SSE/AVX - https://software.intel.com/sites/landingpage/IntrinsicsGuide/

SSE tutorial - https://neilkemp.us/src/sse_tutorial/sse_tutorial.html

MPI Tutorial - https://mpitutorial.com/tutorials/

MPI based fftw - https://www.sandia.gov/~sjplimp/docs/fft/README.html

And to analyze performance PAPI Profiler (Other good ones are also out there) - https://icl.cs.utk.edu/projects/papi/wiki/Main_Page

Hope this helps.

Cheers

Badruddin (Raad)