High-performance Computing in C++ : Open Muti Processing(OpenMP)

Open Multi-Processing:

?Let's consider the parallelization approaches, basically, we can think of imperative Parallelization and Declarative parallelization.

1. ?Imperative Approach: In this case, everything is done by hand which means that we initially separate the data set into several independent parts. Then create threads to process each part separately by using std::thread or pthread. Run the computation on each thread and consolidate the outcome for the final result. In this approach, the main challenge is to divide the data and share it across the thread for computation. Alternatively, we can use some libraries like?Microsoft PPL , Intel TBB etc.
2. ?Declarative Approach: Here we are leaving our sequential code as it is for optimization to the compiler. Decorate code with compiler hints. In this approach, the compiler decides to split which part of the code can be parallelized?into several threads. (eg: For loop)?

In the above case, OpenMP comes into the picture. OpenMP is the standard API?for decorating code for multiprocessing

• Data Parallelism?( eg: loops)
• Task Parallelism?(eg:?running the block of code in a separate thread)

Essentially the OpenMP is a compiler+library solution. Here we completely depend on the compiler to optimize the code but use some library to perform some operations.

In c++, we use #pragma directives to indicate the parallelization. Plenty of compiler clauses support Data Sharing, synchronization, scheduling, etc.?

The following picture gives some details on how the OpenMP works.

• Work Sharing:

Let's see, how that omp for and omp do parallelize the loops to improve the performance and section(s) which assign different code blocks to different threads to optimize the execution. Similarly omp provide the single thread approach, it assigns the single block to a single thread and executes it separately. It implies the barrier at the end of execution. One more concept is "master", in this case, code is always executed by the master thread, and no barrier at end of the execution.?

• Synchronization:

The following techniques are used in omp for synchronization of code execution.

• Critical: block executed by one thread at a time
• Atomic: next memory update is atomic
• Ordered: block executed in the same order as if it were sequential
• Barrier: all threads wait until each one has reached this point
• Nowait: threads can proceed without waiting on other threads.

Let's consider the following code for synchronization.

• Data sharing:
• Shared: item is accessible by all threads simultaneously

??????All variables except the loop counter shared by default

• Private: item is thread-local, not inited or available outside parallel region
• Firstprivate: like private but initiated to the original value
• Lastprivate: like private except original value updated at exit
• Defualt: defines whether, by default, variables are shared or not (none)

?Refer to the following example to check the implementation