Concurrent and Parallel Programming

The above drawing is due to Prof. John Byrne. https://www.irishtimes.com/business/technology/prof-john-byrne-the-man-who-turned-ireland-into-a-tech-world-power-1.4118522

A myriad computers are at work upon the weather of the part of the map where each sits, but each computer attends only to one equation or part of an equation. The work of each region is coordinated by an official of higher rank. Numerous little "night signs" display the instantaneous values so that neighbouring computers can read them. Each number is thus displayed in three adjacent zones so as to maintain communication to the North and South on the map.

?From the floor of the pit a tall pillar rises to half the height of the hall. It carries a large pulpit on its top. In this sits the man in charge of the whole theatre; he is surrounded by several assistants and messengers. One of his duties is to maintain a uniform speed of progress in all parts of the globe. In this respect he is like the conductor of an orchestra in which the instruments are slide-rules and calculating machines. But instead of waving a baton he turns a beam of rosy light upon any region that is running ahead of the rest, and a beam of blue light upon those who are behindhand.

?Four senior clerks in the central pulpit are collecting the future weather as fast as it is being computed, and despatching it by pneumatic carrier to a quiet room. There it will be coded and telephoned to the radio transmitting station. Messengers carry piles of used computing forms down to a storehouse in the cellar.

L. F. Richardson (1912)

This is a pretty good example of a parallel design pattern (Master-Worker).

//

Short note: how to parallelise code

Daniel J. Duffy

2022-2-19

?

Most developers have tinkered with the idea of trying to write concurrent and parallel programs. Writing parallel code is much more difficult than writing serial code. To this end, we gave a rough list of?don’ts and do’s that might provide some guidance on avoiding pitfalls and quagmires.

?Anti-patterns/worst practices

A1. Doing concurrency without the necessary computer science background.

A2. Adding pragmas and decorators (e.g. ?OMP #pragma, numba) to sequential code and hoping for the best.

A3. Thinking sequential instead of thinking parallel.

A4. Writing sequential code and then trying to parallelise it.

A5. Parallel programs that run in tests but fail when shipped to production.

A6. Hubris (“ah, it’s easy”).

?

(Worst-Case) Outcomes

??????????????W1. Trial and error experimentation to improve speedup, and not always successfully.

??????????????W2. Using the wrong paradigm/parallel design pattern/programming language.

??????????????W3. Parallel programs that must be reversed engineered to serial code for reliability reasons.

??????????????W4. Trying to parallelise object-oriented code and libraries.?????????

??????????????W5. Trying to parallelise code whose serial fraction is greater than 10%.

?

Possible work arounds / best practices

B1. Understand parallelism v concurrency and asynchronicity.

B2. Processes v threads v events.

B3. Look for potential concurrency.

B4. Data dependency graph.

B5. Parallel design patterns.

?

No silver bullet.

References

"Weather Prediction by Numerical Processes, Second Edition" L.F. Richardson, Cambridge University Press 2010.

Campbell, C. and Miller, A.?2010 Parallel Programming with Microsoft.NET C++ Microsoft Press.

Campbell, C. and Miller, A.?2011 Parallel Programming with Microsoft Visual C++ Microsoft Press.

Duffy, Daniel J. 2018 Financial Instrument Pricing using C++, Second Edition. John Wiley and Sons Chichester.

Duffy, Daniel J. 2022 Numerical Methods in Computational Finance, a PDE/FDM Approach. ?John Wiley and Sons Chichester.