Ridiculously fast and large-scale Particle Image Velocimetry for rivers in OpenRiverCam

By Hessel Winsemius (Rainbow Sensing) and Udhav Maharaj (GroundTruth)

From today onwards, OpenRiverCam is fast and capable to handle very large and long videos of very large objectives. Do you have drone videos that make you sweat to get your work finished in time? Do you hit memory walls just when you hit 90% of the analysis? That now belongs to the past thanks to a new, much more efficient implementation of the velocimetry algorithms and the frame reading methods. In this blog we highlight and explain the changes made and we put it to the test in some real world use cases using different hardware components. We did so using a video of our pilot site in Limburg, and, with GroundTruth SA, drone footage in South Africa. The video of Limburg with all configuration files needed is shared in this link, so that you can try re-processing it yourself. As we received a lot of requests for a fully working command-line example, we also added a fully commented recipe with the video. Just update to the latest pyOpenRiverCam version and run the batch file!

Over the past years we received lots of feedback from users of the OpenRiverCam (ORC) ecosystem. Especially pyOpenRiverCam (pyORC), the open-source library, API and command-line interface for processing videos from streams and rivers into surface velocity and streamflow, is used more and more for diverse problems and many people are enthusiastic about it. But not all is positive. Especially drone users also gave negative feedback. The typical comments: “the code is too slow for my videos and I run into memory issues all the time!” And they were right!

Acceleration: how?

So acceleration would benefit these users. And even for operational monitoring on river sites this is important. You don’t want a lengthy video processing consuming all your battery life! How then do you accelerate? For you as a user? Very simple! Just open up your terminal now! Activate your python environment holding pyopenrivercam and type:

Easy! This brings you to PyOpenRiverCam version 0.7.1 (or higher if you read this later) and you are ready to go.

How does this work?

Basically two things have happened:

1. Most importantly, the Particle Image Velocimetry (PIV) approach behind OpenRiverCam (ORC) has been replaced! About a month ago, Rainbow Sensing released a new very small and simple library called “Fast and Flexible Particle Image Velocimetry” (FF-PIV). Since today, FF-PIV is embedded in PyORC. Soon it will also be available in the other OpenRiverCam components LiveORC (for web API data collection) and NodeORC (for online or on-site processing of videos). To ensure the PIV problem is optimally solved, the projected frames are subdivided in large chunks. The size of these chunks is determined on run-time by measuring available memory. Hence, this also reduces the chance of running out of memory significantly. If you have little memory, the chunk size will be small, if you have lots, chunks will be bigger.
2. While FF-PIV in theory makes the PIV-part of the processing chain very fast, this also means that pyORC has to produce orthorectified frames faster in order to not have one process waiting long for another. After further profiling the code, we found that the bottleneck was the reading of frames from video. We used to do this in the following way per frame: 1) open video, 2) seek frame start position, 3) read frame, 4) close video. While the reading of the frame is the most important part, the opening, seeking and closing part give a lot of overhead. We therefore changed our video reader into a chunked reader. By default it now reads 20 frames (configurable but usually not necessary) in one go and this makes a humongous difference in speed of the entire pipeline of a velocimetry job.

Details

Let’s give a little more background on the PIV improvements for the interested reader. If you are not interested, just skip this section and go to the examples.

What is PIV?

First of all, you may not realize what Particle Image Velocimetry (PIV) is. PIV is a technique that estimates motion using Eulerian principles applied on a simple video. In common speech, the intensities over a designated area in frame 1, are correlated against intensities in the next frame (2) in a certain search area around the original designated area. The displacement with the highest correlation is where the pattern likely moved to, and so this provides you with an estimate of the displacement. Repeat this for an X amount of frames and you can start addressing things like noise or high frequency variability e. g. due to strong turbulence or due to changes in the external forces on the motion during the video capture period. The science behind this has been encapsulated in a very nice open source project called OpenPIV. This project released a python library which describes and solves the equations using the well-known python numpy library. Since many years, OpenPIV has been the velocimetry backbone of the OpenRiverCam ecosystem.

So why then a new library?

As the name suggests, the new library FF-PIV has two goals: 1) do PIV very fast! 2) ensure that the methods provide a level of flexibility which allows an easy integration into software projects. We consider the science of PIV to be very well encapsulated in OpenPIV, so we borrowed most of the equations and most of the reshaping and restructuring components from OpenPIV, and recoded them to fit better with our implementation and provide flexibility. Therefore a strong acknowledgement for all the hard work done by the OpenPIV team is in place here! Thanks a lot!

There are two issues with the OpenPIV library that we wanted to tackle. Below we describe these and the solution direction:

1. The pure numpy implementation is perhaps very easy to read, but it is also likely to cause a lot of overhead in its execution. Cross-correlation is computed with Fast Fourier Transforms (FFT), which are known to be strongly impacted by overhead in numpy and scipy implementations according to this post. In fact the same post describes an implementation of FFTs in the numba framework called “Rocket-FFT”. This can accelerate the Fourier Transform part by an order of magnitude under ideal circumstances. With some isolated tests on our typical river-related problems, we managed to accelerate image velocimetry by a factor 5 or 6.
2. OpenPIV is implemented with a quite strict use case in mind: compare image 1 with image 2. In most river-related use cases, a user would likely want to process image 1 and 2, and then 2 and 3, 3 and 4, and so on. Likely, the chunking into only single image pairs results in overhead. Also, the main functions of OpenPIV spit out velocity vectors per image pair. However, to reduce noise in the cross-correlation procedure, it can make sense to first process cross-correlations for several frame pairs, average the cross-correlation fields, and then extract velocity vectors from those. This requires more flexibility in the main functions.

We implemented all required functionalities and prepared API main functions that provide this flexibility. We also made both numpy and numba (based on Rocket-FFT) implementations which are 100% interchangeable and give exactly the same results as OpenPIV does if applied in the same manner. You can find the library and working examples in this GitHub link.

Real-world applications

IoT device in Limburg

Typically you want a small device like a Raspberry Pi 4 or 5 to run your videos on the site where you collect them automatically, and send results to a Live Server. This is the NodeORC and LiveORC use case. We tested a video of our test site in Limburg on a Raspberry Pi 5. The video has 125 frames (resolution 1080x1920). It took no more than 42 seconds instead of 97 seconds with our original implementation. The power consumption peaked at about 1.7 Amps (5V) and averaged to about 1.4 Amps during the processing. A raspberry Pi has 8GB of RAM and this also resulted in only 4 chunks to process with the available RAM. We were not running any other processes on the device.

We did the same on a Raspberry Pi 4. On this device processing took much longer, about 1minutes and 45 seconds. The processor speed is much lower, and also the available RAM is much lower so this is quite logical. But when processing with the original OpenPIV engine and the old frame reader, it took about 5 minutes and 23 seconds! So the speedup is massive especially in this example! The current peaked at about 1.3Amps and averaged to about 1 Amps. This demonstrates that in terms of power efficiency, a Raspberry Pi 5, with its larger memory and stronger CPU, and therefore less chunks to process in a shorter time, is so fast, that it will consume less power over the entire job compared to the Raspberry Pi 4.

Drone video of Lions River

The use case mentioned at the start was a drone video. So Udhav Maharaj from GroundTruth and Hessel Winsemius from Rainbow Sensing processed a beautiful video of the Lions river near Pietermaritzburg – South Africa. This problem is beyond capabilities of a small footprint Raspberry Pi. Typically you would process this on a laptop of desktop computer. We decided to go big and perform processing with our NUC Dragon Canyon (processor, Intel core I9 12900, memory: 64GB). We processed 1,800 frames of the video (30Hz) and got the results in 9 minutes and 32 seconds. With the original implementation this took 24 minutes and 9 seconds on the same machine. The 64 GB RAM gave a lot of space for large chunks and therefore the job was subdivided in 16 chunks only, despite the large resolution and long video length. The result is beautiful as shown in the figure below.

We then processed with exactly the same settings on a smaller machine, my laptop with 32GB RAM and an AMD Ryzen 7 7840U processor. This took 10 minutes and 41 seconds. Since the laptop was used for other things at the same time, and only had 32 GB of RAM, the videos were processed in many more (82) but smaller chunks. While this laptop is quite a lot smaller, it still performs very well!

Conclusion

With the new frame reader and implementation of FF-PIV as an accelerator, our velocimetry jobs are now among the fastest and most versatile. Also memory management is much improved, and likely to not cause any issues anymore (please do report an issue with your example if you still face memory crashes). The immense speed up is very good news for field devices that run on limited power availability, as well as very large jobs such as 4 or 5K drone videos, taken at high altitude covering a large space.

We are now working on the implementation in NodeORC and LiveORC so that you can truly benefit from this improvement in all applications.

Want to try this yourself?

We promised to provide a fully working example here. Please download this link to get your own copy of the video, including the prepared camera configuration file, a processing recipe with a full description of what is happening in the recipe, a cross section file (used in the recipe) and batch files for windows and unix/linux to get you started. Just make sure you first upgrade to the latest version! In case you haven’t installed pyorc yet, please make an environment with python 3.12. We are not upgraded to 3.13 yet. Follow the install instructions here.