Python as a support tool for multi-beam data processing or how to fix very problematic data

Python it’s one of the best tools for work with data sets but I guess it’s not so famous in hydrographic data processing sector. So, purpose of this article is to show how to remove MBE seabed detection artifacts from the MBE data on a real example using python with Pandas library.

Story starts on one of my projects where we faced with bottom detection of one MBE - it got 12 cm. artifacts on ideally flat seabed. It was manufacture problem and we had no chance to fix it on board, support didn’t help us. Also we had same issue with spare one – it shown different shape of artifacts on profile but very similar to previous one in values – depth of detected bottom varies up to 12 cm on flat seabed when depth under transducer is 6 m.

So, we had to continue acquisition like it is because errors were in project tolerance and we hoped to fix it by refraction coefficients during post processing. But our expectations were didn’t acquitted – those artifacts became very visible on TDM (see data sample picture below) once we applied tides.

For sure we had to do something, but what? Smooth and lose seabed features or shift middle of each line vertically? Or adjust by refraction corrections? Nothing of the above we couldn't complete in time or it was unacceptable. And I decide to use python to try fix this problem. Idea was to export X, Y, Z, Beam Number information into point cloud .txt files to see if those artifacts are same in all survey lines. I think all modern data processing packages allow to do it (Quimera, Caris for sure).  

 Let’s see what I did with exported comma separated point cloud files ("Date", "Time", "E", "N", "Depth", "Beam") step by step:

1 Define errors profile for each points file:

1.1 Preparations:

1.2 Import data into Pandas data frame:

1.3 Aggregate data by beams:

1.4 Calculation of corrections by beams:

1.5 Plotting results and export profile correction into .txt files:

As results we got follow profiles for each of six files:

- As we can see profiles averaged by beams almost same for each of six lines.

2 Profiles processing and corrections calculation:

2.1 Preparations:

2.2 Load error profiles:

Plotting Out:

- As we can see error profiles are almost same for each of point files.

2.3 Aggregation different profiles in one Data Frame:

2.4 Calculation average values for each beam:

2.5 Calculation of corrections:

2.6 Plotting and export data:

Plotting Out:

- So, here we can see profile of correction for each beam.

Once we get our corrections for each beam we need just apply them to the data.

3. Apply corrections to point cloud files:

3.1 Preparations:

3.2 Apply corrections and export of corrected data into .txt files file by file:

After above operations we get corrected point cloud files and can see the results:

 - Note: data on the screenshot were processed automatically - better results easy reachable with manual cleaning.

So, as we can see finally we removed those terrible artifacts and saved all recorded seabed features with minimal distortion of geometry. Those scripts combining with cleaning and little smoothing helped us to process 30 X 0.6 km corridor within one week.

Exactly used scripts applicable only for flat seabed without of any slant and should be modified in case of use in other conditions. But it shows abilities of python for any data processing tasks considering new upcoming libraries for point clouds and geospatial data processing.

I’m open for any discussion and feedback, please feel free to ask questions or show better way to solve described problem. In the next articles I will try to explore new python libraries for hydrographic data processing. During processing python 3.6 with Anaconda 5.2 package were used.