Road Surface Quality Detection for Autonomous Vehicles

Lets set the context first

Autonomy is a concept that has continued to rapidly advance in the 21st century. However, when the concept is explored, the most common use-case highlighted is that of a self-driving vehicle. Taking many forms, whether it be land, air or sea, the concept of autonomous vehicles continue to grow, with continued and significant success, especially in self-driving cars.

Autonomous vehicles operate in highly complex and diverse environments, be it on-Earth or in-space. The most commons factors leading to this complexity are usually manual components in the overall ecosystem, such as pedestrians, that aren’t always controllable.??

This complexity encompasses (road or vehicular) safety, which is arguably the biggest concern for an autonomous vehicle. This concern however remains, regardless of whether the vehicle is autonomous or manual. One such factor that dictates the safety of a road specifically, is the surface quality.?

Accidents can occur when the road surface quality is sub-standard, however in a manually driven setting, may be avoided (i.e. human vision identifying a hazard). However, in autonomous mode, avoidance must be a feature built into the design of the vehicle itself, leveraging concepts of sensor fusion.?

Negative consequences can range from fatal injuries to the vehicle itself being damaged but can also extend to inefficient routes or excess fuel consumption. This therefore highlights the necessity to remain situationally aware as it relates to driving conditions.?

So, what's the problem?

The concept of autonomous vehicle, or self-driving cars, require a number of factors to be considered and implemented. Only when these factors are in compliance, can a truly autonomous vehicle be fully operational, Level 5. Important to note though, is that these are not always technical or technologically based factors, but also aspects of the wider ecosystem, such as those listed below:?

• Environmental conditions – sound, lighting, weather, etc
• Ecosystem complexity – number of automated and manual cars, pedestrians, etc
• Legal constraints – liability definitions for incidents, etc
• Technology limitations – ability to visualize and localise the entire environment, etc

However, when the concept of navigation is raised, the 'quality enabled navigate-ability' itself is often omitted. Unravelling this concept, and the rationale for deriving a solution to this problem, is concept underlying navigate-ability, or Road Worthiness, or as defined here as Road Surface Quality Detection.

Figure 1: Example pictures of varying road conditions

Having an autonomous vehicle operate over different types of terrain may be a standard operating model in the current day and age. But considering the impact on specific vehicles, such as on-road commercial vehicles, raises significant complications and negative consequences such as damage to the vehicle, caused by dirt, rocks, pot-holes. This in itself drives the need for Road Worthiness.?

Figure 2: Quality scoring example for varying road conditions

Therefore, here I look to employ algorithms (AI/ML) and mechanisms to identify the Road Worthiness through Road Surface Quality Detection across various types of terrain, but also terrain that has been damaged to determine a quality score.

Figure 3: Potential future applications - GPS Routing (not in scope)

The intent would be, that these Quality Scores and embedded into Navigation Software for re-routing if a safer navigate-able path is available, and if not, warn the driver that a risk is nearing.

What are others thinking?

Review of four state of the art approaches have identified the following benefits, challenges and watch points. Each of the literature reviews indicated disparate approaches, two of which have been adopted.

Understanding the most optimal solution to the problem autonomous vehicles faces on the road, that is road surface quality [5], is the first critical step.

Coalescing insights from a number of literary publications has highlighted two distinct approaches, both of which us similar datasets, with similar objectives, albeit from different timeliness aspects, proactive and reactive.

This literature identifies multiple approaches and their validity, each having different success rates, which will be explored in delivering an outcome, and ultimately a proof of concept of those these two approaches may be derived.

Figure 4: State of the Art Literature

Alright, how we do solve this problem?

Based on preliminary research, literature review, and white-boarding, the preliminary approach is as defined below. Largely leveraging a form of Machine Learning to allow the solution (or model in this instance) to learn from a variety of images (and video, if time allows) the identifying features of a good road surface through to a bad road surface.

The model’s ability to learn these scenarios, per datasets available, will be directly correlated to the accuracy of the outcomes. Understanding that a scoring model will need to be enacted (perhaps in relative figures, and not an absolute value), would assist in development of the proposed concept or idea.

The route intended to be followed is below, to be stressed tested, modified and adjusted as the course continues in the event that smarter methods are available to be deployed.

Figure 5: The approach to resolution

The specific approach that is being followed is diagrammatically presented below. The datasets are freely available. The data used for the SmartPhone sensors base case can be found at Kaggle [10], whilst the Imagery based stretch case can be found at the National Library of Medicine [11].

Figure 6: Core algorithm design and methodology

Did it work?

For the purposes of this two parallel approaches have been adopted, the base objective will be designed to utilise of smartphone sensors. This will leverage the data from accelerometers and gyro-meters, within a smartphone. This can help with reactive measures such as reporting on quality, to be stored centrally and uploaded to GPS devices in advancefor intelligent routing.

The stretch objective will be in the form of proactive measures, such as determining the road and path ahead of a vehicle before it ventures on, through imagery taken of the path ahead.

The Base Case

For the purposes of developing the base case model, four files were obtained, extracted and used. These files were split across four scenarios, across two different style of vehicle. In order to make this dataset usable, these files were merged, creating one master dataset.?

For the purposes of building the respective model, an approach to keep is as simplistic as possible, was followed. All numerical fields were converted into Z Scores. Any fields that had missing values were populated with median values.

A number of variables are triggered when driving. For example, when a vehicle hits a pot hole, more Engine Loading is required to push out of the pot hole. This has the effect of increase Engine RPM, Vertical Acceleration and Fuel Consumption, etc.

Figure 8: Results and model configurations

These configurations were determined by assessing the best performing and varying combinations. The outcomes from this ideal and optimal configuration was the production of a model that was classified as 95.33 percent accurate. To confirm this, a Confusion Matrix was performed. The results shown were promising of a well trained model.?

Note: The Logarithmic Values for Time and Epoch were used to ensure ease of readability of the graph. Note that C5 and C8 have the best accuracy scores. Also note that the Score for the C8 is the most optimal, and therefore remains the reason for selecting that specific configuration (i.e., Adam Optimizer, Relu Activation and 1E-6 Learning Rate).

Figure 9: The model produced was of 95.33 percent accuracy.

The Stretch Case

The outcomes from this ideal and optimal configuration was the production of a model that classified a Road Surface. The imagery provide the machine learning model a way of discriminating amongst the 8 data groups above. These have been mapped into a more intuitive manner to highlight whether a road is driveable or not driveable.

The model produced was successfully able to classify a road surface, however similar to the Base Case objective was not successful 100 percent of the time. The imagery highlighted below shows the successful use cases of where the model was able to classify correctly. There are suggestions for further work which may highlight better options to get a more accurate classification more of the time.

Figure 10: Computer vision approach and results

So, where did we land?

Designing a model for Autonomous Cars to use to derive the Road Surface Quality is possible. The key learning is however that each and every road may look different under different conditions and within varying environmental conditions.

It is also important to note that the more driving that is conducted, the more data that can be consumed for a more accurate picture or output. Perhaps the overall message deriving from this is that both proactivity and reactivity are needed to, ultimately, become more proactive in routing an Autonomous Vehicle.

For example, capturing data via a SmartPhone or Vehicle Data Loggers may not be that useful for the immediate vehicle in the driving situation, but can be used to feed data into a Navigation System for the ‘future’ car, or the car that may potentially use the route next. This derives the notion of proactivity. This follows true from an imagery perspective, as it also may be too late for a vehicle to make an alternative manoeuvre at the last minute.?

Whilst this has been successful, the lessons learned indicate that fine tuning a model to achieve 100 percent accuracy may not be plausible, but can provide an indicative suggestion of recommendation for a driver before setting off on a particular path.?

Figure 11: An example of the outcomes from the pre-trained model

Future Work Opportunities

The above focussed on the design, development and operationalisation of two proof of concepts. One of which is related to textual data for the purposes of regression and identifying factors that elude to an outcome as to the Road Surface Quality. The second is the usage of imagery for the purposes of the same objective. Albeit, the first of which forms a reactive measure and the secondary pro-active or real-time.

There exists an opportunity to compare and contrast this and results produced against other approaches that may increase accuracy and speed, and therefore there are areas that may profit from further research such as:

Figure 12: Future work opportunities

These together may highlight an alternative approach for this problem and should be explored as next steps.?

References

[1] L. Claussmann, M. Revilloud, D. Gruyer, and S. Glaser. “A Review of Motion Planning for Highway Autonomous Driving,” IEEE Transactions on Intelligent Transportation Systems, vol. 21, no. 5, pp. 1826 - 1848, May 2020. [Online]. Available: IEEE Xplore, https: //ieeexplore-ieee-org.ezpxy-web-p-u01.wpi.edu/document/8715479.

[2] D. Zhou, Z. Ma, and J. Sun. “Autonomous Vehicles’ Turning Motion Planning for Conflict Areas as Mixed-Flow Intersections,” IEEE Transactions on Intelligent Vehicles, vol. 5, no. 2, pp. 204 - 216, Nov. 25, 2019. [Online]. Available: https://ieeexplore-ieee-org.ezpxy-web-p-u01.wpi.edu/document/8911511.

[3] C. Katrakazas, M. Quddus, W. Chen, and L. Deka. “Real-time motion planning methods for autonomous on-road driving: State-of-theart and future research directions,”transportation Research Part C: Emerging Technologies, vol. 60, pp. 416 - 442, Nov. 2015. [Online] Available: https://www-sciencedirect-com.ezpxy-web-p-u01.wpi.edu/science/article/pii/S0968090X15003447.

[4] H. Prakken. “On the problem of making autonomous vehicles conform to traffic law,” Artificial Intelligence and Law, vol. 25, pp. 341 - 363, 2017. [Online]. Available: https://link-springer-com.ezpxy-web-p-u01.wpi.edu/article/10.1007/s10506-017-9210-0.

[5] Russell, Stuart J. (2018). Artificial intelligence a modern approach. Norvig, Peter (4th ed.). Boston: Pearson. ISBN 978-0134610993. OCLC 1021874142

[6] Roychowdhury, Sohini Zhao, Minming Wallim, Andreas Ohlsson, Niklas Jonasson, Mats. (2018). Machine Learning Models for Road Surface and Friction Estimation using Front-Camera Images. 1-8. 10.1109/IJCNN.2018.8489188.

[7] Sattar S, Li S, Chapman M. Road Surface Monitoring Using Smartphone Sensors: A Review. Sensors (Basel). 2018;18(11):3845. Published 2018 Nov 9. doi:10.3390/s18113845

[8] Afenika, Adhelinia Gunawan, Putu Harry Tarwidi, Dede. (2020). Classification of Road Surface Quality Based on SVM Method. Journal of Physics: Conference Series. 1641. 012064. 10.1088/1742-6596/1641/1/012064.

[9] Rateke, Thiago Justen, Karla Von Wangenheim, Aldo. (2019). Road Surface Classification with Images Captured From Low-cost Camera - Road Traversing Knowledge (RTK) Dataset. 26. 10.22456/2175-2745.91522.

[10] Michele Ruta, Floriano Scioscia, Giuseppe Loseto, Agnese Pinto, Eugenio Di Sciascio. Machine learning in the Internet of Things: A semanticenhanced approach. Semantic Web Journal, Volume 10, Number 1, page 183–204 - 2018.

[11] Bhutad, S., Patil, K. (2022). Dataset of road surface images with seasons for machine learning applications. Data in brief, 42, 108023. https://doi.org/10.1016/j.dib.2022.108023