Steering Angle Generation for Camera Based ADAS Systems using Deep Learning — Behavioral Cloning

There are two different approaches in algorithm development for sensing and control of autonomous vehicles.

Classical approach — Physical and Mathematical models, code is the primary artifact in development.

Deep Learning Approach — Neural Networks , Data Pipeline and data is the primary artifact in development.

Hybrid Approach —The best of both worlds

The problem statement here is quite simple. Based on the car’s surroundings, current and previous positions it needs to steer itself to the desired position in the road.

The classical approach

• Build models, code and simulation
• Use Rapid prototyping to validate the performance on hardware
• Use data logging for validation

Components of the algorithm

• Inputs from Sensing & Localization
• System Safe Operating Area to specify constraints of system operation
• Models of Surroundings/ Observers / History of Measured Controlled Quantities
• State Machines/ Decision Trees.
• P , PI, PID Controllers

The deep learning approach

• Close the loop that you want to automate, manually at first.
• Log all the measurable quantities that are factors that control the decision or value of the output/set of outputs.
• Accumulate data until you are confident that it is representative of all conditions the system will be posed with and is representative of the design space under consideration.
• Pre-process the dataset, train the network

Components of the algorithm

• Data Pipeline
• Neural Network Architecture

The know-how of the system is built in through the data collection techniques and controlling the system manually in the best possible way during data collection itself rather than using thousands of lines of code capturing every obscure condition the system maybe subject to.

This article explains how simple it is to use the deep learning approach and generate steering angles for camera based systems in under 200 lines of code.

Toolchain

• Simulator in training mode as data gathering/ data source
• Offline Anaconda Python Environment for data pre-processing.
• Cloud based ML environment on AWS for Model Training.
• Download model to offline system and run the simulator in autonomous mode

1. Getting started

1. Offline installation — Install Anaconda, use this Environment

2. Cloud Environment — This is a Setup Tutorial written to help you with the cloud based environment setup.

3. Clone this repository to both the cloud and the local environment Behavioral Cloning Repo

4. Simulator Repo — Link

2. Getting around the repo

• Model_Preprocessing.ipynb is the scratchpad that was used to try and experiment with building the model.It helps us extract and preprocess information and combine it.
• Model.py is the Keras Model that contains Model & Image Aug Generators
• It obtains data from complete_data.csv which contains the sum total of all images , steering angles and their paths.
• The data is obatined by running the simulator / stock data given by udacity itself.

3. Data/File Structure

• The data is of the format ** Drive_Log.csv** which contains the path information where the images are sampled from the video and the actual images are stored in IMG. Each image is timestamped.

4. Workflow

• Simulator — Generate Data in Training Mode
• Analyze, Augment and PreProcess Data offline
• Get more Data if required.
• Upload Data to Cloud — AWS Machine Learning system .
• Run the Model Training.
• Download Model and run simulator in autonomous mode.
• Repeat process

5. Summary of Steps

Exploratory Visualization of Dataset

• Pandas + SeaBorn + MatPlotLib to create , load and append dataset from dataframe
• Visualization to understand the distribution and quality of data.
• Distribution Plot to see the spread and quantity of data
• Time Series Plot to understand the quality of data. (To see noise to determine if filters are required)

Data Collection based on Shortcomings

• Visualizing collected data from driving the simulator shows that the dataset looks entirely different for Keyboard and Mouse
• So we pass the data through a Savitzky Golay filter that averages out the samples but maintains the
• Area = (Steering_angle x Time) this effectively filters out the noise without destroying the signal.
• Based on the histogram distribution plots collecting data by using certain driving styles.
• Data from Track 1 — Clock wise and anticlockwise (MAC & Windows)
• Data from Track 2 — Clock wise and anticlockwise (MAC & Windows)
• Smooth Turn from both Tracks (MAC & Windows)
• Recovery driving from both Tracks (MAC & Windows)
• Problem Areas in both tracks (MAC & Windows)
• Keyboard and Mouse
• After initial model save and testing driving and training in problem areas to improve model on subset of data.

Data Augmentation:

• Augmentation using Flipping, Translation from left and right camera images
• Reduce the time spent on data gathering through data augmentation techniques

Data Perturbation to Increase Model Robustness

• Brightness Perturbation : Random perturbation of brightness of the image.
• Gaussian Noise : Blur filter with a random normal distribution across the image.
• Adaptive Histogram Equalization : Can greatly help in the model learning the features quickly
• Colospace inversion : RBG to BGR colorspace change

Sampling and Image Generators

These steps increase the challenge and generalization capability by creating harder images for the model to train on. Below is an example of augmented and perturbed image batch that is linked with the image generator that generates images during model training on the go. Seen below are the distribution of data and images of one random sample generated by the generator.

• Since the distribution is clearly tri modal (peaks around 0 for Center Camera , + 0.25 and -0.25 for left and right cameras respectievely ) it is an unbalanced dataset.
• Although significant efforts have been taken gather more data around turns , there is just simply more data around 0 and +/-0.25
• Best possible option is to do Balancing through DownSampling
• The method used to do downsampling is Weighted Random Normal Sampling .
• Why we choose this is because , the dominant characteristic of the system is to stay around 0/.25 so we make sure we don’t mess with that.
• The steering angles are Discretized i.e made to fall under categories/ Bins
• Counts are taken for each group and the weights are given as 1/ Counts in that group.
• These weights are then Normalized — When summed up they need to be equal to 1
• Then the batch size is used to sample this out of the data frame using the Sampling with weights

Define model architecture

Data Pre-processing steps

• Normalization through feature scaling
• Cropping region of interest
• Resize image to increase model performance

Salient Features of Model

• Batch Normalization before every activation
• Overfitting prevention Dropouts and batch norm
• Dropouts are implemented before the flatten layer and before the output layer with a 50% probability
• Tried by adding multiple dropouts but it did need seem to have an effect on improving validation losses.
• Switched from ReLU to ELU for activations after reading this paper-https://arxiv.org/abs/1511.07289
• NVIDIA End to End Model architecture and train from scratch

Setup Model Training Pipeline

• Hyperparameters: Epochs , Steps per Epoch and Learning Rate decided based on search epochs on subset of data
• Greedy best save and checkpoint implementation.
• Metrics is a purely loss based. Since the label(Steering angle) here is numeric and non-categorical , RMS Loss is used as the loss type.

Save and Deploy Model

• Save using json, hdf5 model.

6. Observation and Learning

• Pandas Dataframe — Sample is extremely handy in picking a weighted random sample.
• Too many dropouts can sometimes be counter-productive.
• Wasted a lot of time in trying to figure out why the model wasn’t performing well and this was due to the Gaussian blur perturbation that converted images to float.
• The keyboard data though noisy didn’t need filtration. Filtering the steering values lead to slower responses in sharp turns.
• Image data generators were a life saver when it came to handling data in batches, sharing load between GPU and CPU.
• Recovery driving training is only partial as we don’t control the direction, throttle or speed just the steering. However that is a topic for the future. To plug in the steering and throttle based closed loop PI based speed controller.

7. Further aspirations

• Optimize the learning process — Use the model driving the vehicle for iterative data generation for further model training.
• Increase the challenge in driving — Single lane driving based on direction, adversarial agents, pedestrians and traffic to understand model performance.