Weekend Hack: Lane Detection, Drift and Self Driving

I had the weekend to myself and so I decided to go back to being a kid and hack something that I would have done if I were a child today. I am really excited about Google self-driving cars in general and especially so now because they are so close to getting it in the hands of customers. So for a hack I was thinking what would be a self driving MVP?

Cruise control was probably the first MVP here. You didn’t need to maintain speed, your car’s computers did that. Then came adaptive or active cruise control. Now you don’t need to maintain distance, the computers could calculate the distance and accelerate or brake appropriately for you all the way to coming to a full stop. The next step was lane detection and drifting close or out of your lane.

If you had lane detection and active cruise control, then you essentially have an incredible product because you can self-drive the lane on the highway by letting the computer steer, accelerate and brake to drive great distances on the lane! I think this would already go a long way in terms of self driving capability. If you are driving 8 hours, most of the time is spent on the highways and maybe just 2 or 3 of them. So if your computer can just stick to a lane (maybe second from the leftmost, say), then there is mostly nothing that you need to do except merge and exit.

So I decided to write some code just for fun. I just wanted to hack for the sake of it. I have no interest in pursuing this as a product. Here are a few tools that I used and a video that I made on Sunday evening of it performing lane detection and alerting.

The entire work was no more than 200 lines of Java code a lot of which was code that I had to write to help me visualize what the computer was doing, i.e., debugging tools. The only investment that went in was a $30 webcam (see below) that I bought from Fry’s Saturday morning. I was initially thinking of using my iPhone of course, but for ease of development - I just used the webcam. 

The primary piece of tool that I used was opencv/javacv which is one of the standards for computer vision. I played around with a few things: converting images to grayscale, threshold functions to smooth out black/white shades that were noisy, Canny edge detector to outline the resulting image and Hough transform to detect linear lines in the resulting image. The lines are in polar co-ordinates (because of some numerical instability during the transform), so I converted them to cartesian  co-ordinates (the usual, rho cos(theta), rho sin(theta)) and had some restrictions on the slopes and intercepts that those lines can have. This is needed since I don’t want it to pick up adjacent lanes, the divider on the highway or even the horizon of the sky. Then I needed to check to see if the triangles induced by the lines are close to isosceles or if the intercept of the lines on the horizontal was out of range. Finally, there were some thresholds to ensure that the drift is real and does not happen all the time so that it does not beep constantly during city driving or even when switching lanes. See the image below for the steps on the transform on an ideal starting image. In reality the images are not even close to this perfect in color and so it is far more noisy. The video shows the drive, the alerts and also what the computer saw during the drive which was surprisingly accurate.

The first three images are the original (a good Google image picture that shows the lanes clearly) then I converted it to grayscale and thresholded it. What the threshold helps us do is remove the black treadmarks, oil marks or even other black marks from the asphalt. We are not interested in those and the human brain doesn’t even recognize them as lines. And we want to computer to ignore them as well. By removing darker than the asphalt shade we can get rid of those. The thresholding functions supported by opencv did not seem adequate and what I wanted was to increase the pixel darkness for everything above a threshold but this does not seem possible natively. I could have written one but in a weekend hack, things like this can add up. So I just used the binary (white or black) threshold which works well here but is trickier in real drives.

The next image is the result of Canny’s edge detection which does a fabulous job of finding the borders of the thresholded image. Lastly we have 6 Houghlines that are just straight lines passing through the most number of points. Again I had some conditions on slopes and intercepts for it to make sure that it was a reasonable candidate to be a lane.

The video above shows me driving from Castro street in Mountain View, merging onto I-85 South and taking the Fremont Avenue exit. I then show how the computer saw the entire drive as well with a solid single white line corresponding to what it detected as the lane.

Hope you enjoyed it. There has never been a better time to be a software engineer. Hack on!