Using AI for Traffic Light Outage Detection: A Key Machine Learning Lesson

Article Technical Level Guide

This article is pitched at a "moderate depth" level. Some sections might be technical.

Traffic in any part of the world is a major issue. With the remote-work pendulum swinging in favour of a return to office, school runs and evening trips to badminton practice, I am finding that roads in Johannesburg are becoming increasingly busy. To add to this issue, traffic lights (also known as "robots" in South Africa) at some intersections do not work. In the area I live in, traffic lights at some intersections have been out of service for months. Let me know in the comments if you experience a similar issue.

As you will note from the map below, in a sample of 16 intersections, only 7 work (at the time of this writing). Now, South Africans are incredibly hardy and most reading this will undoubtedly be thankful for the lights that do work, thinking back to a few months ago when everything was off during loadshedding. As a nation, we are extremely resilient by navigating these intersections as a 4-way stop or initiatives such as the Pointsmen Project from OUTsurance . These are dedicated people who are dispatched to manually direct traffic to help with congestion and as a preventative measure to minimise accidents so there are fewer insurance claims.

Problem Statement

The Johannesburg Roads Agency City of Johannesburg (JRA) is responsible for maintaining thousands of kilometres of road infrastructure. Drawing a comparison to an IT Operations Centre, a critical requirement for support is to provide visibility into the environment.

Can we build a solution that would determine if a traffic light at an intersection is working - without human intervention? Key to highlight is the requirement to not have human involvement in the detection.

Objective

The aim of this project is to use AI to help detect traffic lights that are out of order. We will ultimately use this information to provide near real-time insight or reporting, that could potentially enable the JRA to best direct resources based on priority of location or duration of outage. As a point of departure, we will assume that we are provided with an image of a traffic intersection. We will investigate how AI can help us identify if:

1. the image contains traffic lights; and
2. the traffic light is working

Solution Approach

As you might recall from Project AI Boom Gate: Part 2 (Analyze), I used Amazon Rekognition to detect text and faces. A benefit of a hands-on learning approach is that I found Amazon Rekognition Custom Labels as part of that project and it helped to address this specific use-case. In summary, it provides an "automated machine learning" capability. You provide a set of labelled images and it creates a machine learning model.

Driving around my neighbourhood, I captured images of traffic lights, in different states at different intersections. I used the camera on my mobile phone to take the photos.

Returning home, I downloaded the ±120 images, cropped and categorised (labeled) into red, amber, green and off. With the images uploaded to S3, a dataset created, I kicked off the training for my first model.

Attempt #1

The next morning, I captured images of new intersections to test the model.

After running inference on the images, I found that one (IMG_3370) worked by reporting a result of "red" but another (IMG_3369) reported no labels found. After significant cropping and manipulation of the image, it eventually reported "red".

Based on the defined objective, I considered this attempt as unsuccessful - as significant "human intervention" was required.

Getting Expert Advice

While researching this issue, I was incredibly fortunate to find a YouTube video of a LEGO Brick Sorting Machine by Mike Chambers . I highly recommend you view this demo. Essentially, he used Amazon Rekognition Custom Labels to identify different types of Lego blocks. After more research, I found the following about the video creator, Mike Chambers:

• he is one of the instructors on a DeepLearning.AI course - Generative AI with LLMs and has collaborated with the legendary Andrew Ng!
• is a Senior Developer Advocate for Generative AI at Amazon Web Services (AWS)

I immediately sent him an email explaining the problem statement, approach and the incorrect classifications. A few hours later, he kindly responded with the following suggestion:

"Okay… this is what I would do.? Focus on getting an object detection algorithm working for the head of the traffic light.? This would allow you to crop out just the head from all the noise in the rest of the image.? From there it should be pretty easy to subsequently process the image to see if the lights are working."

Essentially, Mike suggested that I break down the problem into different "steps"...

Attempt #2

Using the feedback, and without any changes to the existing model, or captured images, I applied the sequence outlined below. Images are also provided for further reference.

1. Use Amazon Rekognition to identify the traffic lights in an image. As you will note from the image below, it does this remarkably well - even for South African "robots". It identified the traffic light with a confidence rating of 99.5%.
2. In the JSON response, Rekognition also provides bounding boxes for the objects detected in the image.
3. Using a python script, I was able to crop the "head of the traffic light" from the image.
4. Passing this into my Custom Label model, it correctly predicted "red".

Instead of running inference on the full image, I was able to use the object detection capability of the Rekognition AI service to find the relevant element and process just that part with the custom ML model.

Further testing of additional images and scenarios confirmed that this approach worked reliably and consistently.

Result

To wrap up. In this article, I:

• presented a "real-world" use case that I was hoping to address with Artificial Intelligence.
• found a technology solution that helped create a custom Machine Learning (ML) model.
• showed how my initial use of the model was not successful.
• broke down the problem based on direction and feedback from a GenAI SME.
• used a revised approach to successfully, more accurately and more consistently make a prediction to solve my use case.

As you observed, there was no change to the ML model. To get the solution to work, I had to change the way I used the model.

Coming up...

In the next instalment, I investigate and implement another AI approach to determine if the traffic light is working.

Note: I work at Amazon, but this is my own opinion.