Using Azure Percept to improve onsite workers safety
Ron Dagdag
Engineering Manager @ 7-Eleven | Edge AI, Internet of things, Mixed Reality -> Augmented Intelligence
Here is a how-to article where we explore the use of Azure Percept to improve workers safety and build a proof of concept that connects to a microcontroller to control an alarm. The article describes how to connect Azure Percept to Arduino, set firmware and demonstrates the use of .NET IoT in a docker container.
The importance of Worker Safety
?According to OSHA, the?most common causes of worker deaths on construction sites in America?are the following
Safety is very important on every job site. There are areas the workers must avoid. According to an article from Construction Dive:?Elevator-related construction deaths on the rise
“While the number of total elevator-related deaths among construction and maintenance workers is relatively small when compared to total construction fatalities, the rate of such deaths doubled from 2003 to 2016, from 14 to 28, with a peak of 37 in 2015, according to a report from The Center for Construction Research and Training (CPWR). However, falls are the cause of most elevator-related fatalities, just like in the construction industry at large. More than 53% of elevator-related deaths were from falls, and of those incidents, almost 48% were from heights of more than 30 feet.“
?Looking at these statistics, it’s being at the wrong place at the wrong time. Warning Signs are typically posted around the dangerous area but can be ignored or forgotten.?
What if we can use Azure Percept to monitor these dangerous areas and warn workers to stay away and avoid it. The goal is to have a deterrent, audible sound that reminds them about possible danger. Once worker leave the area and are not detected anymore, the audible sound stops automatically. Also during maintenance period, allow to remotely disable detection with authorizations needed to comply with regulations.?
Here's an explainer video
Getting Started?
We will look at the Hardware, Software requirements and Architecture diagram for this project. Then walk through step-by-step instructions on how to set up and deploy the application.
Hardware
Azure Percept Device Kit https://www.microsoft.com/d/azure-percept-dk/8mn84wxz7xww
ELEGOO UNO R3 Super Starter Kit Compatible With Arduino IDE https://www.elegoo.com/products/elegoo-uno-project-super-starter-kit
Software
Azure Subscription (needed for Azure Percept) https://azure.microsoft.com?
Visual Studio Code?https://code.visualstudio.com/
?Azure IoT Tools Extensions for VS Code https://marketplace.visualstudio.com/items?itemName=vsciot-vscode.azure-iot-tools
?Azure IoT Edge Dev Tools https://github.com/Azure/iotedgedev
?Docker with DockerHub account https://docs.docker.com/get-docker/
?Arduino IDE https://www.arduino.cc/en/software
?
Overall architecture
When the Camera detects a worker that is in the Dangerous Area, an alarm would sound and send telemetry messages to the cloud. Azure Percept Device Kit contains a people detection model that processes frames from the camera. When a worker is detected; it sends a message to the Arduino device to set the buzzer sound and sends the message to IoT Hub.?
?
Instructions
?
Set People Detection Model
Once the Azure Percept Device Kit is connected to the cloud, we can specify to use People Detection Model by going to Vision Tab -> Deploy a Sample Model
Set up Arduino module
I followed the instructions on .NET IoT github page.
?
You can download the Firmata firmware that I used for this project at:
?
Here are the steps:
Send alert to arduino module
Open the Percept Edge Solution project in Visual Studio Code
领英推荐
?This module can run locally on your machine if you have Azure IoT Edge Dev tool and also Azure IoT Tools Extensions for VS Code
To run it locally on your machine, you may have to change this module.json
Change repository to your Dockerhub username.?
"repository": "rondagdag/arduino-percept-module"
?The Arduino Module is a C# application that controls the arduino device. It’s a docker container that uses .NET IoT bindings. Here are the?Nuget packages?I used.??
<PackageReference Include="Microsoft.Azure.Devices.Client" Version="1.38.0" />
<PackageReference Include="Microsoft.Extensions.Configuration" Version="5.0.0" />
<PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" Version="5.0.0" />
<PackageReference Include="Microsoft.Extensions.Configuration.Binder" Version="5.0.0" />
<PackageReference Include="Microsoft.Extensions.Configuration.EnvironmentVariables" Version="5.0.0" />
<PackageReference Include="Microsoft.Extensions.Configuration.FileExtensions" Version="5.0.0" />
<PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="5.0.0" />
<PackageReference Include="System.Runtime.Loader" Version="4.3.0" />
<PackageReference Include="System.IO.Ports" Version="5.0.1" />
<PackageReference Include="System.Device.Gpio" Version="1.5.0" />
<PackageReference Include="Iot.Device.Bindings" Version="1.5.0" />
<PackageReference Include="Microsoft.Extensions.Logging.Console" Version="5.0.0" />
?The module tries to connect to two Arduino USB ports. I noticed that sometimes during reboot it can be either one of these ports that the arduino is connected to.
/dev/ttyACM0
/dev/ttyACM1
?In order to run this locally on your machine, you may have to change the port number to what Arduino IDE determines UNO is connected to (COM3 for Windows or /dev/ttyS1 for Mac)
It connects to 115200 Baud Rate.
The Buzzer is connected to PIN 12 shown below:
As messages are received from the IoT Edge Hub, it can be processed to detect if a worker has been detected. If the payload contains items in the NEURAL NETWORK node, then we can send the alert to Arduino.???
Here’s the?code?to analyze.?
It might be tricky to run this locally on your machine and may need some modifications. You may have to modify?deployment template?and the code to receive the payload coming from Simulated Temperature.
Notice how the ports are mapped on the docker container.?/dev/ttyACM0 and /dev/ttyACM1
"ArduinoModule": {
"version": "1.0",
"type": "docker",
"status": "running",
"restartPolicy": "always",
"settings": {
"image": "${MODULES.ArduinoModule}",
"createOptions": {
"HostConfig": {
"Privileged": true,
"Devices": [
{
"PathOnHost": "/dev/ttyACM0",
"PathInContainer": "/dev/ttyACM0",
"CgroupPermissions": "rwm"
},
{
"PathOnHost": "/dev/ttyACM1",
"PathInContainer": "/dev/ttyACM1",
"CgroupPermissions": "rwm"
}
]
}
}
}
}
Locally the EdgeHub mapping looks like this. The Simulated Temperature sensor is passing the output to the Arduino Module. Then the Arduino Module filters the data, controls the arduino and sends the telemetry up to the cloud.
"$edgeHub": {
"properties.desired": {
"schemaVersion": "1.2",
"routes": {
"sensorToArduinoModule": "FROM /messages/modules/SimulatedTemperatureSensor/outputs/temperatureOutput INTO BrokeredEndpoint(\"/modules/ArduinoModule/inputs/input1\")",
"ArduinoModuleToIoTHub": "FROM /messages/modules/ArduinoModule/outputs/* INTO $upstream"
},
"storeAndForwardConfiguration": {
"timeToLiveSecs": 7200
}
}
You can test this out on the Simulator
Deploy setup
In order to deploy this to Azure Percept, IoT Hub device -> Set Modules -> Add -> IoT Edge Module
Fill in Module Name and Image URI.
Also the Container Create Options to map the correct port.
Specify the Routes. In this case we have to push data coming from Azure Eye Module To Arduino Module.
?Here’s what the mapping would look like
"$edgeHub": {
"properties.desired": {
"routes": {
"AzureSpeechToIoTHub": "FROM /messages/modules/azureearspeechclientmodule/outputs/* INTO $upstream",
"AzureEyeModuleToArduinoModule": "FROM /messages/modules/azureeyemodule/outputs/* INTO BrokeredEndpoint(\"/modules/ArduinoModule/inputs/input1\")",
"ArduinoModuleToIoTHub": "FROM /messages/modules/ArduinoModule/outputs/* INTO $upstream"
},
?
Summary
Keeping workers safe is very important in any job site. Azure Percept can help with people detection and connection with different alerting systems. We’ve demonstrated setting up the Azure Percept Dev Kit to use people detection. We’ve connected Arduino with a buzzer to trigger an audible sound. This actually enables Azure Percept Dev Kit to expand capabilities and broaden use cases. Let me know if this blog helped you in any way by commenting below, I am interested in learning different ideas and use cases.?
Resources
?Reference Code: https://github.com/rondagdag/arduino-percept
?Getting Started With Azure Percept: https://docs.microsoft.com/azure/azure-percept/
?
Join the adventure, Get your own Percept here
?Get Azure https://www.azure.com/account/free
?Purchase Azure Percept –?Build your Azure Percept
Ron Dagdag
Lead Software Engineer
During the day, Ron Dagdag is a Lead Software Engineer with 20+ years of experience working on a number of business applications using a diverse set of frameworks and languages. He currently supports developers at Spacee with their IoT, Cloud and ML development. On the side, Ron Dagdag is an active participant in the community as a Microsoft MVP, speaker, maker and blogger. He is passionate about Augmented Intelligence, studying the convergence of Augmented Reality/Virtual Reality, Machine Learning and the Internet of Things.?https://www.dhirubhai.net/in/rondagdag/