Deploying and Running CVAT with SAM Integration using Docker and Nuclio
Today in this complete guide I will demonstrate in details how to deploy and run CVAT annotation application with Meta Segment Anything Model (SAM) on your device locally using Docker and Nuclio.
First what is CVAT?
CVAT is an interactive video and image annotation tool for computer vision. It is the industry-leading data engine platform for preparing annotated datasets to develop machine learning models in the computer vision domain.
We will leverage SAM model in CVAT as interaction tool to streamline and automate object segmentation and annotation in it. In order to run SAM integration in CVAT we need to use technology called Nuclio.
What is Nuclio?
Nuclio is an open source and managed serverless platform used to minimize development and automate the deployment of data-science based applications. Nuclio follows a serverless model where functions are executed in response to events. A function in Nuclio is a piece of code that gets executed when triggered. Functions are deployed using a Nuclio runtime environment on Docker and Kubernetes.
In our use case we should use SAM Nuclio serverles function to deploy and run SAM in CVAT Docker application.
Requirements
To run CVAT with SAM integration we should have the following:
Nuclio Plugin Installation
We should install nuctl command line tool to build and deploy Nuclio serverless functions
In our case we are using Windows OS so we need to launch WSL terminal to run the commands if you are using Linux OS distribution then you can run the same commands on your Linux terminal
After you open WSL terminal you should run this command to download nuctl version 1.13.0 on your home directory
wget https://github.com/nuclio/nuclio/releases/download/1.13.0/nuctl-1.13.0-linux-amd64
Then we set the required permission and add softlink for it
sudo chmod +x nuctl-1.13.0-linux-amd64
sudo ln -sf $(pwd)/nuctl-1.13.0-linux-amd64 /usr/local/bin/nuctl
We are ready now to use nuctl to deploy and run Nuclio serverless functions on Docker
Deploying CVAT Server with SAM Integration
In order to deploy and run cvat server the first step we need to do is to clone CVAT GitHub repository on our device
In your home directory run the following command
git clone https://github.com/cvat-ai/cvat.git
Now change your directory to the cloned repository
cd cvat/
Before running any docker command make sure if you are using Linux that Docker and docker-compose are installed on your device and on Windows Docker Desktop.
After that to deploy and run cvat with SAM auto annotation tool run the following docker compose file in detach mode
docker compose -f docker-compose.yml -f components/serverless/docker-compose.serverless.yml up -d
You can check if cvat containers are running successfully
docker-compose ps
Now we are ready to deploy and run SAM Nuclio serverless function
First change your directory
cd serverless/
Then run the following command to start SAM Nuclio integration
./deploy_gpu.sh pytorch/facebookresearch/sam/nuclio/
You can check if SAM Nuclio function deployed successfully using this command
nuctl get functions
Then to validate if SAM model running on Docker check your Docker Desktop on Windows
On Linux run this command to validate it
docker ps
Important note here that you have to run them on the same order to make sure integration work successfully:
Now cvat server is ready and we can start working on the CVAT application
Running CVAT Client on Browser
To start CVAT application open your browser and go to the local host URL:
https://localhost:8080/
Then create new account and sign in to it
After you sign in you will be in cvat tasks home page
Then go to projects tab and create new project, you can set project name to Marine Test
After you create the project you can open it and then create new task and select your project name on it after that upload at least one image for the task
After you create the task open the project and add one new label at least to the project
you can name it jellyfish and set the color to red then open group1 task
You only need to open the task Job#1 to start working on it
This is the main UI for CVAT application on the left side you can see the tools panel
Click on the AI tool and on interactor option select Segment Anything (SAM) and on label select jellyfish
Then you can use the SAM interact tool to hover over Jellyfish object and segment it by clicking on mouse left click and wait for seconds for SAM serverless function to run
After that you only need to click on Done to confirm the new segmentation mask with jellyfish label and click on Save to save your project task job
You can see on the right side the labels that we annotated for the segmentation mask
To export job dataset such as segmentation mask and annotations click on Menu and select Export job dataset
On export format select Segmentation mask 1.1 and you can set custom name for the export such as group1
After that export job will start and you can download the export result on your device
Then you can check job segmentation mask in Segmentation Object and Segmentation Class directories.
At the end we have demonstrated on this comprehensive guide how to deploy and run CVAT with SAM auto annotation tool locally on your device using Docker and Nuclio serverless functions.
Follow me on LinkedIn Mohammad Oghli for more interesting technology articles!
Created by Mohamad Oghli