Object Detection Using EfficientNet in Tensorflow 2

In this tutorial, I'll show the necessary steps to create an object detection algorithm using Google Research's EfficientNet, in Tensorflow 2. The paper, EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks, is available here.

First, let's create the Anaconda environment:

conda create -n tf2 python=3.8 anaconda
source activate tf2        

If source activate does not work, add the full path, like: /home/user/anaconda3/bin/activate tf2

Install Tensorflow:

pip install tensorflow-gpu        

Now, go to your tf2 environment inside anaconda3 folder envs and find inside site-packages, where is tensorflow. It will be something like this:

cd /home/user/.conda/envs/tf2/lib/python3.8/site-packages/tensorflow        

Now, inside tensorflow folder, git clone the tensorflow/models repository:

git clone https://github.com/tensorflow/models        

Good, now, cd into the object detection folder:

cd models/research/object_detection        

Now you must configure the object detection API and proto:

cd models/research
protoc object_detection/protos/*.proto --python_out=.
cp object_detection/packages/tf2/setup.py .
python -m pip install .        

After that, create a folder inside the object_detection folder called "demo"

mkdir demo
cd demo        

Now, inside demo folder you must create 5 other folders:

• Annotations for the labelmap.pbtxt and test.record and train.record that will be used by the Tensorflow model while training.
• Images: with two subfolders train and test. In each subfolder you will add the training and test images and the .xml files correspondent to these files with annotations of the rectangles regarding the objects.
• Inference_graph folder: this is where you will save the exported model .pb after training, for inference.
• Output: folder that will store the checkpoints as the model begins training. Important: every time you start a training, this folder must be empty.
• Pre_Trained_Models: this place is where you are going save the trained EfficientNet model and checkpoints, to initialize your model training

Now, let's do the annotation part. The images are located at /demo/images/train and test. You must first install labelImg from GitHub:

git clone https://github.com/tzutalin/labelImg        

In Ubuntu:

sudo apt-get install pyqt5-dev-tool
sudo pip3 install -r requirements/requirements-linux-python3.txt
make qt5py3
python3 labelImg.py        

Then you will type w and annotate (drag to a rectangle) the images with the format PascalVOC, which is a .xml file.

Now, inside the demo folder, add these 6 files from https://github.com/RubensZimbres/Repo-2022/tree/main/Object-Detection-Tensorflow2

Great. These are the steps:

• You annotate the images, generate a .xml for each one.
• With the script xml_to_csv.py you transform the images names and .xml files to a .csv.
• With the .csv in hand you will generate the Tensorflow .record file that will be used for training the neural net, with the file generate_tfrecord.py
• With the file model_main_tf2.py you will train the model.
• With the file exporter_main_v2.py you will export the trained model to a .pb file.
• With the file webcam.py you are able to make inference in real time.

Now it's the tricky part: you have to customize some files in order to succeed.

After applying the xml_to_csv.py file, you will get something like this for train and for test:

Now you must edit generate_tfrecord.py. For instance, if you have one class, change the following part of the code:

def class_text_to_int(row_label):
??? if row_label == 'desod':
??????? return 1
??? else:
??????? return None        

For two classes:

def class_text_to_int(row_label):
??? if row_label == 'desod':
??????? return 1
??? if row_label == 'car':
??????? return 2
??? else:
??????? return None        

Be sure you run these files inside the object_detection folder, as Python will need the utils package (a folder inside object_detection), otherwise you will get the "no module named utils" error. Another issue you may find is "no module named nets". In this case, run:

export PYTHONPATH=$PYTHONPATH=/home/theone/.conda/envs/tf2/lib/python3.8/site-packages/tensorflow/models/research:/home/theone/.conda/envs/tf2/lib/python3.8/site-packages/tensorflow/models/research/slim        

Now let's generate the .record files:

python generate_tfrecord.py --csv_input=images/train_labels.csv --image_dir=images/train --output_path=train.record
python generate_tfrecord.py --csv_input=images/test_labels.csv --image_dir=images/test --output_path=test.record        

Then, let's download the EfficientNet pre-trained weights:

wget https://download.tensorflow.org/models/object_detection/tf2/20200711/efficientdet_d0_coco17_tpu-32.tar.gz

tar -xvf efficientdet_d0_coco17_tpu-32.tar.gz        

Extract this file to the folder Pre_Trained_Models. In side this folder, you will see there is a pipeline.config file. We are going to edit it, this is crucial. At the beggining of the file, chnage to the number of classes you want:

model 
? ssd {
??? num_classes: 1
??? image_resizer {
????? keep_aspect_ratio_resizer {
??????? min_dimension: 512
??????? max_dimension: 512
??????? pad_to_max_dimension: true
????? }        

Then, at train config part, adjust the batch size to some value that fits in the memory of your computer:

train_config 
? batch_size: 4
? data_augmentation_options {
??? random_horizontal_flip {
??? }
? }
? data_augmentation_options {
??? random_scale_crop_and_pad_to_square {
????? output_size: 512
????? scale_min: 0.10000000149011612
????? scale_max: 2.0
??? }
? }        

Then, fill 5 more paths: fine_tune checkpoint to where is the EfficientNet checkpoints, 2 label_map_path to where is your labelmap.pbtxt file (inside Annotations folder) and the path to your train and test.record files (inside Annotations folder). Two more setups:

Fix the number of steps to 5000 and fine_tune_checkpoint_type to "detection". If you keep the fine_tune_checkpoint_type as "classification" the output will be a class (number), but we need a dictionary with the annotated boxes of object detection. This part will look like this:

? fine_tune_checkpoint: "/home/theone/.conda/envs/tf2/lib/python3.8/site-packages/tensorflow/models/research/object_detection/training-demo-tf2/pre-trained-models/efficientdet_d0_coco17_tpu-32/checkpoint/ckpt-0
? num_steps: 5000
? startup_delay_steps: 0.0
? replicas_to_aggreg.ate: 4
? max_number_o_boxes: 100
? unpad_groundtruth_tensors: false
? fine_tune_checkpoint_type: "detection"
? use_bfloat16: true
? fine_tune_checkpoint_version: V2"        

Now, one last setup: in the folder Annotations, edit label_map.pbtxt to contain this (adjust to the number of classes you have):

item {
? id: 1
? name: 'desod'
}        

Now we can train the object detection model (inside /object_detection folder) using the model_main_tf2.py file. Remember, the output folder inside demo folder must be empty in order to avoid errors.

python model_main_tf2.py --pipeline_config_path=./training-demo-tf2/pre-trained-models/efficientdet_d0_coco17_tpu-32/pipeline.config --model_dir=./training-demo-tf2/output --alsologtostderr        

Good, now we can export the inference_graph. Still inside the /object_detection folder:

python exporter_main_v2.py --trained_checkpoint_dir=./training-demo-tf2/output --pipeline_config_path=./training-demo-tf2/pre-trained-models/efficientdet_d0_coco17_tpu-32/pipeline.config --output_directory ./training-demo-tf2/inference_graph        

This will generate a .pb file whose path must be added to the command line of python3 webcam.py file for inference, along with the labelmap.pbtxt path. Be sure to install OpenCV:

pip3 install opencv-python        

Now, if you want, you can deploy the model in the cloud. Add the Python inference file containing the flask app, requirements.txt (libraries required), .yaml files (configurations) and Dockerfile to a folder. The idea is to generate an image in Google Cloud Container Registry and deploy to Cloud Run for inference.

Supposing you are in the folder where the files (requirements.txt, .yaml files) and Dockerfile are, you can create a docker image:

gcloud builds submit --tag gcr.io/your_project_id/image_name .        

Then deploy in Cloud Run:

gcloud run deploy --image gcr.io/your_project_id/image_name        

For GKE (Google Kubernetes Engine):

$ gcloud container clusters create example-gke --num-nodes 1 --enable-basic-auth --issue-client-certificate --enable-autoscaling --min-nodes 1 --max-nodes 3 --region us-central1 --machine-type n1-standard-2 --local-ssd-count 1 --scopes=cloud-platform,storage-full,compute-rw,service-control,cloud-source-repos --service-account [email protected] 

$ gcloud container clusters get-credentials example-gke --region=us-central1

$ kubectl create secret generic my-app-sa-key --from-file service-account.json

$ kubectl get secret

$ kubectl apply -f deployment.yaml

$ kubectl get pods

$ kubectl get service

$ kubectl describe pod example-gke-12345

$ kubectl apply -f allow-all-network-policy.yaml

$ kubectl apply -f service.yaml