Training a Deep Learning model for custom object detection using TensorFlow Object Detection API in Google Colab and converting it to a TFLite model for deploying on mobile devices like Android, iOS, Raspberry Pi, IoT devices using the sample TFLite object detection app from TensorFlow’s GitHub.

IMPORTANT:

This tutorial is for TensorFlow 2.x

THIS IS FOR THE LATEST VERSION TENSORFLOW APP. USE THE FOLLOWING VERSION REFERENCE APP USED IN THIS TUTORIAL:

https://github.com/tensorflow/examples/tree/master/lite/examples/object_detection/android

Roadmap

• – Collect the dataset of images and label them to get their XML files.
• – Install the TensorFlow Object Detection API.
• – Generate the TFRecord files required for training. (need generate_tfrecord.py script and CSV files for this)
• – Edit the model pipeline config file and download the pre-trained model checkpoint.
• – Train and evaluate the model.
• – Export and convert the model into TFlite(TensorFlow Lite) format.
• – Deploy the TFlite model on Android / iOS / IoT devices using the sample TFLite object detection app from TensorFlow’s GitHub.

Objective: Build android app for custom object detection

In this article, I will be training an object detection model for a custom object and converting it to a TFlite model so it can be deployed on Android, iOS, IoT devices. Follow the 21 steps mentioned below. (The first 16 steps are the same as my previous article on training an ML model using TF 2. Since TFLite has support for only SSD models at the moment, we will be using an SSD model here)

( But first Subscribe to my YouTube channel https://bit.ly/3Ap3sdi )

1. Import Libraries
2. Create customTF2, training, and data folders in your google drive
3. Create and upload your image files and XML files
4. Upload the generate_tfrecord.py file to the customTF2 folder in your drive
5. Mount drive and link your folder
6. Clone the TensorFlow models git repository & Install TensorFlow Object Detection API
7. Test the model builder
8.  Navigate to /mydrive/customTF2/data/ and Unzip the images.zip and annotations.zip files into the data folder
9. Create test_labels & train_labels
10. Create CSV and “label_map.pbtxt” files
11. Create ‘train.record’ & ‘test.record’ files
12. Download pre-trained model checkpoint
13. Get the model pipeline config file, make changes to it, and put it inside the data folder
14. Load Tensorboard
15. Train the model
16. Test your trained model
17. Install tensorflow-nightly
18. Export SSD TFlite graph
19. Convert saved model to TFlite model
20. Create TFLite metadata
21. Download the TFLite model with metadata and deploy it on a mobile device

HOW TO BEGIN?

–Open my Colab notebook on your browser.

–Click on File in the menu bar and click on Save a copy in drive. This will open a copy of my Colab notebook on your browser which you can now use.

–Next, once you have opened the copy of my notebook and are connected to the Google Colab VM, click on Runtime in the menu bar and click on Change runtime type. Select GPU and click on save.

LET’S BEGIN !!

1) Import Libraries

import os
import glob
import xml.etree.ElementTree as ET
import pandas as pd
import tensorflow as tf

2) Create customTF2, training, and data folders in your google drive

Create a folder named customTF2 in your google drive.

Create another folder named training inside the customTF2 folder ( training folder is where the checkpoints will be saved during training ).

Create another folder named data inside the customTF2 folder.

3) Create and upload your image files and its their corresponding labeled XML files.

Create a folder named images for your custom dataset images and create another folder named annotations for its corresponding PASCAL_VOC format labeled XML files.

Next, create their zip files and upload them to the customTF2 folder in your drive.

Make sure all the image files have their extension as “.jpg” only. 

Other formats like <.png> , <.jpeg> or even <.JPG> will give errors since the generate_tfrecord and xml_to_csv scripts here have only <.jpg> in them. If you have other format images, then make changes in the scripts accordingly.

For Datasets, you can check out my Dataset Sources at the bottom of this article in the credits section. 

Collecting Images Dataset and labeling them to get their PASCAL_VOC XML annotations.

Labeling your Dataset

Input image example (Image1.jpg)

You can use any software for labeling like the labelImg tool.

I use an open-source labeling tool called OpenLabeling with a very simple UI.

Click on the link below to know more about the labeling process and other software for it:

• Image Dataset Labeling article

NOTE: Garbage In = Garbage Out. Choosing and labeling images is the most important part. Try to find good-quality images. The quality of the data goes a long way towards determining the quality of the result.

The output PASCAL_VOC labeled XML file looks like as shown below:

4) Upload the generate_tfrecord.py file to the customTF2 folder in your drive.

You can find the generate_tfrecord.py file here

5) Mount drive and link your folder

#mount drive

from google.colab import drive
drive.mount('/content/gdrive')

# this creates a symbolic link so that now the path /content/gdrive/My Drive/ is equal to /mydrive

!ln -s /content/gdrive/My Drive/ /mydrive
!ls /mydrive

6) Clone the TensorFlow models git repository & Install TensorFlow Object Detection API

# clone the tensorflow models on the colab cloud vm

!git clone --q https://github.com/tensorflow/models.git

# navigate to /models/research folder to compile protos

%cd models/research

# Compile protos.

!protoc object_detection/protos/*.proto --python_out=.

# Install TensorFlow Object Detection API.

!cp object_detection/packages/tf2/setup.py .
!python -m pip install 

7) Test the model builder

!python object_detection/builders/model_builder_tf2_test.py

8) Navigate to /mydrive/customTF2/data/ and Unzip the images.zip and annotations.zip files into the data folder

%cd /mydrive/customTF2/data/

# unzip the datasets and their contents so that they are now in /mydrive/customTF2/data/ folder

!unzip /mydrive/customTF2/images.zip -d .
!unzip /mydrive/customTF2/annotations.zip -d .

9) Create test_labels & train_labels

Current working directory is /mydrive/customTF2/data/

Divide annotations into test_labels(20%) and train_labels(80%).

10) Create the CSV files and the “label_map.pbtxt” file

Current working directory is /mydrive/customTF2/data/

Run xml_to_csv script below to create test_labels.csv and train_labels.csv

This script also creates the label_map.pbtxt file using the classes mentioned in the xml files.

The 3 files that are created i.e. train_labels.csv, test_labels.csv, and label_map.pbtxt look like as shown below:

The train_labels.csv contains the name of all the train images, the classes in those images, and their annotations.

The test_labels.csv contains the name of all the test images, the classes in those images, and their annotations.

The label_map.pbtxt file contains the names of the classes from your labeled XML files. 

NOTE: I have 2 classes i.e. “with_mask“ and “without_mask“.

Label map id 0 is reserved for the background label.

11) Create train.record & test.record files

Current working directory is /mydrive/customTF2/data/

Run the generate_tfrecord.py script to create train.record and test.record files

#Usage:
#!python generate_tfrecord.py output.csv output_pb.txt /path/to/images output.tfrecords

#For train.record
!python /mydrive/customTF2/generate_tfrecord.py train_labels.csv  label_map.pbtxt images/ train.record

#For test.record
!python /mydrive/customTF2/generate_tfrecord.py test_labels.csv  label_map.pbtxt images/ test.record

The total number of image files is 1370. Since we divided the labels into two categories viz. train_labels(80%) and test_labels(20%), the number of files for “train.record” is 1096, and the number of files for “test.record” is 274.

12) Download pre-trained model checkpoint

Current working directory is /mydrive/customTF2/data/

You can choose any model for training depending upon your data and requirement. Read this blog for more info on which. The official list of detection model checkpoints for TensorFlow 2.x can be found here.

However, since TFLite doesn’t support all models right now, the options for this are limited at the moment. TensorFlow is working towards adding more models with TFLite support. Read more about TFLite compatible models for all ML modules like object detection, image classification, image segmentation, etc here.

Currently, TFLite supports only SSD models (excluding EfficientDet)

In this tutorial, I will use the ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8 model.

# Download the pre-trained model ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8.tar.gz into the data folder & unzip it

!wget http://download.tensorflow.org/models/object_detection/tf2/20200711/ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8.tar.gz

!tar -xzvf ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8.tar.gz

13) Get the model pipeline config file, make changes to it and put it inside the data folder

Current working directory is /mydrive/customTF2/data/

Download ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8.config from /content/models/research/object_detection/configs/tf2. Make the required changes to it and upload it to the /mydrive/custom/data folder.

OR

Edit the config file from /content/models/research/object_detection/configs/tf2 in colab vm and copy the edited config file to the /mydrive/customTF2/data folder.

You can also find the pipeline config file inside the model checkpoint folder we just downloaded in the previous step.

You need to make the following changes:

• change num_classes to the number of your classes.
• change test.record path, train.record path & labelmap path to the paths where you have created these files (paths should be relative to your current working directory while training).
• change fine_tune_checkpoint to the path of the directory where the downloaded checkpoint from step 12 is.
• change fine_tune_checkpoint_type with value classification or detection depending on the type.
• change batch_size to any multiple of 8 depending upon the capability of your GPU. (eg:- 24,128,…,512). The better the GPU capability, the higher you can go. Mine is set to 64. Mine is set to 64.
• change num_steps to the number of steps you want the detector to train.

Max batch size= available GPU memory bytes / 4 / (size of tensors + trainable parameters)

Next, copy the edited config file.

# copy the edited config file from the configs/tf2 directory to the data/ folder in your drive

!cp /content/models/research/object_detection/configs/tf2/ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8.config /mydrive/customTF2/data

The workspace at this point:

There are many data augmentation options that you can add. Check the full list here. For beginners, the above changes are sufficient.

Data Augmentation Suggestions (optional)

First, you should train the model using the sample config file with the above basic changes and see how well it does. If you are overfitting, then you might want to do some more image augmentations.

In the sample config file: random_horizontal_flip & ssd_random_crop are added by default. You could try adding the following as well:

(Note: Each image augmentation will increase the training time drastically)

1. from train_config {}:

data_augmentation_options {
    random_adjust_contrast {
    }
  }
  data_augmentation_options {
    random_rgb_to_gray {
    }
  }
  data_augmentation_options {
    random_vertical_flip {
    }
  }
  data_augmentation_options {
    random_rotation90 {
    }
  }
  data_augmentation_options {
    random_patch_gaussian {
    }
  }

2. In model {} > ssd {} > box_predictor {}: set use_dropout to true This will help you to counter overfitting.

3. In eval_config : {} set the number of testing images you have in num_examples and remove max_eval to evaluate indefinitely

eval_config: {
  num_examples: 274 # set this to the number of test images we divided earlier
  num_visualizations: 20 # the number of visualization to see in tensorboard
}

14) Load Tensorboard

%load_ext tensorboard
%tensorboard --logdir '/content/gdrive/MyDrive/customTF2/training'

15) Train the model

Navigate to the object_detection folder in Colab VM

%cd /content/models/research/object_detection

15 (a) Training using model_main_tf2.py

Here {PIPELINE_CONFIG_PATH} points to the pipeline config and {MODEL_DIR} points to the directory in which training checkpoints and events will be written.

# Run the command below from the content/models/research/object_detection directory

"""
PIPELINE_CONFIG_PATH=path/to/pipeline.config
MODEL_DIR=path to training checkpoints directory
NUM_TRAIN_STEPS=50000
SAMPLE_1_OF_N_EVAL_EXAMPLES=1

python model_main_tf2.py -- \
--model_dir=$MODEL_DIR --num_train_steps=$NUM_TRAIN_STEPS \
--sample_1_of_n_eval_examples=$SAMPLE_1_OF_N_EVAL_EXAMPLES \
--pipeline_config_path=$PIPELINE_CONFIG_PATH \
--alsologtostderr
"""

!python model_main_tf2.py --pipeline_config_path=/mydrive/customTF2/data/ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8.config --model_dir=/mydrive/customTF2/training --alsologtostderr

NOTE :

For best results, you should stop the training when the loss is less than 0.1 if possible, else train the model until the loss does not show any significant change for a while. The ideal loss should be below 0.05 (Try to get the loss as low as possible without overfitting the model. Don’t go too high on training steps to try and lower the loss if the model has already converged viz. if it does not reduce loss significantly any further and takes a while to go down.)

Ideally, we want the loss to be as low as possible but we should be careful so that the model does not over-fit. You can set the number of steps to 50000 and check if the loss goes below 0.1 and if not, then you can retrain the model with a higher number of steps.

The output will normally look like it has “frozen”, but DO NOT rush to cancel the process. The training outputs logs only every 100 steps by default, therefore if you wait for a while, you should see a log for the loss at step 100. The time you should wait can vary greatly, depending on whether you are using a GPU and the chosen value for batch_size in the config file, so be patient.

15 (b) Evaluation using model_main_tf2.py (Optional)

You can run this in parallel by opening another colab notebook and running this command simultaneously along with the training command above (don’t forget to mount the drive, clone the TF git repo and install the TF2 object detection API there as well). This will give you validation loss, mAP, etc so you have a better idea of how your model is performing.

Here {CHECKPOINT_DIR} points to the directory with checkpoints produced by the training job. Evaluation events are written to {MODEL_DIR/eval}.

# Run the command below from the content/models/research/object_detection directory

"""
PIPELINE_CONFIG_PATH=path/to/pipeline.config
MODEL_DIR=path to training checkpoints directory
CHECKPOINT_DIR=${MODEL_DIR}
NUM_TRAIN_STEPS=50000
SAMPLE_1_OF_N_EVAL_EXAMPLES=1

python model_main_tf2.py -- \
--model_dir=$MODEL_DIR --num_train_steps=$NUM_TRAIN_STEPS \
--checkpoint_dir=${CHECKPOINT_DIR} \
--sample_1_of_n_eval_examples=$SAMPLE_1_OF_N_EVAL_EXAMPLES \
--pipeline_config_path=$PIPELINE_CONFIG_PATH \
--alsologtostderr
"""

!python model_main_tf2.py --pipeline_config_path=/mydrive/customTF2/data/ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8.config --model_dir=/mydrive/customTF2/training/ --checkpoint_dir=/mydrive/customTF2/training/ --alsologtostderr

RETRAINING THE MODEL ( in case you get disconnected )

If you get disconnected or lose your session on Colab VM, you can start your training where you left off as the checkpoint is saved on your drive inside the training folder. To restart the training simply run steps 1, 5, 6, 7, 14, and 15.

Note that since we have all the files required for training like the record files, our edited pipeline config file, the label_map file, and the model checkpoint folder, we do not need to create these again.

The model_main_tf2.py script saves the checkpoint every 1000 steps. The training automatically restarts from the last saved checkpoint itself.

However, if you see that it doesn’t restart training from the last checkpoint you can make 1 change in the pipeline config file. Change fine_tune_checkpoint to where your latest trained checkpoints have been written and have it point to the latest checkpoint as shown below:

fine_tune_checkpoint: "/mydrive/customTF2/training/ckpt-X" (where ckpt-X is the latest checkpoint)

Read this TensorFlow Object Detection API tutorial to know more about the training process for TF2.

16) Test your trained model

Export inference graph

Current working directory is /content/models/research/object_detection

!python exporter_main_v2.py --trained_checkpoint_dir=/mydrive/customTF2/training --pipeline_config_path=/content/gdrive/MyDrive/customTF2/data/ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8.config --output_directory /mydrive/customTF2/data/inference_graph

Note: The trained_checkpoint_dir parameter in the above command needs the path to the training directory. There is a file called “checkpoint” which has all the model paths and the latest model checkpoint path saved in it. So it automatically uses the latest checkpoint. In my case, the checkpoint file had ckpt-36 written in it for the latest model_checkpoint_path.

For pipeline_config_path give the path to the edited config file we used to train the model above.

Test your trained Object Detection model on images

Current working directory is /content/models/research/object_detection

This step is optional.

# Different font-type and font-size for labels text

!wget https://freefontsdownload.net/download/160187/arial.zip
!unzip arial.zip -d .

%cd utils/
!sed -i "s/font = ImageFont.truetype('arial.ttf', 24)/font = ImageFont.truetype('arial.ttf', 50)/" visualization_utils.py
%cd ..

Test your trained model

For testing on webcam capture or videos, use this colab notebook.

CONVERTING THE TRAINED SSD MODEL TO TFLITE MODEL

17) Install tf-nightly

(TFLite converter works better with tf-nightly. Using tf-nightly is recommended. You can also try using the latest TensorFlow stable version.)

!pip install tf-nightly

NOTE: If running the above command asks you to restart runtime and you lose all the local variables in your colab VM, you have to run steps 5 & 6 again to mount the drive, clone the TF models repo and install object detection API. After running those steps again run step 18 below.

18) Export SSD TFlite graph

Current working directory is /content/models/research/object_detection

%cd /content/models/research/object_detection

!python export_tflite_graph_tf2.py --pipeline_config_path /content/gdrive/MyDrive/customTF2/data/ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8.config --trained_checkpoint_dir /mydrive/customTF2/training --output_directory /mydrive/customTF2/data/tflite 

19) Convert the TensorFlow saved model to TFlite model

Check input and output tensor names

!saved_model_cli show --dir /mydrive/customTF2/data/tflite/saved_model --tag_set serve --all

Convert to TFlite – Use either Method (a) or Method (b) to convert to TFLite .

Method (a) :- Using the commad-line tool

Convert the saved model to TFLite using the tflite_convert command. This is the simple method used for basic model conversion. For beginners, I recommend using this method first. Although the second method i.e. Python_API is highly recommended everywhere as it has more support and features available, you can start out by using the first command-line tool method for testing, and then you can use the second one later on which also allows us to apply optimizations and post-training quantizations, etc. 

Run the following code block to create the TFLite model using tflite_convert command-line tool.

# The default inference type is Floating-point.

%cd /mydrive/customTF2/data/

!tflite_convert --saved_model_dir=tflite/saved_model --output_file=tflite/detect.tflite

Method (b):- Using Python API 

Convert saved model to TFLite using Python API. This is the better option as per the TensorFlow documentation since we can apply other features and optimizations like post-training quantizations that can reduce the model size and also improve CPU and hardware accelerator latency. You can change the code below depending on your requirements. Read the links just under these code blocks below to get a broader understanding of this and why it is better and also try it once you understand the basics of this entire process of creating a TFLite model.

# Navigate to the data folder

%cd /mydrive/customTF2/data/

Run the following code block to create the TFLite model using Python API. You can use whatever inference type or apply any optimizations you want.

'''*******************************
   FOR FLOATING-POINT INFERENCE
**********************************'''

import tensorflow as tf

saved_model_dir = '/mydrive/customTF2/data/tflite/saved_model'

converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
tflite_model = converter.convert()
open("/mydrive/customTF2/data/tflite/detect.tflite", "wb").write(tflite_model)


'''**************************************************
#  FOR FLOATING-POINT INFERENCE WITH OPTIMIZATIONS
#**************************************************'''

# import tensorflow as tf
# converter = tf.lite.TFLiteConverter.from_saved_model('/mydrive/customTF2/data/tflite/saved_model',signature_keys=['serving_default'])
# converter.optimizations = [tf.lite.Optimize.DEFAULT]
# converter.experimental_new_converter = True
# converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS,  
#                                        tf.lite.OpsSet.SELECT_TF_OPS]
# tflite_model = converter.convert()

# with tf.io.gfile.GFile('/mydrive/customTF2/data/tflite/detect.tflite', 'wb') as f:
#   f.write(tflite_model)


'''**********************************
    FOR DYNAMIC RANGE QUANTIZATION 
*************************************
 The model is now a bit smaller with quantized weights, but other variable data is still in float format.'''


# import tensorflow as tf

# converter = tf.lite.TFLiteConverter.from_saved_model('/mydrive/customTF2/data/tflite/saved_model',signature_keys=['serving_default'])
# converter.optimizations = [tf.lite.Optimize.DEFAULT]
# tflite_quant_model = converter.convert()

# with tf.io.gfile.GFile('/mydrive/customTF2/data/tflite/detect.tflite', 'wb') as f:
#   f.write(tflite_quant_model)


# '''***********************************************************************
#    FOR INTEGER WITH FLOAT FALLBACK QUANTIZATION WITH DEFAULT OPTMIZATIONS 
# **************************************************************************
# Now all weights and variable data are quantized, and the model is significantly smaller compared to the original TensorFlow Lite model.
# However, to maintain compatibility with applications that traditionally use float model input and output tensors, 
# the TensorFlow Lite Converter leaves the model input and output tensors in float'''

# import tensorflow as tf
# import numpy as np

# saved_model_dir = '/mydrive/customTF2/data/tflite/saved_model'

# def representative_dataset():
#     for _ in range(100):
#       data = np.random.rand(1, 320, 320, 3)
#       yield [data.astype(np.float32)]

# converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
# converter.optimizations = [tf.lite.Optimize.DEFAULT]
# converter.representative_dataset = representative_dataset
# tflite_quant_model = converter.convert()

# with open('/mydrive/customTF2/data/tflite/detect.tflite', 'wb') as f:
#   f.write(tflite_quant_model)


# '''*********************************
#   FOR FULL INTEGER QUANTIZATION
# ************************************
# The internal quantization remains the same as previous float fallback quantization method, 
# but you can see the input and output tensors here are also now integer format'''

# import tensorflow as tf
# import numpy as np

# saved_model_dir = '/mydrive/customTF2/data/tflite/saved_model'

# def representative_dataset():
#     for _ in range(100):
#       data = np.random.rand(1, 320, 320, 3)
#       yield [data.astype(np.float32)]

# converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
# converter.optimizations = [tf.lite.Optimize.DEFAULT]
# converter.representative_dataset = representative_dataset
# converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
# converter.inference_input_type = tf.uint8  
# converter.inference_output_type = tf.uint8 
# tflite_quant_model_full_int = converter.convert()

# with open('/mydrive/customTF2/data/tflite/detect.tflite', 'wb') as f:
#   f.write(tflite_quant_model_full_int)

To learn more about the above 2 methods, optimizations, post-training quantization and why we might need them, check out the links below:

TensorFlow Lite converter

Post-training quantization | TensorFlow Lite

Model optimization | TensorFlow Lite

Post-training quantization – Colab Notebook by TensorFlow

NOTE: All the above conversions create “detect.tflite” model. I have used one conversion and commented out the rest. If you run more than one conversion the second one will overwrite the first one. My suggestion is to run one of these conversions first and create the TFLite model with metadata in the next step. Once you have the “detect.tflite” with the metadata for one model, download it, then you can come back and re-run this step for the other model with optimizations and post-training quantization and then also create its TFLite model with metadata. I have written the same name for all the conversions as it is also used in the commands below. If you use a different name, make changes in the next steps accordingly.

20) Create TFLite metadata

The new TensorFlow Lite sample for object detection requires that the final TFLite model should have metadata attached to it in order to run. You can read more about this on the official TensorFlow site here. Run the following steps to get the TFLite model with metadata.

Install tflite_support_nightly package

pip install tflite_support_nightly

Create a separate folder named “tflite_with_metadata” inside the “tflite” folder to save the final TFLite model with metadata added to it.

%cd /mydrive/customTF2/data/
%cd tflite/
!mkdir tflite_with_metadata
%cd ..

Create and Upload “labelmap.txt” file

Create and upload “labelmap.txt” file which we will use inside Android Studio later as well. This file is different than the “label_map.pbtxt” while which we used in Steps 10 & 11. This “labelmap.txt” file only has the names of the classes written in each line and nothing more. Upload this file to /mydrive/customTF2/data folder.The labelmap.txt file looks like as shown below:

Run the code in the code block below to create the TFLite model with metadata.

(NOTE: Change the paths in lines 14, 15 & 16 & 87 in the code block below to your paths. That is only if you’re using different paths for your files. But if you’re following this tutorial you can leave it as it is

• Line 14: The input TFLite model without metadata created above in Step 19.
• Line 15 and 87: The labelmap.txt file.
• Line 16: The location for the final TFLite model output with the metadata added to it.

from tflite_support.metadata_writers import object_detector
from tflite_support.metadata_writers import writer_utils
from tflite_support import metadata
import flatbuffers
import os
from tensorflow_lite_support.metadata import metadata_schema_py_generated as _metadata_fb
from tensorflow_lite_support.metadata.python import metadata as _metadata
from tensorflow_lite_support.metadata.python.metadata_writers import metadata_info
from tensorflow_lite_support.metadata.python.metadata_writers import metadata_writer
from tensorflow_lite_support.metadata.python.metadata_writers import writer_utils

ObjectDetectorWriter = object_detector.MetadataWriter

_MODEL_PATH = "/mydrive/customTF2/data/tflite/detect.tflite"
_LABEL_FILE = "/mydrive/customTF2/data/labelmap.txt"
_SAVE_TO_PATH = "/mydrive/customTF2/data/tflite/tflite_with_metadata/detect.tflite"

writer = ObjectDetectorWriter.create_for_inference(
    writer_utils.load_file(_MODEL_PATH), [127.5], [127.5], [_LABEL_FILE])
writer_utils.save_file(writer.populate(), _SAVE_TO_PATH)

# Verify the populated metadata and associated files.
displayer = metadata.MetadataDisplayer.with_model_file(_SAVE_TO_PATH)
print("Metadata populated:")
print(displayer.get_metadata_json())
print("Associated file(s) populated:")
print(displayer.get_packed_associated_file_list())

model_meta = _metadata_fb.ModelMetadataT()
model_meta.name = "SSD_Detector"
model_meta.description = (
    "Identify which of a known set of objects might be present and provide "
    "information about their positions within the given image or a video "
    "stream.")

# Creates input info.
input_meta = _metadata_fb.TensorMetadataT()
input_meta.name = "image"
input_meta.content = _metadata_fb.ContentT()
input_meta.content.contentProperties = _metadata_fb.ImagePropertiesT()
input_meta.content.contentProperties.colorSpace = (
    _metadata_fb.ColorSpaceType.RGB)
input_meta.content.contentPropertiesType = (
    _metadata_fb.ContentProperties.ImageProperties)
input_normalization = _metadata_fb.ProcessUnitT()
input_normalization.optionsType = (
    _metadata_fb.ProcessUnitOptions.NormalizationOptions)
input_normalization.options = _metadata_fb.NormalizationOptionsT()
input_normalization.options.mean = [127.5]
input_normalization.options.std = [127.5]
input_meta.processUnits = [input_normalization]
input_stats = _metadata_fb.StatsT()
input_stats.max = [255]
input_stats.min = [0]
input_meta.stats = input_stats

# Creates outputs info.
output_location_meta = _metadata_fb.TensorMetadataT()
output_location_meta.name = "location"
output_location_meta.description = "The locations of the detected boxes."
output_location_meta.content = _metadata_fb.ContentT()
output_location_meta.content.contentPropertiesType = (
    _metadata_fb.ContentProperties.BoundingBoxProperties)
output_location_meta.content.contentProperties = (
    _metadata_fb.BoundingBoxPropertiesT())
output_location_meta.content.contentProperties.index = [1, 0, 3, 2]
output_location_meta.content.contentProperties.type = (
    _metadata_fb.BoundingBoxType.BOUNDARIES)
output_location_meta.content.contentProperties.coordinateType = (
    _metadata_fb.CoordinateType.RATIO)
output_location_meta.content.range = _metadata_fb.ValueRangeT()
output_location_meta.content.range.min = 2
output_location_meta.content.range.max = 2

output_class_meta = _metadata_fb.TensorMetadataT()
output_class_meta.name = "category"
output_class_meta.description = "The categories of the detected boxes."
output_class_meta.content = _metadata_fb.ContentT()
output_class_meta.content.contentPropertiesType = (
    _metadata_fb.ContentProperties.FeatureProperties)
output_class_meta.content.contentProperties = (
    _metadata_fb.FeaturePropertiesT())
output_class_meta.content.range = _metadata_fb.ValueRangeT()
output_class_meta.content.range.min = 2
output_class_meta.content.range.max = 2
label_file = _metadata_fb.AssociatedFileT()
label_file.name = os.path.basename("labelmap.txt")
label_file.description = "Label of objects that this model can recognize."
label_file.type = _metadata_fb.AssociatedFileType.TENSOR_VALUE_LABELS
output_class_meta.associatedFiles = [label_file]

output_score_meta = _metadata_fb.TensorMetadataT()
output_score_meta.name = "score"
output_score_meta.description = "The scores of the detected boxes."
output_score_meta.content = _metadata_fb.ContentT()
output_score_meta.content.contentPropertiesType = (
    _metadata_fb.ContentProperties.FeatureProperties)
output_score_meta.content.contentProperties = (
    _metadata_fb.FeaturePropertiesT())
output_score_meta.content.range = _metadata_fb.ValueRangeT()
output_score_meta.content.range.min = 2
output_score_meta.content.range.max = 2

output_number_meta = _metadata_fb.TensorMetadataT()
output_number_meta.name = "number of detections"
output_number_meta.description = "The number of the detected boxes."
output_number_meta.content = _metadata_fb.ContentT()
output_number_meta.content.contentPropertiesType = (
    _metadata_fb.ContentProperties.FeatureProperties)
output_number_meta.content.contentProperties = (
    _metadata_fb.FeaturePropertiesT())

# Creates subgraph info.
group = _metadata_fb.TensorGroupT()
group.name = "detection result"
group.tensorNames = [
    output_location_meta.name, output_class_meta.name,
    output_score_meta.name
]
subgraph = _metadata_fb.SubGraphMetadataT()
subgraph.inputTensorMetadata = [input_meta]
subgraph.outputTensorMetadata = [
    output_location_meta, output_class_meta, output_score_meta,
    output_number_meta
]
subgraph.outputTensorGroups = [group]
model_meta.subgraphMetadata = [subgraph]

b = flatbuffers.Builder(0)
b.Finish(
    model_meta.Pack(b),
    _metadata.MetadataPopulator.METADATA_FILE_IDENTIFIER)
metadata_buf = b.Output()

NOTE: You can add your own name and description for the metadata information in the block above. I have used the general SSD_Detector name and description.

21) Download the TFLite model with metadata and adjust the TFLite Object Detection sample app

(Build the app using Android Studio as described in the link below: https://github.com/tensorflow/examples/blob/master/lite/examples/object_detection/android/README.md)

⦿ Download the TensorFlow Lite examples archive from here and unzip it. You will find an object detection app inside

..\examples-master\lite\examples\object_detection\android

⦿ Next, build the project first to create the assets folder & download all the default models for the object detection sample app inside it.

Select Build -> Make Project and check that the project builds successfully. You will need Android SDK configured in the settings. The build.gradle file will prompt you to download any missing libraries.

NOTE: You can also create the assets folder manually. Since I will also be using the default models in the app I am building the project. If you don’t want to use the default downloaded models and just want your own model in the app, comment out the apply from:’download_models.gradle line in the Android app Module “build.gradle” file and create the assets folder manually inside the main directory location given below.

⦿ Next, copy your custom TFLite models with metadata inside the assets folder in the following location.

object_detection\android\app\src\main\assets

⦿ Next, make changes in the code as mentioned below.

➢ First, set the names of your TFLite model as modelName for each model version you have. You can set different models here but since we are using only floating-point & quantized models of MobileNet for mask detection, I will add 2 more to the model names. You can remove the other models that were downloaded automatically and remove their references. I will be using those as well.

Add the names of your models to the modelName function in the ObjectDetectorHelper.kt file.

val modelName =
            when (currentModel) {
                MY_CUSTOM_MODEL -> "model.tflite"
                MY_CUSTOM_MODEL_QUANTIZED -> "model_quant.tflite"
                MODEL_MOBILENETV1 -> "mobilenetv1.tflite"
                MODEL_EFFICIENTDETV0 -> "efficientdet-lite0.tflite"
                MODEL_EFFICIENTDETV1 -> "efficientdet-lite1.tflite"
                MODEL_EFFICIENTDETV2 -> "efficientdet-lite2.tflite"
                else -> "mobilenetv1.tflite"
            }

Also, add it to the companion object class as shown below. Set the order id number in the same order as the above step. Since I put my custom models at the first 2 lines which is array id 0 & 1, I have set them here to 0 & 1 and changed others accordingly.

companion object {
        const val DELEGATE_CPU = 0
        const val DELEGATE_GPU = 1
        const val DELEGATE_NNAPI = 2
        const val MY_CUSTOM_MODEL = 0
        const val MY_CUSTOM_MODEL_QUANTIZED = 1
        const val MODEL_MOBILENETV1 = 2
        const val MODEL_EFFICIENTDETV0 = 3
        const val MODEL_EFFICIENTDETV1 = 4
        const val MODEL_EFFICIENTDETV2 = 5
    }

➢ Second, Edit your strings.xml file to add your model names to the string array for model names in order to display it on the app page models dropdown list.

<string-array name="models_spinner_titles">
        <item>My Custom Model</item>
        <item>My Custom Model Quantized</item>
        <item>MobileNet V1</item>
        <item>EfficientDet Lite0</item>
        <item>EfficientDet Lite1</item>
        <item>EfficientDet Lite2</item>
    </string-array>

The strings.xml file is inside the res\values directory.

..\object_detection\android\app\src\main\res\values

• ➢ Finally, connect an Android device to the computer and be sure to approve any ADB permission prompts that appear on your phone. Select Run -> Run app.Test your app before making any other new changes or adding more features to it. Now that you’ve made a basic Object detection app, you can try and make changes to your TFLite model or add any customizations to the app. Have fun!

NOTE:

The dataset I have collected for mask detection contains mostly close-up images. For more long-shot images you can search online. There are many sites where you can download labeled and unlabeled datasets. I have given a few links at the bottom under Dataset Sources. I have also given a few links for mask datasets. Some of them have more than 10,000 images.

Though there are certain tweaks and changes we can make to our training config file or add more images to the dataset for every type of object class through augmentation, we have to be careful so that it does not cause overfitting which affects the accuracy of the model.

For beginners, you can start simply by using the config file I have uploaded on my GitHub. I have also uploaded my mask images dataset along with the PASCAL_VOC format labeled text files, which although might not be the best but will give you a good start on how to train your own custom object detector using an SSD model. You can find a labeled dataset of better quality or an unlabeled dataset and label it yourself later.

I have trained this app on a particular scenario for a person wearing or not wearing a face mask and my dataset mostly had close-up images as mentioned above. If we use this app for other scenarios it might give some false positives. You can train the model for your scenario by training on the right dataset. Moreover, if you want to exclude certain objects in your scenario, you can train those objects and then write code in your app to exclude those objects and include only the ones you want.

There are many ways you can customize these ML apps and also handle the false positives in these apps. You can find scripts for that online. This tutorial shows you how to get started with mobile ML. Have fun!

My GitHub

Files for training

Train Object Detection model using TF 2

My mask dataset

https://www.kaggle.com/techzizou/labeled-mask-dataset-pascal-voc-format

My Colab notebook for this

Google Colaboratory Notebook

Check out my Youtube Videos on this!

Part — 1

Part — 2

Part — 3

CREDITS

Documentation / References

• Tensorflow Introduction
• Tensorflow Models Git Repository
• TensorFlow Object Detection API Repository
• TensorFlow Object Detection API tutorial
• TF Object Detection Documentation
• TF2 installation guide
• TensorFlow 2 Detection Model Zoo
• TensorFlow 2 Classification Model Zoo
• Training and Evaluation with TensorFlow 2
• TensorFlow Lite Object Detection Android Demo
• Running TF2 Detection API Models on mobile
• TensorFlow Lite Converter documentation
• TensorFlow Lite sample applications
• Tensorflow tutorials
• Object Detection for Mobile and IoT devices
• Tensorflow Hub
• TensorFlow Hub Object Detection Colab
• TensorFlow Lite converter
• TensorFlow Lite Post-training quantization
• TensorFlow Lite metadata
• TensorFlow Lite hosted models
• TensorFlow Lite Task Library
• TensorFlow Lite Support Library
• TensorFlow Lite Task Library integrate object detectors
• TensorFlow Lite Interpreter Java API
• Object detector tutorial

Dataset Sources

You can download datasets for many objects from the sites mentioned below. These sites also contain images of many classes of objects along with their annotations/labels in multiple formats such as the YOLO_DARKNET txt files and the PASCAL_VOC xml files.

• Open Images Dataset by Google
• Kaggle Datasets
• Roboflow Public Datasets
• VisualData Datasets

Mask Dataset Sources

• Prajnasb Github
• Andrewmvd Kaggle
• X-zhangyang Github

More Mask Datasets

• Prasoonkottarathil Kaggle (20000 images)
• Ashishjangra27 Kaggle (12000 images )

TROUBLESHOOTING:

ERROR 1) OPENCV ERROR

If you get an error for _registerMatType cv2 above, this might be because of OpenCV version mismatches in Colab. Run !pip list|grep opencv to see the versions of OpenCV packages installed i.e. opencv-python, opencv-contrib-python & opencv-python-headless. The versions will be different which is causing this error. This error will go away when colab updates its supported versions. For now, you can fix this by simply uninstalling and installing OpenCV packages.

Check versions:

!pip list|grep opencv

Use the following 2 commands if only the opencv-python-headless is of different version

!pip uninstall opencv-python-headless --y

!pip install opencv-python-headless==4.1.2.30

Or use the following commands if other opencv packages are of different versions. Uninstall and install all with the same version.

!pip uninstall opencv-python --y
!pip uninstall opencv-contrib-python --y
!pip uninstall opencv-python-headless --y

!pip install opencv-python==4.5.4.60
!pip install opencv-contrib-python==4.5.4.60
!pip install opencv-python-headless==4.5.4.60

ERROR 2) DNN Error

DNN library is not found

This error is due to the version mismatches in the Google Colab environment. Here this can be due to 2 reasons. One, as of now since the default TensorFlow version in Google Colab is 2.8.0(as of now) but the default TensorFlow version for the Object Detection API which we install in step 6 is 2.9.0, this causes an error.

Second, the default cuDNN version for Google Colab as of now is 8.0.5 but for TF 2.8 and above it should have 8.1.0. This also causes version mismatches.

This error will go away when Colab updates its packages. But for temporary fixes, after searching on many forums online and looking at responses from members of the Google Colab team, the following are the 2 possible solutions I can recommend:

SOLUTION 1)

This is the easiest fix, however as per the comment of a Google Colab team member on a forum, this is not the best practice and is not safe. This can also cause mismatches with other packages or libraries. But as a temporary workaround here, this will work.

Run the following command before the training step. This will update the cudnn version and you will have no errors after that.

!apt install --allow-change-held-packages libcudnn8=8.1.0.77-1+cuda11.2

SOLUTION 2)

In this method, you can edit the package versions to be installed in the TensorFlow Object Detection API so that it is the same as the default version for Colab.

We divide step 6 into 2 sections.

Section 1:

# clone the tensorflow models on the colab cloud vm
!git clone --q https://github.com/tensorflow/models.git

#navigate to /models/research folder to compile protos
%cd models/research

# Compile protos.
!protoc object_detection/protos/*.proto --python_out=.

The above section 1 will clone the TF models git repository.

After that, you can edit the file at object_detection/packages/tf2/setup.py .
Change the code in the REQUIRED PACKAGES to include the following 4 lines after the pandas package line:

    'tensorflow==2.8.0',
    'tf-models-official==2.8.0',
    'tensorflow_io==0.23.1',
    'keras==2.8.0'

Note: I have written TensorFlow 2.8.0 here above as it is the default Google colab version as of now. 

Next, after that, you can run section 2 of step 6 shown below to install the TF2 OD API with the updated setup.py file.

Section 2:

# Install TensorFlow Object Detection API.
!cp object_detection/packages/tf2/setup.py .
!python -m pip install .

This will install the TensorFlow Object Detection API with TensorFlow 2.8 0 and other required packages with the updated versions we specified in the setup.py file.

Now you will be able to run the training step without any errors.

ERROR 3) TypeError: EndVector() missing 1 required positional argument: ‘vectorNumElems’

This error is caused due to flatbuffers version mismatches. Downgrade the flatbuffers version from 2.0 to 1.12 and it will fix this error.

!pip install flatbuffers==1.12

The post Build Android app for custom object detection appeared first on TECHZIZOU.

IMPORTANT

THIS IS FOR THE LATEST VERSION OF THE TENSORFLOW APP ON GITHUB. YOU CAN SWITCH TO PREVIOUS VERSIONS IN GITHUB. USE THE FOLLOWING VERSION FOR THE IMAGE CLASSIFICATION REFERENCE APP USED IN THIS TUTORIAL:

https://github.com/tensorflow/examples/tree/master/lite/examples/image_classification

Roadmap

• ⁍ Collect the dataset of images.
• ⁍ Open Teachable-Machine’s image project
• ⁍ Train the model.
• ⁍ Evaluate the model.
• ⁍ Export the model
• ⁍ Create TFLite model for Android
• ⁍ Download the sample image classification app from TensorFlow and adjust it for your custom TFLite model.

Image classification or image recognition is a concept in which you showcase an image to the camera sensor of the device and it will tell you what is present in that image or tell us which class it belongs to.

In this tutorial, I will walk you through the custom image classification by training a simple deep learning model with the help of an exciting online tool by google: teachablemachine with google, and then exporting the model to the TensorFlow lite version which is compatible with an Android device. Then, finally, we will deploy this model onto an Android device.

Model

The training platform used for training our custom image classifier is the teachablemachine with google. This is an exciting platform for learning the deep learning training process with just a very few clicks. First, by uploading the different classes of objects from your system or using a webcam, then training it. Finally, after training, you can export the model of your choice. You can choose whatever format you want and download the model. I am training a model for mask image classification. This will be done in the 5 steps mentioned in the section below. The first 3 steps are the same as in the Train image classification model using Teachable Machine tutorial.

1. Upload your dataset
2. Train the model
3. Preview the model / test the model using javascript
4. Export the model in TFLite format
5. Download the TFLite model and add metadata to it
6. Adjust the TensorFlow sample Image classification for your TFLite models and run it

Requirements

• Your custom object image dataset with different classes.
• Android Studio 3.2 + (installed on a Linux, Mac, or Windows machine)
• Android device in developer mode with USB debugging enabled
• A USB cable (to connect the Android device to your computer)

LET’S BEGIN !!

( But first Subscribe to my YouTube channel https://bit.ly/3Ap3sdi )

Objective: Build a custom object Image classification android app using Teachable Machine.

To start, go to the Teachable-Machine site.

NOTE: TeachableMachine only has support for MobileNet model as of now.

Click on Get Started on the homepage and choose Image Project. You can also sign in using your google drive to save your model files otherwise you will lose all your images and model files if you close the tab or refresh the page.

Step 1. Upload your dataset

Create and upload the dataset to the teachablemachine and define the names of the classes accordingly. I am training a model for mask image classification with 2 classes viz., “with_mask” & “without_mask”.

Step 2) Train the image classification model

During training, you can tweak the hyperparameters like:

• No of epochs
• Batch size
• Learning rate

Click on Train Model after setting the values of the hyperparameters. This process will take a while depending on the number of images and epochs.

Important: During training, do not switch tabs as it will stop the training process.

Step 3) Preview your trained model or test the model using javascript

Once the training is over, you can preview your trained model in the preview window. You can either upload images to test or you can test using a webcam. See “preview” test results for both below.

USING WEBCAM

USING IMAGES

You can also test the model using javascript.

You can use your model for browser-based projects using the javascript format.

First, click on Export model. Next, go to the Tensorflow.js tab and click on Upload my model. When you upload your model, Teachable Machine hosts it at the given shareable link for free. You can share that link with anyone if you’d like them to use your model — anyone who has that link can use your model in their projects. Note that your model is published to Google servers, but the examples you used to make the model are not. Just your model — the mathematical program that predicts which class you’re showing it.

You will see the Javascript code snippet with your model’s link written in it. You can use this code to test your model on a browser.

You can also test your model online using the p5.js web editor tool. The web editor for p5.js is a JavaScript library with the goal of making coding accessible to artists, designers, educators, and beginners. To use it for your custom-trained model, select the p5.js code snippet tab and copy the code. Next, click on the p5.js Web Editor link shown in the pic below.

Once you open the p5.js web editor tool you will find the default code there for daytime and nighttime classification. Remove the entire old code and paste the new code you copied from your custom model’s p5.js code snippet and press play. Now, you can test your model using the p5.js web editor tool.

Step 4) Export the model in TFLite format

Finally, export the model in the form of TensorFlow lite format to deploy on mobile devices.

Click on Export Model and select the TensorFlow Lite tab. Next, choose the model format you want. You can download the quantized as well as the floating-point file format of TFLite.

NOTE: Teachable Machine only supports MobileNet model architecture as of now.

The .tflite file should be placed in the asset folder of the android project directory and change the name in the java file which is reading it accordingly(See Step 5)

Step 5) Download the TFLite and add metadata to it

The current latest version does not have a labels.txt file as the classes are added along with the metadata for all the models of the sample image classification app. We need to add metadata to our TFLite model to attach the classes and other necessary information.

You can use this colab notebook below to create metadata:

https://colab.research.google.com/drive/15bXIFWYGlujeSF_UBYkLqP6ia9TpJx23?usp=sharing

Also, check out the following 2 links to learn more about metadata for image classifiers.

• https://www.tensorflow.org/lite/convert/metadatahttps://www.tensorflow.org/lite/convert/metadata_writer_tutorial#image_classifiers

Step 6) Adjust the TensorFlow sample Image classification for your TFLite models and run it

Open the android_java version of the image classification app in Android Studio and adjust it for your custom model

(Build the app using Android Studio as described in the link below: https://github.com/tensorflow/examples/blob/master/lite/examples/image_classification/android/README.md)

⦿ Download the TensorFlow Lite examples archive from here and unzip it. You will find an image classification app inside

NOTE: I am using the android_java version of the image classification app in Android Studio. You can use the android kotlin version too. The changes there are similar to the ones shown below for the java version.

examples-master\lite\examples\image_classification\android_java

⦿ Next, build the project first to create the assets folder & download all the default models for the image classification sample app inside it.

Select Build -> Make Project and check that the project builds successfully. You will need Android SDK configured in the settings. The build.gradle file will prompt you to download any missing libraries.

NOTE: You can also create the assets folder manually. Since I will also be using the default models in the app I am building the project. If you don’t want to use the default downloaded models and just want your own model in the app, comment out the apply from:’download_models.gradle line in the Android app Module “build.gradle” file and create the assets folder manually inside the main directory location given below.

⦿ Next, copy your custom TFLite models with metadata inside the assets folder.

image_classification\android_java\app\src\main\assets

⦿ Next, make changes in the code as mentioned below.

➢ First, set the names of your TFLite model as modelName for each model version you have. You can set different models here but since we are using only floating-point & quantized models for MobileNet, I will add 2 more to the model names. 

Add the names of your models to the modelName function in the ImageClassifierHelper.java file.

        String modelName;
        switch (currentModel) {
            case MY_CUSTOM_MODEL:
                modelName = "model.tflite";
                break;
            case MY_CUSTOM_MODEL_QUANTIZED:
                modelName = "model_quant.tflite";
                break;
            case MODEL_MOBILENETV1:
                modelName = "mobilenetv1.tflite";
                break;
            case MODEL_EFFICIENTNETV0:
                modelName = "efficientnet-lite0.tflite";
                break;
            case MODEL_EFFICIENTNETV1:
                modelName = "efficientnet-lite1.tflite";
                break;
            case MODEL_EFFICIENTNETV2:
                modelName = "efficientnet-lite2.tflite";
                break;
            default:
                modelName = "mobilenetv1.tflite";
        }

Also, add it to the ImageClassifierHelper class as shown below. Set the order id number in the same order as the above step. Since I put my custom models at the first 2 lines which is array id 0 & 1, I have set them here to 0 & 1 and changed others accordingly.

public class ImageClassifierHelper {
    private static final String TAG = "ImageClassifierHelper";
    private static final int DELEGATE_CPU = 0;
    private static final int DELEGATE_GPU = 1;
    private static final int DELEGATE_NNAPI = 2;
    private static final int MY_CUSTOM_MODEL = 0;
    private static final int MY_CUSTOM_MODEL_QUANTIZED = 1;
    private static final int MODEL_MOBILENETV1 = 2;
    private static final int MODEL_EFFICIENTNETV0 = 3;
    private static final int MODEL_EFFICIENTNETV1 = 4;
    private static final int MODEL_EFFICIENTNETV2 = 5;

For Kotlin app version edit the modelName function and the companion object inside the ImageClassifierHelper.kt file.

val modelName =
            when (currentModel) {
                MY_CUSTOM_MODEL -> "model.tflite"
                MY_CUSTOM_MODEL_QUANTIZED -> "model_quant.tflite"
                MODEL_MOBILENETV1 -> "mobilenetv1.tflite"
                MODEL_EFFICIENTDETV0 -> "efficientdet-lite0.tflite"
                MODEL_EFFICIENTDETV1 -> "efficientdet-lite1.tflite"
                MODEL_EFFICIENTDETV2 -> "efficientdet-lite2.tflite"
                else -> "mobilenetv1.tflite"
            }

companion object {
        const val DELEGATE_CPU = 0
        const val DELEGATE_GPU = 1
        const val DELEGATE_NNAPI = 2
        const val MY_CUSTOM_MODEL = 0
        const val MY_CUSTOM_MODEL_QUANTIZED = 1
        const val MODEL_MOBILENETV1 = 2
        const val MODEL_EFFICIENTDETV0 = 3
        const val MODEL_EFFICIENTDETV1 = 4
        const val MODEL_EFFICIENTDETV2 = 5
    }

➢ Second, Edit your strings.xml file to add your model names to the string array for model names in order to display it on the app page models dropdown list.

<string-array name="models_spinner_titles">
        <item>My Custom Model</item>
        <item>My Custom Model Quantized</item>
        <item>MobileNet V1</item>
        <item>EfficientDet Lite0</item>
        <item>EfficientDet Lite1</item>
        <item>EfficientDet Lite2</item>
    </string-array>

The strings.xml file is inside the res\values directory.

..\image_classification\android_java\app\src\main\res\values

• ➢ Finally, connect an Android device to the computer and be sure to approve any ADB permission prompts that appear on your phone. Select Run -> Run app.Test your app before making any other new changes or adding more features to it. Now that you’ve made a basic Image classification app using Teachable Machine, you can try and make changes to your TFLite model or add any customizations to the app. Have fun!

Read these TensorFlow image classification app GitHub Readme and Explore_the_code pages to learn more->

TensorFlow Lite image classification Android example application/Readme

TensorFlow Lite Android image classification example/Explore_the_code

My GitHub

Image Classification sample app

Custom Image Classification

Mask dataset

Prajnasb Github

Check out my Youtube Video on this!

CREDITS

Documentation / References

• ▸ Tensorflow Introduction
• ▸ Tensorflow Models Git Repository
• ▸ TensorFlow 2 Classification Model Zoo
• ▸ TensorFlow Lite sample applications
• ▸ TensorFlow Lite image classification Android example application
• ▸ Tensorflow tutorials
• ▸ Integrate image classifiers
• ▸ Adding metadata to TensorFlow Lite models
• ▸ TensorFlow Lite Metadata Writer API
• ▸ Create metadata command
• ▸ TensorFlow Lite image classification Android example application/Readme
• ▸ TensorFlow Lite Android image classification example/Explore_the_code

♕ TECHZIZOU ♕

The post Build a custom Image classification Android app using Teachable Machine (Latest Version TensorFlow App) appeared first on TECHZIZOU.

In the previous blog “Setup OpenCV-DNN module with CUDA backend support (For Linux)”, we built the OpenCV-DNN module with CUDA backend support on Linux. (I have Ubuntu 18.04). We can use this OpenCV module to run inference on object detection models. The OpenCV-DNN module built with CUDA backend support gives much faster inference as compared to the OpenCV without CUDA backend. All we need to do is add the following two lines to our code and inference the model using this.

net.setPreferableBackend(cv2.dnn.DNN_BACKEND_CUDA)
net.setPreferableTarget(cv2.dnn.DNN_TARGET_CUDA)

The OpenCV-DNN module with CUDA backend support gives much faster inference as compared to the OpenCV without CUDA backend.

Follow the steps below to run inference using the OpenCV-DNN module we have installed:

• First, open Anaconda prompt and activate the virtual environment which you used to install the OpenCV-DNN module.

• Next, Download the following python opencv_dnn.py script and run it using python. Note that this uses the yolov4 model weights and config file. Download them from here.

import cv2
import time

CONFIDENCE_THRESHOLD = 0.2
NMS_THRESHOLD = 0.4
COLORS = [(0, 255, 255), (255, 255, 0), (0, 255, 0), (255, 0, 0)]

class_names = []
with open("classes.txt", "r") as f:
    class_names = [cname.strip() for cname in f.readlines()]

cap = cv2.VideoCapture("video.mp4")

net = cv2.dnn.readNet("yolov4.weights", "yolov4.cfg")
net.setPreferableBackend(cv2.dnn.DNN_BACKEND_CUDA)
net.setPreferableTarget(cv2.dnn.DNN_TARGET_CUDA_FP16)

model = cv2.dnn_DetectionModel(net)
model.setInputParams(size=(416, 416), scale=1/255, swapRB=True)

while cv2.waitKey(1) < 1:
    (grabbed, frame) = cap.read()
    if not grabbed:
        exit()

    start = time.time()
    classes, scores, boxes = model.detect(frame, CONFIDENCE_THRESHOLD, NMS_THRESHOLD)
    end = time.time()

    for (classid, score, box) in zip(classes, scores, boxes):
        color = COLORS[int(classid) % len(COLORS)]
        label = "%s : %f" % (class_names[classid[0]], score)
        cv2.rectangle(frame, box, color, 2)
        cv2.putText(frame, label, (box[0], box[1]-5), cv2.FONT_HERSHEY_SIMPLEX, 0.5, color, 2)
    
    fps = "FPS: %.2f " % (1 / (end - start))
    cv2.putText(frame, fps, (0, 25), cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 0, 255), 2)
    cv2.imshow("output", frame)

You can also download the above python code to test the OpenCV-DNN inference.

IMPORTANT: The OpenCV-DNN-CUDA module only supports inference. So although you will get much faster inference out of it, the training however will be the same as it was for the OpenCV we set up without CUDA backend support.

You can also use the OpenCV-DNN-CUDA module to run inference on models from various other frameworks like TensorFlow, PyTorch, etc.

My YouTube Video on this!

Previous Tutorial:

Setup OpenCV-DNN module with CUDA backend support (For Linux)

The post YOLOv4 inference using OpenCV-DNN-CUDA on Linux appeared first on TECHZIZOU.

In this tutorial, we will be building OpenCV from source with CUDA backend support (OpenCV-DNN-CUDA module).

IMPORTANT: The OpenCV-DNN module only supports inference so although you will get much faster inference out of it, the training however will be the same as was for the OpenCV we set up without CUDA backend support.

STEPS

1. Install CUDA & cuDNN.

2. Install the necessary Ubuntu packages if not already installed.

3. Download and unzip opencv & opencv_contrib

4. Setup python environment

5. Configure OpenCV libraries with CMake commad

6. Build OpenCV from source for your NVidia GPU.

LET’S BEGIN!

STEP 1) Install CUDA & cuDNN

Download and install CUDA and its corresponding cuDNN archive for your system. Go to https://developer.nvidia.com/cuda-downloads to download the latest CUDA Toolkit. You can also download previous versions from Archive of Previous CUDA Releases OR under the Resources section at the cuda-downloads link given above. Next, download and copy the cuDNN folder and its contents to where we installed CUDA. You can download the latest version of cuDNN from https://developer.nvidia.com/cudnn . You can also download previous versions from the cudnn archive at https://developer.nvidia.com/rdp/cudnn-archive .

Next, you need to add the paths to these in your bashrc script.

After that, reboot your system and verify your CUDA installation in a terminal by typing the following 2 commands:

nvidia-smi

nvcc -V

Follow this blog to learn how to install and set up CUDA and CUDNN on Linux.

STEP 2) Install the necessary Ubuntu packages if not already installed.

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install build-essential cmake unzip pkg-config
sudo apt-get install libjpeg-dev libpng-dev libtiff-dev
sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev
sudo apt-get install libv4l-dev libxvidcore-dev libx264-dev
sudo apt-get install libgtk-3-dev
sudo apt-get install libblas-dev liblapack-dev gfortran
sudo apt-get install python3-dev

STEP 3) Download and unzip opencv & opencv_contrib

Navigate to the home directory, download OpenCV archive, unzip it and rename the folder to opencv.

cd ~
wget -O opencv-4.5.5.zip https://github.com/opencv/opencv/archive/4.5.5.zip
unzip -q opencv-4.5.5.zip
mv opencv-4.5.5 opencv
rm -f opencv-4.5.5.zip 

Next, download the opencv_contrib archive, unzip it and rename the folder to opencv_contrib.

Note: Make sure to download the exact same version of the opencv_contrib as the opencv archive you downloaded before that.

wget -O opencv_contrib-4.5.5.zip https://github.com/opencv/opencv_contrib/archive/4.5.5.zip
unzip -q opencv_contrib-4.5.5.zip
mv opencv_contrib-4.5.5 opencv_contrib
rm -f opencv_contrib-4.5.5.zip 

STEP 4) Setup the python environment

Install virtualenv and virtualenvwrapper if you don’t have them on your system already.

sudo pip install virtualenv virtualenvwrapper

Open the bashrc script file using nano.

nano ~/.bashrc

Enter the following export paths to set the path for the virtualenvwrapper at the bottom and save the file. (Press ctrl+x , y and Enter to save the file)

export WORKON_HOME=$HOME/.virtualenvs
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
source /usr/local/bin/virtualenvwrapper.sh

Run source to assert the changes in the system for the bashrc script.

source ~/.bashrc

Create a virtual environment in a terminal. We will be installing the OpenCV-DNN-CUDA module in a virtual environment and not our base environment.

mkvirtualenv opencv_dnn_cuda -p python3

Install NumPy package.

pip install numpy

Activate the virtual environment by entering the following command:

workon opencv_dnn_cuda

STEP 5) Configure OpenCV using the cmake command to install some libraries

Navigate to the opencv folder we created in step 3. Create another folder inside it named build and navigate inside it.

cd ~/opencv
mkdir build
cd build

Run the CMake command to configure settings for the OpenCV you want to build from source with your custom libraries and functions selected.

Change the CUDA_ARCH_BIN parameter in the command below to your GPU’s compute capability. Find your GPU’s cc here. Also, change the virtual environment name in the PYTHON_EXECUTABLE parameter if you are using a different name else leave it as it is.

  cmake -D CMAKE_BUILD_TYPE=RELEASE \
	-D CMAKE_INSTALL_PREFIX=/usr/local \
	-D INSTALL_PYTHON_EXAMPLES=ON \
	-D INSTALL_C_EXAMPLES=OFF \
	-D OPENCV_ENABLE_NONFREE=ON \
	-D WITH_CUDA=ON \
	-D WITH_CUDNN=ON \
	-D OPENCV_DNN_CUDA=ON \
	-D ENABLE_FAST_MATH=1 \
	-D CUDA_FAST_MATH=1 \
	-D CUDA_ARCH_BIN=6.1 \
	-D WITH_CUBLAS=1 \
	-D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib/modules \
	-D HAVE_opencv_python3=ON \
	-D PYTHON_EXECUTABLE=~/.virtualenvs/opencv_dnn_cuda/bin/python \
	-D BUILD_EXAMPLES=ON ..

In the above CMake command, we are compiling OpenCV with both CUDA and cuDNN support enabled (WITH_CUDA and WITH_CUDNN, respectively). We are also enabling the OPENCV_DNN_CUDA parameter to build the DNN module with CUDA backend support. We are also setting ENABLE_FAST_MATH, CUDA_FAST_MATH, and WITH_CUBLAS for optimization purposes.

STEP 6) Lastly, build the OpenCV-DNN module from source with CUDA backend support for your specific NVidia GPU

Run the make command to build the OpenCV with the above-configured settings. (Run nproc in terminal to print your no. of processors. Replace that with 8 below in j8.)

make -j8

This will build the OpenCV-DNN module from source with the CUDA backend. This process can take up to an hour or so.

Next, install the OpenCV on your system once it has finished building the OpenCV in the make command above. Run the following commands:

sudo make install
sudo ldconfig

Next, the final step is to create a symbolic link between the newly installed OpenCV library to our python virtual environment. For that, you need the path where the OpenCV bindings were installed. You can determine that path via the install path configuration in the CMake command step output.

In my case, the install path was lib/python3.6/site-packages/cv2/python-3.6 which means my OpenCV bindings are in /usr/local/lib/python3.6/site-packages/cv2/python-3.6. Check where your “.so” file is.

You can confirm the above path by listing its contents. Run the following ls command:

ls -l /usr/local/lib/python3.6/site-packages/cv2/python-3.6
total 10104
-rw-r--r-- 1 root staff 10345488 Mar 23 07:06 cv2.cpython-36m-x86_64-linux-gnu.so

You will see an output like above confirming the cv2 binding is here. I have Python 3.6 and a 64-bit architecture which explains the file name. Your’s might vary according to your Python and system architecture.

Next, navigate to the virtual environment “site-packages” folder, and finally create a link for the cv2 bindings we found above using the ln -s command. This creates a cv2.so sym-link inside the current working directory which is the site-packages folder in our virtual environment.

cd ~/.virtualenvs/opencv_dnn_cuda/lib/python3.6/site-packages/
ln -s /usr/local/lib/python3.6/site-packages/cv2/python-3.6/cv2.cpython-36m-x86_64-linux-gnu.so cv2.so

Make sure to double-check everything as this command silently fails even if the path is incorrect meaning you will get no warning or error message and the output will not work. So be mindful of this step and check if the cv2.so file is created or not.

That’s it! We have successfully built the OpenCV-DNN module with CUDA backend support.

Check if cv2 is installed correctly.

workon opencv_dnn_cuda
python

Python 3.6.9 (default, Jul 02 2019, 17:25:39)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> cv2.__version__
'4.5.5'
>>>

Check if the OpenCV is installed correctly with CUDA backend support using the test_DNN_CV.py script below. Download it below.

import numpy as np
import cv2 as cv
import time

npTmp = np.random.random((1024, 1024)).astype(np.float32)

npMat1 = np.stack([npTmp,npTmp],axis=2)
npMat2 = npMat1

cuMat1 = cv.cuda_GpuMat()
cuMat2 = cv.cuda_GpuMat()
cuMat1.upload(npMat1)
cuMat2.upload(npMat2)

start_time = time.time()
cv.cuda.gemm(cuMat1, cuMat2,1,None,0,None,1)

print("CUDA using GPU --- %s seconds ---" % (time.time() - start_time))

start_time = time.time()
cv.gemm(npMat1,npMat2,1,None,0,None,1)

print("CPU --- %s seconds ---" % (time.time() - start_time))

Download the test_DNN_CV script below.

My YouTube Video on this!

Next Tutorial:

Run YOLOv4 inference using OpenCV-DNN Module on Linux

The post Setup OpenCV-DNN module with CUDA backend support (For Linux) appeared first on TECHZIZOU.

In the previous blog “Setup OpenCV-DNN module with CUDA backend support (For Windows)”, we built the OpenCV-DNN module with CUDA backend support on Windows. We can use this OpenCV module to run inference on object detection models. The OpenCV-DNN module built with CUDA backend support gives much faster inference as compared to the OpenCV without CUDA backend. All we need to do is add the following two lines to our code and inference the model using this.

net.setPreferableBackend(cv2.dnn.DNN_BACKEND_CUDA)
net.setPreferableTarget(cv2.dnn.DNN_TARGET_CUDA)

Follow the steps below to run inference using the OpenCV-DNN-CUDA module we have installed:

• First, open Anaconda prompt and activate the virtual environment which you used to install the OpenCV-DNN-CUDA module in the previous article “Setup OpenCV-DNN module with CUDA backend support (For Windows)“.

• Next, Download the following python opencv_dnn.py script and run it using python. Note that this uses the yolov4 model weights and config file. Download them from here.

import cv2
import time

CONFIDENCE_THRESHOLD = 0.2
NMS_THRESHOLD = 0.4
COLORS = [(0, 255, 255), (255, 255, 0), (0, 255, 0), (255, 0, 0)]

class_names = []
with open("classes.txt", "r") as f:
    class_names = [cname.strip() for cname in f.readlines()]

cap = cv2.VideoCapture("video.mp4")

net = cv2.dnn.readNet("yolov4.weights", "yolov4.cfg")
net.setPreferableBackend(cv2.dnn.DNN_BACKEND_CUDA)
net.setPreferableTarget(cv2.dnn.DNN_TARGET_CUDA_FP16)

model = cv2.dnn_DetectionModel(net)
model.setInputParams(size=(416, 416), scale=1/255, swapRB=True)

while cv2.waitKey(1) < 1:
    (grabbed, frame) = cap.read()
    if not grabbed:
        exit()

    start = time.time()
    classes, scores, boxes = model.detect(frame, CONFIDENCE_THRESHOLD, NMS_THRESHOLD)
    end = time.time()

    for (classid, score, box) in zip(classes, scores, boxes):
        color = COLORS[int(classid) % len(COLORS)]
        label = "%s : %f" % (class_names[classid[0]], score)
        cv2.rectangle(frame, box, color, 2)
        cv2.putText(frame, label, (box[0], box[1]-5), cv2.FONT_HERSHEY_SIMPLEX, 0.5, color, 2)
    
    fps = "FPS: %.2f " % (1 / (end - start))
    cv2.putText(frame, fps, (0, 25), cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 0, 255), 2)
    cv2.imshow("output", frame)

Download the above script to test the OpenCV-DNN inference here.

IMPORTANT: The OpenCV-DNN-CUDA module only supports inference. So although you will get much faster inference out of it, the training however will be the same as it was for the OpenCV we set up without CUDA backend support.

You can also use the OpenCV-DNN-CUDA module to run inference on models from various other frameworks like TensorFlow, PyTorch, etc.

My YouTube Video on this!

Previous Tutorial:

Setup OpenCV-DNN module with CUDA backend support (For Windows)

The post YOLOv4 inference using OpenCV-DNN-CUDA on Windows appeared first on TECHZIZOU.

In this tutorial, we will be building OpenCV from source with CUDA backend support (OpenCV-DNN-CUDA module).

IMPORTANT: The OpenCV-DNN-CUDA module only supports inference. So although you will get much faster inference out of it, the training however will be the same as for the OpenCV we set up without CUDA backend support.

STEPS

1. Install CUDA & cuDNN.

2. Install CMake GUI.

3. Install Anaconda Individual edition.

4. Install Visual Studio with the “Desktop development for C++” module.

5. Create a virtual environment in Anaconda 

6. Build OpenCV from source for your NVidia GPU.

STEP 1) Install CUDA & cuDNN

Download the latest version of CUDA for your system and its corresponding cuDNN archive. Go to https://developer.nvidia.com/cuda-downloads to download the latest CUDA Toolkit. You can also download previous versions from Archive of Previous CUDA Releases OR under the Resources section at the cuda-downloads link given above. You can download the latest version of cuDNN from https://developer.nvidia.com/cudnn . You can also download previous versions from cudnn archive at https://developer.nvidia.com/rdp/cudnn-archive .

Install CUDA by running the CUDA exe file and next extract and copy the cuDNN folder and its contents to where we installed CUDA. Usually, that is in C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.x. The system also automatically adds this path to environment variables as CUDA_PATH. We need to add a few paths to the Path variable for CMake GUI to find cuDNN. Read about the complete CUDA & cuDNN installation here.

Note: As mentioned on the CUDA & cuDNN setup blog above, you have to add the same paths for the Path system variable as you did for the CUDNN system variable. After adding these paths, reboot the system. This is just to ensure that the CMake GUI software which we will be using next is able to find cuDNN on our system.

STEP 2) Install CMake GUI

Next, we need to install CMake GUI for Windows. (Link given) https://cmake.org/download/

Select the installer for your system. I have Windows (64-bit). Download the setup, run and install it. Also, add it to your PATH during installation.

STEP 3) Install Anaconda Individual Edition For Windows

• Download and run the Anaconda installer from here.
• Select the “Just Me” option while installing Anaconda to avoid any admin privilege issues.
• Select the option to make Anaconda python your default and also select the other option to include Anaconda python in your PATH variable. This ensures that Anaconda python is found first in the system. You can always change this by going to your environment variables and moving the python you want as default to the top above Anaconda python.

.

STEP 4) Install MSVC (Microsoft Visual Studio)

Go to MSVC: https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=Community. This will download the MSVC Community edition setup exe file for you. Run it. Select the “Desktop development with C++” package as shown below and click on install. If you already have another version of the visual studio like visual studio professional or enterprise, make sure the above module is installed.

STEP 5) Create a virtual environment in Anaconda 

NOTE: I am doing this step as I am building OpenCV for a virtual environment. If you want to build for your base environment you can skip this step and move on to Step 6.

● First, open Anaconda prompt and run the following command to create a new Anaconda environment.

conda create -n env_NAME python=3.9

(Where env_NAME is the name of the environment you want to create. I am setting python=3.9)

• To Activate the environment use the following command:

conda activate env_NAME

• Install NumPy for this environment

pip install numpy
#OR
pip install --upgrade numpy

STEP 6) Lastly, build OpenCV from source with CUDA backend support for your specific NVidia GPU.

This is the main step for setting up the OpenCV-DNN-CUDA module i.e. building OpenCV from source with CUDA backend support. This gives us faster inference for object detection models. We will be installing some libraries here.

IMPORTANT: Before we begin, if you are creating this for your base environment, make sure to install NumPy using pip install numpy . Else, if you are doing this for a virtual environment, we already installed numpy in step 5.

● 6) a) First, download OpenCV and opencv_contrib.

Download the latest OpenCV sources files from either the official OpenCV releases page or the GitHub page. Next, download the exact version of opencv_contrib as the OpenCV sources file you downloaded. Use the following links to download both these files. (The latest OpenCV version as of now is 4.5.5)

⇒ https://opencv.org/releases/

⇒ https://github.com/opencv/opencv_contrib

Note: Use the same version for both opencv & opencv_contrib. To select a specific version click on the master dropdown on the top left, click on Tags, and select the version.

● 6) b) Create a new folder in your C drive named opencv. Extract and copy the folders from the previous step to the C:\opencv folder.

● 6) c) Create another folder named build folder inside C:\opencv 

NOTE: Before running CMake, you need to have CUDA, cuDNN installed and configured on your system. Refer to this blog to learn how to install and set up CUDA and CUDNN on Windows.

● 6) d) Open CMake GUI. Select the source as C:\opencv\opencv-4.5.5 and select the destination for building the binaries as C:\opencv\build.

● 6) e) Check the Grouped entries box and click on Configure. Set the “generator for this project” as your Visual Studio version. I have 2017 so I will set Visual Studio 15 2017. And lastly, choose the “Optional platform for generator” as x64.

● 6) f) Once the first configuration is done, select the following modules by typing them in the search box and ticking their boxes.

WITH_CUDA

OPENCV_DNN_CUDA

ENABLE_FAST_MATH

BUILD_opencv_world

BUILD_opencv_dnn 

BUILD_opencv_python3 

(Some of these are usually selected by default but double-check them as these are important)

● 6) g) Next type the following and set the path to the modules folder inside the opencv_contrib folder we extracted in Step 6)b).

OPENCV_EXTRA_MODULES_PATH 

Give the path→ C:\opencv\opencv_contrib-4.5.5\modules

● 6) h) Next, do this step only if you are building OpenCV for the virtual environment we created in Step 5. If not you can skip to Step 6) i). I am doing this as I am installing OpenCV in my virtual environment.

Type PYTHON3 in the search box. You will see all the following python3 path entries pointing to the libraries in the base anaconda environment if you have set anaconda as your default python else you will see the windows python base environment libraries if that is your default python.

We need to set them to point to the virtual environment libraries of the same type. (Note: This is for the virtual environment env_NAME we created in Anaconda prompt in Step 5. Also, here my Anaconda location is C:\anaconda3 . Yours may be different eg: in C:\Program Data\anaconda3 etc.)

PYTHON3_EXECUTABLE 

Change ⇒ C:\anaconda3\python.exe →

to ⇒ C:\anaconda3\envs\env_NAME\python.exe

PYTHON3_INCLUDE_DIR

Change ⇒ C:\anaconda3\include →

to ⇒ C:\anaconda3\envs\env_NAME\include

PYTHON3_LIBRARY

Change ⇒ C:\anaconda3\libs\python39.lib →

to ⇒ C:\anaconda3\envs\env_NAME\libs\python39.lib

PYTHON3_NUMPY_INCLUDE_DIRS

Change ⇒ C:/anaconda3/Lib/site-packages/numpy/core/include →

to ⇒ C:/anaconda3/envs/env_NAME/Lib/site-packages/numpy/core/include

PYTHON3_PACKAGES_PATH

Change ⇒ C:/anaconda3/Lib/site-packages →

to ⇒ C:/anaconda3/envs/env_NAME/Lib/site-packages

● 6) i) Click on Configure again.

● 6) j) Once the second configuration is done, you will see your CUDA and cuDNN in the output. Now, select the following by typing it in the search box and ticking its box.

CUDA_FAST_MATH

● 6) k) Next, set your GPU’s Compute capability as the CUDA_ARCH_BIN. (You can check your GPU’s cc here on this link → https://developer.nvidia.com/cuda-gpus . Go to this link and select your GPU section. For eg: I have a GeForce GPU so I will select the section “CUDA-Enabled GeForce and TITAN products”. This will give a dropdown of all the GPUs with their compute capabilities. Mine is 6.1)

CUDA_ARCH_BIN 6.1

● 6) l) Find CMAKE_CONFIGURATION_TYPES. Remove the Debug option and only set the Release option.

CMAKE_CONFIGURATION_TYPES       Release

● 6) m) Configure one last time.

● 6) n) After the third and last configuration is finished, verify all the outputs and finally click on Generate. This generates the project with these settings which we can use to build our OpenCV.

● 6) o) Next, once the generating is done. Open a command prompt in the C:\opencv directory and run the following command.

"C:\Program Files\CMake\bin\cmake.exe" --build "C:\opencv\build" --target INSTALL --config Release

This will build the OpenCV-DNN module with CUDA backend. This process can take up to an hour or so.

● 6) p) Lastly, set the following paths in your environment variables and reboot your system.

Set a new variable OpenCV_DIR in System Variables to point to the following three. →

• ‣ C:\opencv\build\install\x64\vc15\bin
• ‣ C:\opencv\build\install\x64\vc15\lib
• ‣ C:\opencv\build

Also, add the following path to the Path variable in System Variables. →

• ‣ C:\opencv\build\install\x64\vc15\bin

After adding the paths reboot your system.

That’s it! We have successfully built the OpenCV-DNN module with CUDA backend support.

Check if cv2 is installed on your system. Run the following commands in Anaconda prompt:

conda activate opencv_dnn_cuda
python

Python 3.9.12 (main, Apr  4 2022, 05:22:27) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> cv2.__version__
'4.5.5'
>>>

Check if the OpenCV is installed correctly with CUDA backend support using the test_DNN_CV.py script below.

You can test if the OpenCV-DNN module was installed correctly using the script below. The GPU output will be much faster than the CPU output.

import numpy as np
import cv2 as cv
import time

npTmp = np.random.random((1024, 1024)).astype(np.float32)

npMat1 = np.stack([npTmp,npTmp],axis=2)
npMat2 = npMat1

cuMat1 = cv.cuda_GpuMat()
cuMat2 = cv.cuda_GpuMat()
cuMat1.upload(npMat1)
cuMat2.upload(npMat2)

start_time = time.time()
cv.cuda.gemm(cuMat1, cuMat2,1,None,0,None,1)

print("CUDA using GPU --- %s seconds ---" % (time.time() - start_time))

start_time = time.time()
cv.gemm(npMat1,npMat2,1,None,0,None,1)

print("CPU --- %s seconds ---" % (time.time() - start_time))

Download the test_DNN_CV script below.

My YouTube Video on this!

Next Tutorial:

Run YOLOv4 inference using OpenCV-DNN Module on Windows

The post Setup OpenCV-DNN module with CUDA backend support (For Windows) appeared first on TECHZIZOU.

My YouTube video on this!

In this tutorial, we will be training a custom object detector for mask detection using YOLOv4 and Darknet on our Windows system

FOLLOW THESE 10 STEPS TO TRAIN AN OBJECT DETECTOR USING YOLOv4

( But first Subscribe to my YouTube channel https://bit.ly/3Ap3sdi )

1. Create yolov4 and training folders on your Desktop
2. Open a command prompt and navigate to the “yolov4” folder
3. Create and copy the darknet.exe file
4. Create & copy the files we need for training ( i.e. “obj” dataset, “yolov4-custom.cfg”, “obj.data”, “obj.names” and “process.py” ) to your yolov4 dir
5. Copy the “yolov4-custom.cfg”, “obj.data”, “obj.names”, and “process.py” files and the “obj” dataset folder from the yolov4 directory to the darknet directory
6. Run the process.py python script to create the train.txt & test.txt files
7. Download the pre-trained YOLOv4 weights
8. Train the detector
9. Check performance
10. Test your custom Object Detector

~~~~~~~~~~~~~~ LET’S BEGIN !! ~~~~~~~~~~~~~~

1) Create ‘yolov4’ and ‘training’ folders

Create a folder named yolov4 on your Desktop. Next, create another folder named training inside the yolov4 folder. This is where we will save our trained weights (This path is mentioned in the obj.data file which we will upload later)

2) Open command prompt and navigate to the “yolov4” folder

Navigate to C:\Users\zizou\Desktop\yolov4\darknet folder. 
NOTE: Replace zizou here with your Username

cd C:\Users\zizou\Desktop\yolov4

3) Create and copy the darknet.exe file

Create and copy your darknet folder containing the darknet.exe file into the yolov4 folder. To know how to create the darknet folder containing the darknet.exe file using CMake, go to this blog. Follow all the steps there for YOLO installation on Windows.

This step can be a little tedious but this is the difficult part of this entire process. The rest of the training process here below is fairly simple. So, make sure you follow all the steps carefully in the above article to create darknet and also have CUDA and cuDNN installed and configured on your system. You’ll find all the steps explained in detail in section A of the above-mentioned blog. You can also check out my video on YOLO installation for Windows at the bottom under the YouTube Videos section of the above-mentioned blog.

Once the above process is done, copy the darknet-master folder we get from it and copy it to the yolov4 folder on your desktop. Rename the darknet-master folder to darknet for simplicity.

4) Create & copy the following files which we need for training a custom detector

a. Labeled Custom Dataset
b. Custom cfg file
c. obj.data and obj.names files
d. process.py file (to create train.txt and test.txt files for training)

I have uploaded my custom files for mask detection on my GitHub. I am working with 2 classes i.e. “with_mask” and “without_mask”.

Labeling your Dataset

Input image example (Image1.jpg)

You can use any software for labeling like the labelImg tool.

I use an open-source labeling tool called OpenLabeling with a very simple UI.

Click on the link below to know more about the labeling process and other software for it:

Image Dataset Labeling article

NOTE: Garbage In = Garbage Out. Choosing and labeling images is the most important part. Try to find good-quality images. The quality of the data goes a long way toward determining the quality of the result.

The output YOLO format labeled file looks like as shown below.

4(a) Create and copy the labeled custom dataset “obj” folder to the “yolov4” folder

Put all the input image “.jpg” files and their corresponding YOLO format labeled “.txt” files in a folder named obj. Copy it to the yolov4 folder.

4(b) Create your custom config file and copy it to the ‘yolov4’ folder

Download the yolov4-custom.cfg file from darknet/cfg directory, make changes to it, and copy it to the yolov4 dir.

You can also download the custom config files from the official AlexeyAB Github.

You need to make the following changes in your custom config file:

• change line batch to batch=64

• change line subdivisions to subdivisions=16 or subdivisions=32 or subdivisions=64 (Read Note after this section for more info on these values)

• set network size width=416 height=416 or any value multiple of 32

• change line max_batches to (classes*2000 but not less than the number of training images and not less than 6000), f.e. max_batches=6000 if you train for 3 classes

• change line steps to 80% and 90% of max_batches, f.e. steps=4800,5400

• change [filters=255] to filters=(classes + 5)x3 in the 3 [convolutional] before each [yolo] layer, keep in mind that it only has to be the last [convolutional] before each of the [yolo] layers.

• change line classes=80 to your number of objects in each of 3 [yolo]-layers.

So if classes=1 then it should be filters=18. If classes=2 then write filters=21.

You can tweak other parameter values too like the learning rate, angle, saturation, exposure, and hue once you’ve understood how the basics of the training process work. For beginners, the above changes should suffice.

NOTE: What are subdivisions?

• It is the number of many mini-batches we split our batch into.
• Batch=64 -> loading 64 images for one iteration.
• Subdivision=8 -> Split batch into 8 mini-batches so 64/8 = 8 images per mini-batch and these 8 images are sent for processing. This process will be performed 8 times until the batch is completed and a new iteration will start with 64 new images.
• If you are using a GPU with low memory, set a higher value for subdivisions ( 32 or 64). This will obviously take longer to train since we are reducing the number of images being loaded and also the number of mini-batches.
• If you have a GPU with high memory, set a lower value for subdivisions (16 or 8). This will speed up the training process as this loads more images per iteration.

4(c) Create your “obj.data” and “obj.names” files and copy them to the yolov4 folder

obj.data

The obj.data file has :

• The number of classes.
• The path to train.txt and test.txt files that we will create later.
• The path to “obj.names” file which contains the names of the classes.
• The path to the training folder where the training weights will be saved.

classes = 2
train = data/train.txt
valid = data/test.txt
names = data/obj.names
backup = ../training

obj.names

Has objects’ names — each in a new line. Make sure the classes are in the same order as in the class_list.txt file used while labeling the images so the index id of every class is the same as mentioned in the labeled YOLO text files.

4(d) Copy the <process.py> script file to your yolov4 folder

(To divide all image files into 2 parts. 90% for train and 10% for test)

This process.py script creates the files train.txt & test.txt where the train.txt file has paths to 90% of the images and test.txt has paths to 10% of the images.

You can download the process.py script from my GitHub.

**IMPORTANT: The “process.py” script has only the “.jpg” format written in it, so other formats such as “.png”,“.jpeg”, or even “.JPG”(in capitals) won’t be recognized. If you are using any other format, make changes in the process.py script accordingly.

Now that we have uploaded all the files, our yolov4 folder on our desktop should look like this:

5) Copy all the files from the ‘yolov4' directory to the ‘yolov4/darknet’ directory

You can do all these steps manually through File Explorer.

First, clean the data and cfg folders except for the labels folder inside the data folder which is required for writing label names on the detection boxes. So just remove all the files from the data folder except the labels folder and next, entirely clean the cfg folder as we already have our custom config file in the yolov4 main folder.

Next, copy all the files:

• Copy yolov4-custom.cfg to the darknet/cfg directory.
• Copy obj.names, obj.data and the obj folder to the darknet/data directory.
• Copy process.py to the darknet directory.

OR

Do it using the command prompt with the following commands. The current working directory is C:\Users\zizou\Desktop\yolov4\darknet

Cleaning data and cfg folders.

#change dir to data folder and clean it. Delete all files and folders except the labels folder
cd data/
FOR %I IN (*) DO IF NOT %I == labels DEL %I
cd ..

#delete the cfg dir
rmdir /s cfg
#Are you sure (Y/N)? press y and enter

#create a new cfg folder
mkdir cfg

5(a) Copy the obj folder so that it is now in /darknet/data/ folder

xcopy ..\obj dataobj

# In the following prompt, press D for Directory

** In the prompt, Press D for Directory

5(b) Copy your yolov4-custom.cfg file so that it is now in /darknet/cfg/ folder

copy ..\yolov4-custom.cfg cfg

5(c) Copy the obj.names and obj.data files so that they are now in /darknet/data/ folder

copy ..\obj.names data
copy ..\obj.data  data

5(d) Copy the process.py file into the current darknet directory

copy ..\process.py .

6) Run the process.py python script to create the train.txt & test.txt files inside the data folder

The current working directory is  C:\Users\zizou\Desktop\yolov4\darknet

python process.py

List the contents of the data folder to check if the train.txt & test.txt files have been created.

dir data

The above process.py script creates the two files train.txt and test.txt where train.txt has paths to 90% of the images and test.txt has paths to 10% of the images. These files look as shown below.

7) Download and copy the pre-trained YOLOv4 weights to the “yolov4\darknet” directory.

The current working directory is C:\Users\zizou\Desktop\yolov4\darknet

Here we use transfer learning. Instead of training a model from scratch, we use pre-trained YOLOv4 weights which have been trained up to 137 convolutional layers. Download the YOLOv4 pre-trained weights file from here and copy it to your darknet folder.

yolov4.conv.137 pre-trained weights file

8) Training

Train your custom detector

The current working directory is  C:\Users\zizou\Desktop\yolov4\darknet

For best results, you should stop the training when the average loss is less than 0.05 if possible or at least constantly below 0.3, else train the model until the average loss does not show any significant change for a while.

darknet.exe detector train data/obj.data cfg/yolov4-custom.cfg yolov4.conv.137 -dont_show -map

The map parameter here gives us the Mean Average Precision. The higher the mAP the better it is for object detection. You can remove the -dont_show parameter to see the progress chart of mAP-loss against iterations.

You can visit the official AlexeyAB Github page which gives a detailed explanation on when to stop training. Click on the link below to jump to that section.

AlexeyAB GitHub When to stop training

To restart your training (In case the training does not finish and the training process gets interrupted)

If the training process gets interrupted or stops for some reason, you don’t have to start training your model from scratch again. You can restart training from where you left off. Use the weights that were saved last. The weights are saved every 100 iterations as yolov4-custom_last.weights in the training folder inside the yolov4 dir. (The path we gave as backup in “obj.data” file).

So to restart training run the following command:

darknet.exe detector train data/obj.data cfg/yolov4-custom.cfg ../training/yolov4-custom_last.weights -dont_show -map

9) Check performance

Check the training chart

You can check the performance of all the trained weights by looking at the chart.png file. However, the chart.png file only shows results if the training does not get interrupted i.e. if you do not get disconnected or lose your session. If you restart training from a saved point, this will not work.

Open the chart.png file in the darknet directory.

If this does not work, there are other methods to check your performance. One of them is by checking the mAP of the trained weights.

Check mAP (mean average precision)

You can check mAP for all the weights saved every 1000 iterations for eg:- yolov4-custom_4000.weights, yolov4-custom_5000.weights, yolov4-custom_6000.weights, and so on. This way you can find out which weights file gives you the best result. The higher the mAP the better it is.

Run the following command to check the mAP for a particular saved weights file where xxxx is the iteration number for it.(eg:- 4000,5000,6000,…)

darknet.exe detector map data/obj.data cfg/yolov4-custom.cfg ../training/yolov4-custom_xxxx.weights -points 0

10) Test your custom Object Detector

Make changes to your custom config file to set it to test mode

• change line batch to batch=1
• change line subdivisions to subdivisions=1

You can do it either manually or by simply running the code below

cd cfg
sed -i 's/batch=64/batch=1/' yolov4-custom.cfg
sed -i 's/subdivisions=32/subdivisions=1/' yolov4-custom.cfg
cd ..

NOTE: Set the first value of subdivisions to 16 / 32 / 64 in the command above based on whatever you have set it in the cfg file. Also, to use “sed” or other Unix commands in Windows, you need to install them either using Cygwin or GnuWin32.

Run detector on an image

Upload an image to a folder named mask_test_images on your desktop.

Run your custom detector on an image with this command. (The thresh flag sets the minimum accuracy required for object detection)

darknet.exe detector test data/obj.data cfg/yolov4-custom.cfg ../training/yolov4-custom_best.weights ../../mask_test_images/image1.jpg -thresh 0.3 

The output for this is saved as predictions.jpg inside the darknet folder. You can copy it to any output folder and save it with a different name. I am saving it as output_image.jpg inside the mask_test_images folder itself.

copy predictions.jpg ..\..\mask_test_images\output_image.jpg

Run detector on a video

Upload a video to a folder named mask_test_videos on your desktop.

Run your custom detector on a video with this command. (The thresh flag sets the minimum accuracy required for object detection). This saves the output video inside the same folder as “output.avi“.

darknet.exe detector demo data/obj.data cfg/yolov4-custom.cfg ../training/yolov4-custom_best.weights ../../mask_test_videos/test.mp4 -thresh 0.5 -i 0 -out_filename ../../mask_test_videos/output.avi

Run detector on a live webcam

Run the code below.

darknet.exe detector demo data/obj.data cfg/yolov4-custom.cfg ../training/yolov4-custom_best.weights -thresh 0.5  

NOTE:

The dataset I have collected for mask detection contains mostly close-up images. For more long-shot images you can search online. There are many sites where you can download labeled and unlabeled datasets. I have given a few links at the bottom under Dataset Sources. I have also given a few links for mask datasets. Some of them have more than 10,000 images.

Though there are certain tweaks and changes we can make to our training config file or add more images to the dataset for every type of object class through augmentation, we have to be careful so that it does not cause overfitting which affects the accuracy of the model.

For beginners, you can start simply by using the config file I have uploaded on my GitHub. I have also uploaded my mask images dataset along with the YOLO format labeled text files, which although might not be the best but will give you a good start on how to train your own custom detector model using YOLO. You can find a labeled dataset of better quality or an unlabeled dataset and label it yourself later.

GitHub link

I have uploaded all the files needed for training a custom YOLOv4 detector in Windows on my GitHub link below.

yolov4-custom-training_LOCAL_MACHINE

Labeled Dataset (obj.zip)

Labeled Mask Dataset

CREDITS

References

• Alexey AB GitHub
• pjreddie Github
• theAIGuysCode Github
• Paper YOLOv4
• pjreddie Site

Dataset Sources

You can download datasets for many objects from the sites mentioned below. These sites also contain images of many classes of objects along with their annotations/labels in multiple formats such as the YOLO_DARKNET text files and the PASCAL_VOC XML files.

• Open Images Dataset by Google
• Kaggle Datasets
• Roboflow Public Datasets
• VisualData Datasets

Mask Dataset Sources

I have used these 3 datasets for my labeled dataset:

• Prajnasb Github
• Joseph Nelson Roboflow
• X-zhangyang Github

More Mask Datasets

• Prasoonkottarathil Kaggle (20000 images)
• Ashishjangra27 Kaggle (12000 images )
• Andrewmvd Kaggle

Video Sources

• https://www.pexels.com/

The post Train a custom YOLOv4 object detector on Windows appeared first on TECHZIZOU.

My YouTube video on this!

In this tutorial, we will be training a custom object detector for mask detection using YOLOv4 and Darknet on our Linux(Ubuntu 18.04) system

FOLLOW THESE 10 STEPS TO TRAIN AN OBJECT DETECTOR USING YOLOv4

( But first Subscribe to my YouTube channel https://bit.ly/3Ap3sdi )

1. Create yolov4 and training folders on your Desktop
2. Open a command prompt and navigate to the “yolov4” folder
3. Create and copy the darknet.exe file
4. Create & copy the files we need for training ( i.e. “obj” dataset folder , “yolov4-custom.cfg”, “obj.data”, “obj.names” and “process.py” files) to your yolov4 dir
5. Copy the “yolov4-custom.cfg”, “obj.data”, “obj.names”, and “process.py” files and the “obj” dataset folder from the yolov4 directory to the darknet directory
6. Run the process.py python script to create the train.txt & test.txt files
7. Download the pre-trained YOLOv4 weights
8. Train the detector
9. Check performance
10. Test your custom Object Detector

~~~~~~~~~~~~~~ LET’S BEGIN !! ~~~~~~~~~~~~~~

1) Create ‘yolov4’ and ‘training’ folders

Create a folder named yolov4 on your Desktop. Next, create another folder named training inside the yolov4 folder. This is where we will save our trained weights (This path is mentioned in the obj.data file which we will upload later)

2) Open a terminal and navigate to the “yolov4” folder

Navigate to /Desktop/yolov4 folder using the following command in terminal or just right-click and open it in terminal. 

cd Desktop/yolov4/

3) Create and copy the darknet executable file

Create and copy your darknet folder containing the darknet executable file into the yolov4 folder. To know how to create the darknet folder containing the darknet executable file using make, go to this blog. Follow all the steps there for YOLO-Darknet installation on Linux.

This step can be a little tedious but this is the difficult part of this entire process. The rest of the training process here below is fairly simple. So, make sure you follow all the steps carefully in the above article to create darknet and also have CUDA and cuDNN installed and configured on your system. You’ll find all the steps explained in detail in section C of the above-mentioned blog.

Once the above process is done, copy the darknet folder we get from it and copy it to the yolov4 folder on your desktop.

4) Create & copy the following files which we need for training a custom detector

a. Labeled Custom Dataset
b. Custom cfg file
c. obj.data and obj.names files
d. process.py file (to create train.txt and test.txt files for training)

I have uploaded my custom files for mask detection on my GitHub. I am working with 2 classes i.e. “with_mask” and “without_mask”.

Labeling your Dataset

Input image example (Image1.jpg)

You can use any software for labeling like the labelImg tool.

I use an open-source labeling tool called OpenLabeling with a very simple UI.

Click on the link below to know more about the labeling process and other software for it:

Image Dataset Labeling article

NOTE: Garbage In = Garbage Out. Choosing and labeling images is the most important part. Try to find good-quality images. The quality of the data goes a long way toward determining the quality of the result.

The output YOLO format labeled file looks like as shown below.

4(a) Create and copy the labeled custom dataset “obj” folder to the “yolov4” folder

Put all the input image “.jpg” files and their corresponding YOLO format labeled “.txt” files in a folder named obj. Copy it to the yolov4 folder.

4(b) Create your custom config file and copy it to the ‘yolov4’ folder

Download the yolov4-custom.cfg file from darknet/cfg directory, make changes to it, and copy it to the yolov4 dir.

You can also download the custom config files from the official AlexeyAB Github.

You need to make the following changes in your custom config file:

• change line batch to batch=64

• change line subdivisions to subdivisions=16 or subdivisions=32 or subdivisions=64 (Read Note after this section for more info on these values)

• set network size width=416 height=416 or any value multiple of 32

• change line max_batches to (classes*2000 but not less than the number of training images and not less than 6000), f.e. max_batches=6000 if you train for 3 classes

• change line steps to 80% and 90% of max_batches, f.e. steps=4800,5400

• change [filters=255] to filters=(classes + 5)x3 in the 3 [convolutional] before each [yolo] layer, keep in mind that it only has to be the last [convolutional] before each of the [yolo] layers.

• change line classes=80 to your number of objects in each of 3 [yolo]-layers.

So if classes=1 then it should be filters=18. If classes=2 then write filters=21.

You can tweak other parameter values too like the learning rate, angle, saturation, exposure, and hue once you’ve understood how the basics of the training process work. For beginners, the above changes should suffice.

NOTE: What are subdivisions?

• It is the number of many mini-batches we split our batch into.
• Batch=64 -> loading 64 images for one iteration.
• Subdivision=8 -> Split batch into 8 mini-batches so 64/8 = 8 images per mini-batch and these 8 images are sent for processing. This process will be performed 8 times until the batch is completed and a new iteration will start with 64 new images.
• If you are using a GPU with low memory, set a higher value for subdivisions ( 32 or 64). This will obviously take longer to train since we are reducing the number of images being loaded and also the number of mini-batches.
• If you have a GPU with high memory, set a lower value for subdivisions (16 or 8). This will speed up the training process as this loads more images per iteration.

4(c) Create your “obj.data” and “obj.names” files and copy them to the yolov4 folder

obj.data

The obj.data file has :

• The number of classes.
• The path to train.txt and test.txt files that we will create later.
• The path to “obj.names” file which contains the names of the classes.
• The path to the training folder where the training weights will be saved.

classes = 2
train = data/train.txt
valid = data/test.txt
names = data/obj.names
backup = ../training

obj.names

Has objects’ names — each in a new line. Make sure the classes are in the same order as in the class_list.txt file used while labeling the images so the index id of every class is the same as mentioned in the labeled YOLO text files.

4(d) Copy the <process.py> script file to your yolov4 folder

(To divide all image files into 2 parts. 90% for train and 10% for test)

This process.py script creates the files train.txt & test.txt where the train.txt file has paths to 90% of the images and test.txt has paths to 10% of the images.

You can download the process.py script from my GitHub.

**IMPORTANT: The “process.py” script has only the “.jpg” format written in it, so other formats such as “.png”,“.jpeg”, or even “.JPG”(in capitals) won’t be recognized. If you are using any other format, make changes in the process.py script accordingly.

Now that we have uploaded all the files, our yolov4 folder on our desktop should look like this:

5) Copy all the files from the ‘yolov4' directory to the ‘yolov4/darknet’ directory

You can do all these steps manually through File manager.

Firstly, clean the cfg & data folders. Delete all files inside the data folder except the labels folder which is required for writing label names on the detection boxes and also delete all files inside the cfg folder as we already have our custom config file in the yolov4 main folder.

Next, copy all the files:

• Copy yolov4-custom.cfg to the darknet/cfg directory.
• Copy obj.names, obj.data and the obj folder to the darknet/data directory.
• Copy process.py to the darknet directory.

OR

Do the above steps using a terminal if you’re a terminal geek . Change current working directory to  /Desktop/yolov4/darknet

cd darknet/

Cleaning data and cfg folders.

find ./data -mindepth 1 ! -regex '^./data/labels(/.*)?' -delete

rm -rf cfg/
mkdir cfg

5(a) Copy the obj dataset folder so that it is now in /darknet/data/ folder

cp -R ../obj data/

5(b) Copy your yolov4-custom.cfg file so that it is now in /darknet/cfg/ folder

cp ../yolov4-custom.cfg cfg

5(c) Copy the obj.names and obj.data files so that they are now in /darknet/data/ folder

cp ../obj.names data
cp ../obj.data  data

5(d) Copy the process.py file into the current darknet directory

cp ../process.py .

6) Run the process.py python script to create the train.txt & test.txt files inside the data folder

The current working directory is /Desktop/yolov4/darknet.

python process.py

List the contents of the data folder to check if the train.txt & test.txt files have been created.

ls data/

The above process.py script creates the two files train.txt and test.txt where train.txt has paths to 90% of the images and test.txt has paths to 10% of the images. These files look as shown below.

7) Download and copy the pre-trained YOLOv4 weights to the “yolov4/darknet” directory.

The current working directory is /Desktop/yolov4/darknet

Here we use transfer learning. Instead of training a model from scratch, we use pre-trained YOLOv4 weights which have been trained up to 137 convolutional layers. Download the YOLOv4 pre-trained weights file from here and copy it to your darknet folder.

OR

Just run the command below.

wget https://github.com/AlexeyAB/darknet/releases/download/darknet_yolo_v3_optimal/yolov4.conv.137

8) Training

Train your custom detector

The current working directory is /Desktop/yolov4/darknet

For best results, you should stop the training when the average loss is less than 0.05 if possible or at least constantly below 0.3, else train the model until the average loss does not show any significant change for a while.

./darknet detector train data/obj.data cfg/yolov4-custom.cfg yolov4.conv.137 -dont_show -map

The map parameter here gives us the Mean Average Precision. The higher the mAP the better it is for object detection. You can remove the -dont_show parameter to see the progress chart of mAP-loss against iterations.

You can visit the official AlexeyAB Github page which gives a detailed explanation on when to stop training. Click on the link below to jump to that section.

AlexeyAB GitHub When to stop training

To restart your training (In case the training does not finish and the training process gets interrupted)

If the training process gets interrupted or stops for some reason , you don’t have to start training your model from scratch again. You can restart training from where you left off. Use the weights that were saved last. The weights are saved every 100 iterations as yolov4-custom_last.weights in the training folder inside the yolov4 dir. (The path we gave as backup in “obj.data” file).

So to restart training run the following command:

./darknet detector train data/obj.data cfg/yolov4-custom.cfg ../training/yolov4-custom_last.weights -dont_show -map

9) Check performance

Check the training chart

You can check the performance of all the trained weights by looking at the chart.png file. However, the chart.png file only shows results if the training does not get interrupted i.e. if you do not get disconnected or lose your session. If you restart training from a saved point, this will not work.

Open the chart.png file in the darknet directory.

If this does not work, there are other methods to check your performance. One of them is by checking the mAP of the trained weights.

Check mAP (mean average precision)

You can check mAP for all the weights saved every 1000 iterations for eg:- yolov4-custom_4000.weights, yolov4-custom_5000.weights, yolov4-custom_6000.weights, and so on. This way you can find out which weights file gives you the best result. The higher the mAP the better it is.

Run the following command to check the mAP for a particular saved weights file where xxxx is the iteration number for it.(eg:- 4000,5000,6000,…)

./darknet detector map data/obj.data cfg/yolov4-custom.cfg ../training/yolov4-custom_xxxx.weights -points 0

10) Test your custom Object Detector

Make changes to your custom config file to set it to test mode

• change line batch to batch=1
• change line subdivisions to subdivisions=1

You can do it either manually or by simply running the code below

cd cfg
sed -i 's/batch=64/batch=1/' yolov4-custom.cfg
sed -i 's/subdivisions=32/subdivisions=1/' yolov4-custom.cfg
cd ..

NOTE: Set the first value of subdivisions to 16 / 32 / 64 above in the command above based on whatever you have set it in the cfg file.

Run detector on an image

Upload an image to a folder named mask_test_images on your desktop.

Run your custom detector on an image with this command. (The thresh flag sets the minimum accuracy required for object detection)

./darknet detector test data/obj.data cfg/yolov4-custom.cfg ../training/yolov4-custom_best.weights ../../mask_test_images/image1.jpg -thresh 0.3 

The output for this is saved as predictions.jpg inside the darknet folder. You can copy it to any output folder and save it with a different name. I am saving it as output_image.jpg inside the mask_test_images folder itself.

cp predictions.jpg ../../mask_test_images/output_image1.jpg

Run detector on a video

Upload a video to a folder named mask_test_videos on your desktop.

Run your custom detector on a video(test.mp4) with this command. (The thresh flag sets the minimum accuracy required for object detection). This saves the output video inside the same folder as “output.avi“.

./darknet detector demo data/obj.data cfg/yolov4-custom.cfg ../training/yolov4-custom_best.weights ../../mask_test_videos/test.mp4 -thresh 0.5 -i 0 -out_filename ../../mask_test_videos/output.avi

Run detector on a live webcam

Run the code below.

./darknet detector demo data/obj.data cfg/yolov4-custom.cfg ../training/yolov4-custom_best.weights -thresh 0.5  

NOTE:

The dataset I have collected for mask detection contains mostly close-up images. For more long-shot images you can search online. There are many sites where you can download labeled and unlabeled datasets. I have given a few links at the bottom under Dataset Sources. I have also given a few links for mask datasets. Some of them have more than 10,000 images.

Though there are certain tweaks and changes we can make to our training config file or add more images to the dataset for every type of object class through augmentation, we have to be careful so that it does not cause overfitting which affects the accuracy of the model.

For beginners, you can start simply by using the config file I have uploaded on my GitHub. I have also uploaded my mask images dataset along with the YOLO format labeled text files, which although might not be the best but will give you a good start on how to train your own custom detector model using YOLO. You can find a labeled dataset of better quality or an unlabeled dataset and label it yourself later.

GitHub link

I have uploaded my custom mask dataset and all the other files needed for training a custom YOLOv4 detector in Windows on my GitHub link below.

yolov4-custom-training_LOCAL_MACHINE

Labeled Dataset (obj.zip)

Labeled Mask Dataset

YouTube video on this!

CREDITS

References

• Alexey AB GitHub
• pjreddie Github
• theAIGuysCode Github
• Paper YOLOv4
• pjreddie Site

Dataset Sources

You can download datasets for many objects from the sites mentioned below. These sites also contain images of many classes of objects along with their annotations/labels in multiple formats such as the YOLO_DARKNET text files and the PASCAL_VOC XML files.

• Open Images Dataset by Google
• Kaggle Datasets
• Roboflow Public Datasets
• VisualData Datasets

Mask Dataset Sources

I have used these 3 datasets for my labeled dataset:

• Prajnasb Github
• Joseph Nelson Roboflow
• X-zhangyang Github

More Mask Datasets

• Prasoonkottarathil Kaggle (20000 images)
• Ashishjangra27 Kaggle (12000 images )
• Andrewmvd Kaggle

Video Sources

• https://www.pexels.com/

The post Train a custom YOLOv4 object detector on Linux appeared first on TECHZIZOU.

My YouTube video on this!

In this tutorial, we will be training a custom object detector for mask detection using YOLOv4-tiny and Darknet on our Windows system

FOLLOW THESE 10 STEPS TO TRAIN AN OBJECT DETECTOR USING YOLOv4-tiny

( But first Subscribe to my YouTube channel https://bit.ly/3Ap3sdi )

1. Create yolov4-tiny and training folders on your Desktop
2. Open a command prompt and navigate to the “yolov4-tiny” folder
3. Create and copy the darknet.exe file
4. Create & copy the files we need for training ( i.e. “obj” dataset, “yolov4-tiny-custom.cfg”, “obj.data”, “obj.names” and “process.py” ) to your yolov4-tiny dir
5. Copy the files “yolov4-tiny-custom.cfg”, “obj.data”, “obj.names”, and “process.py” and the “obj” dataset folder from the yolov4-tiny directory to the darknet directory
6. Run the process.py python script to create the train.txt & test.txt files
7. Download the pre-trained YOLOv4-tiny weights
8. Train the detector
9. Check performance
10. Test your custom Object Detector

~~~~~~~~~~~~~~ LET’S BEGIN !! ~~~~~~~~~~~~~~

1) Create ‘yolov4-tiny’ and ‘training’ folders

Create a folder named yolov4-tiny on your Desktop. Next, create another folder named training inside the yolov4-tiny folder. This is where we will save our trained weights (This path is mentioned in the obj.data file which we will upload later)

2) Open command prompt and navigate to the “yolov4-tiny” folder

Navigate to C:\Users\zizou\Desktop\yolov4-tiny folder. 
NOTE: Replace zizou here with your Username

cd C:\Users\zizou\Desktop\yolov4-tiny

3) Create and copy the darknet.exe file

Create and copy your darknet folder containing the darknet.exe file into the yolov4-tiny folder. To know how to create the darknet folder containing the darknet.exe file using CMake, go to this blog. Follow all the steps there for YOLO installation on Windows.

This step can be a little tedious but this is the difficult part of this entire process. The rest of the training process here below is fairly simple. So, make sure you follow all the steps carefully in the above-mentioned article to create darknet and also have CUDA and cuDNN installed and configured on your system. You’ll find all the steps explained in detail in section A of the above-mentioned blog. You can also check out my video on YOLO installation for Windows at the bottom under the YouTube Videos section of the above-mentioned blog.

Once the above process is done, copy the darknet-master folder we get from it and copy it to the yolov4-tiny folder on your desktop. Rename the darknet-master folder to darknet for simplicity.

4) Create & copy the following files which we need for training a custom detector

a. Labeled Custom Dataset
b. Custom cfg file
c. obj.data and obj.names files
d. process.py file (to create train.txt and test.txt files for training)

I have uploaded my custom files for mask detection on my GitHub. I am working with 2 classes i.e. “with_mask” and “without_mask”.

Labeling your Dataset

Input image example (Image1.jpg)

You can use any software for labeling like the labelImg tool.

I use an open-source labeling tool called OpenLabeling with a very simple UI.

Click on the link below to know more about the labeling process and other software for it:

Image Dataset Labeling article

NOTE: Garbage In = Garbage Out. Choosing and labeling images is the most important part. Try to find good-quality images. The quality of the data goes a long way toward determining the quality of the result.

The output YOLO format labeled file looks like as shown below.

4(a) Create and copy the labeled custom dataset “obj” folder to the “yolov4-tiny” folder

Put all the input image “.jpg” files and their corresponding YOLO format labeled “.txt” files in a folder named obj.

Copy it to the yolov4-tiny folder.

4(b) Create your custom config file and copy it to the ‘yolov4-tiny’ folder

Download the yolov4-tiny-custom.cfg file from darknet/cfg directory, make changes to it, and copy it to the yolov4-tiny dir.

You can also download the custom config files from the official AlexeyAB Github.

You need to make the following changes in your custom config file:

• change line batch to batch=64

• change line subdivisions to subdivisions=16 or subdivisions=32 or subdivisions=64 (Read Note after this section for more info on these values)

• set network size width=416 height=416 or any value multiple of 32

• change line max_batches to (classes*2000 but not less than the number of training images and not less than 6000), f.e. max_batches=6000 if you train for 3 classes

• change line steps to 80% and 90% of max_batches, f.e. steps=4800,5400

• change [filters=255] to filters=(classes + 5)x3 in the 2 [convolutional] before each [yolo] layer, keep in mind that it only has to be the last [convolutional] before each of the [yolo] layers.

• change line classes=80 to your number of objects in each of 2 [yolo]-layers.

So if classes=1 then it should be filters=18. If classes=2 then write filters=21.

You can tweak other parameter values too like the learning rate, angle, saturation, exposure, and hue once you’ve understood how the basics of the training process work. For beginners, the above changes should suffice.

NOTE: What are subdivisions?

• It is the number of many mini-batches we split our batch into.
• Batch=64 -> loading 64 images for one iteration.
• Subdivision=8 -> Split batch into 8 mini-batches so 64/8 = 8 images per mini-batch and these 8 images are sent for processing. This process will be performed 8 times until the batch is completed and a new iteration will start with 64 new images.
• If you are using a GPU with low memory, set a higher value for subdivisions ( 32 or 64). This will obviously take longer to train since we are reducing the number of images being loaded and also the number of mini-batches.
• If you have a GPU with high memory, set a lower value for subdivisions (16 or 8). This will speed up the training process as this loads more images per iteration.

4(c) Create your “obj.data” and “obj.names” files and copy them to the yolov4-tiny folder

obj.data

The obj.data file has :

• The number of classes.
• The path to train.txt and test.txt files that we will create later.
• The path to “obj.names” file which contains the names of the classes.
• The path to the training folder where the training weights will be saved.

classes = 2
train = data/train.txt
valid = data/test.txt
names = data/obj.names
backup = ../training

obj.names

Has objects’ names — each in a new line. Make sure the classes are in the same order as in the class_list.txt file used while labeling the images so the index id of every class is the same as mentioned in the labeled YOLO text files.

4(d) Copy the <process.py> script file to your yolov4-tiny folder

(To divide all image files into 2 parts. 90% for train and 10% for test)

This process.py script creates the files train.txt & test.txt where the train.txt file has paths to 90% of the images and test.txt has paths to 10% of the images.

You can download the process.py script from my GitHub.

**IMPORTANT: The “process.py” script has only the “.jpg” format written in it, so other formats such as “.png”,“.jpeg”, or even “.JPG”(in capitals) won’t be recognized. If you are using any other format, make changes in the process.py script accordingly.

Now that we have uploaded all the files, our yolov4-tiny folder on our desktop should look like this:

5) Copy all the files from the ‘yolov4-tiny' directory to the ‘yolov4-tiny/darknet’ directory

You can do all these steps manually through File Explorer.

First, clean the data and cfg folders except for the labels folder inside the data folder which is required for writing label names on the detection boxes. So just remove all the files from the data folder except the labels folder and next, entirely clean the cfg folder as we already have our custom config file in the yolov4-tiny main folder.

Next, copy all the files:

• Copy yolov4-tiny-custom.cfg to the darknet/cfg directory.
• Copy obj.names, obj.data and the obj folder to the darknet/data directory.
• Copy process.py to the darknet directory.

OR

Do it using the command prompt with the following commands. The current working directory is  C:\Users\zizou\Desktop\yolov4-tiny\darknet

Cleaning data and cfg folders.

#change dir to data folder and clean it. Delete all files and folders except the labels folder
cd data/
FOR %I IN (*) DO IF NOT %I == labels DEL %I
cd ..

#delete the cfg dir
rmdir /s cfg
#Are you sure (Y/N)? press y and enter

#create a new cfg folder
mkdir cfg

5(a) Copy the obj folder so that it is now in /darknet/data/ folder

xcopy ..\obj data\obj

# In the following prompt, press D for Directory

** In the prompt, Press D for Directory

5(b) Copy your yolov4-tiny-custom.cfg file so that it is now in /darknet/cfg/ folder

copy ..\yolov4-tiny-custom.cfg cfg

5(c) Copy the obj.names and obj.data files so that they are now in /darknet/data/ folder

copy ..\obj.names data
copy ..\obj.data  data

5(d) Copy the process.py file into the current darknet directory

copy ..\process.py .

6) Run the process.py python script to create the train.txt & test.txt files inside the data folder

The current working directory is  C:\Users\zizou\Desktop\yolov4-tiny\darknet

python process.py

List the contents of the data folder to check if the train.txt & test.txt files have been created.

dir data
#OR
ls data

The above process.py script creates the two files train.txt and test.txt where train.txt has paths to 90% of the images and test.txt has paths to 10% of the images. These files look as shown below.

7) Download and copy the pre-trained YOLOv4-tiny weights to the “yolov4-tiny\darknet” directory.

The current working directory is  C:\Users\zizou\Desktop\yolov4-tiny\darknet

Here we use transfer learning. Instead of training a model from scratch, we use pre-trained YOLOv4-tiny weights which have been trained up to 29 convolutional layers. Download the YOLOv4-tiny pre-trained weights file from here and copy it to your darknet folder.

yolov4-tiny.conv.29 pre-trained weights file

8) Training

Train your custom detector

The current working directory is  C:\Users\zizou\Desktop\yolov4-tiny\darknet

For best results, you should stop the training when the average loss is less than 0.05 if possible or at least constantly below 0.3, else train the model until the average loss does not show any significant change for a while.

darknet.exe detector train data/obj.data cfg/yolov4-tiny-custom.cfg yolov4-tiny.conv.29 -dont_show -map

The map parameter here gives us the Mean Average Precision. The higher the mAP the better it is for object detection. You can remove the -dont_show parameter to see the progress chart of mAP-loss against iterations.

You can visit the official AlexeyAB Github page which gives a detailed explanation on when to stop training. Click on the link below to jump to that section.

AlexeyAB GitHub When to stop training

To restart your training (In case the training does not finish and the training process gets interrupted)

If the training process gets interrupted or stops for some reason, you don’t have to start training your model from scratch again. You can restart training from where you left off. Use the weights that were saved last. The weights are saved every 100 iterations as yolov4-tiny-custom_last.weights in the training folder inside the yolov4-tiny dir. (The path we gave as backup in “obj.data” file).

So to restart training run the following command:

darknet.exe detector train data/obj.data cfg/yolov4-tiny-custom.cfg ../training/yolov4-tiny-custom_last.weights -dont_show -map

9) Check performance

Check the training chart

You can check the performance of all the trained weights by looking at the chart.png file. However, the chart.png file only shows results if the training does not get interrupted i.e. if you do not get disconnected or lose your session. If you restart training from a saved point, this will not work.

Open the chart.png file in the darknet directory.

If this does not work, there are other methods to check your performance. One of them is by checking the mAP of the trained weights.

Check mAP (mean average precision)

You can check mAP for all the weights saved every 1000 iterations for eg:- yolov4-tiny-custom_4000.weights, yolov4-tiny-custom_5000.weights, yolov4-tiny-custom_6000.weights, and so on. This way you can find out which weights file gives you the best result. The higher the mAP the better it is.

Run the following command to check the mAP for a particular saved weights file where xxxx is the iteration number for it.(eg:- 4000,5000,6000,…)

darknet.exe detector map data/obj.data cfg/yolov4-tiny-custom.cfg ../training/yolov4-tiny-custom_xxxx.weights -points 0

10) Test your custom Object Detector

Make changes to your custom config file to set it to test mode

• change line batch to batch=1
• change line subdivisions to subdivisions=1

You can do it either manually or by simply running the code below

cd cfg
sed -i 's/batch=64/batch=1/' yolov4-tiny-custom.cfg
sed -i 's/subdivisions=32/subdivisions=1/' yolov4-tiny-custom.cfg
cd ..

NOTE: Set the first value of subdivisions to 16 / 32 / 64 above in the command above based on whatever you have set it in the cfg file. Also, to use “sed” or other Unix commands in Windows, you need to install them either using Cygwin or GnuWin32.

Run detector on an image

Upload an image to a folder named mask_test_images on your desktop.

Run your custom detector on an image with this command. (The thresh flag sets the minimum accuracy required for object detection)

darknet.exe detector test data/obj.data cfg/yolov4-tiny-custom.cfg ../training/yolov4-tiny-custom_best.weights ../../mask_test_images/image1.jpg -thresh 0.3 

The output for this is saved as predictions.jpg inside the darknet folder. You can copy it to any output folder and save it with a different name. I am saving it as output_image.jpg inside the mask_test_images folder itself.

copy predictions.jpg ..\..\mask_test_images\output_image.jpg

Run detector on a video

Upload a video to a folder named mask_test_videos on your desktop.

Run your custom detector on a video with this command. (The thresh flag sets the minimum accuracy required for object detection). This saves the output video inside the same folder as “output.avi“.

darknet.exe detector demo data/obj.data cfg/yolov4-tiny-custom.cfg ../training/yolov4-tiny-custom_best.weights ../../mask_test_videos/test.mp4 -thresh 0.5 -i 0 -out_filename ../../mask_test_videos/output.avi

Run detector on a live webcam

Run the code below.

darknet.exe detector demo data/obj.data cfg/yolov4-tiny-custom.cfg ../training/yolov4-tiny-custom_best.weights -thresh 0.5  

NOTE:

The dataset I have collected for mask detection contains mostly close-up images. For more long-shot images you can search online. There are many sites where you can download labeled and unlabeled datasets. I have given a few links at the bottom under Dataset Sources. I have also given a few links for mask datasets. Some of them have more than 10,000 images.

Though there are certain tweaks and changes we can make to our training config file or add more images to the dataset for every type of object class through augmentation, we have to be careful so that it does not cause overfitting which affects the accuracy of the model.

For beginners, you can start simply by using the config file I have uploaded on my GitHub. I have also uploaded my mask images dataset along with the YOLO format labeled text files, which although might not be the best but will give you a good start on how to train your own custom detector model using YOLO. You can find a labeled dataset of better quality or an unlabeled dataset and label it yourself later.

GitHub link

I have uploaded all the files needed for training a custom YOLOv4-tiny detector in Windows on my GitHub link below.

yolov4-tiny-custom-training_LOCAL_MACHINE

Labeled Dataset (obj.zip)

Labeled Mask Dataset

CREDITS

References

• Alexey AB GitHub
• pjreddie Github
• theAIGuysCode Github
• Paper YOLOv4
• pjreddie Site

Dataset Sources

You can download datasets for many objects from the sites mentioned below. These sites also contain images of many classes of objects along with their annotations/labels in multiple formats such as the YOLO_DARKNET text files and the PASCAL_VOC XML files.

• Open Images Dataset by Google
• Kaggle Datasets
• Roboflow Public Datasets
• VisualData Datasets

Mask Dataset Sources

I have used these 3 datasets for my labeled dataset:

• Prajnasb Github
• Joseph Nelson Roboflow
• X-zhangyang Github

More Mask Datasets

• Prasoonkottarathil Kaggle (20000 images)
• Ashishjangra27 Kaggle (12000 images )
• Andrewmvd Kaggle

Video Sources

• https://www.pexels.com/

The post Train a custom YOLOv4-tiny object detector on Windows appeared first on TECHZIZOU.

My YouTube video on this!

In this tutorial, we will be training a custom object detector for mask detection using YOLOv4-tiny and Darknet on our Linux(Ubuntu 18.04) system

FOLLOW THESE 10 STEPS TO TRAIN AN OBJECT DETECTOR USING YOLOv4-tiny

( But first Subscribe to my YouTube channel https://bit.ly/3Ap3sdi )

1. Create yolov4-tiny and training folders on your Desktop
2. Open a command prompt and navigate to the “yolov4-tiny” folder
3. Create and copy the darknet.exe file
4. Create & copy the files we need for training ( i.e. “obj” dataset folder , “yolov4-tiny-custom.cfg”, “obj.data”, “obj.names” and “process.py” files) to your yolov4-tiny dir
5. Copy the “yolov4-tiny-custom.cfg”, “obj.data”, “obj.names”, and “process.py” files and the “obj” dataset folder from the yolov4-tiny directory to the darknet directory
6. Run the process.py python script to create the train.txt & test.txt files
7. Download the pre-trained YOLOv4-tiny weights
8. Train the detector
9. Check performance
10. Test your custom Object Detector

~~~~~~~~~~~~~~ LET’S BEGIN !! ~~~~~~~~~~~~~~

1) Create ‘yolov4-tiny’ and ‘training’ folders

Create a folder named yolov4-tiny on your Desktop. Next, create another folder named training inside the yolov4-tiny folder. This is where we will save our trained weights (This path is mentioned in the obj.data file which we will upload later)

2) Open a terminal and navigate to the “yolov4-tiny” folder

Navigate to /Desktop/yolov4-tiny folder using the following command in terminal or just right-click and open it in terminal. 

cd Desktop/yolov4-tiny/

3) Create and copy the darknet executable file

Create and copy your darknet folder containing the darknet executable file into the yolov4-tiny folder. To know how to create the darknet folder containing the darknet executable file using make, go to this blog. Follow all the steps there for YOLO-Darknet installation on Linux.

This step can be a little tedious but this is the difficult part of this entire process. The rest of the training process here below is fairly simple. So, make sure you follow all the steps carefully in the above article to create darknet and also have CUDA and cuDNN installed and configured on your system. You’ll find all the steps explained in detail in section C of the above-mentioned blog.

Once the above process is done, copy the darknet folder we get from it and copy it to the yolov4-tiny folder on your desktop.

4) Create & copy the following files which we need for training a custom detector

a. Labeled Custom Dataset
b. Custom cfg file
c. obj.data and obj.names files
d. process.py file (to create train.txt and test.txt files for training)

I have uploaded my custom files for mask detection on my GitHub. I am working with 2 classes i.e. “with_mask” and “without_mask”.

Labeling your Dataset

Input image example (Image1.jpg)

You can use any software for labeling like the labelImg tool.

I use an open-source labeling tool called OpenLabeling with a very simple UI.

Click on the link below to know more about the labeling process and other software for it:

Image Dataset Labeling article

NOTE: Garbage In = Garbage Out. Choosing and labeling images is the most important part. Try to find good-quality images. The quality of the data goes a long way toward determining the quality of the result.

The output YOLO format labeled file looks like as shown below.

4(a) Create and copy the labeled custom dataset “obj” folder to the “yolov4-tiny” folder

Put all the input image “.jpg” files and their corresponding YOLO format labeled “.txt” files in a folder named obj. Copy it to the yolov4-tiny folder.

4(b) Create your custom config file and copy it to the ‘yolov4-tiny’ folder

Download the yolov4-tiny-custom.cfg file from darknet/cfg directory, make changes to it, and copy it to the yolov4-tiny dir.

You can also download the custom config files from the official AlexeyAB Github.

You need to make the following changes in your custom config file:

• change line batch to batch=64

• change line subdivisions to subdivisions=16 or subdivisions=32 or subdivisions=64 (Read Note after this section for more info on these values)

• set network size width=416 height=416 or any value multiple of 32

• change line max_batches to (classes*2000 but not less than the number of training images and not less than 6000), f.e. max_batches=6000 if you train for 3 classes

• change line steps to 80% and 90% of max_batches, f.e. steps=4800,5400

• change [filters=255] to filters=(classes + 5)x3 in the 2 [convolutional] before each [yolo] layer, keep in mind that it only has to be the last [convolutional] before each of the [yolo] layers.

• change line classes=80 to your number of objects in each of 2 [yolo]-layers.

So if classes=1 then it should be filters=18. If classes=2 then write filters=21.

You can tweak other parameter values too like the learning rate, angle, saturation, exposure, and hue once you’ve understood how the basics of the training process work. For beginners, the above changes should suffice.

NOTE: What are subdivisions?

• It is the number of many mini-batches we split our batch into.
• Batch=64 -> loading 64 images for one iteration.
• Subdivision=8 -> Split batch into 8 mini-batches so 64/8 = 8 images per mini-batch and these 8 images are sent for processing. This process will be performed 8 times until the batch is completed and a new iteration will start with 64 new images.
• If you are using a GPU with low memory, set a higher value for subdivisions ( 32 or 64). This will obviously take longer to train since we are reducing the number of images being loaded and also the number of mini-batches.
• If you have a GPU with high memory, set a lower value for subdivisions (16 or 8). This will speed up the training process as this loads more images per iteration.

4(c) Create your “obj.data” and “obj.names” files and copy them to the yolov4-tiny folder

obj.data

The obj.data file has :

• The number of classes.
• The path to train.txt and test.txt files that we will create later.
• The path to “obj.names” file which contains the names of the classes.
• The path to the training folder where the training weights will be saved.

classes = 2
train = data/train.txt
valid = data/test.txt
names = data/obj.names
backup = ../training

obj.names

Has objects’ names — each in a new line. Make sure the classes are in the same order as in the class_list.txt file used while labeling the images so the index id of every class is the same as mentioned in the labeled YOLO text files.

4(d) Copy the <process.py> script file to your yolov4-tiny folder

(To divide all image files into 2 parts. 90% for train and 10% for test)

This process.py script creates the files train.txt & test.txt where the train.txt file has paths to 90% of the images and test.txt has paths to 10% of the images.

You can download the process.py script from my GitHub.

**IMPORTANT: The “process.py” script has only the “.jpg” format written in it, so other formats such as “.png”,“.jpeg”, or even “.JPG”(in capitals) won’t be recognized. If you are using any other format, make changes in the process.py script accordingly.

Now that we have uploaded all the files, our yolov4-tiny folder on our desktop should look like this:

5) Copy all the files from the ‘yolov4-tiny' directory to the ‘yolov4-tiny/darknet’ directory

You can do all these steps manually through File manager.

Firstly, clean the cfg & data folders. Delete all files inside the data folder except the labels folder which is required for writing label names on the detection boxes and also delete all files inside the cfg folder as we already have our custom config file in the yolov4-tiny main folder.

Next, copy all the files:

• Copy yolov4-tiny-custom.cfg to the darknet/cfg directory.
• Copy obj.names, obj.data and the obj folder to the darknet/data directory.
• Copy process.py to the darknet directory.

OR

Do the above steps using a terminal if you’re a terminal geek . Change current working directory to  /Desktop/yolov4-tiny/darknet

cd darknet/

First, clean the data and cfg folders except for the labels folder inside the data folder which is required for writing label names on the detection boxes.

Cleaning data and cfg folders.

find ./data -mindepth 1 ! -regex '^./data/labels(/.*)?' -delete

rm -rf cfg/
mkdir cfg

5(a) Copy the obj dataset folder so that it is now in /darknet/data/ folder

cp -R ../obj data/

5(b) Copy your yolov4-tiny-custom.cfg file so that it is now in /darknet/cfg/ folder

cp ../yolov4-tiny-custom.cfg cfg

5(c) Copy the obj.names and obj.data files so that they are now in /darknet/data/ folder

cp ../obj.names data
cp ../obj.data  data

5(d) Copy the process.py file into the current darknet directory

cp ../process.py .

6) Run the process.py python script to create the train.txt & test.txt files inside the data folder

The current working directory is /Desktop/yolov4-tiny/darknet.

python process.py

List the contents of the data folder to check if the train.txt & test.txt files have been created.

ls data/

The above process.py script creates the two files train.txt and test.txt where train.txt has paths to 90% of the images and test.txt has paths to 10% of the images. These files look as shown below.

7) Download and copy the pre-trained YOLOv4-tiny weights to the “yolov4-tiny/darknet” directory.

The current working directory is /Desktop/yolov4-tiny/darknet

Here we use transfer learning. Instead of training a model from scratch, we use pre-trained YOLOv4-tiny weights which have been trained up to 29 convolutional layers. Download the YOLOv4-tiny pre-trained weights file from here and copy it to your darknet folder.

OR

Just run the command below.

wget https://github.com/AlexeyAB/darknet/releases/download/darknet_yolo_v4_pre/yolov4-tiny.conv.29

8) Training

Train your custom detector

The current working directory is /Desktop/yolov4-tiny/darknet

For best results, you should stop the training when the average loss is less than 0.05 if possible or at least constantly below 0.3, else train the model until the average loss does not show any significant change for a while.

./darknet detector train data/obj.data cfg/yolov4-tiny-custom.cfg yolov4-tiny.conv.29 -dont_show -map

The map parameter here gives us the Mean Average Precision. The higher the mAP the better it is for object detection. You can remove the -dont_show parameter to see the progress chart of mAP-loss against iterations.

You can visit the official AlexeyAB Github page which gives a detailed explanation on when to stop training. Click on the link below to jump to that section.

AlexeyAB GitHub When to stop training

To restart your training (In case the training does not finish and the training process gets interrupted)

If the training process gets interrupted or stops for some reason , you don’t have to start training your model from scratch again. You can restart training from where you left off. Use the weights that were saved last. The weights are saved every 100 iterations as yolov4-tiny-custom_last.weights in the training folder inside the yolov4-tiny dir. (The path we gave as backup in “obj.data” file).

So to restart training run the following command:

./darknet detector train data/obj.data cfg/yolov4-tiny-custom.cfg ../training/yolov4-tiny-custom_last.weights -dont_show -map

9) Check performance

Check the training chart

You can check the performance of all the trained weights by looking at the chart.png file. However, the chart.png file only shows results if the training does not get interrupted i.e. if you do not get disconnected or lose your session. If you restart training from a saved point, this will not work.

Open the chart.png file in the darknet directory.

If this does not work, there are other methods to check your performance. One of them is by checking the mAP of the trained weights.

Check mAP (mean average precision)

You can check mAP for all the weights saved every 1000 iterations for eg:- yolov4-tiny-custom_4000.weights, yolov4-tiny-custom_5000.weights, yolov4-tiny-custom_6000.weights, and so on. This way you can find out which weights file gives you the best result. The higher the mAP the better it is.

Run the following command to check the mAP for a particular saved weights file where xxxx is the iteration number for it.(eg:- 4000,5000,6000,…)

./darknet detector map data/obj.data cfg/yolov4-tiny-custom.cfg ../training/yolov4-tiny-custom_xxxx.weights -points 0

10) Test your custom Object Detector

Make changes to your custom config file to set it to test mode

• change line batch to batch=1
• change line subdivisions to subdivisions=1

You can do it either manually or by simply running the code below

cd cfg
sed -i 's/batch=64/batch=1/' yolov4-tiny-custom.cfg
sed -i 's/subdivisions=32/subdivisions=1/' yolov4-tiny-custom.cfg
cd ..

NOTE: Set the first value of subdivisions to 16 / 32 / 64 above in the command above based on whatever you have set it in the cfg file.

Run detector on an image

Upload an image to a folder names mask_test_images on your desktop.

Run your custom detector on an image with this command. (The thresh flag sets the minimum accuracy required for object detection)

./darknet detector test data/obj.data cfg/yolov4-tiny-custom.cfg ../training/yolov4-tiny-custom_best.weights ../../mask_test_images/image1.jpg -thresh 0.3 

The output for this is saved as predictions.jpg inside the darknet folder. You can copy it to any output folder and save it with a different name. I am saving it as output_image.jpg inside the mask_test_images folder itself.

cp predictions.jpg ../../mask_test_images/output_image1.jpg

Run detector on a video

Upload a video to a folder named mask_test_videos on your desktop.

Run your custom detector on a video(test.mp4) with this command. (The thresh flag sets the minimum accuracy required for object detection). This saves the output video inside the same folder as “output.avi“.

./darknet detector demo data/obj.data cfg/yolov4-tiny-custom.cfg ../training/yolov4-tiny-custom_best.weights ../../mask_test_videos/test.mp4 -thresh 0.5 -i 0 -out_filename ../../mask_test_videos/output.avi

Run detector on a live webcam

Run the code below.

./darknet detector demo data/obj.data cfg/yolov4-tiny-custom.cfg ../training/yolov4-tiny-custom_best.weights -thresh 0.5  

NOTE:

The dataset I have collected for mask detection contains mostly close-up images. For more long-shot images you can search online. There are many sites where you can download labeled and unlabeled datasets. I have given a few links at the bottom under Dataset Sources. I have also given a few links for mask datasets. Some of them have more than 10,000 images.

Though there are certain tweaks and changes we can make to our training config file or add more images to the dataset for every type of object class through augmentation, we have to be careful so that it does not cause overfitting which affects the accuracy of the model.

For beginners, you can start simply by using the config file I have uploaded on my GitHub. I have also uploaded my mask images dataset along with the YOLO format labeled text files, which although might not be the best but will give you a good start on how to train your own custom detector model using YOLO. You can find a labeled dataset of better quality or an unlabeled dataset and label it yourself later.

GitHub link

I have uploaded my custom mask dataset and all the other files needed for training a custom YOLOv4-tiny detector in Windows on my GitHub link below.

yolov4-tiny-custom-training_LOCAL_MACHINE

Labeled Dataset (obj.zip)

Labeled Mask Dataset

YouTube video on this!

CREDITS

References

• Alexey AB GitHub
• pjreddie Github
• theAIGuysCode Github
• Paper YOLOv4
• pjreddie Site

Dataset Sources

You can download datasets for many objects from the sites mentioned below. These sites also contain images of many classes of objects along with their annotations/labels in multiple formats such as the YOLO_DARKNET text files and the PASCAL_VOC XML files.

• Open Images Dataset by Google
• Kaggle Datasets
• Roboflow Public Datasets
• VisualData Datasets

Mask Dataset Sources

I have used these 3 datasets for my labeled dataset:

• Prajnasb Github
• Joseph Nelson Roboflow
• X-zhangyang Github

More Mask Datasets

• Prasoonkottarathil Kaggle (20000 images)
• Ashishjangra27 Kaggle (12000 images )
• Andrewmvd Kaggle

Video Sources

• https://www.pexels.com/

The post Train a custom YOLOv4-tiny object detector on Linux appeared first on TECHZIZOU.