Counting Vehicles - Deploy to DeepLens - Part 4

In this part we will export our TensorFlow model, optimize it with the Intel OpenVino Toolkit and deploy it to the DeepLens device.

Export and Optimizing the TensorFlow model

Once we are satisified with the results of our model the next step is to export the model so that it can be optimized by the Intel OpenVino Toolkit. When I first began this project I followed Amazon's guide on importing an externally trained model and wasn't able to get it to work properly. The model optimization process on the device kept failing with the error.

  File "/opt/awscam/intel/deeplearning_deploymenttoolkit/deployment_tools/model_optimizer/mo/", line 222, in main
    return driver(argv)
  File "/opt/awscam/intel/deeplearning_deploymenttoolkit/deployment_tools/model_optimizer/mo/", line 190, in driver
  File "/opt/awscam/intel/deeplearning_deploymenttoolkit/deployment_tools/model_optimizer/mo/pipeline/", line 143, in tf2nx
  File "/opt/awscam/intel/deeplearning_deploymenttoolkit/deployment_tools/model_optimizer/mo/middle/passes/", line 55, in partial_infer
    nodes = nx.topological_sort(graph)
  File "/usr/local/lib/python3.5/dist-packages/networkx/algorithms/", line 157, in topological_sort
    raise nx.NetworkXUnfeasible("Graph contains a cycle.")
networkx.exception.NetworkXUnfeasible: Graph contains a cycle.

Therefore, we will optimize the model before loading it onto the device. First, download the Intel OpenVino Toolkit. Version 2018 R3 and 2018 R4 should both work, however, if you do choose to use R4 make sure to use the flag --generate_deprecated_IR_V2 when invoking the python script below, the DeepLens only supports IR V2 and will fail with the error that it doesn't support model version 3 without that flag. Also, if you deploy the optimized model and the bounding boxes are not correct or the results are not very good try downgrading back to the 2018 R3 version. Once downloaded and installed navigate to the computer_vision_sdk/deployment_tools/model_optimizer/ directory. Within there I chose to create a new pipenv environment using Python 3 and installed TensorFlow version 1.9, since we will not be training inside this virtual environment we do not need the TensorFlow GPU version. There are other Python requirements within requirements.txt that pipenv should pick up. If you encounter an error like

TypeError: new() got an unexpected keyword argument 'serialized_options'

You may need to update the version of the protocol buffers library within the virtual enivronment as described in this GitHub issue Once that is setup run the script, after copying it to models/research, updating the file directories and specifying the correct checkpoint file. Be sure to run this script while in the shell for the virtual environment for the OpenVino toolkit. This script will give us a frozen_inference_graph.pb file that we will optimize.

While still in the virtual environment navigate back to the computer_vision_sdk/deployment_tools/model_optimizer/ directory and execute the following (append the flag --generate_deprecated_IR_V2 if using the 2018 R4 release)

python --log_level=DEBUG --input_model /mnt/storage/development/count-cars/models/export-1.9/frozen_inference_graph.pb --tensorflow_use_custom_operations_config extensions/front/tf/ssd_v2_support.json --tensorflow_object_detection_api_pipeline_config /mnt/storage/development/count-cars/models/model/pipeline.config --reverse_input_channels

You can read more information about the different options to the script by looking at Intel's documentation on optimizing TensorFlow Object Detection models. If that command executes successfully you should get a frozen_inference_graph.bin and frozen_inference_graph.xml file representing the optimized model.

Deploying and Running on the DeepLens

Since we manually optimized our model in order to have the DeepLens use them we have to transfer them to the device and store them in the /opt/awscam/artifacts/ directory. The next step is to create a Lambda function that will be responsible for actually doing the inference and to create and deploy a DeepLens project with that Lambda function. For assistance with these two steps refer back to Part 2, the steps will be exactly the same, with the only difference being the code that is used within the Lambda function. The code we will use is contained within Most of the code is the same except for a few key parts. We will also define an environment variable named VEHICLE_DETECTION_BUCKET where we we store our inference results, using the same S3 bucket created early is advised.

Loading the model

Since we already optimized the model we only need to load it from the filesystem.

model_path = '/opt/awscam/artifacts/mobilenet/frozen_inference_graph.xml'
model = awscam.Model(model_path, {'GPU': 1})

Doing Inference

Inference is fairly straightforward, take the model and call parseResult with the frame that we resized to the model's input shape.

parsed_inference_results = 
        model.parseResult(model_type, model.doInference(frame_resize))

Analyzing results

We use a for loop to go over the inference results and for each detection that passes the threshold that we defined we draw it on the frame and collect results to upload to S3.

DeepLens Inference Result This is an image capture of the result labeled with a bounding box

[{"ymax": 882, "probability": 0.9631285667419434, "label": "car right", "xmax": 1183, "xmin": 675, "ymin": 696}]

This is a subset of the data extracted that tells us the probability and the coordinates of the bounding box.

Counting Vehicle Script

I had this DeepLens vehicle detector run for a day generating results and uploading them to S3. Later on I used awscli to copy the data from S3 to my local computer and ran a simple script to count the vehicles. Since we capture a vehicle driving by, we may capture it multiple times, to avoid double counting it we use simple heuristics to de-duplicate. In reality we could have used machine learning to classify the duplicate vehicles, however, I found using a simple image hash, along with the time between detections and the vehihcle direction and type I was able to get it to be fairly accurate. Here are the results of my analysis.

Vehicle Class Amount
Car 82
Truck 67
SUV 87
Van 5
Motorcycle 0
Person 0
Total 241


Overall the model does a fairly good job of predicting a correct label if it does detect a vehicle. However, it does have trouble detecting some vehicles. I believe the issue is the lighting, the initial batch of training data I collected was from the beginnig of September, so the sun was at a higher angle in the sky providing a different lighting pattern than in November. One way to alleviate this would to add some training examples from this time of year or take advantage of more data augemntation opportunities. There are also some landscaping differences, mostly the addition of snow, Christmas decorations and the completion of the road work.

Some areas that I would like to improve upon are

  1. More resiliance to lighting changes to improve the recall ability, either through data augmentation or more training data
  2. Improve the inference speed, I am able to only get about 1-2 frames per second, meaning if a vehicle is driving fast we may miss detecting it
  3. Detect the color of the vehicle
  4. Calculate the speed of the vehicle (relies on faster inference speed)

Thank you for reading.





Setup TensorFlow Training


Model Improvements