Why TensorRT ONNX parser fails, while parsing the ONNX model? Tips and tricks to win

Well, till now you all know about ONNX (ONNX is an open format built to represent machine learning models). My previous article on ONNX can be seen here.

ONNX is cool, but sometimes it’s too complicated

The above line actually means a lot when you start getting TensroRT ONNX parser error. Below are the two most common errors (but not limited to) we get while parsing the ONNX model in TensroRT.

1.  Please ensure the network was created using the EXPLICIT_BATCH NetworkDefinitionCreationFlag."
2. Network validation failed.

The first error is more related to the dynamic batch size, for solving this problem, you can see my previous article. In addition to this, you can also open your ONNX file in notepad and manually put {?} [yes, a question mark sign] at the location of batch size. A more code-savvy audience can use the below code to do the same thing-

network_creation_flag = 1 <<        int(trt.NetworkDefinitionCreationFlag.EXPLICIT_PRECISION)

network_creation_flag |= 1 <<int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH)

self.network = self.builder.create_network(network_creation_flag)

Now problem 2 needs more in-depth analysis, as this is related to the ONNX format, opset version, etc…

So, let’s tame this problem

Simplify the ONNX transported model (ONNX Simplifier)

Let’s assume you exported your ONNX model, but have you ever seen the model graph of it? If not then, please do it once! You will find that ONNX model graph is way too complicated even for simple operations and when we talk about a complex model it’s like a cobweb.

Hence, our target is to simplify this complexity first.

ONNX Simplifier [GitHub] – It’s an open-source library which helps in simplifying this complex exported ONNX model. And this simplification, most of the time, solves the second problem of model parsing, stated above. For python lovers, you can directly install it using pip command –

pip3 install onnx-simplifier

ONNX Simplifier (Web version) - It works out of the box and doesn't need any installation. Just open the webpage, choose ONNX as the output format, check the ONNX simplifier, and then select your model to simplify. Note that the web version is in its very early stage, if the web version doesn't work well for you, you can install the Python version.

An example

Let see how a simple operation in PyTorch, when exported with ONNX gets complicated.

import torch
 
 
class JustReshape(torch.nn.Module):
    def __init__(self):
        super(JustReshape, self).__init__()
 
    def forward(self, x):
        return x.view((x.shape[0], x.shape[1], x.shape[3], x.shape[2]))
 
 
net = JustReshape()
model_name = 'just_reshape.onnx'
dummy_input = torch.randn(2, 3, 4, 5)
torch.onnx.export(net, dummy_input, model_name, input_names=['input'], output_names=['output'])

The above code takes a tensor and just reshape it.

Now, let’s visualize the graph using Netron. Netron is a viewer for neural networks, deep learning, and machine learning models.

Now we simplify the ONNX model and then again visualize it. This can be achieved by below lines of code

import onnx

from onnxsim import simplify


# load your predefined ONNX model

model = onnx.load('just_reshape' + '.onnx')


# convert model

model_simp, check = simplify(model)

onnx.save(model_simp,'simplified.onnx')

 And below is the simplified graph -

And this is what exactly, I was expecting my operation to be. While the default ONNX model looks so complicated, against the simplified ONNX model. This complication affects the parsing engine in TensorRT

So now you know that even simple operations, when exported can create complex operation graphs, which causes parsing failure while using TensorRT ONNX model parser engine.

The problems discussed above are a select one, there can be other reasons as well, I welcome any suggestion and ready to render my help in this respect.

Articles in this series

1- Model optimization for Fast Inference and Quantization

2- TensorRT Installation: Problems and "Way Arounds"

3 - ONNX Model: Export Using Pytorch, Problems, and Solutions

Till then enjoy oranges (mandarini)…!!

Thanks