Repository navigation
Why is the TensorRT input binding set to FP16 for FP16 engines by default?Β #6777
Description
Search before asking
- I have searched the YOLOv5 issues and found no similar bug report.
YOLOv5 Component
No response
Bug
Right now, when exporting a model to TensorRT using export.py with the --half argument, the input binding is set to datatype "half" while the output is set to datatype "float":
TensorRT: Network Description:
TensorRT: input "images" with shape (1, 3, 640, 640) and dtype DataType.HALF
TensorRT: output "output" with shape (1, 25200, 6) and dtype DataType.FLOAT
However, this means that whenever the resulting engine is used, its inputs also need to be converted to FP16, as can be seen in detect.py:
Line 118 in cea994b
Now I was wondering: is this intentional? TensorRT can handle an input in FP32 without problem even if the remaining engine is in half precision. However, having TensorRT handle the conversion means less overhead for the developer, while most likely also being faster as the conversion can be handled by the GPU instead of CPU. The datatype of the input binding when creating a new engine can be easily changed with a single line of code:
network.get_input(0).dtype = trt.DataType.FLOAT
Are you willing to submit a PR?
- Yes I'd like to help by submitting a PR!
Activity
glenn-jocher commentedon Feb 25, 2022
@DavidBaldsiefen thanks for raising this issue!
The basic idea is that
--halfexport will create a TRT model expecting--halfinputs, i.e.If you want FP32 inputs simply use default settings:
Does this resolve your question or do you still think there's an optimization/simplification we could do in the workflow?
DavidBaldsiefen commentedon Feb 25, 2022
I believe that it makes more sense to set the default input binding to float even for FP16 engines. As mentioned above, I do belive that a) it is faster to have TensorRT handle the input conversion and b) it is more intuitive for developers. For example, C++ does not have any FP16 datatype by default, which means that the engine exported by
export.pycannot be used for C++ inference without modifications. In addition, it would no longer be necessary to append the--halfoption toval.pyordetect.py. However, I am not sure how changing the input-tensor to float precision would affect ONNX models that are converted to FP16glenn-jocher commentedon Feb 25, 2022
@DavidBaldsiefen hmm interesting. I just tested a new workflow without issue, exporting at --half and running inference with FP32 inputs. The TRT model seems to convert on it's own (so flag already appears unnecessary).
glenn-jocher commentedon Feb 25, 2022
@DavidBaldsiefen BTW the ONNX exports made during TRT export are just a byproduct of the TRT exports, so I wouldn't worry about them, they likely won't be used by users. You can always export them directly with
--include onnxalso.DavidBaldsiefen commentedon Feb 25, 2022
How so? When I create an engine using the default
export.py, I need to set the--halfflag fordetect.pyandval.py, as otherwise nothing is detected. If I modifyexport.pyso that the input binding is FP32, the opposite is true.glenn-jocher commentedon Feb 25, 2022
@DavidBaldsiefen ah yes, you are correct! Silent errors (no detections) are produced with improper FP16-FP32 pairing, but no error is shown unfortunately.
glenn-jocher commentedon Feb 25, 2022
I get these results when testing all 4 combinations:
DavidBaldsiefen commentedon Feb 26, 2022
The results make sense. A start could be to check the engines input datatype in
detect.py(orcommon.py) and print a warning if it does not match the presence (or absence) of--half. That would prevent issues like #6766glenn-jocher commentedon Feb 26, 2022
@DavidBaldsiefen do you know of a way to determine the quantization of a TRT model after loading? Perhaps we should have an assert but I don't know how to get the actual value, i.e.:
Assert engine precision ultralytics#6777
DavidBaldsiefen commentedon Feb 27, 2022
@glenn-jocher Yes, I have created a pull request that would do the assertion in
val.pyanddetect.py: #6798glenn-jocher commentedon Feb 28, 2022
@DavidBaldsiefen I was thinking after seeing the above issues perhaps we want to default TRT exports to FP16 and then add the line in your original post since you said this could accommodate both FP32 and FP16 inputs?
Part of what we want to do is default to the best settings, and based on the above speed results --half TRT export seems like the best choice. I still need to do a mAP comparison, but usually (i.e. with PyTorch) FP32 and FP16 inference return almost identical accuracies.
EDIT: where exactly would we slot the above line into export.py?
glenn-jocher commentedon Mar 4, 2022
@DavidBaldsiefen any comment on what I asked earlier in #6777 (comment) ?
Would it make sense to default to FP16 and add your line to be robust to FP32 inputs by default?
DavidBaldsiefen commentedon Mar 4, 2022
@glenn-jocher I agree that it makes sense to default TensorRT exports to FP16; I suppose that in most cases, TensorRT deployment is performed for the sole purpose of increasing performance through that method. I actually tested YOLOv5s and YOLOv5m TensorRT engines recently on the validation dataset with these results (please ignore YOLOv4-CSP):
This would indicate a accuracy loss of just around 1% at a performance gain of around 100% (for batch size one, throughput more than doubled while latency halved. Tested on a NVIDIA Jetson AGX).
Do you mean that you would default the engine to FP16 and make the input precision depend on the
--halfflag?2 remaining items
DavidBaldsiefen commentedon Mar 4, 2022
@glenn-jocher I have updated my pull request to include these changes
glenn-jocher commentedon Mar 4, 2022
@DavidBaldsiefen awesome! Ran some benchmarks of my own on Colab. See no reason not to default to FP16 as long as we can run inference without the --half flag (so users don't get confused on crash without it).
YOLOv5 π v6.1-13-g601dbb8 torch 1.10.0+cu111 CUDA:0 (Tesla V100-SXM2-16GB, 16160MiB)
Automatically adjust detect.py input precision ultralytics#6777
Automatically adjust val.py input precision ultralytics#6777
Default FP16 TensorRT export (#6798)
Update Neuralmagic-YoloV5 integration to v6.1 (#29)
henry-zwart commentedon May 4, 2022
I've run into the above-mentioned issue when working with the model programmatically (exported at half precision, inputs at full precision, no detections found). I'm currently passing my inputs as a list of PIL.Image objects, and it doesn't seem that there's a simple way to cast these inputs to the right dtype. Do you have any recommendations?
glenn-jocher commentedon May 4, 2022
@henry-zwart π hi, thanks for letting us know about this possible problem with YOLOv5 π. TensorRT export and PIL inference with PyTorch Hub works correctly in our tests:
We've created a few short guidelines below to help users provide what we need in order to start investigating a possible problem.
How to create a Minimal, Reproducible Example
When asking a question, people will be better able to provide help if you provide code that they can easily understand and use to reproduce the problem. This is referred to by community members as creating a minimum reproducible example. Your code that reproduces the problem should be:
For Ultralytics to provide assistance your code should also be:
git pullorgit clonea new copy to ensure your problem has not already been solved in master.If you believe your problem meets all the above criteria, please close this issue and raise a new one using the π Bug Report template with a minimum reproducible example to help us better understand and diagnose your problem.
Thank you! π
Default FP16 TensorRT export (ultralytics#6798)