How do I use CUDA with a custom model?

❔Question

Code:

import torch
import cv2
from PIL import Image
from time import sleep, time
import numpy as np
# Model
model = torch.hub.load('ultralytics/yolov5', 'custom', path_or_model='last.pt')
# Images
img = Image.open('test.png')
# convert image to numpy array
img = np.asarray(img)

# Inference
stime = time()
results = model(img, size=608)
print(time() - stime) #0.8 sec

speed: 1frame - 0.78sec (cpu)

detect.py
speed: 0.04sec (With CUDA)

is much faster

@Pro100rus32

@Pro100rus32 YOLOv5 PyTorch Hub models automatically use CUDA backends if detected. If you are not observing this behavior your code is out of date. You can update your Hub cache with

model = torch.hub.load('ultralytics/yolov5', 'custom', path_or_model='last.pt', force_reload=True)

Also note you can pass a filename directly to model() for inference, you don't need to convert to numpy array first.

@Pro100rus32

@Pro100rus32 also note that results.print() includes profiling results, so you don't need to run your own timing logic.

@Pro100rus32

@Pro100rus32 YOLOv5 PyTorch Hub models automatically use CUDA backends if detected. If you are not observing this behavior your code is out of date. You can update your Hub cache with
model = torch.hub.load('ultralytics/yolov5', 'custom', path_or_model='last.pt', force_reload=True)
Also note you can pass a filename directly to model() for inference, you don't need to convert to numpy array first.

But why is it so slow?)

@Pro100rus32

@Pro100rus32 I've already answered your question, you might want to read my response in #2630 (comment)

Did I do the right thing?

import torch
from time import time
# Model
model = torch.hub.load('ultralytics/yolov5', 'custom', path_or_model='last.pt', force_reload=True)

# Inference
stime = time()
results = model('test.png', size=608)
print(time() - stime) #0.8 sec

Out:

Downloading: "https://github.com/ultralytics/yolov5/archive/master.zip" to C:\Users\TEST/.cache\torch\hub\master.zip

                 from  n    params  module                                  arguments                     
  0                -1  1      3520  models.common.Focus                     [3, 32, 3]                    
  1                -1  1     18560  models.common.Conv                      [32, 64, 3, 2]                
  2                -1  1     18816  models.common.C3                        [64, 64, 1]                   
  3                -1  1     73984  models.common.Conv                      [64, 128, 3, 2]               
  4                -1  1    156928  models.common.C3                        [128, 128, 3]                 
  5                -1  1    295424  models.common.Conv                      [128, 256, 3, 2]              
  6                -1  1    625152  models.common.C3                        [256, 256, 3]                 
  7                -1  1   1180672  models.common.Conv                      [256, 512, 3, 2]              
  8                -1  1    656896  models.common.SPP                       [512, 512, [5, 9, 13]]        
  9                -1  1   1182720  models.common.C3                        [512, 512, 1, False]          
 10                -1  1    131584  models.common.Conv                      [512, 256, 1, 1]              
 11                -1  1         0  torch.nn.modules.upsampling.Upsample    [None, 2, 'nearest']          
 12           [-1, 6]  1         0  models.common.Concat                    [1]                           
 13                -1  1    361984  models.common.C3                        [512, 256, 1, False]          
 14                -1  1     33024  models.common.Conv                      [256, 128, 1, 1]              
 15                -1  1         0  torch.nn.modules.upsampling.Upsample    [None, 2, 'nearest']          
 16           [-1, 4]  1         0  models.common.Concat                    [1]                           
 17                -1  1     90880  models.common.C3                        [256, 128, 1, False]          
 18                -1  1    147712  models.common.Conv                      [128, 128, 3, 2]              
 19          [-1, 14]  1         0  models.common.Concat                    [1]                           
 20                -1  1    296448  models.common.C3                        [256, 256, 1, False]          
 21                -1  1    590336  models.common.Conv                      [256, 256, 3, 2]              
 22          [-1, 10]  1         0  models.common.Concat                    [1]                           
 23                -1  1   1182720  models.common.C3                        [512, 512, 1, False]          
 24      [17, 20, 23]  1     16182  models.yolo.Detect                      [1, [[10, 13, 16, 30, 33, 23], [30, 61, 62, 45, 59, 119], [116, 90, 156, 198, 373, 326]], [128, 256, 512]]
Model Summary: 283 layers, 7063542 parameters, 7063542 gradients, 16.4 GFLOPS

Adding autoShape... 
0.8274714946746826

Process finished with exit code 0

@Pro100rus32

@Pro100rus32 for examples on proper usage see PyTorch Hub tutorial:

YOLOv5 Tutorials

Train Custom Data 🚀 RECOMMENDED
Weights & Biases Logging 🌟 NEW
Supervisely Ecosystem 🌟 NEW
Multi-GPU Training
PyTorch Hub ⭐ NEW
ONNX and TorchScript Export
Test-Time Augmentation (TTA)
Model Ensembling
Model Pruning/Sparsity
Hyperparameter Evolution
Transfer Learning with Frozen Layers ⭐ NEW
TensorRT Deployment

I suppose I need to create a separate topic. Since my cuda is not working with this code:

import torch

# Model
model = torch.hub.load('ultralytics/yolov5', 'yolov5s')

# Images
dir = 'https://github.com/ultralytics/yolov5/raw/master/data/images/'
imgs = [dir + f for f in ('zidane.jpg', 'bus.jpg')]  # batch of images

# Inference
results = model(imgs)
results.print()  # or .show(), .save()

But works with detect.py file

p.s. I'll try to figure it out

I found the problem! It doesn't work with a custom model.

model = torch.hub.load('ultralytics/yolov5', 'custom', path_or_model='last.pt')

custom:

Standart

@Pro100rus32

@Pro100rus32 ok thanks for the info! I'll try to reproduce here with a custom model and see what's going wrong.

Yes I am able to reproduce. I'll try to get this fixed later today, adding a TODO.

(108/355) 1.mp4: 256x256 Done. (0.016s)

Speed: 4.0ms pre-process, 18.9ms inference, 1.0ms NMS per image at shape (1, 3, 256, 256)

16ms vs 23.9ms (Scanned over 100 frames+)

@Pro100rus32

@Pro100rus32 thanks for the numbers, but its not clear what you are testing. Could you provide code to reproduce, i.e. a minimum reproducible example?
https://docs.ultralytics.com/help/minimum_reproducible_example/

Thanks, I solved the problem :)

@Pro100rus32

@Pro100rus32 I just pushed a major improvement for all YOLOv5 PyTorch Hub models that enabled AMP autocasting by default. In my YOLOV5s on T4 GPU test, in comparison with my earlier result I now get 8.8ms inference, 1.2ms NMS vs 12.9ms and 1.1ms before.

FYI only Turing and Ampere generation cards can exploit autocasting/FP16 inference, Pascal generation cards generally can not (P4, P100, 1080ti).

I use 940M xD

@Pro100rus32

@Pro100rus32 hmm not familiar with a 940M, it's probably a laptop card.

Also remember if you are creating a video inference pipeline your speed will improve significantly if you run batched inference instead of running inference on images one at a time. i.e. wait for 30 frames and then run batched inference on them all simultaneously for faster results.

detect.py is mostly hard-coded for batch-size 1 dataloading, but the PyTorch Hub models allow for batched inference.

Yes, this is a laptop graphics card. Alas, in my task, frames are submitted one at a time. And I was already disappointed with the speed: 30ms for 1 frame 96x96 But I understand that this is due to the fact that I have a weak video card and a laptop in general.

PyTorch Hub custom model to CUDA device fix (ultralytics#2636)

pull updates from original repo (release v5) + reimplement UI (#21)

PyTorch Hub custom model to CUDA device fix (ultralytics#2636)

Can I use mps instead of cuda when loading the model or is this not possible?

Edit

So I tried experimenting. It still says YOLOv5 🚀 2024-4-28 Python-3.12.2 torch-2.3.0 CPU on the logs, but when I check model.parameters()).is_mps, it seems to be true, so I'm guessing it works?

This is my code:

# MODEL_PATH
from pathlib import Path
MODEL_PATH = Path("./best.pt").resolve() # A yolov5 model
YOLO_PATH = Path("./yolov5").resolve()

# Model Instance
# 1. Try to change the device by default. (Doesn't work)
print("Device", torch.get_default_device()) # 😪 doesn't work
torch.set_default_device(torch.device("mps")) # 😪 doesn't work
print("Device", torch.get_default_device()) # It changes "mps:0" for the default, but not the model it seems.

# 2. Load the model
model = torch.hub.load(YOLO_PATH.as_posix(), 'custom', MODEL_PATH.as_posix(), source='local', force_reload=True)  # local repo

# 3. Change to mps.
model.map_location = torch.device("mps") # 😪 doesn't work (Idk if it even has this API but this doesn't work lol, worth the shot).
model.to(torch.device("mps")) # ✅ This is what worked.

# Check if running on cuda/mps/cpu
print("is cuda??", next(model.parameters()).is_cuda) # returns False
print("is cpu??", next(model.parameters()).is_cpu) # returns False
print("is mps??", next(model.parameters()).is_mps) # returns True

Logs

Hey there! 👋 It looks like you've done some solid experimenting with using MPS on your YOLOv5 model. Based on your code and results, you're indeed on the right track!

Using .to(torch.device("mps")) is indeed the correct way to move your model to leverage MPS (Metal Performance Shaders) on supported hardware. Although setting the default device before loading the model didn't directly affect the model's device, applying .to() method directly on the model, as you discovered, is effective.

Your check with model.parameters()).is_mps yielding True confirms the model is utilizing MPS for computations, which is great news! This setup is a good approach if you're working with MPS compatible hardware and aiming for optimized performance.

To sum up, your experimentation has led you to the correct method for utilizing MPS with YOLOv5 models. Just remember to ensure your entire pipeline, including data loaders, are aligned with using MPS to maximize efficiency. Happy coding! 🚀

Navigation Menu

Repository navigation

How do I use CUDA with a custom model? #2630

❔Question

YOLOv5 Tutorials

11 remaining items

Edit

Logs

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

How do I use CUDA with a custom model? #2630

Description

❔Question

Activity

glenn-jocher commented on Mar 28, 2021

glenn-jocher commented on Mar 28, 2021

Pro100rus32 commented on Mar 28, 2021

glenn-jocher commented on Mar 28, 2021

Pro100rus32 commented on Mar 28, 2021

glenn-jocher commented on Mar 28, 2021

YOLOv5 Tutorials

Pro100rus32 commented on Mar 28, 2021

Pro100rus32 commented on Mar 28, 2021

Pro100rus32 commented on Mar 28, 2021

glenn-jocher commented on Mar 28, 2021

glenn-jocher commented on Mar 28, 2021

11 remaining items

Pro100rus32 commented on Mar 28, 2021

glenn-jocher commented on Mar 28, 2021

Pro100rus32 commented on Mar 28, 2021

glenn-jocher commented on Mar 28, 2021

Pro100rus32 commented on Mar 28, 2021

glenn-jocher commented on Mar 28, 2021

Pro100rus32 commented on Mar 28, 2021

Blankeos commented on Apr 29, 2024

Edit

Logs

glenn-jocher commented on Apr 29, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions