Repository navigation
YOLOv5 Tutorial Colab COCO test-dev2017 issue #2649
Description
❔Question
I'm using the Colab YOLOv5 Tutorial jupyter notebook, i'm in the section COCO test-dev2017 and i'm getting this output when i run the following command !python test.py --weights yolov5s.pt --data coco.yaml --task test:
Namespace(augment=False, batch_size=32, conf_thres=0.001, data='./data/coco.yaml', device='', exist_ok=False, img_size=640, iou_thres=0.6, name='exp', project='runs/test', save_conf=False, save_hybrid=False, save_json=True, save_txt=False, single_cls=False, task='test', verbose=False, weights=['yolov5s.pt'])
YOLOv5 🚀 v4.0-169-g1e8ab3f torch 1.8.0+cu101 CUDA:0 (Tesla T4, 15109.75MB)
Fusing layers...
Model Summary: 224 layers, 7266973 parameters, 0 gradients, 17.0 GFLOPS
test: Scanning '../coco/test-dev2017' images and labels... 0 found, 20288 missing, 0 empty, 0 corrupted: 100% 20288/20288 [00:17<00:00, 1142.06it/s]
test: WARNING: No labels found in ../coco/test-dev2017.cache. See https://github.com/ultralytics/yolov5/wiki/Train-Custom-Data
test: New cache created: ../coco/test-dev2017.cache
Class Images Labels P R mAP@.5 mAP@.5:.95: 100% 634/634 [03:38<00:00, 2.90it/s]
all 20288 0 0 0 0 0
Speed: 3.0/3.0/6.0 ms inference/NMS/total per 640x640 image at batch-size 32
/usr/local/lib/python3.7/dist-packages/seaborn/matrix.py:194: RuntimeWarning: All-NaN slice encountered
vmin = np.nanmin(calc_data)
/usr/local/lib/python3.7/dist-packages/seaborn/matrix.py:199: RuntimeWarning: All-NaN slice encountered
vmax = np.nanmax(calc_data)
Evaluating pycocotools mAP... saving runs/test/exp2/yolov5s_predictions.json...
loading annotations into memory...
Done (t=0.96s)
creating index...
index created!
Loading and preparing results...
pycocotools unable to run: Results do not correspond to current coco set
Results saved to runs/test/exp
I think the problem is in this line:
test: Scanning '../coco/test-dev2017' images and labels... 0 found, 20288 missing, 0 empty, 0 corrupted: 100% 20288/20288
because the ../coco/test-dev2017 does not exist in the coco folder that gets downloaded by following the jupyter notebook line by line.
I've noticed that the test-dev2017 lables aren't present in the coco folder mentioned before, maybe that's the reason why i'm getting this other error too:
pycocotools unable to run: Results do not correspond to current coco set
Probably since it was a competition the test-dev2017 lables are not publicly available and so the results must be submitted to the evaluation server.
If it's how i said can you explain me how to submit them to the correct evaluation server since i can't figure it out myself?
In addition i've found another error in the same section:
(Original)
torch.hub.download_url_to_file('https://github.com/ultralytics/yolov5/releases/download/v1.0/coco2017labels.zip', 'tmp.zip')
!unzip -q tmp.zip -d ../ && rm tmp.zip # unzip labels
!f="test2017.zip" && curl http://images.cocodataset.org/zips/$f -o $f && unzip -q $f && rm $f # 7GB, 41k images
%mv ./test2017 ./coco/images && mv ./coco ../ # move images to /coco and move /coco next to /yolov5
(Resolved)
torch.hub.download_url_to_file('https://github.com/ultralytics/yolov5/releases/download/v1.0/coco2017labels.zip', 'tmp.zip')
!unzip -q tmp.zip -d ../ && rm tmp.zip # unzip labels
!f="test2017.zip" && curl http://images.cocodataset.org/zips/$f -o $f && unzip -q $f && rm $f # 7GB, 41k images
%mv ./test2017 ../coco/images # move images to /coco
Additional context
I'm using free version of Google Colab for funning the jupyter
Activity
github-actions commentedon Mar 29, 2021
👋 Hello @NicoloPosta, thank you for your interest in 🚀 YOLOv5! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.
If this is a 🐛 Bug Report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.
If this is a custom training ❓ Question, please provide as much information as possible, including dataset images, training logs, screenshots, and a public link to online W&B logging if available.
For business inquiries or professional support requests please visit https://www.ultralytics.com or email Glenn Jocher at glenn.jocher@ultralytics.com.
Requirements
Python 3.8 or later with all requirements.txt dependencies installed, including
torch>=1.7. To install run:Environments
YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):
Status
If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training (train.py), testing (test.py), inference (detect.py) and export (export.py) on MacOS, Windows, and Ubuntu every 24 hours and on every commit.
glenn-jocher commentedon Mar 29, 2021
@NicoloPosta for COCO test-dev2017 results:
glenn-jocher commentedon Mar 29, 2021
@NicoloPosta I see the bug in cell you mentioned, I think your fix looks good, I will update the notebook with this, thank you for the recommendation!
NicoloPosta commentedon Mar 29, 2021
@glenn-jocher Thanks for your replies, i have another problem.
On the submission server (the link you gave me), i go on the Participate tab, i select test-dev2019 (bbox) and click the Submit button, then after choosing the json file to submit the server gives me this error
Invalid file type (application/json).Am i submitting the json file to the wrong section/server or am i fillowing the wrong procedure to evaluate the results? Sorry for bothering you.
Created using Colaboratory (#2649)
glenn-jocher commentedon Mar 29, 2021
@NicoloPosta I've updated Colab notebook now per your feedback in 1b100cd
You may need to update the json file name to conform to the conventions they require, but I'm not sure. What site are you going to and what steps are you doing exactly?
NicoloPosta commentedon Mar 29, 2021
@glenn-jocher I've seen the commint, it's not a major bug but i hope my contribution helped to make the Tutorial better for other people.
The site i'm going to is: https://competitions.codalab.org/competitions/20794
yolov5s_predictions.jsonfile that i saved on my computer from the test-dev2017 results obtained from the Colab Tutorial.Invalid file type (application/json).I even tried renaming the json file as you suggested but i can't find any standard.
I tryed with
test-dev2017_yolov5s_predictions.jsonbut i get the same error message as before.Thanks for the patience and the help.
glenn-jocher commentedon Mar 29, 2021
@NicoloPosta I see. If I remember I think you might need to zip the JSON file before upload? If this doesn't work let me know and I'll add a TODO to investigate this week. Thanks!
glenn-jocher commentedon Mar 29, 2021
@NicoloPosta your JSON filename might need to follow this format (not sure, but this is what we use), and then I confirm the upload must be a zip file:
NicoloPosta commentedon Mar 29, 2021
@glenn-jocher It worked!
The problem was the required zip extension, thanks a lot for the help, i admire your work on the Yolo models and the support you gave me.
In my opinion if your team can put a few lines on how to submit the results and the extension they need to be in the Colab Tutorial it might be helpfull for others in the future, a part from that thanks again for the help!
glenn-jocher commentedon Mar 29, 2021
That's a good idea! Should we update the text like this?
COCO test-dev2017
Download COCO test2017 dataset (7GB - 40,000 images), to test model accuracy on test-dev set (20,000 images). Results are saved to a
*.jsonfile which can be submitted to the evaluation server at https://competitions.codalab.org/competitions/20794.COCO test-dev2017
Download COCO test2017 dataset (7GB - 40,000 images), to test model accuracy on test-dev set (20,000 images, no labels). Results are saved to a
*.jsonfile which should be zipped and submitted to the evaluation server at https://competitions.codalab.org/competitions/20794.NicoloPosta commentedon Mar 29, 2021
The second version seems more clear on how to commit it, i would go with that.
glenn-jocher commentedon Mar 30, 2021
@NicoloPosta updated in 1b475c1. Thank you for your contributions!
Created using Colaboratory (ultralytics#2649)
pull updates from original repo (release v5) + reimplement UI (#21)
Created using Colaboratory (ultralytics#2649)
Created using Colaboratory (ultralytics#2649)
Created using Colaboratory (ultralytics#2649)