Heritrix status code report #747

Current version of Heritrix enables to generate an HTTP response code report from the crawllog: responsecode-report.txt displaying the number of URLs downloaded per status code. This file is created by the StatisticsTracker bean and is written at the end of the crawl.

Sample output:

[#urls] [rescode]
1306 200
31 404
1 1

To get a better view on possible issues (for instance DNS lookup or HTTP connection failures), we propose to generate a Heritrix status report from the crawllog: heritrixstatuscode-report.txt which will group and count URLs based on H3 status codes. The different status codes are listed here: https://heritrix.readthedocs.io/en/latest/glossary.html#status-codes.

Sample output :

[#urls] [rescode]
12698 1
123 -1
126 -2
10 -5000