Skip to content

Extend SourceTags report with status code - #758

Merged
ato merged 1 commit into
internetarchive:masterfrom
bnfleb:issue748
Aug 4, 2026
Merged

ato merged 1 commit into
internetarchive:masterfrom
bnfleb:issue748

Conversation

@bnfleb

@bnfleb bnfleb commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Related to issue #748

To use the extended version of SourceTags, which lists for each seed and each host the number of urls by status code, you have to enable the parameter includeResCode in the bean SourceTagsReport.

<bean id="statisticsTracker" class="org.archive.crawler.reporting.StatisticsTracker" autowire="byName">
        <property name="intervalSeconds" value="20" />
        <property name="reports">
          <list>
                 <bean id="crawlSummaryReport" class="org.archive.crawler.reporting.CrawlSummaryReport" />
               <bean id="seedsReport" class="org.archive.crawler.reporting.SeedsReport" />
               <bean id="hostsReport" class="org.archive.crawler.reporting.HostsReport">
                   <property name="maxSortSize" value="-1" />
                   <property name="suppressEmptyHosts" value="false" />
               </bean>
            <bean id="sourceTagsReport" class="org.archive.crawler.reporting.SourceTagsReport" >
              <property name="includeResCode" value="true" />
            </bean>
            <bean id="mimetypesReport" class="org.archive.crawler.reporting.MimetypesReport" />
               <bean id="responseCodeReport" class="org.archive.crawler.reporting.ResponseCodeReport" />
               <bean id="processorsReport" class="org.archive.crawler.reporting.ProcessorsReport" />
               <bean id="frontierSummaryReport" class="org.archive.crawler.reporting.FrontierSummaryReport" />
               <bean id="frontierNonemptyReport" class="org.archive.crawler.reporting.FrontierNonemptyReport" />
               <bean id="toeThreadsReport" class="org.archive.crawler.reporting.ToeThreadsReport" />
          </list>
        </property>
    </bean>

The report is sorted, for each seed, by host then by number of urls.

[source] [host] [rescode] [#urls]
https://presses.enssib.fr/sites/presses.enssib.fr/files/css/css_NN1sj6420pSCVqQ6X632iDwJGg7mMdkYYyJTA3EXJg0.css dns: 1 1
https://presses.enssib.fr/sites/presses.enssib.fr/files/css/css_NN1sj6420pSCVqQ6X632iDwJGg7mMdkYYyJTA3EXJg0.css presses.enssib.fr 200 9
https://bloomevents.fr/ bloomevents.fr 200 9
https://bloomevents.fr/ dns: 1 3
https://bloomevents.fr/ fonts.googleapis.com 200 3
https://bloomevents.fr/ fonts.googleapis.com 404 1
https://bloomevents.fr/ fonts.gstatic.com 200 8
https://bloomevents.fr/ fonts.gstatic.com 404 1
https://fr.wix.com/ assets.abovedomains.com 200 1
https://fr.wix.com/ assets.abovedomains.com 404 1
https://fr.wix.com/ axios-http.com 301 3
https://fr.wix.com/ axios.rest 200 1
https://fr.wix.com/ axios.rest 404 1

@ato
ato merged commit 26e8891 into internetarchive:master Aug 4, 2026
3 checks passed
@ato ato linked an issue Aug 5, 2026 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

New or extension of current SourceTags report

2 participants