[bnfleb]

See #699
Add sizeOnDisk in sizeTotalsReport tag in the job status api.

  <sizeTotalsReport>
    <dupByHash>0</dupByHash>
    <dupByHashCount>0</dupByHashCount>
    <novel>0</novel>
    <novelCount>0</novelCount>
    <notModified>0</notModified>
    <notModifiedCount>0</notModifiedCount>
    <total>0</total>
    <totalCount>0</totalCount>
    <sizeOnDisk>6011832</sizeOnDisk>
  </sizeTotalsReport>

[ato]

I've merged this but I have some lingering concerns about this one:

• This obviously won't work if you've reconfigured the WARC output directory.
• Will enumerating all the WARCs cause performance problems for a large crawl with many files or slower storage? (e.g. remote filesystem mounted over NFS or something)
• Could WriterPoolProcessor.getTotalBytesWritten() produce the same result without having to read the filesystem? Or are we intentionally adding this because the existing WARCWriter statistics are inaccurate?

[bnfleb]

• If there is a parameter to configure the WARC output directory, we can cahnge the code to use it (but I didn't find it).
• At the BnF, we have been using this feature for three years on a large scale whitout any performance issue.
• In principle, WriterPoolProcessor.getTotalBytesWritten() corresponds to the number of uncompressed bytes written. sizeOnDisk is the total size of the compressed warcs.