Download distribution zip (or tar.gz)

Full Changelog | Javadoc | Maven Central

Security

• Web UI: Added protection against cross-site request forgery (CSRF). POST, PUT and DELETE requests that a
  browser sends on behalf of another site are now rejected, based on the Sec-Fetch-Site header. To allow requests
  from another site, list its origin in the heritrix.trustedOrigins system property, e.g.
  -Dheritrix.trustedOrigins=https://dashboard.example.org.
• Web UI: Template output is now HTML-escaped by default, fixing cases where values were included in pages without
  escaping. #780

New features

• Paged log viewer: #780
  • Added a search box for filtering log lines, using a small query language, e.g.
    status:4xx host:example.com -type:image size>1MB. Any log can be searched for text; crawl.log also supports
    fields such as status, size, duration, depth, hops, url, host, via, type and annot. Matches
    are highlighted, and a progress bar is shown while searching large logs.
  • Added an "include checkpoint logs" link to view a log together with the generations of it rotated at each
    checkpoint, as if they were one file.
• BotBlockDetector: Added detection of AWS WAF, Kasada, Nginx Lua Anti-DDoS and PerimeterX. BotBlockDetector is
  now enabled in the default job profiles. #778

Bug fixes

• FetchHTTP: The payload digest of chunked responses is now calculated after transfer decoding, so it no longer
  includes the chunk framing. #770
• ExtractorJson: URIs nested inside JSON arrays are now extracted. #771
• ExtractorJS: JavaScript Unicode code-point escapes (\u{...}) in extracted URLs are now decoded. #772
• AMQPUrlReceiver: Improved parsing of URL messages that contain null values. #767
• Web UI: The copy job and delete job dialogs now work in browsers without support for invoker commands, such as
  Firefox ESR. #773
• Paged log viewer: Rotated crawl.log files now get status code highlighting, and unusual status codes no longer
  cause an error. #780

Changes

• Startup: heritrix_dmesg.log is now only written when Heritrix is started in the background by bin/heritrix,
  making it possible to run with a read-only $HERITRIX_HOME. The path can be set with the heritrix.dmesg system
  property. #779

Dependency upgrades

• amqp-client: 5.35.0 → 5.36.0
• codemirror__commands: 6.10.4 → 6.11.0
• codemirror__state: 6.7.1 → 6.7.4
• codemirror__view: 6.43.7 → 6.43.11
• groovy-bom: 5.1.1 → 6.0.0
• jackson-bom: 2.22.2 → 2.22.3
• jetty (jetty-bom, jetty-ee10-bom): 12.0.38 → 12.0.39
• lz4-java: 1.11.2 → 1.12.0
• marijn__find-cluster-break: 1.0.3 → 1.0.4
• slf4j-bom: 2.0.18 → 2.0.20
• webjars-locator-lite: 1.1.4 → 1.1.5

Download distribution zip (or tar.gz)

Full Changelog | Javadoc | Maven Central | Docker Hub

Bug fixes

• FetchHTTP: SSL handshake failures now use the dedicated S_SSL_ERROR (-9) fetch status instead of S_CONNECT_FAILED, reducing unlikely retries. #765
• SourceTagsReport: Extended reports now include Heritrix status codes for failed, retried, and disregarded URIs. #766

Dependency upgrades

• freemarker: 2.3.34 → 2.3.35
• groovy-bom: 5.1.0 → 5.1.1

Download distribution zip (or tar.gz)

Full Changelog | Javadoc | Maven Central | Docker Hub

New features

• SourceTagsReport: Added an includeResCode property to include HTTP response codes in the report. #758
• ResponseCodeReport: Heritrix error codes (negative response codes) are now included in the report. #757
• BotBlockDetector: Added a new processor that detects responses from common bot-blocking services (Akamai, Anubis, Cloudflare, DataDome, and Incapsula) and annotates affected URIs with botblock:<service>. #760

Bug fixes

• BrowserProcessor and LocalWebDriverBiDi: #750
  • Automatically restart the browser if it crashes during a visit.
  • Added null checks and improved error handling when closing tabs and continuing intercepted requests.
  • Fixed UnsupportedOperationException when mapping Void results in BiDiJson.
  • Serialized WebSocket sends to prevent IllegalStateException from concurrent sendText calls.
  • Added timeout and cleanup logic when the browser fails to start.
  • Ensured close() reliably terminates the browser process and fails pending commands when the connection is lost.
• MetadataRecordBuilder: Added a null check for curi.getVia() to prevent NullPointerException during WARC metadata generation. #751
• Reports: Report generation now returns a clear message instead of failing when BDB is not running. #759

Dependency upgrades

• amqp-client: 5.33.0 → 5.35.0
• codemirror__autocomplete: 6.20.1 → 6.20.3
• codemirror__commands: 6.10.2 → 6.10.4
• codemirror__language: 6.12.2 → 6.12.4
• codemirror__lint: 6.9.5 → 6.9.7
• codemirror__state: 6.5.4 → 6.7.1
• codemirror__view: 6.39.16 → 6.43.7
• commons-codec: 1.22.0 → 1.22.1
• crelt: 1.0.6 → 1.0.7
• groovy-bom: 5.0.6 → 5.1.0
• jackson-bom: 2.22.0 → 2.22.2
• java-socks-proxy-server: 4.1.2 → 4.2.0
• jetty (jetty-bom, jetty-ee10-bom): 12.0.36 → 12.0.38
• jsch: 2.28.3 → 2.28.7
• junit-jupiter: 6.1.1 → 6.1.3
• lezer__common: 1.5.1 → 1.5.2
• lezer__lr: 1.4.8 → 1.4.10
• lz4-java: 1.11.0 → 1.11.2
• marijn__find-cluster-break: 1.0.2 → 1.0.3
• pdfbox: 3.0.7 → 3.0.8
• spring (spring-beans, spring-context, spring-core, spring-expression): 7.0.8 → 7.0.9
• webarchive-commons: 3.0.4 → 3.0.5
• webjars-locator-lite: 1.1.3 → 1.1.4

Download distribution zip (or tar.gz)

Full Changelog | Javadoc | Maven Central

Distribution packages moved to GitHub releases

Due to recently introduced size limits, the Heritrix distribution packages (-dist.zip and -dist.tar.gz) can no longer be published to Maven Central. Instead, they will be made available on the GitHub release page.

The individual module binary, Javadoc, and source JARs will continue to be published to Maven Central.

New features

• PaginationBehavior: Added a new behavior for BrowserProcessor that repeatedly clicks the "next-page" link and extracts links, enabling crawling of client-side paginated sites. #736

Bug fixes

• BrowserProcessor: #742
  • Improved subresource request failure handling.
  • Implemented recording truncation based on length and timeout limits.
  • Fixed content digest calculation for subresources.
  • Ensured the User-Agent header is set correctly for browser requests.
  • Handle WebDriver BiDi Cookie.sameSite value default.
  • Fixed a bug in where POST requests were recorded with doubled headers and trailing nulls. #737
• Error logging: The exception is now logged when creating a new job fails.
• Groovy default profile: Removed a stray closing brace }.

Dependency upgrades

• amqp-client: 5.30.0 → 5.33.0
• central-publishing-maven-plugin: 0.10.0 → 0.11.0
• dnsjava: 3.6.4 → 3.6.5
• groovy-bom: 5.0.5 → 5.0.6
• jackson-bom: 2.21.3 → 2.22.0
• jaxb-runtime: 4.0.7 → 4.0.9
• jetty (jetty-bom, jetty-ee10-bom): 12.0.34 → 12.0.36
• jsch: 2.28.2 → 2.28.3
• junit-jupiter: 6.0.3 → 6.1.1
• kafka-clients: 4.2.0 → 4.3.1
• maven-surefire-plugin: 3.5.5 → 3.5.6
• slf4j-bom: 2.0.17 → 2.0.18
• spring (spring-beans, spring-context, spring-core, spring-expression): 7.0.7 → 7.0.8

Download distribution zip (or tar.gz)

Full Changelog | Javadoc | Maven Central

New features

• Profile selector: You can now select a profile from a dropdown list when creating a new job.
  #725
• Groovy config editor: The job config editor now supports Groovy job configs. This includes auto-completion
  with documentation hints for bean and property names (but not for arbitrary Groovy code). Accepting completion of
  bean class names inserts imports statements if needed. #728

Bug fixes

• Groovy profiles: Fixed discovery of job directories containing profile-crawler-beans.groovy
• ExtractorHTML: Capped dynamic attribute-name scanning to avoid pathological regex behavior on pages with extremely
  large attribute values. #733

Changes

• Removed obsolete vendored JavaScript libraries from the UI #732
  • Replaced Foundation JS with native HTML dialogs and simple vanilla JS click handlers.
  • Removed jQuery and Zepto as they were only used by Foundation JS.
  • Removed Internet Explorer compatibility shims (modernizr and IE conditional comments).

Download distribution zip (or tar.gz)

Full Changelog | Javadoc | Maven Central

Bug fixes

• FetchHTTP2: Fixed an IllegalArgumentException when the host contains underscores. #718
• FetchHTTP2: Removed HttpClient default authentication protocol handlers to avoid failures on 401/407 responses with large response bodies or missing *-Authenticate headers. #719
• Crawl job XML parsing: Enabled XXE protection when parsing crawl job XML. #713
• URI precedence warnings: Heritrix now logs a warning when URI precedence exceeds the maximum supported value of 127 and is clipped. #721

Dependency upgrades

• amqp-client: 5.28.0 → 5.29.0
• codemirror__autocomplete: 6.20.0 → 6.20.1
• codemirror__commands: 6.10.1 → 6.10.2
• codemirror__language: 6.12.1 → 6.12.2
• codemirror__lint: 6.9.2 → 6.9.5
• codemirror__view: 6.39.11 → 6.39.16
• commons-net: 3.12.0 → 3.13.0
• groovy-bom: 5.0.4 → 5.0.5
• jackson-bom: 2.21.0 → 2.21.2
• jakarta.xml.bind-api: 4.0.4 → 4.0.5
• jaxb-runtime: 4.0.6 → 4.0.7
• jetty (jetty-bom, jetty-ee10-bom): 12.0.32 → 12.0.34
• jsch: 2.27.7 → 2.28.0
• junit-jupiter: 6.0.2 → 6.0.3
• kafka-clients: 4.1.1 → 4.2.0
• lezer__common: 1.5.0 → 1.5.1
• lezer__lr: 1.4.7 → 1.4.8
• lz4-java: 1.10.3 → 1.10.4
• pdfbox: 3.0.6 → 3.0.7
• spring (spring-beans, spring-context, spring-core, spring-expression): 7.0.3 → 7.0.6

Download distribution zip (or tar.gz)

Full Changelog | Javadoc | Maven Central

New features

• JSON API responses: You can now request JSON responses from the REST API with a Accept: application/json request header. #703
• Job deletion: You can now delete jobs via both the UI and the API. #707
• ExtractorYoutubeDL: Added a new skipSuperResolution option to skip upscaled YouTube videos. #709
• FetchHTTP2: Added SOCKS5 proxy support. #710

Bug fixes

• Improved usability of the engine page when many jobs are active by making the "Exit Java" joblist scrollable. #704

Dependency upgrades

• amqp-client: 5.27.1 → 5.28.0
• codemirror__autocomplete: 6.18.6 → 6.20.0
• codemirror__commands: 6.8.1 → 6.10.1
• codemirror__language: 6.11.3 → 6.12.1
• codemirror__lint: 6.8.5 → 6.9.2
• codemirror__state: 6.5.2 → 6.5.4
• codemirror__view: 6.38.1 → 6.39.11
• commons-codec: 1.20.0 → 1.21.0
• crawler-commons: 1.5 → 1.6
• dnsjava: 3.6.3 → 3.6.4
• groovy-bom: 5.0.2 → 5.0.4
• jackson-bom: 2.20.1 → 2.21.0
• jetty (jetty-bom, jetty-ee10-bom): 12.0.30 → 12.0.32
• junit-jupiter: 6.0.1 → 6.0.2
• lezer__common: 1.2.3 → 1.5.0
• lezer__highlight: 1.2.1 → 1.2.3
• lezer__lr: 1.4.2 → 1.4.7
• lz4-java: 1.10.1 → 1.10.3
• spring (spring-beans, spring-context, spring-core, spring-expression): 7.0.1 → 7.0.3
• style-mod: 4.1.2 → 4.1.3
• webarchive-commons: 3.0.2 → 3.0.3
• webjars-locator-lite: 1.1.2 → 1.1.3

Download distribution zip (or tar.gz)

Full Changelog | Javadoc | Maven Central

New features

• Config editor: IDE-style completions for bean names and Spring XML (powered by the new bean docs generator). #684
• Job status API: The sizeTotalsReport now includes a sizeOnDisk value totaling the size of the files in latest/warcs. #700
• ExtractorJson: New extractor that extracts URI strings from JSON documents. #701

Bug fixes

• AbstractCookieStore: Fixed cookies with leading dot (.example.com) being ignored #691
• ExtractorHTML: Fixed attribute values longer than 2048 characters causing extraction of truncated strings. #697
• ClientFTP: Fixed MalformedServerReplyException when FTP sends a response with only an error code and no message. #694
• BdbMultipleWorkQueues: Added null checks, type validation, and warning logs in BdbMultipleWorkQueues.delete() to improve frontier stability in the case of corrupted or partially persisted CrawlURIs. #693
• BeanDocProcessor: Fixed compiler IllegalArgumentException when IntelliJ runs the annotation processor with a ProcessingEnvironment wrapper.

Dependency upgrades

• amqp-client: 5.27.0 → 5.27.1
• commons-cli: 1.10.0 → 1.11.0
• commons-codec: 1.19.0 → 1.20.0
• commons-io: 2.20.0 → 2.21.0
• jackson: 2.20.0 → 2.20.1
• jetty: 12.0.29 → 12.0.30
• jsch: 2.27.4 → 2.27.7
• junit-jupiter: 6.0.0 → 6.0.1
• kafka-clients: 4.1.0 → 4.1.1
• lz4-java: 1.8.0 → 1.10.1
• spring-framework: 6.2.12 → 7.0.1
• webarchive-commons: 3.0.1 → 3.0.2

Download distribution zip (or tar.gz)

Full Changelog | Javadoc | Maven Central

New features

• ConfigurableExtractorJS: Regex rules to skip extracting <script> tags when their attributes match. #672

Bug fixes

• Docs: Switch bean docs generation to an annotation processor, fixing the bean reference broken by Java language changes. #683
• StatisticsTracker: Don’t restore crawlEndTime when resuming from a checkpoint. #669
• ExtractorJS: Fix overriding the strict setting in sheets. #670
• Berkeley DB: Handle more shutdown interrupts gracefully. #671

Dependency upgrades

• amqp-client: 5.26.0 → 5.27.0
• groovy: 4.0.28 → 5.0.2
• jaxb-runtime: 4.0.5 → 4.0.6
• jetty: 12.0.27 → 12.0.29
• jsch: 2.27.3 → 2.27.4
• junit-jupiter: 5.13.4 → 6.0.0
• kafka-clients: 3.9.1 → 4.1.0
• pdfbox: 3.0.5 → 3.0.6
• rethinkdb-driver: 2.3.3 → 2.4.4
• spring: 6.2.11 → 6.2.12
• webarchive-commons: 3.0.0 → 3.0.1
• webjars-locator-lite: 1.1.0 → 1.1.2

Download distribution zip (or tar.gz)

Full Changelog | Javadoc | Maven Central

New features

• KnowledgableExtractorJS now extends ConfigurableExtractorJS for its additional options. #668

Bug fixes

• Invalid characters are now stripped from the XML REST API output. Log file truncation after an unclean shutdown can sometimes introduce such characters. #667

Dependency upgrades

• codemirror@language: 6.11.2 → 6.11.3
• jakarta.xml.bind-api: 4.0.2 → 4.0.4
• jetty: 12.0.25 → 12.0.27
• jsch: 2.27.2 → 2.27.3
• gson: 2.13.1 → 2.13.2
• spring: 6.2.10 → 6.2.11