Repository navigation
Log warning when URI precedence exceeds maximum 127 - #721
Merged
ato merged 1 commit intoApr 6, 2026
Merged
Conversation
BdbMultipleWorkQueues.calculateInsertKey() silently clips precedence values to 127 for queue ordering. This can cause unexpected behavior for custom cost assignment policies that return values above 127. Log a warning when clipping occurs and fix the misleading comment in CrawlURI.holderCost that said "should not exceed 255" (the actual limit is 127, not 255). Fixes internetarchive#502
simons-hub
marked this pull request as ready for review
April 1, 2026 06:02
Collaborator
|
Thanks. Released in 3.14.1. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
BdbMultipleWorkQueues.calculateInsertKey()silently clips URI precedence values to 127 for queue ordering. Custom cost assignment policies that return values above 127 get their precedence silently truncated with no indication anything is wrong.This change:
CrawlURI.holderCostthat said "should not exceed 255" — the actual limit is 127 (7-bit signed byte, not 8-bit unsigned)Details
The precedence byte in the BDB insert key is packed as a signed byte, so the effective maximum is 127 (
Math.min(curi.getPrecedence(), 127)). The original author noted this in the method javadoc: "Dangers here are: (1) priorities or precedences over 2^7 (signed byte comparison)" — but no runtime warning was emitted.A user with a custom
CostAssignmentPolicyreturning values up to 10000 (as described in #502) would have all precedence values silently flattened to 127, making their policy ineffective with no feedback.Changes
BdbMultipleWorkQueues.javaWARNINGwhenprecedence > 127before clippingCrawlURI.javaTesting
Full test suite passes (96 tests, 0 failures). Specifically:
BdbMultipleWorkQueuesTest— 2/2 (directly exercisescalculateInsertKeyacross all ordinal ranges)BdbFrontierTest— passFixes #502