Repository navigation
Crawl log search - #780
Merged
Merged
Crawl log search#780
Conversation
Checkpointing renames each log in place (crawl.log becomes crawl.log.cp00001-<timestamp>) and starts a new one, so earlier entries could only be viewed one file at a time. The paged viewer now offers an "include N checkpoint logs" link (all=y) that shows the log and its rotated generations in the current launch as one virtual file, oldest first, with each run of lines labelled with its file. Positions in the virtual file stay valid across checkpoints, since a rotated file keeps its place and the new log continues after it, and they are converted when toggling so the view stays where it was. Gzipped generations can't be read at arbitrary positions and are left out. LogSeries opens the files and fixes their lengths up front, so a rotation mid-request doesn't change what is read. Paging uses a new FilteredLineScanner over a LogSeries instead of FileUtils.pagedLines, which can only read a single file. Results are the same for every position the navigation links generate; a hand-edited pos in the middle of a line now starts at that line rather than the next one. Lines are capped at 256 KiB so a file with few or no newlines can't exhaust the heap. Also fixes in the crawl.log rendering: a rotated crawl.log gets the same status highlighting, status codes unknown to Jetty (e.g. 999) and numbers too long to be a status no longer fail the page, and the file name, status titles and navigation links are HTML escaped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The paged log viewer gains a search box taking a search-engine style query, passed as the q parameter: status:4xx host:example.com -type:image size>1MB duration>10s - Terms are ANDed; '-' negates; double quotes allow spaces. - 'field:value' is a case-insensitive substring match with '*' wildcards, 'field=value' is exact and 'field~regex' is a Java regex. Numeric fields take the same values with ':' or '=', and no regex; a comparison needs no operator, as in size>1MB. - Bare words and the 'line' field search the whole line in any log. crawl.log also supports status (404, 4xx, <0, 500..599, 404,410), size (>1MB, 100KB..10MB; units are powers of 1024), duration (>10s, 500..2000 in ms), depth (hops from the seed: 0, >5), hops (the hop path, e.g. hops:*E), url, host (includes subdomains), via and viahost (the URI it was discovered from, and its host), type and annot. - Bare words and line terms are highlighted in the results. Filtering keeps the viewer's stateless, byte-position paging and works across checkpoint logs: each request scans forward or backward from pos until it has enough matches or reaches the start or end, streaming a progress bar to the browser. A scan stops after 5 minutes, offering a link to continue from where it stopped, or as soon as the client disconnects. Regex matching is aborted at the same deadline to guard against catastrophic backtracking. The query is carried through all navigation links. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Enables autoescaping in Freemarker and adds some missing escaping where we generate HTML in Java methods.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds two things I've always really wanted in the crawl log viewer. Firstly, an option to view the current log file concatenated with rotated checkpoint log files. So no more having to dig in multiple files because a checkpoint just happened.
Second, search filtering. For any viewed file you can do a simple string search:
But for crawl logs it has a little query language that makes it easy to filter for all sorts of things:
status:4xxstatus<0status:500..599 host:example.com-status:2xx,3xxtype:pdf size>10MBduration>30sdepth>10 hops:*Etype:image -host:cdn.example.comurl~(/[^/]+)\1\1via:example.com/newsannot:botblockIf the log is really large you'll see a progress indicator (cancel by hitting the browser's stop button):
There's also a help popup for the query language: