[ushnish-de]

Serve filings from the local SEC corpus when edgar_search found them

edgar_search queries a local full-text index built from a downloaded SEC
corpus, but reads of its results did not come from that corpus. To turn a filing
URL into a file on disk, _lookup_dump rebuilt a path out of _filings_cache,
which only sec_filing_search populates. After an edgar_search hit there was
nothing to rebuild from, so the read went to sec.gov.

When sec_filing_search had run earlier in the same rollout, the rebuilt path
was worse than useless for exhibits — the separate documents filed alongside a
filing, such as an EX-99.2 press release. A rebuilt path can only ever name a
filing's main document, so the read returned that instead of the exhibit, and
reported success.

Changes

• edgar_search records each hit's source_path from the index, keyed by
  canonical URL, and filing reads consult it before falling back to the rebuilt
  path. The index holds a row per document, so this is the only route that can
  name an exhibit.
• Filing reads are counted by source (cache, sec-corpus, live). The totals
  are logged on the first read, then at most once every 30 minutes, and on
  shutdown, at warning level so they appear in job logs with no logging
  configuration.

Metrics

The fix changes how a filing URL is turned into a file path. There are two ways
to do that, and the table compares them on the same documents:

• the exact path the index recorded for the document the search returned
• a path rebuilt from filing metadata, which can only name a filing's main
  document

8 queries against a 3-year, 5-ticker demo index returned 138 distinct documents
(62 exhibits, 76 main documents). Each was read both ways, and the bytes that
came back were compared against the document the search had actually returned:

how the file path is chosen	right document	wrong document	no file found
exact path from the index — after this change	138 (100%)	0	0
rebuilt from filing metadata — before this change	64 (46%)	56 (41%)	18 (13%)

All 56 wrong reads were requests for an exhibit, where we previously returned the filing's primary document instead of the exhibit itself. None of the 76 primary-document reads is wrong, which is the shape the mechanism predicts: a rebuilt path can only ever name primary-document.html, so it is right for a filing's own document and wrong for every exhibit in that filing.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[cmunley1]

/ok to test dce9d45

[cmunley1]

/claude review

[claude]

SHIP — LGTM, no reliability concerns.

Reviewed all 4 changed files (finance_sec_search serving path + tests). This adds a search-driven dump-path index so edgar_search hits can be read from the local SEC corpus (including exhibits, which the convention-based path could never name) instead of hitting sec.gov, plus a per-source read counter.

Why this is low-risk:

• No verifier/scoring surface touched. This only changes where filing text is sourced. Dump, cache, and live all return the same document, so a bug here degrades to a live fetch or the existing convention path — never a wrong score.
• Correct document mapping. _dump_paths is keyed on canonical_url_key(url) and populated from the index row matching that exact URL key, so a recorded path can only ever serve the document its URL requested. Missing dump files fall through to live (covered by test_missing_dump_file_falls_through_to_live).
• Path traversal guarded. _lookup_dump rejects absolute paths and .. components before reading under sec_dump_path.
• Async-safe. SQLite reads go through asyncio.to_thread / run_in_executor; no httpx, no ray.get, no missing awaits. The _dump_paths dict is only mutated at await boundaries in the single-threaded loop; concurrent edgar_search calls at worst redo identical resolves.
• Backward compatible. supports_dump_paths degrades cleanly on indexes built before the source_path/canonical_url_key columns existed (test_dump_paths_are_unavailable_without_the_columns). New imports are stdlib only.
• Well tested. New tests exercise precedence, exhibit serving, live fallback, and the read-source counter.

One soft coupling worth noting (not blocking): canonical_url_key must stay byte-identical to the index builder's key normalization (unpadded CIK / dashless accession / lowercased filename). If they ever diverge, lookups silently miss and fall back to convention/live — degradation, not corruption. The code comment already flags this; keeping the builder and this function in sync is the maintenance obligation.

[cmunley1]

If a recorded document path exists but can’t be read, should we return None here instead of falling back? For exhibit URLs, the fallback below reads primary-document.html, so a missing exhibit can silently return the wrong document.