Repository navigation
perf(livecodebench): keep unit tests compressed until verification - #2806
Merged
Merged
Conversation
jubick1337
force-pushed
the
perf/lcb-compressed-unit-tests
branch
from
August 26, 2026 21:05
a241368 to
8c7169b
Compare
LiveCodeBench ships its hidden tests base64+zlib+pickle compressed. prepare decoded them and embedded the plaintext in every materialized row, which made the materialized-inputs file 12.1 GB against 4-6 MB for every other benchmark (p50 70 KB, p90 20 MB, max 189 MB per row). That file is read in full and held for the duration of the run, so it dominated eval-client footprint and slowed every tool that walks rollouts. Across a 22-checkpoint sweep it cost roughly 139 GB. Store the tests packed and decode them in the verifier, immediately before use, so the plaintext exists for one verification rather than for the whole run. Measured 12.1 GB -> 5.8 GB per task. Score-neutral by construction: the verifier decodes the same bytes back before validating, so the tests it runs are identical. Verified on the real v6 set -- all 1362 rows round-trip to non-empty tests with matching input/output counts (median 42 tests per problem). The verifier also still accepts the old plaintext shape, so artifacts prepared before this change continue to verify. Signed-off-by: Matvei Novikov <mnovikov@nvidia.com>
jubick1337
force-pushed
the
perf/lcb-compressed-unit-tests
branch
from
August 26, 2026 21:08
8c7169b to
adc6b6d
Compare
jubick1337
requested review from
bxyu-nvidia and
kbhardwaj-nvidia
and removed request for
bxyu-nvidia
August 26, 2026 21:19
github-actions
Bot
added
the
sla:review-overdue
bxyu-nvidia
approved these changes
Sep 10, 2026
jkyi-nvidia
pushed a commit
that referenced
this pull request
Sep 11, 2026
…2806) LiveCodeBench ships its hidden tests base64+zlib+pickle compressed. prepare decoded them and embedded the plaintext in every materialized row, which made the materialized-inputs file 12.1 GB against 4-6 MB for every other benchmark. That file is read in full and stored for the duration of the whole run. Store the tests packed and decode them in the verifier, immediately before use, so the plaintext exists for one verification rather than for the whole run. Measured 12.1 GB -> 5.8 GB per task. Signed-off-by: Matvei Novikov <mnovikov@nvidia.com> Signed-off-by: Junkeun Yi <jkyi@nvidia.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LiveCodeBench ships its hidden tests base64+zlib+pickle compressed. prepare decoded them and embedded the plaintext in every materialized row, which made the materialized-inputs file 12.1 GB against 4-6 MB for every other benchmark.
That file is read in full and stored for the duration of the whole run.
Store the tests packed and decode them in the verifier, immediately before use, so the plaintext exists for one verification rather than for the whole run. Measured 12.1 GB -> 5.8 GB per task.