Repository navigation
zstd: stream a small buffer when its in-memory decode exceeds the size limit - #1207
Conversation
…e limit Decoder.Reset decodes a source with Bytes and Len shorter than WithDecodeBuffersBelow in one go through DecodeAll, where WithDecoderMaxMemory caps the total decoded size. A Reader is documented with the streaming contract, where the option caps the window instead, so the same Reader followed one rule or the other depending on input length: a 1740-byte frame with no content size that expands past a 20 MiB limit failed over a bytes.Buffer and decoded over any other io.Reader. When the shortcut's DecodeAll reports ErrDecoderSizeExceeded, fall through to the normal streaming decode of the same reader, whose bytes the shortcut has not consumed. Memory stays bounded as documented: the in-memory attempt stops at the cap, its buffer is dropped, and streaming holds a window. Found by a differential fuzz harness while fuzzing klauspost#1204; the stock fuzzers share the threshold on both sides and cannot see it. Fixes klauspost#1206 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (2)
Included review availability: Your plan provides up to 8 included reviews per hour; 7 remain after this review. 📝 WalkthroughWalkthroughThe decoder now falls back from ChangesDecoder shortcut fallback
Estimated code review effort: 2 (Simple) | ~10 minutes Merge Risk: ⚪ Minimal · up to Small buffered zstd Reader inputs that exceed the in-memory decode limit now continue with bounded streaming decode, matching plain-reader behavior. The targeted regression coverage supports merge readiness with no remaining actionable risk. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Fixes #1206.
Decoder.Resetdecodes aBytes()/Len()source shorter thanWithDecodeBuffersBelowin one go throughDecodeAll, whereWithDecoderMaxMemorycaps the total decoded size. A Reader is documented with the streaming contract, where the option caps the window, so the same Reader followed one rule or the other depending on input length.Change: when the shortcut's
DecodeAllreportsErrDecoderSizeExceeded, fall through to the normal streaming decode of the same reader (the shortcut has not consumed its bytes). Memory stays bounded as documented: the in-memory attempt stops at the cap, its buffer is dropped rather than kept for reuse, and streaming holds a window.Test:
TestDecoderBufferShortcutFallsBackToStreamingencodes 4 MiB with a 64 KiB window (a few hundred compressed bytes), sets a 1 MiB limit, assertsDecodeAllstill fails as documented, and asserts a Reader over abytes.Bufferand over a plainio.Readerboth produce the full output. It fails on master withdecompressed size exceeds configured limiton the buffer case. The original 1740-byte fuzz reproducer from #1206 also passes the differential harness with this change.🤖 Generated with Claude Code
Summary by CodeRabbit