Expose GZIP/DEFLATE decompressor state (sliding window & bit-offsets) to enable zran / ratarmount style index generation

[unxed]

I am developing a pure-Go advanced TAR library (github.com/unxed/tar) that provides $O(1)$ random access to compressed .tar.gz archives. To achieve this, we aim for binary compatibility with industry-standard indexes used by Python's ratarmount and gztool, which are based on Mark Adler's zran.c (GZIDX format).

Currently, we can successfully read and seek using existing GZIP indexes in pure Go. We achieve this by loading the 32KB dictionary checkpoint from the index into flate.NewReaderDict and handling bit-alignment via a custom bit-shifting reader wrapper.

However, we are unable to generate these indexes on-the-fly during sequential decompression. This limitation exists because the internal state of the flate decompressor is encapsulated, making it impossible to capture the necessary checkpoint metadata.

What is needed to generate a zran-compatible index:

During sequential decompression, we need to periodically (e.g., every 1MB or 4MB of uncompressed data) capture a checkpoint containing:

1. Uncompressed offset (in bytes).
2. Compressed offset (the exact byte offset in the input stream).
3. Bit offset (the remaining 0–7 bits inside the current byte where the next DEFLATE block begins).
4. The sliding window history (the exact 32KB dictionary state at that moment).

Proposed API Changes / Ideas

It would be extremely helpful to have an API or hook in github.com/klauspost/compress/gzip (or the underlying flate package) to retrieve this state.

For example, a callback hook or an interface on the Reader:

type GzipCheckpoint struct {
    UncompressedOffset int64
    CompressedOffset   int64 // Byte offset in the compressed stream
    BitOffset          uint8 // 0-7 bits
    Window             []byte // 32KB sliding window dictionary
}

// Possible approaches:
// 1. A callback registered on the reader
gzipReader.OnCheckpoint(interval int64, func(cp GzipCheckpoint) { ... })

// 2. Or a method to manually dump the current state
cp, err := gzipReader.DumpState()

I would be glad to discuss how such an API could be designed or to contribute to a prototype if this feature aligns with the roadmap of the repository.

References:
https://github.com/madler/zlib/blob/master/examples/zran.c
https://github.com/mxmlnkn/ratarmount
https://github.com/circulosmeos/gztool
https://github.com/unxed/tar

[klauspost]

There are several options available through the API:

A) Use Flush once you've reached the desired checkpoint. The current block will end and all output will have been written to output. You simply attach a counter to the output. Once Flush has returned you have your compressed offset. And you already control the input.

B) Use an approach similar to pgzip that splits input and uses ResetDict to reset with a provided window. You can use concurrent compression of these blocks. It does 1MiB blocks with 16KiB history provided to each.

Both will have blocks ending at full byte offsets.

You can track the window easily yourself. Though if you are going to store it, you might as well start a clean block.

[unxed]

Hi @klauspost,

Thank you for the detailed suggestions! Both Options A and B are excellent techniques when I am creating (compressing) archives within our library. I will definitely look into incorporating Flush() at interval checkpoints to produce highly optimized, byte-aligned, and easily seekable .tar.gz files when writing archives.

However, the primary challenge I am trying to solve is the reading (indexing and random-access decompression) of pre-existing, arbitrary .tar.gz archives that were created by external, standard tools (such as GNU tar, Python, or the standard gzip CLI).

When our library acts as a reader for these third-party archives:

1. I have no control over how they were compressed. They do not contain custom Flush boundaries or block-aligned resets.
2. The compressed blocks in standard DEFLATE streams end at arbitrary bit offsets (0–7).
3. To build a random-access index on-the-fly during the first sequential read of such a third-party archive, I must be able to capture the exact decompressor state (the 32KB sliding window/dictionary, exact byte offset, and the bit offset) at selected intervals.

Currently, I achieve this on the read path by using Go's reflect package to extract the unexported fields of flate.decompressor (hist, wrPos, nb, and step). While this works, it is fragile and highly dependent on the internal implementation details of the standard library or klauspost/compress.

This is why a stable, read-side API (e.g., a method to dump the current sliding window, bit offset, and stream position from an active gzip.Reader or flate.Reader) would be incredibly valuable. It would allow pure-Go libraries to index and mount any standard GZIP file with $O(1)$ random access in a safe, production-ready manner.

[klauspost]

Ah, you are looking to index existing streams. I will look at it tomorrow if possible.

[klauspost]

See #1154

[unxed]

Hi Klaus,

Thank you for implementing this! This is exactly what I needed.

By introducing WithEobCallback and ResetCP / WithResumeFrom directly in the flate package, I can replace the fragile reflect workaround I previously used to extract the internal decompressor state (the sliding window history, bit offsets, and stream positions).

This native API provides a clean and stable way to generate and resume from zran/ratarmount-compatible checkpoints in pure Go. I will adapt unxed/tar to utilize it once it is available.

Best regards!

[klauspost]

Merged. You can reference the commit in master. A release is probably about 2 weeks out, since I have some other stuff I want to finish up first.