Skip to content

Postgres simple-query path materializes the entire result set in memory before sending (OOM on a large SELECT) #6679

Closed

Description

@ruispereira

Summary

The Postgres simple-query ('Q') path materializes the entire result set into a List<Result> in memory before writing the first DataRow, so a large SELECT over the simple protocol can OOM the server. The Bolt executor streams; the Postgres extended protocol is bounded by the portal max-rows; only the simple-query path is unbounded.

Where (HEAD 86a704526)

postgresw/src/main/java/com/arcadedb/postgres/PostgresNetworkExecutor.java:670 → final List<Result> cachedResultSet = browseAndCacheResultSet(resultSet, 0); (limit 0 = unlimited) then writeRowDescription(...) / writeDataRows(cachedResultSet, ...). All row objects are held simultaneously before the first byte is sent.

Repro

A plain SELECT over the simple query protocol (e.g. psql -c "SELECT FROM HugeType") against a large table → the whole result is buffered into an ArrayList server-side before streaming, risking OutOfMemoryError. The Bolt path (handlePull) streams row-by-row and does not exhibit this (verified).

Suggested fix

For the 'Q' path, iterate the ResultSet and write DataRows incrementally instead of pre-caching with limit 0; or cap with a configurable fetch bound.

Scope / confidence

High that everything is materialized (read at HEAD). Honest caveat: this is a long-standing, documented design (the browse-cache predates the #4197 audit and is deliberate for portal semantics), so it may be considered known/intended — reporting because it is a real "whole result materialized → OOM on a big simple-query SELECT" gap and the sibling transports avoid it.

Activity

  1. added this to the 26.9.1 milestone on Aug 24, 2026
  2. added
    in progressThe team is actively working on this issue
    on Aug 24, 2026
  3. self-assigned this
    on Aug 24, 2026
  4. added 4 commits that reference this issue on Aug 24, 2026
  5. removed
    in progressThe team is actively working on this issue
    on Aug 24, 2026
  6. lvca commented on Aug 24, 2026

    @lvca
    Member

    Fixed in #6693 (merged to main).

    A very large SELECT over the Postgres wire protocol (simple query mode) could use unbounded memory. There's now a configurable limit (arcadedb.postgres.simpleQueryMaxRows, default 1,000,000 rows) - a result larger than that is refused with a clear error instead of risking a server crash. For very large result sets, use the extended query protocol with a bounded fetch size instead.


    ❤️ Thanks for helping make ArcadeDB better! ArcadeDB is free and open source, sustained by its community. If it's useful to you or your company, please consider becoming a sponsor to keep development going.

  7. ruispereira commented on Sep 2, 2026

    @ruispereira
    ContributorAuthor

    Re-validated against main at c50338bd8: the 'Q' path is properly bounded now. The extended-protocol sibling (:415, :530) still buffers unbounded, and the new config doc-text recommends that path as the remedy - filed as a follow-up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions