S3 resource needs a raw-bytes / streaming read

💬 Feature Requests
,
1

Summary

The provided S3 resource library cannot return the raw bytes of a binary object. For any object whose content type isn't on Retool's internal "binary" list, the resource UTF‑8‑decodes the body server-side and returns it as a Body: string with IsBase64: false. Once that decode happens, the original bytes are unrecoverable—invalid UTF‑8 sequences are replaced with U+FFFD and the returned string is shorter than the source object.

We need a way to get the raw bytes—ideally as a stream—out of the S3 client without Retool performing any text decoding on them.

Reproduction

Read any binary S3 object whose Content-Type is not on Retool's binary-content-type list (e.g., application/zstd). The returned Body is a string, IsBase64 is false, and Body.length is less than
ContentLength due to UTF‑8 replacement characters embedded in place of invalid byte sequences. There is no way to reconstruct the original bytes from the value the resource returns.

Same result via download and readStreamRaw

  • download — same Body: string with the same UTF‑8 replacement behaviour.
  • readStreamRaw — the streamed chunks are the JSON envelope of the same Body ⁣string. The Body field of that JSON has already been UTF‑8‑decoded server-side (replacement chars embedded directly in the JSON string literal), so reassembling the stream yields the exact same lossy payload as the buffered read.

In other words, the *StreamRaw variant does not stream the underlying S3 object bytes—it streams the JSON envelope around an already-decoded Body, so it can't be used as a workaround.

Request

Please add one (or more) of the following to the internal Retool API for the
S3 resource. Any of these would unblock us; option 1 is the cleanest:

  1. A truly raw *StreamRaw variant of read / download that yields the S3 object bytes themselves (not a JSON envelope around a decoded Body). Concretely, on the backend the AsyncIterable should be chunk-by-chunk passthrough of the S3 GetObject response stream, with no UTF‑8 decoding and no JSON wrapping. Returning the iterable directly. from a serverless function (as the *StreamRaw contract already describes) would then expose it to the frontend hook's .stream as AsyncIterable<Uint8Array> of file bytes.

  2. A per-call option to force base64 encoding, e.g. s3.read({ fileKey, responseType: 'base64' }) or { binary: true }. Set IsBase64: true and return Body as a base64 string. This matches the existing documented behaviour of IsBase64 for binary objects, just opt‑in instead of relying on content-type sniffing.

  3. Surface the raw GetObjectCommandOutput.Body stream from the underlying @aws-sdk/client-s3 call as an opt-in (raw: true or a new readRaw method). Anything that exposes the SDK's existing readable stream untouched would work.