"Read file from S3" returns binary/octet-stream

App Building
1

I am trying to read a CSV file in S3 that was previously found with another query. The first query runs just fine as seen below.

Screen Shot 2023-01-05 at 6.31.56 PM
Screen Shot 2023-01-05 at 6.31.56 PM2532×1636 454 KB

When I run my second query using the table selection, I get a response, but it is ContentType: "binary/octet-stream".

Screen Shot 2023-01-05 at 6.34.57 PM
Screen Shot 2023-01-05 at 6.34.57 PM1920×1287 221 KB

I was able to make another query that downloads this same file, and Retool is able to download this properly.

Someone else had this issue before, but their solution was not applicable to my case.

I am running self-hosted Retool on my company's premises.

2

I wasn't aware of this, but there is a metadata tag on these files in S3 that specifies the content type as binary/octet-stream. Is it possible in Retool to request a file as a particular ContentType?

3

If the file uploaded with Retool?

4

@jonathanbredo The file is not uploaded via Retool. It is uploaded to S3 via another service at our company.

I am requesting that there be a field in the Read a file from S3 tool to specify a certain filetype.

5

Hey @cander!

It looks like Retool automatically encodes binary/octet-stream files into base64 data. Can you try using atob on the data returned by your read query along with something like the built-in Papa Parse library (e.g. Papa.parse(atob(get_s3.data.Body), {header: true}))?

Let me know if that works!

1 Like
6

Thanks @Kabirdas for that insight. I was able to read my binary/octet-stream files after doing atob on the payload.

In my case, I was working with a zstd-compressed file, and had to convert that ASCII string into an UInt8Array for decompression, but OP likely has a different use case.

Do we know why this behavior occurs?

7

I can't speak fully to what's going on in the background (mostly because I'm not super familiar with it :sweat_smile: ) I believe data that can be passed between components/queries follows some of the restrictions of Window.postMessage and data that isn't in JSON format often gets converted to a base64 string to get moved around.

If I can find a more definitive answer I'll pass it along!

1 Like