How do I read all files from S3, not just one?

I would like to read all CSV files from S3 and do some calculations on them. However, I cannot figure out how to do this.

The "List all files in a bucket" doesn't include the CSV data.
The "Read a file from S3" only reads one file, not all files in the bucket.

I've spent days trying to figure this out.

Hey @nutmeg!

How many CSV files is this? Since Retool is a frontend application, all that data would need to be loaded into your browser and iterated over with javascript to parse the CSVs. If this is a significant amount of files, I’m a little concerned this approach might overload your browser and crash the app.

If it is not a massive amount of data, you could take an approach like this JS example:

You would swap out the REST query for individual “Read a file” queries from the “List all files” return.

Keep in mind that if you are ever loading “All” of something and the data set keeps growing, eventually it will be too much to load in a browser and your performance will keep degrading over time.

1 Like