Question about Retool possibilities for my requirements

Discussion
1

Hey there,

i am going to need to process huge amount of data/files in around 5-50GB (something around that).

How would this work with Retool, since i was reading that there is a limitation with the CDN with 100MB.

So if i use the inputs i will basically upload the file somewhere and receive their URL, right?

But what i need is like a simple HTML input with the option for multiple files, and only process it if i submit it.

Maybe i misunderstand something but that’s the basic idea and i would appreciate if someone can give me some clarification.

Many thanks in advance.

2

Hello @allinjoe!

Unfortunately Retool is most likely not the best option for uploading and processing huge amounts of data and files :sweat_smile:

Perhaps Dropbox or a similar service could better meet your needs.

There are limits on how big of a file you can process in Retool, you could try chunking your data and folders/files into smaller amounts. You can read more about how other users uploaded files with custom HTML components to an AWS S3 bucket on this forum post here.

Could you elaborate on the 'processing' you are looking to do on submit? The issue with upload files and not sending them to a database "until you submit' is that while they are being held in the frontend/state on your app/browser, they could be lost from a browser refresh, networking issues, etc.

If you send the data to a database you can then access it to manipulate it and save it back to the Resource with additional queries.