File fetching is slow in Retool from S3 (pdf, image files). It takes time randomly, ranging from a minimum of 4 seconds to sometimes 20 seconds or more. File size is usually 200 ~ 1020 kb. for image files fetching is even slow.
To optimize speed for fetching files from S3, I have enabled the transfer acceleration for the S3 bucket, where I have to use the accelerated endpoint as shown below:
awsbucket.s3-accelerate.amazonaws.com
In the Retool S3 resource query configuration, there is no provision to use the accelerated endpoint as shown in the image below. I have tried updating the endpoint under the Custom S3 Endpoint field as shown in the image, but I am getting the error Base URL is not properly formatted.
Please guide me on how to configure the S3 accelerated endpoint in the Retool S3 resource. Or How to increase the file fetching speed from s3 in retool.
@victoria @justin
have you tried with CORS configured? does the console or browser dev tools provide more or different errors? you might need to delete the S3 bucket name field since the base url specifies the bucket name... the resulting request might be trying to access a nested bucket or something.
if you put the normal non-accelerated url in the Base URL field does it work? if not, I'd guess permissions are actually the issue here.
i guess a workaround might be to use a JS Query and manually use fetch() or whatever instead of the S3 resource when you want to use the accelerated endpoint
just a couple things i noted from the aws docs:
- After Transfer Acceleration is enabled, it can take up to 20 minutes for you to realize the performance benefit. However, the accelerate endpoint is available as soon as you enable Transfer Acceleration.
I just notcied this though:
Transfer Acceleration is only supported on virtual-hosted style requests. For more information about virtual-hosted style requests
which might mean enabling force S3 URLs to use path style could be causing problems, since the api expects virtual-hosted style only trying to read a path style url as such would lead you to an invalid endpoint/bucket.
Yes. you are right. It worked for me after removing the bucket name custom url.
https://s3-accelerate.amazonaws.com
1 Like
