Concurrent workflow

Workflows
1

Hello,

I'm planning to upload a large number of photos to the Retool DB using a workflow. I have a few questions:

  1. Uploading takes a considerable amount of time. Is there a way to parallelize the uploading process?
  2. Which approach is better: running a workflow for each photo individually or running one workflow for the entire batch with a loop inside?
  3. If the photos are too large, my browser crashes. Is there a way to handle this issue?

Thank you.

2

Have you confirmed that the size limitations for Retool DB and size limits for the file input won't ultimately become a limiting factor? There are also timeout and size limits for workflows, so you should review that as well.

Not knowing the full scope of the size (both in terms of items and bytes) of the expected upload list, and not having specifically tried to get through something like this myself, I can only suggest trying a few different approaches and seeing which is scaling better.

One thought would be to take the list of files from the input array, and split that by index into subsets, sending files 0-4 (for example) to one loop, 5-9 to a second loop, etc. I would then include a block that connects to all the loops you create to re-join the results from all of the loops into a workflow body response.

When you say your browser crashes, where is that happening? In the app when loading a huge list of files? If so, you can limit the number in the app. In the workflow when trying to develop it? If so, switch to the tree view, which tends to behave nicer in my experience.

Good luck with this - would love to hear back how you solved for the issue.

1 Like
3

Thank you, I'll update soon.