I have a workflow that has a loop block that loops over 10,000 records, and in the block calls a multi-step function making an API call and database save. When I run for a 100 records I get no issues, over for the full number of records it fails on the loop. If I specifically just use one record, e.g. one it failed on, the workflow runs correctly.
This suggests that at some point the loop has issues when there are too many records. Is there a guide on how many it can cope with?
Yes, you’re probably running up against a Workflow limit. According to the official docs, a synchronous workflow run (or the synchronous portion up to the first webhook-response block) has a 15-minute timeout.
If you want to share a few screenshots of your workflow configuration and recent run history I can help pinpoint what limit you hit (timeout, memory, concurrency, etc.).
There are a number of common techniques to stay within those limits while processing large sets of records. For instance: batching/chunking data, splitting the job across multiple workflows, or scheduling to process in smaller increments (e.g. run a subset now, then resume later).
Hi @klautier ,
Retool have problem with running large chunks of datapoint at once insted loading all of them at once you can try batch processing.
Split the Dataset into small batches (e.g., 200 each)
Loop over each batch
Inside the loop, process only those 200 items
Workflow finishes safely without timeout/memory issues
Thanks for reaching out on the Retool community with your question. I believe both Shawn and Vishal have some good suggestions to what could be the problem and some ways to mitigate this
If you would like to explain a bit more about what the Workflows whole goal is, what these 10k records are, some details on the error you are getting in the UI, and how this is all configured, we can try and troubleshoot further what else could be wrong
I’m not sure there is a solution presented here. I also experience this pretty often. I have a workflow to perform file exporting from one system, uploading to a file storage and updating a db with file ids. I use a loop block set to batch and I process 10 items per batch. On some files that are large, meaning about 20-40mb, the loop will crash with an error of “an internal error occurred…” If I locate the problem file by doing smaller batch sizes until I am certain which file is tripping up the system, I isolate that file and run the block on an array with just that one item. The loop completes like a champ. The issue is really flaky and I think it has something to do with the loop block accumulating memory on its iterations rather then clearing its memory on each one.
The key question is whether you're running your batches sequentially or in parallel. The latter can actually be quite memory-intensive, as the results of each iteration accumulate in the same shared sandbox.
After discussion with support, there are hard-coded time limits when running loops with multi-step functions. There is no work-around other than designing to avoid these loops if the time-limit is likely to be exceeded.
Any extra details would be greatly appreciated