I’m running into an issue with a workflow that retrieves historical orders for multiple e-commerce merchants. The challenge isn’t the 139 error from a single block but rather the workflow’s overall memory usage accumulating over execution, eventually hitting the 1GB limit. Some merchants have only a few orders, while others have thousands, leading to inconsistent failures.
Is there any way to clear memory or handle garbage collection in Workflows? In Retool apps, we can use function.reset() to free up memory (at least I assume it would), and something similar here would be incredibly useful—ideally, a way to discard JSON data that’s no longer needed after processing or writing to the database.
Are there any best practices, workarounds, or solutions I should consider? Would love any insights!
Great question, I would definitely like to hear from other members in the community about their strategies and best practices for managing memory for large amounts of data in workflows.
Along with calling sub-workflows for tasks inside of larger workflows, as these will execute in a separate sandboxed environment with more memory available.
To clarify on your specific question about freeing up memory inside of a workflow before it runs once the data in no longer needed, I can definitely make a feature request for that!
Also for self-hosted users, there are a lot more options for controlling query/block run times and data limits as well which can go higher than what we allow for cloud users!
I haven’t done it yet but, I am considering completely changing my approach and start using js blocks for controlling a lot of the data flow in a workflow. I also experience a lot of weird crashing that seems related to memory overages or something but its just really unclear where it stems from. There might be away to really pinpoint what’s causing the crashing but, its really not obvious or feels like I would need a degree in something I don’t have to be able to pick through and understand what logs might exist somewhere. I use retool so I don’t need a degree in those things so I guess part of this comment is to hint that retool really should implement something that helps people see better what they are doing to crash a workflow. Just break up tasks in smaller sizes is generally what is said but obviously that isn’t working. I think I am going to start calling blocks from a js block and create my own loops so I can .reset() the queries when I am done storing the results into a variable within the block. I often have to do things where I get file data from somewhere, then send it out to possibly be converted, and then upload the converted files somewhere. Every time these files leave a block to go to another block, the data use is doubling. Its seems incredibly inefficient. For example, I just ran a workflow that would have processed two approx. 20mb files in this way. The workflow crashed with an error that the workflow exceeded 2500mb!!!??? How could that possibly be? To me, there is something under the hood that is severely bloating data usage withing a workflow.
We have an open ticket for handling memory violation errors more gracefully!
And to your later point, @Shawn_Crocker, there's currently almost no visibility into memory usage during a workflow. One subtlety - the Node environment and bundled code both consume memory, meaning larger third-party libraries can drastically reduce the amount of "usable" RAM.
You've already seen how file processing can hog space, as well, often because it's parsed as base64. The recommended pattern is to pass an S3 URL instead of the file itself.
The other thing that trips people is up is unnecessary data duplication. Because output from every upstream block is passed into the activity sandbox, it makes sense to consolidate queries + data transformations whenever possible.
I'm kind of interested in publishing a more thorough guide, if folks would find it helpful. Loop blocks are another common failure point with a bunch of nuances.
How could that possibly be? To me, there is something under the hood that is severely bloating data usage withing a workflow.