Workflow run out of memory at 256mb instead of 1Gb

💬 Workflows
1

Good afternoon,

I am experimenting with Retool Workflow to handle some ETL tasks.

When using the « Python Code Block » I encounter an error « Ran out of memory while returning results. Only 256mb total memory allowed, including data returned » (I attached a screenshot).

This task is responsible to download a 50mb file (which is far less than 256mb) and moreover, in your online documentation, it says that workflows have a memory of 1Gb (I also attached a screenshot).

Could you highlight me how to resolve this error ?

Capture d’écran 2024-10-10 à 15.25.39
Capture d’écran 2024-10-10 à 15.25.392879×1324 325 KB

Capture d’écran 2024-10-10 à 15.25.37
Capture d’écran 2024-10-10 à 15.25.372822×1478 372 KB

Regards,

1 Like
2

I'm still stuck on this. Any ideas from the community on a solution or workaround here?

3

Hey @axel_lcdp! Welcome back to the community.

Thanks for flagging this inconsistency. :+1: I checked with the team and there is currently an undocumented, block-level limit of 256mb enforced specifically when executing Python code. We'll be updating the public docs accordingly!

Even then, though, it's not immediately clear why you are hitting this limit as there is little to no visibility into the block's memory usage. You can see the entire workflow's network usage in the run history, which provides some insight, but there is also an ambiguous baseline memory cost associated with executing code in a Python environment.

image

The sum of those two things is what's exceeding the 256mb limit, in this case. Any information you can share about the contents of the code block, including the data you're fetching and any imported libraries, would be helpful in figuring out whether this behavior is expected or not!

4

Hi Darren,

Thank you for your response. Do you know if there is such limit for the JavaScript code block as well ?

I attached you the network analysis, but there is nothing more.

Capture d’écran 2024-10-16 à 09.48.06
Capture d’écran 2024-10-16 à 09.48.062866×956 316 KB

For the actual code, we have not imported any libraries (only using requests which is already present). After that, this block runs an HTTP request and return its content (We cannot use a Retool's Resource for this API Rest because of some weird authentication mechanism that even "Custom Auth" cannot handle).

The response is pretty heavy, between 1 MB and 5 MB of JSON data. However, 5 MB response is far less than 256 MB, so I expected it to work.

Here is the Python code template:

import requests

auth_response = requests.post(
  url = ...,
  json = { ... }
)

auth_response.raise_for_status()
access_token = auth_response.json()['access_token']

response = requests.request(
  method = 'POST',
    url = ...,
    headers = {
        'Authorization': access_token,
    },
    params = ...,
    json = ...,
)

response.raise_for_status()
content = response.json()

return content

When I do the same thing using a JavaScript block (using node-fetch library), it works, it can handle the 5 MB response.

Could you highlight me on the response size limit for Python and JavaScript block ? It would help us determine if we can use Retool for some basic ETL processing.

Thank you, have a nice day.

5

Yes - I can confirm that individual JS code blocks share the same 1GB memory limit as the entire workflow. I'm not sure why there is a more strict limitation enforced for Python blocks. :thinking:

In this case, I'm pretty sure you're not seeing the total network cost because the workflow failed to run to completion. Looking at the contents of your block, though, I'm even more mystified as to how you're hitting the memory limit. If it makes sense for your use case, I'd probably recommend using JS blocks for making manual API requests.

To your last point, there isn't a separate limit to the size of the payload returned by a given block. It would just need to respect any block-level limitations - 256MB for Python and 1024 MB for JS - and the 1GB memory ceiling for the entire workflow run. The other thing to keep in mind is that the body of individual resource requests are generally limited to ~100MB. I'm not 100% sure if this would apply to requests made from within a code block but I'm fairly sure it would.

It's worth noting that all the above limits can be adjusted or removed entirely if you are running a self-hosted instance!

1 Like