RAM exceeded When Running A Workflow

Workflows
1

Hi,

We're getting this error when attempting to run a workflow. Can anyone lead me to where I can check how to adjust our infrastructure to resolve this? The workflow works fine with 15k transactions, if I adjust it to 25k transactions it fails with this error.

We're currently hosting this on AWS ECS and we used the CF template provided by Retool.

"RAM exceeded 1610612736 bytes".

3 Likes
2

Hi Stefano! Welcome to the Retool community!

It looks like you are hitting the limit determined by the environment variable shown here. Can you confirm that WORKFLOW_MONITOR_PROCESS_ENABLED is set to true and what you have set for WORKFLOW_MEMORY_LIMIT_MBS as well as how much memory you have allocated to the code_executor service to try and diagnose where you are hitting a limit? Also curious if the code block you are running is JS or Python. Thanks!

2 Likes
3

Thanks for the response. I will check these.

4

@Taylor_M I don't see either of these ENV vars set....which ECS service would require them? I believe the CF template we used to deploy Retool is this one and I don't see the ENV vars in the template.

https://raw.githubusercontent.com/tryretool/retool-onpremise/master/cloudformation/retool-workflows.fargate.yaml

We are using JS for all of our workflows.

5

Thanks for the response and referencing the template that you used. Very helpful!

It looks like that CF template is missing the service that those variables are used in which is the code executor. I am looking into the sections that would need to be added to the template to setup code executor and will follow up with you.

1 Like
6

Hey @steveno_411!

The environment variables that Taylor mentioned above would typically be relevant to the code-executor service.

We haven't yet published an official CloudFormation template that includes this service, but there is a Terraform module that you can reference here! The two formats aren't directly interchangeable, but it should give you a good place to start, at the very least. There are also some decent AI tools out there for converting, if you want to try that.

8

Thx, I'll give this a try

1 Like
9

@Darren I was able to get the Code Executor service deployed. Do I need to do anything special to have it register with Retool?

Also, do I need to update my Workflow to use the Code Executor?

10

Nice! Glad to hear it. You shouldn't need to make any changes to the workflow itself, just set the appropriate environment variables. I'm not nearly as familiar with deploying on ECS and the required variables, but I imagine it's at least necessary to set CODE_EXECUTOR_INGRESS_DOMAIN, as shown here.

11

@Darren I've added the CODE_EXECUTOR_INGRESS_DOMAIN variable to all services but I'm still having the problem. When I run a workflow, I also don't see any logs from the code executor. How can I verify everything is wired up to use the code executor service?

12

@Darren Think I'm getting closer.....now getting these errors when I try to run the Workflow. Any idea on whatelse I need to check?

Error: hit an unexpected error\n    at lj.handleCodeExecutorError (/retool_backend/bundle/main.js:679:6625)\n    at lj.runBlock (/retool_backend/bundle/main.js:679:4258)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async /retool_backend/bundle/main.js:2576:41288

hit error communicating with code executor {\"status\":\"validation_error\",\"error\":{\"runBlockParams.retoolContextParameters\":\"Required\"}}

Error running the workflow