Crash Loop in Retool Workflows Worker (Temporal Connection / DNS Issue)

💬 Workflows
,
1

We are experiencing an intermittent issue with our self-hosted Retool (on-premise) deployment where the workflows worker container enters a crash loop. The issue occurs sporadically and requires manual intervention (removing the container and re-running docker compose up -d) to recover.

Environment

  • Retool self-hosted (on-premise, Docker Compose)

  • Temporal-based workflows enabled

  • External PostgreSQL database

Logs:

/retool_backend/node_modules/.pnpm/@temporalio+worker@1.11.6_@swc+helpers@0.5.3_metro@0.80.9_encoding@0.1.13_metro-minify-terser@0.80.9_/node_modules/@temporalio/worker/lib/connection.js:58
                throw new core_bridge_1.TransportError(err.message);
                      ^

TransportError: tonic::transport::Error(Transport, ConnectError(ConnectError("dns error", Custom { kind: Uncategorized, error: "failed to lookup address information: Temporary failure in name resolution" })))
    at NativeConnection.connect (/retool_backend/node_modules/.pnpm/@temporalio+worker@1.11.6_@swc+helpers@0.5.3_metro@0.80.9_encoding@0.1.13_metro-minify-terser@0.80.9_/node_modules/@temporalio/worker/lib/connection.js:58:23)
    at async $W (/retool_backend/bundle/main.js:2959:17389)
    at async uts (/retool_backend/bundle/main.js:11238:9746)
    at async Object.GsT (/retool_backend/bundle/main.js:11238:14445)

Node.js v20.18.1
1 Like
4

Hi @milad.za! Happy to help here. If you could confirm a few things for me I may be able to help you debug:

  1. The version of your on-prem Retool deployment.
  2. Are you using Retool managed Temporal, or self-hosted Temporal?
  3. Is this a new deployment, or has it worked in the past and only recently became an issue?

Also appreciate you sharing the error logs! It could be useful to see what the worker container was up right before it crashed. Would you be able to share the the preceding logs as well docker logs --tail 50 <container_name>? Feel free to DM me the logs if you’re concerned about sharing them on this thread.

Look forward to hearing back!

Best,

Altai || Retool Support

1 Like
5

Have you had a chance to revisit this, @milad.za?