Can't connect to workflows service in k8s self-hosted install

Self Hosted Retool
,
1

Hi! Faced with problems.
Retool image "3.33.30-stable", Helm chart version: 6.2.0
All pods started. During the run any query in workflows get the error: {"data":null,"metadata":{},"error":"Internal Error running a block: Error: An internal server error occurred"}
In retool pod logs :
{"error":{"message":"FetchError: request to http://retool-workflow-backend/api/workflow/runQueryForSandboxed failed, reason: getaddrinfo EAI_AGAIN retool-workflow-backend","stacktrace":"FetchError: request to http://retool-workflow-backend/api/workflow/runQueryForSandboxed failed, reason: getaddrinfo EAI_AGAIN retool-workflow-backend\n at ClientRequest. (/tmp/node_modules/node-fetch/lib/index.js:1501:11)\n at ClientRequest.emit (node:events:517:28)\n at E.emit (/tmp/src/runBlockHandler.js:20:343331)\n at Socket.socketErrorListener (node:_http_client:501:9)\n
From pod to this curl to k8s service retool-workflow-backend is working.
Nslookup is working (shows the correct internal k8s IP)
During debug with tcpdump I've found only request for another k8s service - code-executor

08:37:05.117350 eth0 Out IP retool-9b4c78dd6-r9m8g.56076 > 169.254.25.10.53: 22135+ A? retool-code-executor.retool.svc.cluster.local. (63)
08:37:05.117428 eth0 Out IP retool-9b4c78dd6-r9m8g.56076 > 169.254.25.10.53: 65386+ AAAA? retool-code-executor.retool.svc.cluster.local. (63)
08:37:05.118701 eth0 In IP 169.254.25.10.53 > retool-9b4c78dd6-r9m8g.56076: 65386*- 0/1/0 (156)
08:37:05.118790 eth0 In IP 169.254.25.10.53 > retool-9b4c78dd6-r9m8g.56076: 22135*- 1/0/0 A 10.233.4.229 (124)
08:37:05.135899 eth0 Out IP retool-9b4c78dd6-r9m8g.45015 > 169.254.25.10.53: 54622+ PTR? 10.25.254.169.in-addr.arpa. (44)
08:37:05.137453 eth0 In IP 169.254.25.10.53 > retool-9b4c78dd6-r9m8g.45015: 54622 NXDomain* 0/1/0 (123)
08:39:26.857037 eth0 Out IP retool-9b4c78dd6-r9m8g.35765 > 169.254.25.10.53: 52930+ A? retool-code-executor.retool.svc.cluster.local. (63)
08:39:26.857105 eth0 Out IP retool-9b4c78dd6-r9m8g.35765 > 169.254.25.10.53: 27132+ AAAA? retool-code-executor.retool.svc.cluster.local. (63)
08:39:26.857680 eth0 In IP 169.254.25.10.53 > retool-9b4c78dd6-r9m8g.35765: 52930*- 1/0/0 A 10.233.4.229 (124)
08:39:26.857932 eth0 In IP 169.254.25.10.53 > retool-9b4c78dd6-r9m8g.35765: 27132*- 0/1/0 (156)

Can you suggest about how to solve this problem?

2

Hey there! Sorry to hear you're running into this. Is this the first time you're deploying workflows, or do you have existing workflows in production that just stopped working today?

3

If you're able to provide screenshots of the query blocks and the error logs you see in the workflows UI, that will help us troubleshoot. This error comes up when we can't persist the the query block results. A common cause is that there are too many operations running at once. I've seen this happen before with a loop block that was missing an await statement so the queries were firing off too quickly. If you're testing with complex block logic, I would recommend trying this again with a very simple code block (i.e. return 1 + 2).

4

Hi Chris!
It first time setup.
Very simple example GET to https://docs.retool.com/changelog/feed.json or SELECT * FROM. All queries produced similar errors in UI and pod.

image
image852×434 33.6 KB

5

@Chris_Cheasty Maybe you found an answer? )

6

short update. I've disabled codeExecutor pods and at the moment queries can run. Could you explain how it affects the working process?