I just deployed a self hosted Temporal server in AWS via cloudformation template to pair with our self hosted Retool instance and I'm running into an internal error using one of the workflows from templates. I'm using the 'Notification and alerting' template and instead of showing the API call I get an error. When I curl the same API call from my local terminal and also from the base Retool query library, it's successful and returns json, so I'm assuming some Temporal related service can't make API calls and can't get out to the internet some how.
Here is the exact error and screenshot attached below: Internal Error running a block: Error: An internal server error occurred
Something that isn't helping is there are ZERO results when searching for "error" and "warning" in each of the ECS logs, making tracking down exactly what is going wrong here a bit hard. Any help is much appreciated.
3/29 update: found the verbose error in the logs, looks like a variable somewhere isn't getting defined or able to be pulled because the address listed has null.undefined at the beginning. here's a snippet from the logs:
"error": {
"message": "FetchError: request to https://null.undefined/api/workflow/runQueryForSandboxed failed, reason: getaddrinfo ENOTFOUND null.undefined",
"stacktrace": "FetchError: request to https://null.undefined/api/workflow/runQueryForSandboxed failed, reason: getaddrinfo ENOTFOUND null.undefined\n at ClientRequest.<anonymous> (/node_modules/node-fetch/lib/index.js:1501:11)\n at ClientRequest.emit (node:events:517:28)\n at E.emit (/retool_backend/bundle/workflowCodeExecutor/dist/runBlockHandler.js:20:343329)\n at TLSSocket.socketErrorListener (node:_http_client:501:9)\n at TLSSocket.emit (node:events:529:35)\n at TLSSocket.emit (/retool_backend/bundle/workflowCodeExecutor/dist/runBlockHandler.js:20:370113)\n at emitErrorNT (node:internal/streams/destroy:151:8)\n at emitErrorCloseNT (node:internal/streams/destroy:116:3)\n at process.processTicksAndRejections (node:internal/process/task_queues:82:21)",
"type": "FetchError"
},
"jobId": "f3c2820b-a38d-4a5f-97ea-4b908e1150a8",
"level": "error",
"message": "Error executing block:",
I found that the issue was with how I was populating the environment variables with !Ref and dropping a built parameter for the cloudmap namespace reference. I needed to use !Sub to reference the cloudmap namespace that it could be looked up properly.
Basically, if I make a parameter called TemporalHost and make it "temporal.retoolsvc-${PostfixTag}", when I use !Ref the ECS service sees this as a string and doesn't try to make a DNS look up to the cloudmap resource. I was seeing things like https://temporal.retoolsvc-staging:7233 in the logs and knew something wasn't right. So, instead I just made a parameter for PostfixTag and used !Sub to pass it that way instead and it worked.
