I'm running into an issue with a workflow that uses a Python script to send an email (with a base64-encoded attachment) via Azure Communication Services.
When I run the workflow as a whole (server-side), I get this error at the Python block:
Error evaluating sendEmail: {"error":true,"message":"Internal server error"}
All preceding steps complete successfully, and the error only occurs at the Python step.
However, when I manually run each block step-by-step (client-side), including the Python block, everything works fine and the email is sent successfully.
I found a similar thread here: "Urgent: Internal Server Error" (Urgent - Internal Server Error). It looks like the issue was resolved, but there’s no detailed explanation how.
Has anyone found a fix or workaround for this behavior?
Sorry for this issue, very odd that it works fine when run manually but fails when the workflow is triggered and run independently.
It is unfortunate that the error is so vague. Are you self hosted? You might be able to check the container logs to see some more details on that error message
I’m cloud-hosted (Retool version 3.248.0), not self-hosted, so unfortunately I don’t have access to container logs. I’m also puzzled by the inconsistent behavior - manual execution works fine, but the full workflow fails with that vague error.
I’ll try to troubleshoot further and see if I can narrow it down.
If you share the workflow's run-id from one of the recent times that it fails its run I should be able to look up the run history in our cloud's data dog to see if I can get some more clues on this error
Thanks for offering to take a look! Here are two recent workflow run IDs from different workflows that both use the same Python block to send emails and return the internal server error:
01987972-6744-739b-b7f1-d8cee172320b and 01985122-e6ae-75b5-bd05-bd2ea3f79a66
Maybe it helps to compare the logs between these runs, since they share the same failing component.
Let me know if you’re able to find anything useful in the logs. Appreciate your help!
I was able to find the logs for both, with 01987972-6744-739b-b7f1-d8cee172320b giving me an error message saying it could not find a specific file/folder in the directory of the sandbox it was being executed in.
This makes me think that it could be a permissions related error where the environment that the python code block is being executed in does not have access to the permissions to the Azure Communication Services it needs to send the email.
Where the code is being executed in a different environment when you run it manually where it is able to access the auth credentials from the Retool app and dispatch the email. I will do some more research to figure out what could be done to remedy this.
Could you share with me how you are authenticating into Azure Communication Services? Is this a resource you are manually authenticating into in your app? My hunch is that the sandboxed environment where the python code is being run by the workflow doesn't have access to the permissions
One important note about workflow code blocks is that you won't be able to access the current_user object in the sandboxed environment. So if there are any auth processes that are using that object, that could be the cause of the block failing.
Just to clarify - I’m currently using an access key directly within the Python block and I don’t really believe this is a authentication issue. The exact same Python code is used in another workflow where the automatic run works perfectly (e.g., Workflow Run IDs: 0198c011-31ac-7521-bf41-e1f5627a874f and 0198ab79-5d5d-7275-9480-de1f80076336).
I’m currently investigating whether attachment size limits in Azure Communication Services might be a contributing factor, as one of the workflows could be generating larger attachments, but I don’t think that’s the root cause either.
I’ll send you the Python code via private message so you can get a better sense of how it’s executed.
Just wanted to check in and see if the attachment size limits could be playing a role in why the emails are sending in some cases but not in the production version of the app.
I don’t think the issue is related to attachment size. I’ve tested the workflow with data well below the limits - including files around 2–4 MB. Even after factoring in the ~33% overhead for Base64-encoded messages (as required by Azure Communication Services), the total size remained within acceptable thresholds, yet it still didn’t work in the production version.
I was doing some digging and it seems that this issue is likely related to a known bug that the engineers believed they had fully patched.
For some reason the root cause is from the python block and the loop block not playing nicely. I was able to dig into the datadog errors from the workflow runs you shared and the root error was related to a file path not existing for the code executor when trying to run one of the workflow blocks.
I am waiting to hear back from the engineers on next steps after I explained your situation. A short term work around would be to use a JS block to run a for statement loop instead of the loop block. Will let you know more about what the engineers tell me.
