Workflow run error logs suggest out of order block execution

I've got a workflow which has been in use for a while and functioning as expected. Last edit and deploy was April 5 2024. This workflow is used to save user uploaded files to our Retool Storage and database. The workflow is linear with no conditionals, just a few blocks one after another - 5 blocks total including trigger and return with no orphaned blocks. The main blocks are loops. I'm using Retool Cloud v3.53.0-432fed2 (Build 172663).

Here is the workflow with all blocks visible:

A few days ago we had a few failed runs several times in a row. The user was trying to upload two files and attempted several times without success. A few minutes later the workflow succeeded with the same inputs. I haven't been able to find anything wrong with the workflow or the inputs yet.

In the workflow run history each of the runs with errors has the same log showing what looks like blocks executing out of order:

The correct order should be:

  1. startTrigger
  2. extractFilesFromInput
  3. addToStorage
  4. addToDatabase
  5. webhookReturn1

Additionally, skips and errors shown visually in the "blocks" section don't match the logs - addToStorage is skipped in logs but shows as errored in blocks, and addToDatabase shows the opposite, skipped in blocks but errored in logs. I'm not sure if the order of logs here is guaranteed since several occur at close to the same time. Even if not, failures of later steps caused skipping of earlier steps (seen in the last log --- Hit failure, skipping queries: addToDatabase, extractFilesFromInput)

Clicking into the skipped and errored blocks gives no additional data. The "Inputs", "JSON", and "Logs" tabs for individual blocks are blank and the "Data" for them shows undefined for skipped steps and "undefined null" for errored steps.

Edit: I noticed a similar execution order issue in another workflow around the same time:

I've reviewed past community posts related to workflows + logs but none I found mentioned this behavior.

Our workflow is currently functioning without errors and no changes have been made. But I'm curious if there is anything that I can do to prevent these errors.

Side note:
You can see what I think are two other bugs in the screenshots. One is minor: It is possible to deploy a workflow version even when no change have been made ("Deploy" button is blue, clickable, and deployment possible despite Latest version deployed). The other is that the previously selected keys of the key/value pairs in addToDatabase block are not being shown, despite this block currently being functional when the workflow runs. These don't seem related but might be.

1 Like

Hello @benp!

Thank you for bringing these to our attention. Very odd behavior as from you screen shots you have everything looking correctly setup and it sounds like this issue was not occurring before and you had not made any changes.

My guess would be that there was something else going on behind the scenes on our end, maybe a code patch to fix another bug had spilled over and caused out of order workflows.

I would guess that if the errors/behavior has gone away that our team caught the bug that was causing this and fixed it :sweat_smile: if it comes back or can be consistently be reproduced then we can file a bug report for the eng team. You shouldn't need to do anything on your end to prevent these.

In regards to your side note, I believe we have a request to either make the deploy workflow button unclickable/differently colored to limit or reduce no change deploys.

The issue with the key/value pairs not showing up from a past bug is also a common linting error that we have logged to our eng team, where until the block runs and its state in the app is given data/values to be read/previewed in other blocks it will not show up/throw a message but still work fine when all the events run in the correct order successfully.

Hope things keep working smoothly!

1 Like

Thanks Jack! I figured there might be something happening behind the scenes that resolved itself, glad to hear that's what you're thinking too. I haven't seen any more issues like this, but I'll update here if I do. Appreciate you responding!

1 Like