Workflow Loop Error With no Error Logs

[daviswray]

Hey There!

I'm working on a workflow that get's 1,000 records from an API then uses a loop to add each component to a ReTool database table.

It's been running with no issue over 95% of the time, but occasionally the workflow will fail, and I cant find anything in the logs that indicate why it might have errored.

Here's a screenshot showing a successful run and the run blocks.

Here's a screenshot showing the failed run.

Im particularly perplexed by the blocks and logs on this failed run. The "Blocks" do not list the loop, but the logs do indicate that the loop and went through some iterations before suddenly stopping. No errors were logged however.

Here's a screenshot of the loop component itself.

The js in the loop just logs the data being sent to the database, calls a function to append the data, and logs the success or failure.

And I'm not sure if relevant but here's a screenshot of the function the js loop calls.

Anyone know why this might be erroring? Or if there's any additional logging I should add / setting to change that will give me more insight onto workflow failures in the future?

[WidleStudioLLP]

Hello @daviswray

Can you share the error that comes from the workflow. For error checking , check the input you can view the data that input by user , from the data you can check the data that comes , from the json you can check the json format of the data and from the logs you can view the status of each block to see where a failure may have occurred.

For example:

In my SMS_Scheduler there is a error, the error might be caused by a specific query in the workflow. Could you identify the query that's causing the issue? If needed, I can help debug the workflow further.

Workflow doc : Retool Workflows documentation | Retool Docs

[Darren]

Hey @daviswray! Thanks for reaching out.

My initial thought is that your query is likely timing out, but I'm not sure why the retries you've configured wouldn't handle the resulting exception. It's possible that the wrapping catch block is interfering with it.

Would you mind sending me the specific ID for each workflow run so that I can check our backend logs for any additional insights? Thanks!

[mascaritas]

Hi , I am planning a similar workflow, therefore interested on the outcome of this, since history is important for debugging etc,

I have a sidequestion; how do you store 1000 records from an API call, , do you run a pagination loop and write them in a database or is there a more elegant way ?

thanks

[Darren]

It depends a bit on the type of database you're using, but I'd usually recommend chunking the results off the API call with lodash and then running a bulk insert operation in a loop!

[mascaritas]

thanks a lot. So first fetching paginated payloads, then chunking it and batch insert into a db or processing in a workflow. that strategy will help