I'm using Workflows to insert records into my SQL database. The workflow retrieves some data from an API, then it transforms the data so that it can be inserted into SQL, and then with the loop component is inserted to the database.
The issue I'm having is that each time the array of objects I want to send has a length bigger than 500 (sometimes it can be down to 100), the loop component that has the task of inserting into a database returns the following error:
Internal Error running a block: TypeError: NetworkError when attempting to fetch resource.
This issue is breaking completely my pipeline automation since sometimes when it executes automatically it doesn't insert the total amount of records.
Is this issue related to a configuration of my database? What can be causing this error?
What are your resources used in each code block? particularly in the loop block? is it an API to insert record/s or the native database resource that you're using. The type of error seems related to CORS and I have not encountered this so far but happy to dig deeper on this issue.
Just noting as well that it's good to check the constraints/limits that Workflows have in the docs and your own resources.
Just want to add on here that looping through a large number of records can hit database and API rate limits as well, and there can also be race condition issues. There's an example of how to add a delay to your loop block at the bottom of this post. That might resolve your issue and can also be good to try just to get additional context for debugging the issue. It may be worth exploring batching your data as well!
