Am following along with this thread closely.
I am also having a similar issue where when scheduled, the workflow is failing but when individually running the components in order it works completely fine with no issues.
Some context to the workflow if it helps:
- Similar to @Ross_JWHI Ross, am pulling from an API. Happens once a day and grabs transactions that occurred the day before and writes to a postgres DB
- The error that comes up is indicating that ALL transactions are duplicating when trying to write to our DB for storage even though there are only a fraction of records trying to be written that day.
- The fact that the workflow works when manually executing each element but fails when scheduled indicates to me that it may be out of my control
- The workflow was working with no issues a week ago with no issues however now consistently fails each day.
Happy to offer more information if it helps the investigation.