Workflow Iteration and API Limit Issue in Resume Processing

[Kshitij_Srivastava]

Dear Retool Team,

I am encountering a challenge in processing batches of 50 resumes within a 10-minute time window, which is the limit for processing each batch. I am iterating through the resumes in batches of 50 and processing them sequentially. However, once the 10-minute block times out, I track the iteration number to resume processing from where it left off and continue with the next batch of 50.

The issue arises when attempting to carry forward the iteration number to resume processing in a new cycle. This creates a loop, causing the system to cycle back and preventing further batch processing. I considered using triggers to handle batches of 50 in different workflows, but this approach leads to reaching the API limit when handling multiple batches.

Could you please advise on how to solve this issue? Specifically, I need guidance on:

1. How to efficiently resume processing the next batch of 50 resumes after the time block expires without creating a cycle or hitting the API limit.
2. Whether there is a way to maintain the iteration number or state between processing cycles to avoid starting over.

Thank you for your assistance.

This is the workflow link : https://peopleboxai.retool.com/workflows/7b9850db-5ecd-4326-9bd7-7ba4d9a0765a

Best regards,
Kshitij Srivastava
Software Engineering Intern,Peoplebox

[Jack_T]

Hello @Kshitij_Srivastava!

You should be able to run your code in a loop block as outlined here in our docs.

You can definitely keep a counter variable to track how many iterations you have done as well as keep track of while number on the batch of 50 you are currently on.

This value can then be passed to another block of the same workflow or pass the counter as a parameter into another workflow which should have its own separate time window and can pick up where you left off to keep iterating through from where you left off