Hey everyone,
I wanted to share a method we’ve been using to loop through multiple pages of a REST API. This approach leverages Retool’s option to run workflows directly from within a workflow, so there’s no need to use the webhook trigger. I found some of the other solutions offered on the forum a bit convoluted for my use case, so I'm sharing this in hopes it helps someone in a similar situation.
How It Works
-
Initial API Call: We start with an API call to grab the data.
-
Handling the Response: When we get the response, we run an if-statement:
- Check for Items: First, we check if any items are returned.
- Check for Next Page Token: We then check for a next page token.
-
Stopping Condition:
- If no items are returned, the workflow stops—meaning all pages have been fetched and the response is empty.
- If there is a next page token, we grab it and feed it back into the same workflow, creating a loop.
-
Recursive Workflow: This creates a simple, clean recursive workflow that continues fetching pages until no next page token is found. It automatically stops when it reaches the end.
Why This Method?
This approach is cleaner and simpler than others I’ve seen, which often involve using two different workflows with different triggers. It’s straightforward and stops automatically when there are no more pages to fetch.
I’ve come across a couple of other forum posts that discuss iterating through paginated REST API responses, but they didn’t quite hit the mark for me (e.g., looping through 2 workflows). They use a more complex setup, while I found this method keeps things tidy with just one workflow.
I’ve included an annotated screenshot below that gives a high-level overview of the workflow blocks in action:
Hope this helps! Let me know if you have any questions, suggestions to modify this, or other methods that have worked well for you. For example, some REST APIs offer options to combine HTTP requests using multipart, but for a routine database update, this seemed much simpler.
