REST Pagination in Retool Workflow

Workflows
1

I have a REST API which return 100 records max. If more it will return an offset which i have to send in a new request. So i have to call the endpoint until no offset will return to make sure i have all data and i have to aggregate the data from all calls to proceed it in the next block. How can i do it in a Workflow?

2

Hey @Timo,

Welcome to the community!

To create a workflow in Retool to iterate over a REST API that returns a maximum of 100 records and returns an offset to retrieve the next page of results, you can follow these steps:

  1. Create a new workflow and add a Call API block.
  2. In the Call API block, configure the following:
    • Method: GET
    • URL: The URL of your REST API endpoint
    • Headers: Any headers that your REST API requires
    • Query Parameters: If your REST API requires any query parameters, add them here.
    • Body: If your REST API requires a body, add it here.
  3. Add a Set Variable block after the Call API block.
  4. In the Set Variable block, configure the following:
    • Variable Name: offset
    • Value: The offset value returned in the response from the Call API block.
  5. Add a Conditional block after the Set Variable block.
  6. In the Conditional block, configure the following:
    • Condition: Check if the offset value is empty.
    • If True: Add a Call API block and repeat steps 2-4.
    • If False: Add a Combine Results block to aggregate the data from all of the API calls.

The following is an example of a Retool workflow that iterates over a REST API that returns a maximum of 100 records and returns an offset to retrieve the next page of results:

# Call the API to retrieve the first page of results.
response = workflow.call_api("https://my-api.com/endpoint")

# Set the offset variable to the offset value returned in the response.
workflow.set_variable("offset", response.json()["offset"])

# Loop until the offset variable is empty, which indicates that there are no more results to retrieve.
while workflow.get_variable("offset") is not None:

    # Call the API to retrieve the next page of results.
    response = workflow.call_api("https://my-api.com/endpoint", query_parameters={"offset": workflow.get_variable("offset")})

    # Update the offset variable to the offset value returned in the response.
    workflow.set_variable("offset", response.json()["offset"])

    # Combine the results from the current API call with the results from the previous API calls.
    workflow.combine_results(response.json()["data"])

# Once the loop has finished, the combined results will be available in the `results` variable.
results = workflow.get_variable("results")

# Do something with the results.
print(results)

This is just a basic example, and you may need to modify it to meet your specific needs. For example, you may need to add additional logic to handle errors or to process the results in a different way.

Hope this helps you get started.

:grinning:

Patrick

3

Hey Patrick,

Thank you very much for your help.
I get the idea but I have no clue how to add a combine result block in Workflow.
I understand you code but I don't know how to do it in the Workflow editor. Because I would need a global variable to store the results. But I don't see how I can do this. I just have code block but these lose their information after each run.

Timo

4

Hey @Timo,

Sure, I understand, no problem. Maybe I got a little over excited with my answer. :sweat_smile:

Now, to combine the results from multiple API calls, you can also use a JavaScript function. The following JavaScript function combines the results from two arrays into a single array:

function combineResults(results1, results2) {
  return [...results1, ...results2];
}

To use the combineResults function in Retool Workflow, you can add a JavaScript block to your workflow. In the JavaScript block, add the following code:

const results1 = workflow.getVariable("results1");
const results2 = workflow.getVariable("results2");

const combinedResults = combineResults(results1, results2);

workflow.setVariable("results", combinedResults);

Once you have added the JavaScript block to your workflow, you can connect it to the two API blocks that are returning the results that you want to combine. Then, you can run the workflow and the combined results will be available in the results variable.

:grinning:

Patrick

1 Like
5