Recursively call a paginated API query to get all results

Queries and Resources
1

I've been trying to follow other forum instructions the past few days but haven't been able to get my query to work, so turning to this post.

I have a REST API resource I'm using to make a query within my app. The query is simple and just passes URL parameters (and authorization stored in the Resource) like /endpoint/v2/slug?offset=100&per_page=100&where[created_at][gte]=2024-03-01T13:37:38.230Z&order=-created_at

If the result set is greater than the per_page value, then the API response includes a "next" url in the links section as well as a "next.offset" value in the meta section of the output.

So I need to trigger the query, check if data.meta.next.offset exists, and if it does, trigger the query again but with the new offset value, then join all these query data requests together into one data set.

This is the closest I feel like I've gotten, but there is nothing in the output and I feel like there might be a simpler way.



function getAllDonationsRecursively(offset = 0, allDonations = []) { 
    return new Promise((resolve, reject) => {
        getDonations.trigger({
            onSuccess: (data) => {
                const currentDonations = data.data;
                allDonations.push(...currentDonations);

               const nextPage = data.meta.next.offset;
                if (nextPage) {
                    const nextOffset = new URL(nextPage);
                    getAllDonationsRecursively(nextOffset, allDonations).then(resolve).catch(reject);
                } else {
                    resolve(allDonations);
                }
            },
            onFailure: (error) => {
                reject(error);
            },
            additionalScope: {
                offset
            }
        });
    });
}

return getAllDonationsRecursively();

Any tips or ideas on how to get this working? Below is also an example of the data response to see what I have access to (shortened for simplicity):

{
    "links": {
        "self": "https://api.domain.com/endpoint/v2/slug?per_page=100",
        "next": "https://api.domain.com/endpoint/v2/slug?offset=100&per_page=100"
    },
    "data": [
        {
            "type": "Donation",
            "id": "123456789"
            },
        {
            "type": "Donation",
            "id": "987654321"
            }
    ],
    "meta": {
        "total_count": 234,
        "count": 100,
        "next": {
            "offset": 100
        }
}
2

There is definitely a way to do this using the links.next URL but I am a bit confused by these lines:

const nextPage = data.meta.next.offset;
...
const nextOffset = new URL(nextPage);

This doesn't appear to actually build a new endpoint URL but only seems to be supplying the offset value to your function.

When there are less results in the response than the per page value, does the links.next property exist? I feel like that is a better mechanism for making recursive calls.

1 Like
3

You are correct, when there are less results than per_page, links.next does not exist, only links.self.

Since the only thing changing in the URL was the offset, I had started down the road of just updating the offset value of the original URL (basically just changing that one URL param) to get the next page.

4

With a lot of trial and error (and some help from a friendly GPT assistant...) I was able to get it to work! Spelling it out here in case others have a similar situation and need to do it this way.

Here is my JS Query

function getAllItemsRecursively(offset = 0, allItems = []) { 
    return new Promise((resolve, reject) => {
        myQuery.trigger({
            onSuccess: (data) => {
                const currentItems = data.data;
                allItems.push(...currentItems);

                const nextPage = data.links.next;
                if (nextPage) {
                    const url = new URL(nextPage);
                    const nextOffset = url.searchParams.get('offset');
                    if (nextOffset !== offset) { // Check if nextOffset is different from the current offset
                        getAllItemsRecursively(nextOffset, allItems).then(resolve).catch(reject);
                    } else {
                        resolve(allItems); // End recursion if the offset doesn't change
                    }
                } else {
                    resolve(allItems); // End recursion if there is no next page
                }
            },
            onFailure: (error) => {
                reject(error);
            },
            additionalScope: {
                offset // Passing the current offset to the API call
            }
        });
    });
}
 // Additional step I did to get the data into the format I needed for the table
function formatItemsData(items) {
    return items.map(item => ({
      id: item.id,
      type: item.type,
      created_at: item.attributes.created_at,
      payment_status: item.attributes.payment_status,
      received_at: item.attributes.received_at
    }));
}

function getAllItems() {
    return getAllItemsRecursively().then(data => {
        // Remove duplicate items based on unique identifier (ID)
        const uniqueItems = data.filter((item, index, self) => self.findIndex(i => i.id === item.id) === index);
        // Format items data
        const formattedItems = formatItemsData(uniqueItems);
        return formattedItems;
    });
}

return getAllItems();

For this to work, I had to update my myQuery query and for the URL Parameter value for offset, I put {{offset}} so it can use this additional scope value.

Once I had this additional scope on the myQuery and would trigger the JS Query, it would run and get all tables of data, then do an extra step to make sure there aren't any duplicates by ID as a safety net, then format the data as needed.

2 Likes
5

Thanks for sharing!