Error with returning all results for a cursor-based paginated API (infinite loop)

[jordanp]

Hi everyone!

It seems to be a common request on this forum, but I'd like to return all results for a cursor-based paginated API. The thread here: Returning all results for a cursor-based paginated API seems to be the best bet, but I can't seem to get it working on my app.

I've created two queries:

1. API Query to fetch my first n results
   

   

2. JavaScript Query, which calls the API Query as many times as we need to. In this case, we're using pagination_token as our cursor.

// Cursor-based pagination. This function recursively calls the API as long as the cursor from the previous request exists. 
const fetchAll = (pagination_token, records) => {

  // Base case: we've reached the end, and there are no more cursors.
  if (pagination_token == null) return records
  
  // Wrap the query result in a promise
  return new Promise(resolve => {
    return query9.trigger({
    	additionalScope: {
      	pagination_token
    	},
    	onSuccess: queryResult => {
        // Add the records from this query to all the records we have so far
        const newResults = records.concat(queryResult.records)
        
        return resolve(fetchAll(queryResult.pagination_token, newResults))
    	}
  	})
  })
}

return fetchAll(0, [])

Unfortunately, when I run the above JS query, my program gets stuck in an infinite loop. Could someone help debug?

Thank you!

[Kabirdas]

Hey @jordanp! Would you mind posting a screenshot of an example result you're getting from that API? Based on Twitter's pagination docs it looks like you should be receiving a next_token in the result that you might want to pass as your pagination_token, i.e. running something like

return resolve(fetchAll(queryResult.next_token, newResults))

[jordanp]

hey @Kabirdas! thanks so much for your help here. i actually got it to work thanks to some other Retool team members. i wasn't retrieving the next_token property correctly in the JS query

if you have any tips for this question: Group by functionality in table - #4 by ScottR -- let me know! no worries if not, however. this team is incredibly helpful regardless.

[nickaus]

Hi all,
I thought I would reply on this thread in the spirit of sharing/giving back. I have had amazing help from @Kabirdas getting some of my JS stuff to work (I am a real JS beginner).

I had a challenge today of fetching paginated Tickets (checking for updates) from FreshDesk for a given date. I started with this very basic JS Resource calling an API resource (with page=1 fixed as a URL Parameter):

const response = await filterTickets.trigger()
return response.results;

This is fine, but FreshDesk only returns 30 results for this particular endpoint:

https://a_company.freshdesk.com/api/v2/search/tickets/?query="updated_at:%27{{dateForTesting.value}}%27"

The endpoint returns (for a given day) {"results:[n], "total": x}. This is the adapted JS (with assistance from GPT (3.5) to get additional pages (if needed):

let allResults = [];
const fetchData = async () => {
  const resultsPerPage = 30;
  let page = 1;
  while (true) {
    const response = await filterTickets.trigger({ additionalScope: { page } });
    // Check if there are results in the current page
    if (response.results && response.results.length > 0) {
      allResults = allResults.concat(response.results);
    } else {
      break;
    }
    // If number of results < less than the results per page, @ last page
    if (response.results.length < resultsPerPage) {
      break;
    }
    page++;
  }
  return allResults; // Return final aggregated results
};
const finalResults = await fetchData();
return finalResults;

Maybe someone will find this helpful (especially if they need to deal with the vagaries of FreshDesk API - pagination for the three main endpoints I need is handled differently .

Also, if anyone has a more elegant approach/solution I am very keen to learn

Regards,
Nick