Getting all results from a page-based pagination on API

I'm new here, and a complete outsider to JS and mostly to SQL too. Please be gentle.

I have an API (ClickUp) with page-based pagination. It doesn't tell you the amount of results, and returns up to 100, so I have to iterate pages until one returns <100.

I have already had a look at How to pull a all data from a paged REST API and Returning all results for a cursor-based paginated API . Both very similar but not exact and being clear enough for me to follow.

I have no clue how to write it or where, but I understand the logic and it shouldn't be too difficult. To increase the page number by 1 until clientstaskpage.tasks <100.

Any help is greatly appreciated. Thanks!

1 Like

Hi @Chaos!

I think you may be on the right track here with cursor-based pagination. Can you try setting up your table as follows (it should be all green in your case):

You should be able to access the page parameter you're passing to your query in clientstaskspage.rawData.args, we make sure to convert that to a number by adding the + in front of it, then add 1. This gets stored in the table as the next cursor to be passed to the query, and you can reference it as follows:

How exactly this is configured depends on your API, you might need to pass something like {{table1.nextAfterCursor || 0}} in case the endpoint always needs to see a number in the page parameter (here the logical or || is used so that if nextAfterCursor is empty the transformer returns 0).

Finally, you can check to see if there's a next page by seeing if 100 results are returned, as you mentioned, using {{ clientstaskspage.data.tasks.length === 100 }}.

Let me know if that works or raises any further issues!

Hey @Kabirdas, I tried to use .args but it's not working. Do you know why?

image

Hi @yeftanma can we see the query response when you query the first page of results?