How does auto-pagination work for queries?

I’m not really understanding how auto-pagination for queries works. Or if it even supports fetching paginated results in the way I need.
I’m trying to get all the results from an Airtable spreadsheet. The Airtable API only returns 100 rows at a time, and supports getting additional rows by passing an ‘offset’ parameter in the request URL with a UUID from the previous JSON response. Is this possible in Retool without setting up a bunch of chained queries?

Hi Christopher! so auto-pagination specifically works with retool tables. a retool table has the following helpful properties: pageSize and paginationOffset. it’s meant to be exactly analogous to sql’s limit and offset statements
looking at airtable’s api, you can pass in pageSize directly. offset is a bit trickier since they use cursor based pagination (as opposed to index-based), but you can pass in something like[ - 1].id
for the offset. does that make sense? or would it be helpful if i wrote out a full example?

Hi Abdul. A full example would be great. The part I’m confused about is if I need to write multiple queries in Retool (and have the first query call the next, passing in the offset value) or if there’s a way I can do it with a single query setup (this would be ideal, since I wouldn’t have to add more queries whenever my Airtable spreadsheet grows).

sure gimme a couple mins. i think you just need one query

Hey Abdul. Were you able to see if this would work with just one query?

hey man, really sorry about the delay. turns out cursor based pagination on REST is pretty tricky! especially if you wanna be able to page backwards

but i didn’t forget about it! i’ll get you an example soon

Okay! Thanks for the update!

alright so, you can do paging forward with 1 query like this:

note that there’s a small hack for not passing in the offset for the first page (the table1.afterCursor && 'offset' part)

paging backwards is actually really hard (do-able, but you have to manually store all the previous cursors, and have logic to figure out whether or not you should be using table1.afterCursor or table1.beforeCursor)

we should probably have a built-in pagination option that makes this easy, since right now it’s unfortunately pretty hard :frowning:

i’ll bring up this use-case to the team, and we’ll build this hopefully soon!

Thanks! This is very helpful

@abdul-nimeri Are there any updates on this, we just stumbled upon a similar use case where we want to paginate over an Rest API with offset and limit.
Are there any work arounds ?
How can we display all the data on that we go on a single call, it show only 7 rows out of 100 and when clicking next it pulling new data and show 7 out of 100?

@christopher How are you guys dealing with this ?

I have the exact same problem and I think Retool would be an awesome no-code tool in combination with Airtable. Please let me know if there’s any update regarding this!

Also very keen on this functionality! Would be great to add a resource query into an app and optionally set it as a paginated query that Retool would loop through.

I'm sure it can get a little complex, especially if the original response is not an array, you might need to specify which path on the response is the array that needs to grow with every page, OR you end up with dataArray (instead of data) and you can do your own mapping across each response.

Hey marcusjroberts! Retool does allow for server side pagination of queries. You can set it up on our table component. The docs here go into detail of how to set it up with a SQL query.

The information starts about 25% of the way down the page.