My REST API paginates responses by using the Link header defined by RFC5988. Is there any built-in support for this? If not how would I implement it? There doesn't seem any obvious way to plug a complete link back to the original query.
Thanks!
Hi @jje !
Depending on the API you're querying, you should be able to use the server-side pagination functionality described here. Rather than plugging in a complete query link, you'll use some of the table properties (pageSize & selectedPageIndex) to update the query params when the page changes. I've attached the app JSON of an example using the Github Search API. The screenshots show how the query and table are configured for server-side pagination.
You'll notice the example provided does not use the Link header. If you do need to retrieve values from the Link header, you can do so using {{query.metadata.headers.link}}. Unfortunately, the header string is not the easiest value to parse, so the example app also includes a parsedLinkHeader transformer (h/t Josh Frank) for returning Link headers as JSON.
I hope you find this helpful! If you're able to share more details about the API you're working with, we can get into the details about the specific type of server-side pagination to use.
Let me know how it goes!
- Jane
Link Header Pagination Demo.json (11.5 KB)
Hi there @jane
Thanks for the response! The Link parsing stuff wasn't my problem, but finding a way to plug it back into my query component was. Is that included in the demo? (sorry I can't see it right now as I'm not using a laptop with the Retool server installed). Or can you briefly explain here?
As far as the API goes, it's one that I wrote for an internal application I manage. The business logic of paging is entirely controlled by complex decisions on the server, so simple client-side solutions are a bad idea in general, and I think coupling the client to the server like this is bad design.
The limit/offset model is also a terrible way to page unless you have a very small amount of data, as SQL servers generally will materialise the entire result set anyway before slicing it to the window required.
Hence, I am using Link headers! It's much easier for clients to simply follow a rel=next link to get the next page.
Cheers.
@jane One more thing, the example you have shows a 1-1 tie between the transformer and the query component. But I want a generic transformer (or whatever else runs code) that I can attach to a query and effect the next page transition.
Just to chime in, I just bumped into the same challenge. I'm a Retool newb to be sure, but I'm using the Basecamp 3 API and it returns long query responses (for example, getAllPeople) paginated using the Link header. The first reply contains 15 users, then follow the link in the Link header to get the next page. My queries aren't so massively long that I care about paginating to be honest, I'd be happy to harvest all the data from the links and display the complete results in the table. But... it's tricky 
Replying to myself, but it seems like a combo of the info @jane posted plus this thread on returning all results for a cursor-based paginated API may get me where I want to be. Will work with it some more. Not sure if this will help you @jje.
Hi there!
My problem is that the pagination is not a simple single cursor, it's a complex relationship between multiple response fields, that could potentially change in the future, which is why I went with a Link header so the client doesn't have to worry about it.
The queries can also have a few optional URL params which adds to the complication, as they need to be passed back in the next query along with the multiple extra "cursor" fields that the server adds in the Link (the server does this for you automatically).
I feel at this point I'm on a hiding to nothing trying to implement this in Retool. The simplest way forward is for it to implement RFC5988 and then the application can literally just click that as an option and it will "just work".
Cheers.
Hey @jje!
This looks to be possible using relay-based cursor pagination. You can set your REST resource to have a blank base URL allowing you to pass the entire link generated by your API as the URL for your query. I've used a generic RESTQuery resource here that has authentication tokens passed to its headers:
Here the rel="next" and rel="previous" links are parsed into the Previous cursor and Next cursor fields respectively. Since table.afterCursor and table.beforeCursor are alternately defined based on whether you're paginating forward or backward passing {{table.beforeCursor || table.afterCursor || defaultUrl}} to your table will select the correct URL each time.
Does that work?
Edit: The code for this was done only using limit as an extra parameter of the default query but you can build out the base URL to include as many additional parameters as you'd like.
1 Like