Hi @polymath!
Thank you for sharing the added details and the screen shot, definitely helpful
It looks like you did everything correct, I just pulled up Airtable's documentation on their GET all records route.
It looks like the request route will "Return one page of records at a time. Each page will contain pageSize records, which is 100 by default.
If there are more records, the response will contain an offset.
To fetch the next page of records, include offset in the next request's parameters.
Pagination will stop when you've reached the end of your table. If the maxRecords parameter is passed, pagination will stop once you've reached this maximum."
As they mention in their docs here!
So it seems that to get all the records, you would either need to change the default from 100 records to a very large page size (not sure if it's possible to have all records be on a single page but that would be my first try!)
The other option if you run into a limit on how many records can be on one page and queried, would be to run a loop that keeps incrementing up the page offset number until no more pages of records can be found.
You can use a Javascript block to run some logic to do this!
You would likely just need a way to increment up the page number that is being used as the "offset" in the REST API URL, which would be something like while query1.data.length
then increment the offset number and run query1.trigger.
You might need to pass in the offset number as 'additional scope' to then get it into the REST API URL to get the next page of records.
The other option would be a loop block in a workflow, we just released a v2 of this feature for users to demo, more info about that here!
Server side pagination is a feature you can toggle on for a table, this works best when the query grabs a ton of data as this will work to limit down the amount of data coming in. But in your situation you have the opposite problem and need to grab more data via either a bigger request or more consecutive requests.