Connecting to an Airtable base as a data source

- Goal: I want to connect my Airtable base to retool in real-time so that I can build AI RAG applications on top of the data. I currently have a connection established, however it only pulls in 100 records where as there are over 1000+ records in my Airtable DB.

- Steps: I used the following docs to make the connection: Connect to Airtable | Retool Docs

I am not really seeing any errors but I am only seeing 100 records rather 1000+. How can I fix this and also make it real time?

Thank you in advance :slight_smile:

1 Like

Hello @polymath!

How are you fetching the records from your Airtable DB?

Are you running queries or using a workflow?

If you are putting the records/rows of data in a table with 'server side pagination' turned on, it will limit the number or rows displayed on the table to increase rendering speed. In which case you could turn this off to grab all.

If you can share a screen shot of the query that is getting the data I can better assist!

Hello @Jack_T ! Thanks for following up.

  • I created a resource using these docs to pull the records from my Airtable DB: Connect to Airtable | Retool Docs

  • I have not written any query. All I did was create a resource using my Airtable BASE URL and auth info. I am doing a GET request to pull the data from Airtable and am currently just displaying it in a table.

  • Where can I find the 'server side pagination' toggle?

Let me know if this is helpful.

Hi @polymath!

Thank you for sharing the added details and the screen shot, definitely helpful :slightly_smiling_face:

It looks like you did everything correct, I just pulled up Airtable's documentation on their GET all records route.

It looks like the request route will "Return one page of records at a time. Each page will contain pageSize records, which is 100 by default.

If there are more records, the response will contain an offset.

To fetch the next page of records, include offset in the next request's parameters.

Pagination will stop when you've reached the end of your table. If the maxRecords parameter is passed, pagination will stop once you've reached this maximum."

As they mention in their docs here!

So it seems that to get all the records, you would either need to change the default from 100 records to a very large page size (not sure if it's possible to have all records be on a single page but that would be my first try!)

The other option if you run into a limit on how many records can be on one page and queried, would be to run a loop that keeps incrementing up the page offset number until no more pages of records can be found.

You can use a Javascript block to run some logic to do this!

You would likely just need a way to increment up the page number that is being used as the "offset" in the REST API URL, which would be something like while query1.data.length then increment the offset number and run query1.trigger.

You might need to pass in the offset number as 'additional scope' to then get it into the REST API URL to get the next page of records.

The other option would be a loop block in a workflow, we just released a v2 of this feature for users to demo, more info about that here!

Server side pagination is a feature you can toggle on for a table, this works best when the query grabs a ton of data as this will work to limit down the amount of data coming in. But in your situation you have the opposite problem and need to grab more data via either a bigger request or more consecutive requests.